Skip to content

AshishBagdane/go-report-engine

🚀 go-report-engine

A production-grade, modular reporting engine for Go with comprehensive error handling, thread-safe registries, enterprise-grade architecture, and complete YAML/JSON configuration support.

Built using Strategy, Factory, Builder, Template Method, and Chain of Responsibility patterns.

Fetch → Process → Format → Output — fully customizable.

Go Version Build Status Test Coverage License Built in Public


✨ Features

Core Features

  • 🔌 Pluggable Providers - Fetch data from any source (DB, CSV, API, etc.)
  • ♻️ Processing Pipeline - Chain of Responsibility for data transformation
  • 🧾 Multiple Formatters - JSON, CSV, YAML output formats
  • 📤 Flexible Outputs - Console, File, API, Slack, Email delivery
  • 🧱 SOLID Principles - Clean, testable, extensible architecture
  • 🧪 Test-Driven - 95%+ test coverage with comprehensive test suite

Production-Ready Features

  • 🔒 Thread-Safe Registries - Concurrent access with sync.RWMutex
  • 🚨 Comprehensive Error Handling - Context-rich errors with classification
  • 🔄 Intelligent Retry Logic - Automatic retries for transient failures
  • 📊 Error Classification - Transient, Permanent, Configuration, Validation, Resource
  • 🎯 Component-Specific Errors - Specialized errors for debugging
  • 🛡️ Panic Recovery - Graceful handling with RunWithRecovery()
  • Input Validation - Comprehensive validation across all components
  • 🏗️ Builder Pattern - Fluent API for engine construction
  • ⚙️ Config-Driven Setup - YAML/JSON configuration support
  • 📝 Structured Logging - slog integration with metrics tracking
  • 🔍 Observable Pipeline - Every stage logged with performance metrics
  • 🌍 Environment Overrides - Runtime configuration via environment variables
  • 🎁 Configuration Presets - Default, Development, Production, Testing presets
  • 📦 Integration Helpers - One-step load-and-build functions
  • 🌱 Built in Public - Follow the real-time development journey

📦 Installation

go get github.com/AshishBagdane/go-report-engine

🧠 Architecture Overview

Provider → Processor Chain → Formatter → Output

Pipeline Components

Component Purpose Examples
Provider Fetch data from sources Mock, CSV, Database, API
Processor Transform data step-by-step Filter, Validate, Transform
Formatter Convert to output format JSON, CSV, YAML
Output Deliver the final report Console, File, Slack, Email

🧰 Quick Start

1. Using Configuration Files (Recommended)

Create a config.yaml file:

provider:
  type: mock
  params: {}

processors:
  - type: min_score_filter
    params:
      min_score: "90"

formatter:
  type: json
  params:
    indent: "2"

output:
  type: console
  params: {}

Load and run:

package main

import (
    "log"
    "github.com/AshishBagdane/go-report-engine/internal/config"
)

func main() {
    // One-step load and build
    engine, err := config.LoadAndBuild("config.yaml")
    if err != nil {
        log.Fatalf("Failed to create engine: %v", err)
    }

    // Run the engine
    if err := engine.Run(); err != nil {
        log.Fatalf("Execution failed: %v", err)
    }
}

2. Using Builder Pattern

package main

import (
    "context"
    "fmt"
    "log"
    "github.com/AshishBagdane/go-report-engine/internal/engine"
    "github.com/AshishBagdane/go-report-engine/internal/provider"
    "github.com/AshishBagdane/go-report-engine/internal/processor"
    "github.com/AshishBagdane/go-report-engine/internal/formatter"
    "github.com/AshishBagdane/go-report-engine/internal/output"
    "github.com/AshishBagdane/go-report-engine/internal/registry"
)

func init() {
    // Register components
    registry.RegisterProvider("mock", func() provider.ProviderStrategy {
        return provider.NewMockProvider([]map[string]interface{}{
            {"id": 1, "name": "Alice", "score": 95},
            {"id": 2, "name": "Bob", "score": 88},
        })
    })

    registry.RegisterFormatter("json", func() formatter.FormatStrategy {
        return formatter.NewJSONFormatter("  ")
    })

    registry.RegisterOutput("console", func() output.OutputStrategy {
        return output.NewConsoleOutput()
    })
}

func main() {
    // Build engine with builder pattern
    eng, err := engine.NewEngineBuilder().
        WithProviderType("mock").
        WithFormatterType("json").
        WithOutputType("console").
        Build()

    if err != nil {
        log.Fatalf("Failed to build engine: %v", err)
    }

    // Run with context
    ctx := context.Background()
    if err := eng.RunWithContext(ctx); err != nil {
        fmt.Println("Error during execution:", err)
    }
}

3. Using Configuration Presets

package main

import (
    "log"
    "github.com/AshishBagdane/go-report-engine/internal/config"
)

func main() {
    // Use built-in preset configurations

    // For development
    engine, err := config.BuildFromDevelopment()
    if err != nil {
        log.Fatalf("Failed: %v", err)
    }

    // Or for production
    // engine, err := config.BuildFromProduction()

    // Or for testing
    // engine, err := config.BuildFromTesting()

    engine.Run()
}

⚙️ Configuration

Configuration File Formats

The engine supports both YAML and JSON configuration formats:

# config.yaml
provider:
  type: mock
  params:
    data_source: "test"

processors:
  - type: filter
    params:
      field: "score"
      min: "80"

formatter:
  type: json
  params:
    indent: "2"

output:
  type: console
  params: {}
{
  "provider": {
    "type": "mock",
    "params": {}
  },
  "processors": [],
  "formatter": {
    "type": "json",
    "params": {
      "indent": "2"
    }
  },
  "output": {
    "type": "console",
    "params": {}
  }
}

Environment Variable Overrides

Override configuration at runtime using environment variables:

# Set environment variables
export ENGINE_PROVIDER_TYPE=postgres
export ENGINE_PROVIDER_PARAM_HOST=localhost
export ENGINE_PROVIDER_PARAM_PORT=5432
export ENGINE_FORMATTER_TYPE=json
export ENGINE_OUTPUT_TYPE=file

# Load config with overrides
engine, err := config.LoadAndBuildWithEnv("config.yaml")

Supported Environment Variables:

  • ENGINE_PROVIDER_TYPE - Override provider type
  • ENGINE_FORMATTER_TYPE - Override formatter type
  • ENGINE_OUTPUT_TYPE - Override output type
  • ENGINE_PROVIDER_PARAM_<KEY> - Override provider parameters
  • ENGINE_FORMATTER_PARAM_<KEY> - Override formatter parameters
  • ENGINE_OUTPUT_PARAM_<KEY> - Override output parameters

Configuration Presets

The engine includes built-in configuration presets:

Preset Provider Formatter Output Use Case
Default mock json console Development/testing
Development mock json console Local development
Production mock json file Production deployment
Testing mock json console Unit/integration tests
// Use presets
engine := config.MustBuildFromDefault()
engine := config.MustBuildFromDevelopment()
engine := config.MustBuildFromProduction()
engine := config.MustBuildFromTesting()

Integration Functions

Convenient one-step functions for common patterns:

// Load config file and build engine
engine, err := config.LoadAndBuild("config.yaml")

// Load with environment overrides
engine, err := config.LoadAndBuildWithEnv("config.yaml")

// Build from raw bytes
yamlBytes := []byte(`provider: {type: mock}...`)
engine, err := config.BuildFromBytes(yamlBytes, "yaml")

// Load with fallback to default
cfg, err := config.LoadOrDefault("config.yaml")
engine, err := config.ValidateAndBuild(*cfg)

// Must variants (panic on error - for init functions)
engine := config.MustLoadAndBuild("config.yaml")
engine := config.MustBuildFromDefault()

📁 Project Structure

go-report-engine/
├── cmd/
│   └── example/
│       └── main.go                         # ✅ Example usage
├── pkg/
│   └── api/
│       └── interfaces.go                   # ✅ Public API
├── internal/
│   ├── config/                             # ✅ Configuration system
│   │   ├── loader.go                       # ✅ YAML/JSON loading
│   │   ├── loader_test.go                  # ✅ Loader tests
│   │   ├── defaults.go                     # ✅ Default configs & presets
│   │   ├── defaults_test.go                # ✅ Defaults tests
│   │   ├── integration.go                  # ✅ Integration helpers
│   │   └── integration_test.go             # ✅ Integration tests
│   ├── engine/
│   │   ├── builder.go                      # ✅ Builder pattern
│   │   ├── builder_test.go                 # ✅ Builder tests
│   │   ├── config.go                       # ✅ Configuration structs
│   │   ├── config_test.go                  # ✅ Config tests
│   │   ├── engine.go                       # ✅ Core engine
│   │   ├── engine_test.go                  # ✅ Engine tests
│   │   └── options.go                      # ✅ Functional options
│   ├── errors/                             # ✅ Complete error system
│   │   ├── errors.go                       # ✅ Core error infrastructure
│   │   ├── errors_test.go                  # ✅ Core error tests
│   │   ├── provider_errors.go              # ✅ Provider-specific errors
│   │   ├── provider_errors_test.go         # ✅ Provider error tests
│   │   ├── processor_errors.go             # ✅ Processor-specific errors
│   │   ├── processor_errors_test.go        # ✅ Processor error tests
│   │   ├── formatter_errors.go             # ✅ Formatter-specific errors
│   │   ├── output_errors.go                # ✅ Output-specific errors
│   │   └── formatter_output_errors_test.go # ✅ Formatter/Output tests
│   ├── registry/                           # ✅ Thread-safe registries
│   │   ├── formatter_registry.go           # ✅ Formatter registry
│   │   ├── formatter_registry_test.go      # ✅ Formatter registry tests
│   │   ├── output_registry.go              # ✅ Output registry
│   │   ├── output_registry_test.go         # ✅ Output registry tests
│   │   ├── processor_registry.go           # ✅ Processor registry
│   │   ├── processor_registry_test.go      # ✅ Processor registry tests
│   │   ├── provider_registry.go            # ✅ Provider registry
│   │   └── provider_registry_test.go       # ✅ Provider registry tests
│   ├── logging/                            # ✅ Structured logging
│   │   ├── logger.go                       # ✅ Logger implementation
│   │   ├── logger_test.go                  # ✅ Logger tests
│   │   ├── context.go                      # ✅ Context helpers
│   │   └── context_test.go                 # ✅ Context tests
│   ├── provider/
│   │   ├── provider.go                     # ✅ Provider interface
│   │   ├── mock.go                         # ✅ Mock implementation
│   │   ├── mock_test.go                    # ✅ Mock provider tests
│   │   └── mock_logging_test.go            # ✅ Logging tests
│   ├── processor/
│   │   ├── processor.go                    # ✅ Processor interface
│   │   ├── base.go                         # ✅ Base processor
│   │   ├── base_test.go                    # ✅ Base processor tests
│   │   ├── wrappers.go                     # ✅ Type-safe wrappers
│   │   └── wrappers_test.go                # ✅ Wrapper tests
│   ├── formatter/
│   │   ├── formatter.go                    # ✅ Formatter interface
│   │   ├── json.go                         # ✅ JSON formatter
│   │   └── json_test.go                    # ✅ JSON formatter tests
│   ├── output/
│   │   ├── output.go                       # ✅ Output interface
│   │   ├── console.go                      # ✅ Console output
│   │   └── console_test.go                 # ✅ Console output tests
│   └── factory/
│       ├── engine_factory.go               # ✅ Engine factory
│       ├── engine_factory_test.go          # ✅ Engine factory tests
│       ├── processor_chain_factory.go      # ✅ Processor chain factory
│       └── processor_chain_factory_test.go # ✅ Chain factory tests
├── examples/                               # ✅ Complete examples
│   ├── configs/                            # ✅ Example configs
│   │   ├── config.yaml                     # ✅ Complete YAML config
│   │   ├── config.minimal.yaml             # ✅ Minimal config
│   │   ├── config.json                     # ✅ JSON config
│   │   └── README.md                       # ✅ Config documentation
│   ├── scripts/                            # ✅ Helper scripts
│   │   └── env_override_demo.sh            # ✅ Environment demo
│   ├── config_loading/                     # ✅ Config loading examples
│   │   └── main.go
│   ├── defaults_usage/                     # ✅ Presets examples
│   │   └── main.go
│   ├── integration_patterns/               # ✅ Integration patterns
│   │   └── main.go
│   └── README.md                           # ✅ Examples documentation
├── go.mod                                  # ✅ Module definition
├── go.sum                                  # ✅ Dependencies
└── README.md                               # ✅ This file

🧪 Testing

Run Tests

# Run all tests
go test ./... -v

# Run with race detector
go test ./... -race

# Check coverage
go test ./... -cover

# Generate coverage report
go test ./... -coverprofile=coverage.out
go tool cover -html=coverage.out

# Run benchmarks
go test ./... -bench=. -benchmem

Current Test Statistics

  • 271+ test functions across all packages
  • 52+ benchmarks for performance validation
  • 95%+ code coverage on core components
  • Race detector clean - safe for concurrent use
  • Zero flaky tests - reliable and deterministic

Test Files by Package

Package Test Functions Benchmarks Coverage
engine 25 4 95%
config 35 6 95%
errors 38 6 95%
registry 48 12 100%
logging 24 7 95%
provider 23 6 100%
processor 28 4 95%
formatter 14 4 100%
output 13 3 100%
factory 20 3 95%

🔌 Available Components

Providers

  • MockProvider - In-memory test data
  • CSVProvider - Read from CSV files
  • DBProvider - SQL database support (PostgreSQL, MySQL)
  • APIProvider - REST API integration

Processors

  • BaseProcessor - Pass-through processor
  • FilterWrapper - Filter data rows with FilterStrategy
  • ValidatorWrapper - Validate data with ValidatorStrategy
  • TransformWrapper - Transform data with TransformerStrategy
  • AggregateProcessor - Aggregate data (Group By, Sum, Avg)
  • DeduplicateProcessor - Remove duplicate records
  • 🚧 SanitizeProcessor - Coming soon

Formatters

  • JSONFormatter - JSON output with indentation
  • CSVFormatter - CSV output
  • YAMLFormatter - YAML output
  • 🚧 HTMLFormatter - Coming soon
  • 🚧 HTMLFormatter - Coming soon
  • 🚧 XMLFormatter - Coming soon

Outputs

  • ConsoleOutput - Terminal/stdout output
  • FileOutput - File system output
  • 🚧 S3Output - AWS S3 output
  • 🚧 SlackOutput - Slack webhook
  • 🚧 EmailOutput - Email delivery

📖 Examples

The examples/ directory contains comprehensive examples demonstrating different usage patterns:

1. Config Loading (examples/config_loading/)

Learn how to load configuration from YAML and JSON files:

cd examples/config_loading
go run main.go

2. Default Configs (examples/defaults_usage/)

Explore preset configurations and builder patterns:

cd examples/defaults_usage
go run main.go

3. Integration Patterns (examples/integration_patterns/)

Production-ready integration patterns:

cd examples/integration_patterns
go run main.go

Environment Variable Demo

cd examples/scripts
./env_override_demo.sh

See examples/README.md for detailed examples documentation.


🗺️ Roadmap

Phase 1 - FoundationCOMPLETED

  • ✅ Core architecture and interfaces
  • ✅ Thread-safe registries with sync.RWMutex
  • ✅ Comprehensive error handling system
  • ✅ Builder and factory patterns
  • ✅ Input validation across all components
  • ✅ Structured logging with slog
  • ✅ Observable pipeline with metrics tracking
  • ✅ 95%+ test coverage on core components
  • ✅ Complete documentation with examples
  • ✅ 271+ unit tests + 52+ benchmarks

Phase 2 - Additional ComponentsCOMPLETED

  • ✅ CSV Provider implementation
  • ✅ Database Provider (PostgreSQL, MySQL)
  • ✅ REST API Provider
  • ✅ CSV Formatter
  • ✅ YAML Formatter
  • ✅ File Output implementation
  • ✅ Additional processor types (Aggregate, Deduplicate)

Phase 3 - Configuration & IntegrationCOMPLETED

  • ✅ YAML/JSON config file loading
  • ✅ Environment variable overrides
  • ✅ Configuration presets (Default, Dev, Prod, Testing)
  • ✅ Integration helper functions
  • ✅ Complete examples directory
  • ✅ Configuration documentation
  • ✅ Must variants for initialization
  • ✅ Fallback patterns

Phase 4 - PerformanceCOMPLETED

  • Concurrent processing in chains
  • Worker pools for bounded concurrency
  • Memory pooling for efficiency (sync.Pool implementation)
  • Streaming for large datasets
  • Performance benchmarks and profiling

Benchmark Results (10,000 Records)

Mode Time/Op Memory/Op Allocations/Op Improvement
Batch ~11.8ms 11.9 MB 200,114 Baseline
Streaming (No Pool) ~10.7ms 10.1 MB 200,104 15% Memory savings
Streaming (Pooled) ~10.3ms 6.7 MB 180,140 43% Memory savings

Streaming with memory pooling significantly reduces GC pressure for large datasets.

Phase 5 - EnterpriseCOMPLETED

  • Resource cleanup and lifecycle management
  • Retry mechanisms with exponential backoff
  • Metrics and observability (MetricsCollector interface)
  • Circuit breakers for resilience
  • Distributed tracing (Tracer interface, decorators)
  • Health check endpoints (Checker interface, engine.Health() API)

Phase 6 - DevOpsCOMPLETED

  • CI/CD Pipeline (GitHub Actions)
  • Linting & Testing Automation
  • Build Verification

Future - Advanced (Planned)

  • Dashboard UI
  • Scheduling and cron jobs
  • AI-powered data enrichment
  • Caching layer
  • BigQuery / Snowflake providers
  • Webhooks and event-driven processing

📊 Progress

Category Status Coverage Tests
Core Engine ✅ Complete 95% 25
Configuration Loading ✅ Complete 95% 35
Error Handling ✅ Complete 95% 38
Thread-Safe Registries ✅ Complete 100% 48
Input Validation ✅ Complete 95% 15
Builder Pattern ✅ Complete 95% 12
Factory Pattern ✅ Complete 95% 20
Base Providers ✅ Complete 100% 12
Processors ✅ Complete 95% 28
Parallel Processing ✅ Complete 100% 15
Resource Cleanup ✅ Complete 100% 8
Base Formatters ✅ Complete 100% 14
Base Outputs ✅ Complete 100% 13
Structured Logging ✅ Complete 95% 24
Context Support ✅ Complete 100% 8
Examples & Documentation ✅ Complete 100% -
DevOps ✅ Complete 100% -

Overall Progress: All Planned Phases Complete (100%)


🎯 Design Principles

  1. SOLID Principles - Single Responsibility, Open/Closed, Liskov Substitution, Interface Segregation, Dependency Inversion
  2. Design Patterns - Strategy, Factory, Builder, Chain of Responsibility, Registry, Template Method
  3. Testability - Every component is interface-driven and mockable
  4. Concurrency - Thread-safe by design with proper locking
  5. Error Handling - Comprehensive context and classification
  6. Performance - Optimized for production use with benchmarks
  7. Validation - Input validation at all boundaries
  8. Documentation - Comprehensive godoc for all exports
  9. Observability - Structured logging with performance metrics
  10. Configuration - Flexible, environment-aware, preset-based

💡 Advanced Usage

Structured Logging

The engine includes comprehensive structured logging with slog:

// Logging output example
{"time":"2024-11-24T10:30:45Z","level":"INFO","component":"engine","msg":"starting report generation","request_id":"req-123"}
{"time":"2024-11-24T10:30:45Z","level":"INFO","component":"provider.mock","msg":"fetch completed","provider_type":"mock","duration_ms":0,"duration_us":42,"record_count":2}
{"time":"2024-11-24T10:30:45Z","level":"INFO","component":"formatter.json","msg":"formatting completed","formatter_type":"json","record_count":2,"output_size_bytes":156,"duration_ms":1}

Context Support

Use context for request tracking and cancellation:

import (
    "context"
    "github.com/AshishBagdane/go-report-engine/internal/logging"
)

// Add request ID to context
ctx := logging.WithRequestID(context.Background(), "req-123")
ctx = logging.WithCorrelationID(ctx, "corr-456")

// Run with context
if err := engine.RunWithContext(ctx); err != nil {
    log.Printf("Failed: %v", err)
}

Panic Recovery

// Run with automatic panic recovery
err := engine.RunWithRecovery()
if err != nil {
    // Panics are converted to errors
    log.Printf("Pipeline failed: %v", err)
}

Error Context Extraction

if engineErr, ok := err.(*errors.EngineError); ok {
    fmt.Printf("Stage: %s\n", engineErr.Stage)
    fmt.Printf("Type: %s\n", engineErr.ErrorType)
    fmt.Printf("Component: %s\n", engineErr.Component)

    // Check if retriable
    if engineErr.IsRetriable() {
        // Implement retry logic
    }
}

Custom Processors

Implement your own processing logic:

type MinScoreFilter struct {
    MinScore int
}

func (f *MinScoreFilter) Keep(row map[string]interface{}) bool {
    if score, ok := row["score"].(int); ok {
        return score >= f.MinScore
    }
    return false
}

func (f *MinScoreFilter) Configure(params map[string]string) error {
    minScoreStr, ok := params["min_score"]
    if !ok {
        return api.ErrMissingParam("min_score")
    }
    score, err := strconv.Atoi(minScoreStr)
    if err != nil {
        return fmt.Errorf("min_score must be an integer: %w", err)
    }
    f.MinScore = score
    return nil
}

// Register the custom processor
func init() {
    registry.RegisterFilter("min_score_filter", &MinScoreFilter{})
}

🤝 Contributing

We welcome contributions! This project is built in public and we're actively developing new features.

Ways to Contribute

Please open:

  • Issues for bugs or feature requests
  • Discussions for ideas and questions
  • PRs for improvements and new features

How to Contribute

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Follow SOLID principles and existing patterns
  4. Add tests (maintain 95%+ coverage)
  5. Add comprehensive documentation
  6. Run tests and linters: go test ./... -race && go vet ./...
  7. Submit a PR with clear description

Development Guidelines

  • Work on ONE component at a time
  • Write godoc for all exports
  • Use table-driven tests
  • Follow Go best practices
  • Run go fmt, go vet, and golangci-lint
  • Ensure race detector passes: go test ./... -race
  • Add benchmarks for performance-critical code

Code Review Checklist

  • Godoc comments on all exports
  • Error handling with proper context
  • Input validation at boundaries
  • Thread-safety considered
  • Tests written and passing (>95% coverage)
  • Benchmarks for critical paths
  • No data races (-race clean)
  • SOLID principles followed
  • Documentation updated

Join the #buildinpublic journey! 🎉


📖 Documentation


🪪 License

MIT License — free for personal & commercial use.

See LICENSE for details.


⭐ Support the Project

If you find this useful:

  • ⭐ Star the repo on GitHub
  • 🐦 Share on Twitter/X
  • 🤝 Contribute code or documentation
  • 💬 Join discussions and provide feedback
  • 🐛 Report bugs and suggest features

📞 Follow the Journey


🏆 Project Highlights

  • 271+ Test Functions - Comprehensive test coverage
  • 52+ Benchmarks - Performance validation
  • 95%+ Coverage - High-quality codebase
  • Zero Race Conditions - Thread-safe implementation
  • SOLID Design - Professional architecture
  • Production-Ready - Enterprise-grade error handling
  • Observable Pipeline - Structured logging with metrics
  • Config-Driven - YAML/JSON with environment overrides
  • Complete Examples - Production patterns & integration guides
  • Well-Documented - Complete godoc and example coverage
  • Built in Public - Transparent development process

Built with ❤️ in Go | Production-Ready | Enterprise-Grade | 95%+ Test Coverage

Last Updated: November 27, 2024

About

A Go framework for building reporting pipelines with replaceable components and clean interfaces. Easy to extend. Easy to test.

Topics

Resources

License

Code of conduct

Contributing

Stars

Watchers

Forks

Packages

 
 
 

Contributors