Development Guide

This guide covers setting up a development environment, understanding the codebase structure, and contributing to the Open Resource Broker.

Development Environment Setup

Prerequisites

Python 3.10+: Required for the application
Git: For version control
AWS CLI: For AWS provider functionality (optional)
Docker: For containerized development (optional)

Local Development Setup

Clone the Repository

git clone https://github.com/finos/open-resource-broker.git
cd open-resource-broker

Create Virtual Environment

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

Install Dependencies

# Install with all development dependencies
pip install -e ".[dev]"

# Or use make target
make dev-install

# Or install specific feature groups
pip install -e ".[cli,api,monitoring,dev]"

Setup Git Hooks

# Configure version-controlled hooks (required for beads integration)
./dev-tools/scripts/setup-hooks.sh

Configure Environment

# Initialize configuration
orb init

# Or copy example configuration
cp config/config.example.json config/config.json
vim config/config.json

Run Tests

# Run all tests
make test

# Run with coverage
make test-cov

Start Development Server

# CLI mode
orb templates list

# API mode (requires [api] extras)
python -m src.bootstrap --config config/config.json --log-level DEBUG

Installing from Local Wheel (Testing)

When testing local changes before publishing:

# Build wheel
python -m build --wheel

# Install minimal (base only)
pip install dist/orb_py-*.whl

# Install with CLI colors
pip install "dist/orb_py-*.whl[cli]"

# Install with API server
pip install "dist/orb_py-*.whl[api]"

# Install with monitoring
pip install "dist/orb_py-*.whl[monitoring]"

# Install everything
pip install "dist/orb_py-*.whl[all]"

Note: Quotes are required when using brackets in bash!

Optional Dependencies

The package supports several optional feature groups:

[cli]: Rich console output with colors
- Adds: rich, rich-argparse
- Use when: You want colored CLI output
[api]: REST API server mode
- Adds: fastapi, uvicorn, jinja2
- Use when: Running as API server
[monitoring]: Observability features
- Adds: opentelemetry-*, prometheus-client, psutil
- Use when: Need distributed tracing and metrics
[dev]: Development tools
- Adds: pytest, ruff, mypy, etc.
- Use when: Contributing to the project
[all]: All optional features
- Installs: [cli,api,monitoring]
- Use when: You want all features

Development Configuration

Create a config/dev-config.json file for development:

{
  "aws": {
    "region": "us-east-1",
    "profile": "default"
  },
  "logging": {
    "level": "DEBUG",
    "file_path": "logs/dev.log",
    "console_enabled": true,
    "format": "detailed"
  },
  "database": {
    "type": "sqlite",
    "name": "dev_database.db"
  },
  "REPOSITORY_CONFIG": {
    "type": "json",
    "json": {
      "storage_type": "single_file",
      "base_path": "data/dev",
      "filenames": {
        "single_file": "dev_database.json"
      }
    }
  },
  "development": {
    "auto_reload": true,
    "debug_mode": true,
    "mock_providers": false
  }
}

Project Structure

High-Level Architecture

The project follows Domain-Driven Design (DDD) with clean architecture principles:

open-resource-broker/
+--- src/                     # Source code
|   +--- domain/              # Domain layer (business logic)
|   +--- application/         # Application layer (use cases)
|   +--- infrastructure/      # Infrastructure layer (technical concerns)
|   +--- providers/           # Provider implementations
|   +--- api/                 # API layer
+--- tests/                   # Test suite
+--- config/                  # Configuration files
+--- scripts/                 # Shell scripts
+--- docs/                    # Documentation
+--- requirements*.txt        # Dependencies

Domain Layer (`src/domain/`)

Contains pure business logic with no external dependencies:

domain/
+--- base/                    # Shared kernel
|   +--- entity.py           # Base entities and aggregates
|   +--- value_objects.py    # Common value objects
|   +--- events.py           # Domain events
|   +--- exceptions.py       # Domain exceptions
|   +--- repository.py       # Repository interfaces
+--- template/               # Template bounded context
+--- machine/                # Machine bounded context
+--- request/                # Request bounded context

Key Principles:

No dependencies on infrastructure or external libraries
Rich domain models with business logic
Domain events for state changes
Value objects for data integrity

Application Layer (`src/application/`)

Orchestrates domain operations and coordinates with infrastructure:

application/
+--- base/                   # Base application components
+--- dto/                    # Data transfer objects
+--- interfaces/             # Application interfaces
+--- commands/               # Command handlers (CQRS)
+--- queries/                # Query handlers (CQRS)
+--- template/               # Template use cases
+--- machine/                # Machine use cases
+--- request/                # Request use cases

Key Principles:

Thin orchestration layer
CQRS pattern for complex operations
Service pattern for simple CRUD operations
DTO objects for data transfer

Infrastructure Layer (`src/infrastructure/`)

Implements technical concerns and external integrations:

infrastructure/
+--- interfaces/             # Technical interfaces
+--- ports/                  # External system ports
+--- storage/                # Data storage
+--- events/                 # Event infrastructure
+--- logging/                # Logging utilities
+--- config/                 # Configuration management
+--- di/                     # Dependency injection

Key Principles:

Implements domain interfaces
Handles external system integration
Provides technical services
Configurable implementations

Provider Layer (`src/providers/`)

Cloud provider-specific implementations:

providers/
+--- aws/                    # AWS provider
    +--- domain/             # AWS domain extensions
    +--- application/        # AWS application services
    +--- infrastructure/     # AWS infrastructure
    +--- managers/           # AWS resource managers

Key Principles:

Provider-agnostic domain layer
Cloud-specific implementations
Extensible for multiple providers
Clean separation of concerns

Development Workflow

Feature Development

Create Feature Branch

git checkout -b feature/your-feature-name

Write Tests First (TDD)

# Create test file
touch tests/test_your_feature.py

# Write failing tests
make test tests/test_your_feature.py -v

Implement Feature
- Start with domain layer (business logic)
- Add application layer (use cases)
- Implement infrastructure layer (technical details)
- Update provider layer if needed

Run Tests

# Run specific tests
make test FILE=tests/test_your_feature.py

# Run all tests
make test

# Check coverage
make test-cov

Update Documentation

# Update relevant documentation
vim docs/docs/user_guide/your-feature.md

# Build documentation
cd docs && mkdocs serve

Commit Changes

git add .
git commit -m "feat: add your feature description"

Code Style and Standards

Python Code Style

Follow PEP 8 with these specific guidelines:

# Use type hints with appropriate flexibility
def create_request(template_id: str, machine_count: int) -> str:
    """Create a new machine request."""
    pass

# Use flexible typing for CLI argument handling
from typing import Any

def convert_cli_args_to_hostfactory_input(self, operation: str, args: Any) -> Dict[str, Any]:
    """Convert CLI arguments to HostFactory JSON input format.

    Uses Any type for args parameter to support different argument sources
    including argparse.Namespace, dict, or other argument containers.
    """
    pass

# Use dataclasses for DTOs
@dataclass
class RequestDto:
    request_id: str
    template_id: str
    machine_count: int

# Use enums for constants
class RequestStatus(Enum):
    PENDING = "PENDING"
    IN_PROGRESS = "IN_PROGRESS"
    COMPLETED = "COMPLETED"
    FAILED = "FAILED"

# Use dependency injection
class RequestService:
    def __init__(self, repository: RequestRepository):
        self._repository = repository

Naming Conventions

Classes: PascalCase (RequestService, MachineRepository)
Functions/Methods: snake_case (create_request, get_machine_status)
Variables: snake_case (request_id, machine_count)
Constants: UPPER_SNAKE_CASE (MAX_MACHINE_COUNT, DEFAULT_TIMEOUT)
Private Members: Leading underscore (_repository, _logger)

Documentation Standards

class RequestService:
    """Service for managing machine requests.

    This service provides high-level operations for creating,
    updating, and querying machine requests.
    """

    def create_request(self, template_id: str, machine_count: int) -> str:
        """Create a new machine request.

        Args:
            template_id: ID of the template to use
            machine_count: Number of machines to request

        Returns:
            The ID of the created request

        Raises:
            TemplateNotFoundError: If template doesn't exist
            ValidationError: If parameters are invalid
        """
        pass

Testing Guidelines

Test Structure

class TestRequestService:
    """Test suite for RequestService."""

    @pytest.fixture
    def service(self):
        """Create service instance for testing."""
        repository = Mock(spec=RequestRepository)
        return RequestService(repository)

    def test_create_request_success(self, service):
        """Test successful request creation."""
        # Arrange
        template_id = "template-1"
        machine_count = 2

        # Act
        result = service.create_request(template_id, machine_count)

        # Assert
        assert result is not None
        service._repository.save.assert_called_once()

    def test_create_request_invalid_template(self, service):
        """Test request creation with invalid template."""
        # Arrange
        service._repository.get_template.return_value = None

        # Act & Assert
        with pytest.raises(TemplateNotFoundError):
            service.create_request("invalid-template", 2)

Test Categories

Unit Tests: Test individual components in isolation
Integration Tests: Test component interactions
End-to-End Tests: Test complete workflows
Performance Tests: Test performance characteristics

Running Tests

### Testing Examples

```bash
# Run all tests (default: quick test suite)
make test

# Run specific test file
make test tests/unit/test_request_service.py

# Run specific directory
make test tests/unit

# Run with verbose output
make test -v

# Run specific file with verbose output
make test tests/unit/test_request_service.py -v

# Run tests matching pattern
make test -k "request"

# Run specific test method
make test tests/unit/test_request_service.py -k "test_create_request"

# Run tests with coverage
make test-cov

# Run tests in parallel
make test-parallel

# Run only unit tests
make test-unit

# Run unit tests in specific directory
make test-unit tests/unit/domain

# Run integration tests with pattern
make test-integration -k "workflow"

Debugging

Logging

The application uses structured logging:

from src.infrastructure.logging import get_logger

logger = get_logger(__name__)

def create_request(template_id: str, machine_count: int) -> str:
    logger.info("Creating request", extra={
        "template_id": template_id,
        "machine_count": machine_count
    })

    try:
        # Business logic
        request_id = "req-123"

        logger.info("Request created successfully", extra={
            "request_id": request_id
        })

        return request_id

    except Exception as e:
        logger.error("Failed to create request", extra={
            "template_id": template_id,
            "machine_count": machine_count,
            "error": str(e)
        })
        raise

Debug Configuration

Enable debug mode in configuration:

{
  "logging": {
    "level": "DEBUG",
    "console_enabled": true,
    "format": "detailed"
  },
  "development": {
    "debug_mode": true,
    "auto_reload": true
  }
}

Debugging Tools

# Enable debug logging
export HF_LOG_LEVEL=DEBUG

# Run with Python debugger
python -m pdb -m src.bootstrap

# Use IPython for interactive debugging
pip install ipython
ipython -m src.bootstrap

Performance Optimization

Profiling

# Profile application startup
python -m cProfile -o profile.stats -m src.bootstrap

# Analyze profile results
python -c "import pstats; pstats.Stats('profile.stats').sort_stats('cumulative').print_stats(20)"

# Memory profiling
pip install memory-profiler
python -m memory_profiler src/bootstrap.py

Performance Guidelines

Database Queries: Use appropriate indexes and query optimization
Caching: Implement caching for frequently accessed data
Async Operations: Use async/await for I/O operations
Batch Processing: Process multiple items together when possible
Connection Pooling: Reuse connections to external services

Security Considerations

Secure Coding Practices

# Input validation
def create_request(template_id: str, machine_count: int) -> str:
    if not template_id or not template_id.strip():
        raise ValidationError("template_id is required")

    if machine_count <= 0 or machine_count > 1000:
        raise ValidationError("machine_count must be between 1 and 1000")

# Secure credential handling
import os
from src.infrastructure.config import get_config

def get_aws_credentials():
    # Never hardcode credentials
    config = get_config()
    return {
        'access_key': os.environ.get('AWS_ACCESS_KEY_ID'),
        'secret_key': os.environ.get('AWS_SECRET_ACCESS_KEY'),
        'region': config.aws.region
    }

# SQL injection prevention (using parameterized queries)
def get_requests_by_status(status: str) -> List[Request]:
    query = "SELECT * FROM requests WHERE status = ?"
    return database.execute(query, (status,))

Security Checklist

Input validation on all user inputs
Parameterized database queries
Secure credential storage
Appropriate error handling (don't leak sensitive info)
Access control and authorization
Secure communication (HTTPS, TLS)
Regular dependency updates

Contributing

Release Management

The project uses automated release management with semantic versioning:

# Create releases
make release-minor-alpha     # Start new feature
make promote-stable         # Final release

# Test without changes
DRY_RUN=true make release-patch

For complete release documentation, see Release Management Guide.

Pull Request Process

Fork the Repository

git clone <your-fork-url>
cd open-resource-broker

Create Feature Branch
```
git checkout -b feature/your-feature
```
Make Changes
- Follow coding standards
- Write tests
- Update documentation
Test Changes
```
make test
make ci-quality
```

Commit Changes

git add .
git commit -m "feat: add your feature"

Push and Create PR

git push origin feature/your-feature
# Create pull request on GitHub

Commit Message Format

Use conventional commits format:

type(scope): description

[optional body]

[optional footer]

Types:

feat: New feature
fix: Bug fix
docs: Documentation changes
style: Code style changes
refactor: Code refactoring
test: Test changes
chore: Build/tooling changes

Examples:

feat(request): add request priority support
fix(aws): handle EC2 throttling errors
docs(api): update API documentation
refactor(domain): simplify request aggregate

Code Review Guidelines

For Authors

Keep PRs small and focused
Write clear commit messages
Include tests for new functionality
Update documentation
Respond to feedback promptly

For Reviewers

Review for correctness and design
Check test coverage
Verify documentation updates
Consider performance implications
Be constructive in feedback

Troubleshooting

Common Development Issues

Import Errors

# Check Python path
python -c "import sys; print('\n'.join(sys.path))"

# Install in development mode
pip install -e .

# Check for circular imports
python -m src.bootstrap --check-imports

Test Failures

# Run tests with verbose output
make test -v

# Run specific failing test
make test tests/test_specific.py -k "test_method" -v

# Check test dependencies
pip list | grep pytest

Configuration Issues

# Validate configuration
python -m src.infrastructure.config.validate

# Check environment variables
env | grep HF_

# Test configuration loading
python -c "from src.infrastructure.config import get_config; print(get_config())"

Database Issues

# Check database connectivity
python -m src.infrastructure.persistence.database.check_connection

# Reset database
rm data/database.db
python -m src.infrastructure.persistence.database.init_db

# Check database schema
python -m src.infrastructure.persistence.database.show_schema

Getting Help

Check Documentation: Review relevant documentation sections
Search Issues: Look for similar issues in the repository
Ask Questions: Create an issue with the "question" label
Join Discussions: Participate in repository discussions

Next Steps

Architecture: Understand the system architecture
CQRS: Learn about command and query patterns
Events: Understand the event system
Providers: Learn about provider integration
API Reference: Explore the API documentation

FilesExpand file tree

development.md

Latest commit

History

development.md

File metadata and controls

Development Guide

Development Environment Setup

Prerequisites

Local Development Setup

Installing from Local Wheel (Testing)

Optional Dependencies

Development Configuration

Project Structure

High-Level Architecture

Domain Layer (src/domain/)

Application Layer (src/application/)

Infrastructure Layer (src/infrastructure/)

Provider Layer (src/providers/)

Development Workflow

Feature Development

Code Style and Standards

Python Code Style

Naming Conventions

Documentation Standards

Testing Guidelines

Test Structure

Test Categories

Running Tests

Debugging

Logging

Debug Configuration

Debugging Tools

Performance Optimization

Profiling

Performance Guidelines

Security Considerations

Secure Coding Practices

Security Checklist

Contributing

Release Management

Pull Request Process

Commit Message Format

Code Review Guidelines

For Authors

For Reviewers

Troubleshooting

Common Development Issues

Import Errors

Test Failures

Configuration Issues

Database Issues

Getting Help

Next Steps

Domain Layer (`src/domain/`)

Application Layer (`src/application/`)

Infrastructure Layer (`src/infrastructure/`)

Provider Layer (`src/providers/`)