|
| 1 | +# CLAUDE.md |
| 2 | + |
| 3 | +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
| 4 | + |
| 5 | +## Repository Overview |
| 6 | + |
| 7 | +This is the Dagster Community Integrations repository - a collection of community-built and maintained integrations for the Dagster data orchestration platform. The repository provides a centralized place for custom integrations without requiring individual maintainers to manage their own build and release processes. |
| 8 | + |
| 9 | +## Repository Structure |
| 10 | + |
| 11 | +- `libraries/` - Contains all integration packages (e.g., dagster-anthropic, dagster-polars, etc.) |
| 12 | +- `libraries/_template/` - Template for creating new integrations |
| 13 | +- `pipes/implementations/` - Dagster Pipes implementations for Java, Rust, and TypeScript |
| 14 | + |
| 15 | +## Development Commands |
| 16 | + |
| 17 | +All integrations use standardized Makefiles with these commands: |
| 18 | + |
| 19 | +```bash |
| 20 | +# Install dependencies |
| 21 | +make install # or: uv sync |
| 22 | + |
| 23 | +# Build package |
| 24 | +make build # or: uv build |
| 25 | + |
| 26 | +# Run tests |
| 27 | +make test # or: uv run pytest |
| 28 | + |
| 29 | +# Format and lint code |
| 30 | +make ruff # Runs: uv run ruff check --fix && uv run ruff format |
| 31 | + |
| 32 | +# Type checking |
| 33 | +make check # Runs: uv run pyright |
| 34 | +``` |
| 35 | + |
| 36 | +### Running Individual Tests |
| 37 | + |
| 38 | +```bash |
| 39 | +# Run a specific test file |
| 40 | +uv run pytest dagster_<package>_tests/test_specific.py |
| 41 | + |
| 42 | +# Run a specific test function |
| 43 | +uv run pytest dagster_<package>_tests/test_specific.py::test_function_name |
| 44 | + |
| 45 | +# Run tests with verbose output |
| 46 | +uv run pytest -v |
| 47 | +``` |
| 48 | + |
| 49 | +## Creating New Integrations |
| 50 | + |
| 51 | +1. Navigate to `libraries/` directory |
| 52 | +2. Copy the template: `cp -r _template dagster-<my-package>` |
| 53 | +3. Replace all instances of `example-integration` with your package name |
| 54 | +4. Update version in `dagster_<package>/__init__.py` |
| 55 | +5. Create GitHub Actions workflows from templates in `.github/workflows/template-*` |
| 56 | + |
| 57 | +## Code Quality Standards |
| 58 | + |
| 59 | +- **Python Version**: Default to Python 3.11 |
| 60 | +- **Package Management**: Use `uv` exclusively (never pip, poetry, or conda) |
| 61 | +- **Formatting**: Enforced with Ruff (`ruff format`) |
| 62 | +- **Linting**: Enforced with Ruff (`ruff check --fix`) |
| 63 | +- **Type Checking**: Required with Pyright (`pyright`) |
| 64 | +- **Testing**: Required with pytest |
| 65 | + |
| 66 | +## Release Process |
| 67 | + |
| 68 | +```bash |
| 69 | +# Create a release tag |
| 70 | +./release.sh dagster-<package> X.X.X |
| 71 | + |
| 72 | +# Tags follow pattern: <integration-name>-X.X.X |
| 73 | +``` |
| 74 | + |
| 75 | +## Integration Architecture |
| 76 | + |
| 77 | +Each integration follows this structure: |
| 78 | +``` |
| 79 | +dagster-<package>/ |
| 80 | +├── Makefile # Standard development commands |
| 81 | +├── README.md # Documentation with usage examples |
| 82 | +├── pyproject.toml # Package configuration and dependencies |
| 83 | +├── uv.lock # Locked dependencies |
| 84 | +├── dagster_<package>/ # Main package code |
| 85 | +│ ├── __init__.py # Contains __version__ |
| 86 | +│ └── ... # Integration modules |
| 87 | +└── dagster_<package>_tests/ |
| 88 | + └── test_*.py # Test files |
| 89 | +``` |
| 90 | + |
| 91 | +### Key Architectural Patterns |
| 92 | + |
| 93 | +1. **Dagster Resources**: Integrations typically implement Dagster resources for external services |
| 94 | +2. **IO Managers**: For data storage integrations, implement custom IO managers |
| 95 | +3. **Ops and Assets**: Provide pre-built ops and assets for common operations |
| 96 | +4. **Type Annotations**: Use type hints throughout the codebase |
| 97 | +5. **Error Handling**: Wrap external API calls with appropriate error handling |
| 98 | + |
| 99 | +## Testing Local Changes |
| 100 | + |
| 101 | +```bash |
| 102 | +# Run GitHub Actions locally with act |
| 103 | +act -W .github/workflows/quality-check-dagster-<package>.yml |
| 104 | + |
| 105 | +# For testcontainers support |
| 106 | +act --env TESTCONTAINERS_HOST_OVERRIDE=`ipconfig getifaddr en0` -W .github/workflows/quality-check-dagster-<package>.yml |
| 107 | +``` |
| 108 | + |
| 109 | +## Important Notes |
| 110 | + |
| 111 | +- All tests, formatting, and type checking must pass before merging |
| 112 | +- Follow existing patterns in similar integrations |
| 113 | +- Include comprehensive documentation with usage examples |
| 114 | +- Version is managed in `__init__.py` as `__version__ = "X.X.X"` |
| 115 | +- Dependencies are specified in `pyproject.toml` and locked with `uv.lock` |
0 commit comments