diff --git a/.agents/skills/_legacy/dignified-python-local/SKILL.md b/.agents/skills/_legacy/dignified-python-local/SKILL.md new file mode 100644 index 0000000..8a36f9b --- /dev/null +++ b/.agents/skills/_legacy/dignified-python-local/SKILL.md @@ -0,0 +1,294 @@ +--- +name: dignified-python-313 +description: This skill should be used when editing Python code in the erk codebase. Use when writing, reviewing, or refactoring Python to ensure adherence to LBYL exception handling patterns, Python 3.13+ type syntax (list[str], str | None), pathlib operations, ABC-based interfaces, absolute imports, and explicit error boundaries at CLI level. Also provides production-tested code smell patterns from Dagster Labs for API design, parameter complexity, and code organization. Essential for maintaining erk's dignified Python standards. +--- + +# Dignified Python - Python 3.13+ Coding Standards + +Write explicit, predictable code that fails fast at proper boundaries. + +--- + +## Quick Reference - Check Before Coding + +| If you're about to write... | Check this rule | +| --------------------------------------------- | -------------------------------------------------------------------------------------------------------- | +| `try:` or `except:` | → [Exception Handling](#1-exception-handling---never-for-control-flow-) - Default: let exceptions bubble | +| `from __future__ import annotations` | → **FORBIDDEN** - Python 3.13+ doesn't need it | +| `List[...]`, `Dict[...]`, `Union[...]` | → Use `list[...]`, `dict[...]`, `X \| Y` | +| `dict[key]` without checking | → Use `if key in dict:` or `.get()` | +| `path.resolve()` or `path.is_relative_to()` | → Check `path.exists()` first | +| `typing.Protocol` | → Use `abc.ABC` instead | +| `from .module import` | → Use absolute imports only | +| `__all__ = ["..."]` in `__init__.py` | → See references/core-standards.md#code-in-**init**py-and-**all**-exports | +| `print(...)` in CLI code | → Use `click.echo()` | +| `subprocess.run(...)` | → Add `check=True` | +| `@property` with I/O or expensive computation | → See references/core-standards.md#performance-expectations | +| Function with many optional parameters | → See references/code-smells-dagster.md | +| `repr()` for sorting or hashing | → See references/code-smells-dagster.md | +| Context object passed everywhere | → See references/code-smells-dagster.md | +| Function with 10+ local variables | → See references/code-smells-dagster.md | +| Class with 50+ methods | → See references/code-smells-dagster.md | + +--- + +## CRITICAL RULES (Top 6) + +### 1. Exception Handling - NEVER for Control Flow 🔴 + +**ALWAYS use LBYL (Look Before You Leap), NEVER EAFP** + +```python +# ✅ CORRECT: Check before acting +if key in mapping: + value = mapping[key] +else: + handle_missing_key() + +# ❌ WRONG: Using exceptions for control flow +try: + value = mapping[key] +except KeyError: + handle_missing_key() +``` + +**Details**: See `references/core-standards.md#exception-handling` for complete patterns + +### 2. Type Annotations - Python 3.13+ Syntax Only 🔴 + +**FORBIDDEN**: `from __future__ import annotations` + +```python +# ✅ CORRECT: Modern Python 3.13+ syntax +def process(items: list[str]) -> dict[str, int]: ... +def find_user(id: int) -> User | None: ... + +# ❌ WRONG: Legacy syntax +from typing import List, Dict, Optional +def process(items: List[str]) -> Dict[str, int]: ... +``` + +**Details**: See `references/core-standards.md#type-annotations` for all patterns + +### 3. Path Operations - Check Exists First 🔴 + +```python +# ✅ CORRECT: Check exists first +if path.exists(): + resolved = path.resolve() + +# ❌ WRONG: Using exceptions +try: + resolved = path.resolve() +except OSError: + pass +``` + +**Details**: See `references/core-standards.md#path-operations` + +### 4. Dependency Injection - ABC Not Protocol 🔴 + +```python +# ✅ CORRECT: Use ABC +from abc import ABC, abstractmethod + +class MyOps(ABC): + @abstractmethod + def operation(self) -> None: ... + +# ❌ WRONG: Using Protocol +from typing import Protocol +``` + +**Details**: See `references/core-standards.md#dependency-injection` + +### 5. Imports - Module-Level and Absolute 🔴 + +**ALL imports must be at module level unless preventing circular imports** + +```python +# ✅ CORRECT: Module-level, absolute imports +from erk.config import load_config +from pathlib import Path +import click + +# ❌ WRONG: Inline imports (unless for circular import prevention) +def my_function(): + from erk.config import load_config # WRONG unless circular import + return load_config() + +# ❌ WRONG: Relative imports +from .config import load_config +``` + +**Exception**: Inline imports are ONLY acceptable when preventing circular imports. Always document why: + +```python +def create_context(): + # Inline import to avoid circular dependency with tests + from tests.fakes.gitops import FakeGitOps + return FakeGitOps() +``` + +**Details**: See `references/core-standards.md#imports` + +### 6. No Silent Fallback Behavior 🔴 + +```python +# ❌ WRONG: Silent fallback +try: + result = primary_method() +except: + result = fallback_method() # Untested, brittle + +# ✅ CORRECT: Let error bubble up +result = primary_method() +``` + +**Details**: See `references/core-standards.md#anti-patterns` + +--- + +## When to Load References + +### Load `references/core-standards.md` when: + +- Writing exception handling code (LBYL patterns) +- Working with type annotations (Python 3.13+ syntax) +- Implementing path operations (exists() checks) +- Creating ABC interfaces (dependency injection) +- Organizing imports (absolute imports, module-level) +- Working with CLI code (Click patterns) +- Using dataclasses and immutability +- Avoiding anti-patterns (silent fallback, exception swallowing) +- Implementing `@property` or `__len__` (performance expectations) + +### Load `references/code-smells-dagster.md` when: + +- Designing function APIs (default parameters, keyword arguments) +- Managing parameter complexity (parameter anxiety, invalid combinations) +- Refactoring large functions/classes (god classes, local variables) +- Working with context managers (assignment patterns) +- Using `repr()` programmatically (string representation abuse) +- Passing context objects (context coupling) +- Dealing with error boundaries (early validation) + +### Load `references/patterns-reference.md` when: + +- Developing CLI commands with Click +- Working with file I/O and pathlib +- Implementing dataclasses and frozen structures +- Managing subprocess operations +- Reducing code nesting (early returns, helper functions) + +--- + +## Progressive Disclosure Guide + +This skill uses a three-level loading system: + +1. **This file (SKILL.md)**: Core rules and navigation (~350 lines) +2. **Reference files**: Detailed patterns and examples (loaded as needed) +3. **Quick lookup**: Use the tables above to find what you need + +The agent loads reference files only when needed based on the current task. The reference files contain: + +- **`core-standards.md`**: Foundational Python patterns from this skill +- **`code-smells-dagster.md`**: Production-tested anti-patterns from Dagster Labs +- **`patterns-reference.md`**: Common implementation patterns and examples + +--- + +## Philosophy + +**Write dignified Python code that:** + +- Fails fast at proper boundaries (not deep in the stack) +- Makes invalid states unrepresentable (use the type system) +- Expresses intent clearly (LBYL over EAFP) +- Minimizes cognitive load (explicit over implicit) +- Enables confident refactoring (test what you build) + +**Default stances:** + +- Let exceptions bubble up (handle at boundaries only) +- Break APIs and migrate immediately (no unnecessary backwards compatibility) +- Check conditions proactively (LBYL) +- Use modern Python 3.13+ syntax + +--- + +## Quick Decision Tree + +**About to write Python code?** + +1. **Using `try/except`?** + - Can you use LBYL instead? → Do that + - Is this an error boundary? → OK to handle + - Otherwise → Let it bubble + +2. **Using type hints?** + - Use `list[str]`, `str | None`, not `List`, `Optional` + - NO `from __future__ import annotations` + +3. **Working with paths?** + - Check `.exists()` before `.resolve()` + - Use `pathlib.Path`, not `os.path` + +4. **Writing CLI code?** + - Use `click.echo()`, not `print()` + - Exit with `raise SystemExit(1)` + +5. **Too many parameters?** + - See `references/code-smells-dagster.md#parameter-anxiety` + +6. **Class getting large?** + - See `references/code-smells-dagster.md#god-classes` + +--- + +## Checklist Before Writing Code + +Before writing `try/except`: + +- [ ] Can I check the condition proactively? (LBYL) +- [ ] Is this at an error boundary? (CLI/API level) +- [ ] Am I adding meaningful context or just hiding the error? + +Before using type hints: + +- [ ] Am I using Python 3.13+ syntax? (`list`, `dict`, `|`) +- [ ] Have I removed all `typing` imports except essentials? + +Before path operations: + +- [ ] Did I check `.exists()` before `.resolve()`? +- [ ] Am I using `pathlib.Path`? +- [ ] Did I specify `encoding="utf-8"`? + +Before adding backwards compatibility: + +- [ ] Did the user explicitly request it? +- [ ] Is this a public API? +- [ ] Default: Break and migrate immediately + +--- + +## Common Patterns Summary + +| Scenario | Preferred Approach | Avoid | +| --------------------- | ----------------------------------------- | ------------------------------------------- | +| **Dictionary access** | `if key in dict:` or `.get(key, default)` | `try: dict[key] except KeyError:` | +| **File existence** | `if path.exists():` | `try: open(path) except FileNotFoundError:` | +| **Type checking** | `if isinstance(obj, Type):` | `try: obj.method() except AttributeError:` | +| **Value validation** | `if is_valid(value):` | `try: process(value) except ValueError:` | +| **Path resolution** | `if path.exists(): path.resolve()` | `try: path.resolve() except OSError:` | + +--- + +## References + +- **Core Standards**: `references/core-standards.md` - Detailed LBYL patterns, type annotations, imports +- **Code Smells**: `references/code-smells-dagster.md` - Production-tested anti-patterns +- **Pattern Reference**: `references/patterns-reference.md` - CLI, file I/O, dataclasses +- Python 3.13 docs: https://docs.python.org/3.13/ diff --git a/.agents/skills/context-doc-maintainer/SKILL.md b/.agents/skills/context-doc-maintainer/SKILL.md new file mode 100644 index 0000000..a64fd75 --- /dev/null +++ b/.agents/skills/context-doc-maintainer/SKILL.md @@ -0,0 +1,50 @@ +--- +name: context-doc-maintainer +description: Automated agent workflow for reviewing and updating project agent context files when code changes are made. +license: MIT +--- + +# Context Documentation Maintainer + +Use this skill when code changes require updates to agent-facing context files, project maps, skill instructions, PR templates, or repository guidance under `.agents/`. + +## Scope + +- Keep `.agents/` context aligned with current architecture, commands, and conventions. +- Update skill files only when code changes alter the workflow the skill describes. +- Preserve concise, task-oriented context; avoid duplicating detailed docs already available elsewhere. +- Do not update generated artifacts, secrets, local environment files, or unrelated documentation. + +## Workflow + +1. **Inspect changes** + - Run `git status --short`. + - Compare against the PR base with `git diff ...HEAD --stat` and `git diff ...HEAD --name-status` when a base is known. + - Categorize changes by domain: Dagster, dbt, API, frontend, tests, infrastructure, or docs. + +2. **Map context impact** + - Update `.agents/skills/*/SKILL.md` when a workflow, command, path, or convention changed. + - Update `.agents/skills/*/references/*.md` when detailed domain guidance changed. + - Update `.agents/AGENTS.md` or repo-level `AGENTS.md` only for broad project conventions or architecture shifts. + - Update `.agents/skills/context-doc-maintainer/templates/pr_template.md` only when the PR handoff structure changes. + +3. **Edit context** + - Keep each `SKILL.md` lean and directly actionable. + - Prefer linking to reference files over embedding long examples. + - Use `.agents/` paths for project agent context. + - Avoid stale product-specific wording unless the skill is explicitly about that product. + +4. **Validate** + - Check frontmatter includes `name` and `description`. + - Search for stale product/path references with ripgrep before handoff. + - Check for prohibited auxiliary docs: `find .agents/skills -name README.md -o -name CHANGELOG.md -o -name INSTALLATION_GUIDE.md -o -name QUICK_REFERENCE.md`. + - Run `git diff --check`. + +5. **Handoff** + - Summarize which context files changed and why. + - Call out validation that could not run and the exact blocker. + - Do not create commits, PRs, or merges unless the user explicitly asks. + +## PR Template + +Use `templates/pr_template.md` when creating a documentation-only PR for context updates. diff --git a/.agents/skills/context-doc-maintainer/templates/pr_template.md b/.agents/skills/context-doc-maintainer/templates/pr_template.md new file mode 100644 index 0000000..879c75c --- /dev/null +++ b/.agents/skills/context-doc-maintainer/templates/pr_template.md @@ -0,0 +1,36 @@ +# Update Agent Context Documentation + +## Summary + +Automated documentation updates based on recent code changes. + +## Code Changes Analyzed + +**Commit Range**: {{BASE_BRANCH}}...{{CURRENT_BRANCH}} +**Files Changed**: {{FILES_CHANGED}} +**Additions**: +{{ADDITIONS}} lines +**Deletions**: -{{DELETIONS}} lines + +### Key Code Changes + +{{CHANGE_SUMMARY}} + +## Documentation Updates + +{{UPDATE_DETAILS}} + +## Validation Results + +{{VALIDATION_SUMMARY}} + +## Review Checklist + +- [ ] Documentation accurately reflects code changes +- [ ] No sensitive information exposed +- [ ] Markdown formatting is correct +- [ ] Section placements are appropriate +- [ ] Project structure tree matches filesystem + +--- + +🤖 Generated with an agent-assisted workflow. diff --git a/.agents/skills/create-custom-dagster-component/SKILL.md b/.agents/skills/create-custom-dagster-component/SKILL.md new file mode 100644 index 0000000..8a8f560 --- /dev/null +++ b/.agents/skills/create-custom-dagster-component/SKILL.md @@ -0,0 +1,439 @@ +--- +name: create-custom-dagster-component +description: Create a custom Dagster Component with demo mode support, realistic asset structure, and optional custom scaffolder using the dg CLI. Use this skill if there is no Component included in an existing integration or if Dagster does not have the integration. +license: MIT +--- + +# Create a custom Component + +## Overview + +This skill automates the creation and validation of a new custom Dagster component using the `dg` CLI tool with uv as a package manager. It incorporates demo mode functionality for creating realistic demonstrations that can run locally without external dependencies. The documentation for creating good components can be found here https://docs.dagster.io/guides/build/components/creating-new-components/creating-and-registering-a-component and here https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/components/dbt_project/component.py for a complex example of a component. + +## What This Skill Does + +When invoked, this skill will: + +1. ✅ Create a new Dagster Component project using `dg scaffold component ComponentName` +2. ✅ Fills in the component logic in the `build_defs()` function with both real and demo mode implementations +3. ✅ Implement a `demo_mode` boolean flag in the component YAML for toggling between real and local demo implementations +4. ✅ Create 3-5 realistic assets with proper dependencies and technology kinds +5. ✅ Instantiate a component YAML and fill it in using the `dg scaffold defs my_module.components.ComponentName my_component` command +6. ✅ Optionally create a custom scaffolder if requested +7. ✅ Validate that the component loaded correctly using `dg check defs` and `dg list defs` to ensure that the expected component instances are all loaded. +8. ✅ Provide clear next steps for development + +## Prerequisites + +Before running this skill, ensure: + +- `uv` is installed (check with `uv --version`) +- You have a component name in mind (or will use the default) +- You're in a Dagster project directory with the dg CLI available +- You have inspected and understand how to create multi-asset integrations (see this guide: https://docs.dagster.io/integrations/guides/multi-asset-integration) + +## Skill Workflow + +### Step 1: Get Component Name and Demo Mode Preference + +Ask the user for: + +1. A component name, or use a sensible default like `MyDagsterComponent`. Validate that: + - The name starts with a letter + - Contains only alphanumeric characters, hyphens, or underscores + - The component doesn't already exist (or ask to overwrite) +2. Whether they want demo mode support (default: yes for demonstration projects) +3. Whether they want to create a custom scaffolder (see Step 5) + +### Step 2: Create Component + +Use `dg` to create the component + +```bash +uv run dg scaffold component +``` + +This will: + +- Scaffold a new Dagster Component in `defs/components` +- Create a `component_name.py` file + +### Step 3: Implement Component Logic with Demo Mode + +Fill in the `build_defs()` function in the component file. The component should: + +1. **Accept a `demo_mode` parameter** in the component params (default: False) +2. **Create 3-5 realistic assets** based on the chosen technologies +3. **Implement dual logic paths:** + - Real implementation: connects to actual systems (the body should include connections to real systems) + - Demo mode: uses local data/mocked behavior for demonstrations + - Important Configuration in Pydantic + YAML fields: All configuration (a new pipeline or asset) should be configurable in the Component and _not_ hard coded in the Component Python. **Demo mode assets are the exception that _should_ be hard coded.** + - All Resources should be configured outside of the Component in the `defs/` folder in a `resources.py` file, using `dg scaffold defs dagster.resource resources.py` + - All Components that invoke a pipeline or API that might trigger multiple assets should allow for an `assets` field in the YAML that describes what assets are used in the underlying component. See https://dagster.io/blog/dsls-to-the-rescue for best practices in how to design a good DSL. Refer to https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/components/dbt_project/component.py and https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/components/workspace_component/component.py for two reference architectures for good component design with mutli-assets. +4. **Set proper asset metadata:** + - Use the `kinds` argument to indicate technologies in use + - Add descriptive names and documentation + - Establish proper dependencies between assets +5. **Design asset keys for downstream integration** (CRITICAL): + - Consider what components will consume your assets + - Choose key structures that minimize downstream configuration + - See "Design Asset Keys for Integration" section below + +Example asset structure: + +- Raw data ingestion asset +- Data transformation/cleaning asset +- Business logic/aggregation asset +- ML model or analytics asset (if applicable) +- Output/export asset + +### Design Asset Keys for Integration + +**CRITICAL:** When creating a custom component, consider **what will consume** your component's assets. The asset keys you generate should align with downstream component expectations to avoid requiring per-asset configuration. + +#### Key Principle: Upstream Defines, Downstream Consumes + +Your component (upstream) should generate asset keys in a structure that downstream components naturally reference. This eliminates the need for `meta.dagster.asset_key` or complex translation configuration. + +#### Common Downstream Consumers + +**If dbt will consume your assets:** +- Use pattern: `["", ""]` +- Example: `["fivetran_raw", "customers"]` or `["api_raw", "users"]` +- This allows dbt sources to reference naturally: `source('fivetran_raw', 'customers')` + +**If custom Dagster assets will consume them:** +- Match the key structure those assets expect in their `deps` +- Minimize nesting when possible (prefer 2 levels: `["category", "name"]`) +- Avoid deeply nested keys like `["system", "subsystem", "type", "name"]` unless necessary + +**If another integration component will consume them:** +- Check that component's expected input key structure +- Align your keys to match, or provide clear mapping documentation + +**If your assets are intermediate and consumed by your own component:** +- Use clear, hierarchical keys that reflect data flow +- Example: `["raw", "table"]` → `["processed", "table"]` → `["enriched", "table"]` + +#### Example: Creating an API Ingestion Component + +```python +import dagster as dg + +class APIIngestionComponent(dg.Component, dg.Model, dg.Resolvable): + """Ingests data from REST APIs.""" + + api_endpoint: str + tables: list[str] + demo_mode: bool = False + + def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions: + assets = [] + + for table in self.tables: + # Design key for dbt consumption: ["api_raw", "table_name"] + # NOT: ["api", "ingestion", "raw", "table_name"] + @dg.asset( + key=dg.AssetKey(["api_raw", table]), # ← Flattened for easy downstream reference + kinds={"api", "python"}, + ) + def ingest_table(context: dg.AssetExecutionContext): + if self.demo_mode: + context.log.info(f"Demo mode: Mocking API call for {table}") + return {"status": "demo", "rows": 100} + else: + # Real API call + pass + + assets.append(ingest_table) + + return dg.Definitions(assets=assets) +``` + +**Result:** dbt can reference these assets naturally: +```yaml +# sources.yml +sources: + - name: api_raw + tables: + - name: customers # Matches ["api_raw", "customers"] +``` + +#### Verification + +Always verify asset keys align with downstream dependencies: + +```bash +# Check asset keys and their dependencies +uv run dg list defs --json | uv run python -c " +import sys, json +assets = json.load(sys.stdin)['assets'] +print('\\n'.join([f\"{a['key']}: deps={a.get('deps', [])}\" for a in assets])) +" +``` + +**What to verify:** +- Downstream assets list your assets in their `deps` array +- No duplicate keys with different structures +- Keys are simple and descriptive (typically 2 levels: `["category", "name"]`) + +#### Anti-Patterns to Avoid + +❌ **Too deeply nested:** `["company", "team", "project", "environment", "table"]` +- Hard for downstream to reference +- Requires complex mapping + +❌ **Inconsistent structure:** Some assets with 2 levels, others with 4 +- Confusing for consumers +- Unpredictable references + +❌ **Generic names:** `["data", "table1"]`, `["output", "result"]` +- Not clear what system they're from +- Conflicts with other components + +✅ **Good patterns:** +- `["source_system", "entity"]`: `["fivetran_raw", "customers"]` +- `["integration", "object"]`: `["salesforce", "accounts"]` +- `["stage", "table"]`: `["staging", "orders"]` + +#### Critical: Asset Keys Must Be Identical in Demo and Production Mode + +**IMPORTANT:** Asset keys should be **exactly the same** whether `demo_mode` is True or False. Only the asset implementation (the function body) should differ between modes. + +**Why this matters:** +- Downstream components reference assets by key +- Dependencies are established based on keys +- If keys differ between modes, dependencies break when switching modes +- Testing in demo mode won't accurately reflect production behavior + +**Example - CORRECT approach:** + +```python +def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions: + @dg.asset( + key=dg.AssetKey(["fivetran_raw", "customers"]), # ← Same key in both modes + kinds={"fivetran"}, + ) + def customers_sync(context: dg.AssetExecutionContext): + if self.demo_mode: + # Demo implementation - mock data + context.log.info("Demo mode: Creating empty table") + # ... create mock table + else: + # Production implementation - real Fivetran sync + context.log.info("Production: Syncing from Fivetran") + # ... call Fivetran API + + return dg.Definitions(assets=[customers_sync]) +``` + +**Example - INCORRECT approach:** + +```python +def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions: + if self.demo_mode: + @dg.asset( + key=dg.AssetKey(["demo", "customers"]), # ❌ Different key! + ) + def demo_customers(): + pass + return dg.Definitions(assets=[demo_customers]) + else: + @dg.asset( + key=dg.AssetKey(["fivetran_raw", "customers"]), # ❌ Different key! + ) + def prod_customers(): + pass + return dg.Definitions(assets=[prod_customers]) +``` + +**Reference Documentation:** + +- Cross-reference https://docs.dagster.io/llms.txt for up-to-date titles and descriptions +- Use https://docs.dagster.io/llms-full.txt for full API details +- Check available integrations with: + +```bash +uv run dg docs integrations --json +``` + +## Important: Always Add Kinds to Assets + +When creating assets in your component, **ALWAYS add the `kinds` parameter** to properly categorize assets by their technology/integration type. This helps with: +- Filtering and organizing assets in the Dagster UI +- Understanding the technology stack at a glance +- Grouping assets by integration type + +**Common integration kinds:** +- `kinds={"fivetran"}` for Fivetran assets +- `kinds={"dbt"}` for dbt assets +- `kinds={"census"}` for Census assets +- `kinds={"sling"}` for Sling assets +- `kinds={"powerbi"}` for PowerBI assets +- `kinds={"looker"}` for Looker assets +- `kinds={"airbyte"}` for Airbyte assets +- `kinds={"python"}` for custom Python processing +- `kinds={"snowflake"}` for Snowflake assets + +You can verify kinds are showing correctly by running: +```bash +uv run dg list defs +``` +The "Kinds" column should show the integration type for each asset. + +**Example Component Structure:** + +```python +from dagster import asset, Definitions, AssetExecutionContext +from pydantic import BaseModel + +class MyComponentParams(BaseModel): + demo_mode: bool = False + # ... other params + +class MyComponent(Component): + params_schema = MyComponentParams + + def build_defs(self, context: ComponentLoadContext) -> Definitions: + params = self.params + + @asset( + kinds={"fivetran"}, # ← REQUIRED: Add the integration kind + ) + def raw_data(context: AssetExecutionContext): + if params.demo_mode: + # Demo implementation - local/mocked data + context.log.info("Running in demo mode with local data") + pass + else: + # Real implementation - connect to actual systems + context.log.info("Running with real data source") + pass + + @asset( + deps=[raw_data], + kinds={"dbt"}, # ← REQUIRED: Add the integration kind + ) + def processed_data(context: AssetExecutionContext): + if params.demo_mode: + context.log.info("Processing demo data") + pass + else: + context.log.info("Processing real data") + pass + + # ... more assets + + return Definitions(assets=[raw_data, processed_data, ...]) +``` + +### Step 4: Create Component Instance YAML + +Use `dg scaffold defs` to create the component instance: + +```bash +uv run dg scaffold defs my_module.components.ComponentName my_component +``` + +This creates a YAML file that should include the `demo_mode` parameter: + +```yaml +type: my_module.components.ComponentName +attributes: + demo_mode: true # Set to true for local demos, false for real deployments + # ... other params +``` + +### Step 5: Create Custom Scaffolder (Optional) + +If the user requested a custom scaffolder in Step 1, follow the directions here: +https://docs.dagster.io/guides/build/components/creating-new-components/component-customization#customizing-scaffolding-behavior + +Customize the scaffolder to provide a better developer experience for creating instances of this component. + +### Step 6: Validate Setup and Asset Key Alignment + +Run these commands to ensure everything works: + +```bash +# Check that definitions load without errors +uv run dg check defs + +# List all assets to verify they were created +uv run dg list defs +``` + +Verify that: + +- ✅ All expected assets are listed +- ✅ The component instance is properly configured +- ✅ No errors or warnings are shown +- ✅ The `demo_mode` flag toggles between implementations correctly +- ✅ The `demo_mode: false` implementation uses realistic resources and is a production implementation + +**CRITICAL: Verify Asset Key Alignment** + +Check that asset dependencies are correct by running: + +```bash +uv run dg list defs --json | uv run python -c " +import sys, json +data = json.load(sys.stdin) +assets = data.get('assets', []) +print('Asset Dependencies:\n') +for asset in assets: + key = asset.get('key', 'unknown') + deps = asset.get('deps', []) + if deps: + print(f'{key}') + for dep in deps: + print(f' ← {dep}') + else: + print(f'{key} (no dependencies)') + print() +" +``` + +**What to verify:** +- ✅ Downstream assets list upstream assets in their `deps` array +- ✅ No missing dependencies +- ✅ Asset keys are simple and descriptive (typically 2 levels: `["category", "name"]`) +- ✅ Asset keys work consistently in both demo mode and production mode + +**Key Principle:** Asset keys should be **identical** between demo mode and production mode. Only the asset implementation (the function body) should differ. This ensures: +- Dependencies work the same in both modes +- You can switch between modes without reconfiguring downstream components +- Testing in demo mode accurately reflects production behavior + +### Step 7: Test Demo Mode + +If demo mode was implemented: + +1. Ensure the component YAML has `demo_mode: true` +2. Run `dg check defs` to verify it works locally +3. Document how to switch between demo and real modes + +## Success Criteria + +The component is complete when: + +- ✅ Component scaffolding is created +- ✅ `build_defs()` is implemented with proper asset logic +- ✅ Demo mode flag is working (if applicable) +- ✅ Non demo mode has realistic connections to the database or APIs implemented +- ✅ 3-5 realistic assets are created with proper dependencies +- ✅ Assets have appropriate `kinds` metadata +- ✅ Component YAML instance is created and configured +- ✅ Custom scaffolder is implemented (if requested) +- ✅ `dg check defs` passes without errors +- ✅ `dg list defs` shows all expected assets + +## Next Steps + +After completion, inform the user: + +1. The component has been created and validated +2. Location of the component files +3. How to toggle demo mode (if applicable) +4. How to customize the component further +5. How to create additional instances using the scaffolder \ No newline at end of file diff --git a/.agents/skills/dagster-development/SKILL.md b/.agents/skills/dagster-development/SKILL.md new file mode 100644 index 0000000..7b0991e --- /dev/null +++ b/.agents/skills/dagster-development/SKILL.md @@ -0,0 +1,463 @@ +--- +name: dagster-development +description: Expert guidance for Dagster data orchestration including assets, resources, schedules, sensors, partitions, testing, and ETL patterns. Use when building or extending Dagster projects, writing assets, configuring automation, or integrating with dbt/dlt/Sling. +--- + +# Dagster Development Expert + +This skill is **project-specific** and complements `/dagster-expert`. Use `/dagster-expert` for dg CLI commands and general Dagster concepts, and this skill for patterns specific to this repo. + +## Quick Reference + +| If you're writing... | Check this section/reference | +| ------------------------------------- | ------------------------------------------------------------- | +| `@dg.asset` | [Assets](#assets-quick-reference) or `references/assets.md` | +| `ConfigurableResource` | [Resources](#resources-quick-reference) or `references/resources.md` | +| `@dg.schedule` or `ScheduleDefinition`| [Automation](#automation-quick-reference) or `references/automation.md` | +| `@dg.sensor` | [Sensors](#sensors-quick-reference) or `references/automation.md` | +| `PartitionsDefinition` | [Partitions](#partitions-quick-reference) or `references/automation.md` | +| Tests with `dg.materialize()` | [Testing](#testing-quick-reference) or `references/testing.md` | +| `@asset_check` | `references/testing.md#asset-checks` | +| `@dlt_assets` or `@sling_assets` | `references/etl-patterns.md` | +| `@dbt_assets` | [dbt Integration](#dbt-integration) or `dbt-development` skill | +| `Definitions` or code locations | `references/project-structure.md` | + +--- + +## Core Concepts + +**Asset**: A persistent object (table, file, model) that your pipeline produces. Define with `@dg.asset`. + +**Resource**: External services/tools (databases, APIs) shared across assets. Define with `ConfigurableResource`. + +**Job**: A selection of assets to execute together. Create with `dg.define_asset_job()`. + +**Schedule**: Time-based automation for jobs. Create with `dg.ScheduleDefinition`. + +**Sensor**: Event-driven automation that watches for changes. Define with `@dg.sensor`. + +**Partition**: Logical divisions of data (by date, category). Define with `PartitionsDefinition`. + +**Definitions**: The container for all Dagster objects in a code location. + +--- + +## Assets Quick Reference + +### Basic Asset + +```python +import dagster as dg + +@dg.asset +def my_asset() -> None: + """Asset description appears in the UI.""" + # Your computation logic here + pass +``` + +### Asset with Dependencies + +```python +@dg.asset +def downstream_asset(upstream_asset) -> dict: + """Depends on upstream_asset by naming it as a parameter.""" + return {"processed": upstream_asset} +``` + +### Asset with Metadata + +```python +@dg.asset( + group_name="analytics", + key_prefix=["warehouse", "staging"], + description="Cleaned customer data", +) +def customers() -> None: + pass +``` + +**Naming**: Use nouns describing what is produced (`customers`, `daily_revenue`), not verbs (`load_customers`). + +--- + +## Resources Quick Reference + +### Define a Resource + +```python +from dagster import ConfigurableResource + +class DatabaseResource(ConfigurableResource): + connection_string: str + + def query(self, sql: str) -> list: + # Implementation here + pass +``` + +### Use in Assets + +```python +@dg.asset +def my_asset(database: DatabaseResource) -> None: + results = database.query("SELECT * FROM table") +``` + +### Register in Definitions + +```python +dg.Definitions( + assets=[my_asset], + resources={"database": DatabaseResource(connection_string="...")}, +) +``` + +--- + +## Automation Quick Reference + +### Schedule + +```python +import dagster as dg +from my_project.defs.jobs import my_job + +my_schedule = dg.ScheduleDefinition( + job=my_job, + cron_schedule="0 0 * * *", # Daily at midnight +) +``` + +### Common Cron Patterns + +| Pattern | Meaning | +| ------------- | -------------------------- | +| `0 * * * *` | Every hour | +| `0 0 * * *` | Daily at midnight | +| `0 0 * * 1` | Weekly on Monday | +| `0 0 1 * *` | Monthly on the 1st | +| `0 0 5 * *` | Monthly on the 5th | + +--- + +## Sensors Quick Reference + +### Basic Sensor Pattern + +```python +@dg.sensor(job=my_job) +def my_sensor(context: dg.SensorEvaluationContext): + # 1. Read cursor (previous state) + previous_state = json.loads(context.cursor) if context.cursor else {} + current_state = {} + runs_to_request = [] + + # 2. Check for changes + for item in get_items_to_check(): + current_state[item.id] = item.modified_at + if item.id not in previous_state or previous_state[item.id] != item.modified_at: + runs_to_request.append(dg.RunRequest( + run_key=f"run_{item.id}_{item.modified_at}", + run_config={...} + )) + + # 3. Return result with updated cursor + return dg.SensorResult( + run_requests=runs_to_request, + cursor=json.dumps(current_state) + ) +``` + +**Key**: Use cursors to track state between sensor evaluations. + +--- + +## Partitions Quick Reference + +### Time-Based Partition + +```python +weekly_partition = dg.WeeklyPartitionsDefinition(start_date="2023-01-01") + +@dg.asset(partitions_def=weekly_partition) +def weekly_data(context: dg.AssetExecutionContext) -> None: + partition_key = context.partition_key # e.g., "2023-01-01" + # Process data for this partition +``` + +### Static Partition + +```python +region_partition = dg.StaticPartitionsDefinition(["us-east", "us-west", "eu"]) + +@dg.asset(partitions_def=region_partition) +def regional_data(context: dg.AssetExecutionContext) -> None: + region = context.partition_key +``` + +### Partition Types + +| Type | Use Case | +| ---- | -------- | +| `DailyPartitionsDefinition` | One partition per day | +| `WeeklyPartitionsDefinition` | One partition per week | +| `MonthlyPartitionsDefinition` | One partition per month | +| `StaticPartitionsDefinition` | Fixed set of partitions | +| `MultiPartitionsDefinition` | Combine multiple partition dimensions | + +--- + +## Testing Quick Reference + +### Direct Function Testing + +```python +def test_my_asset(): + result = my_asset() + assert result == expected_value +``` + +### Testing with Materialization + +```python +def test_asset_graph(): + result = dg.materialize( + assets=[asset_a, asset_b], + resources={"database": mock_database}, + ) + assert result.success + assert result.output_for_node("asset_b") == expected +``` + +### Mocking Resources + +```python +from unittest.mock import Mock + +def test_with_mocked_resource(): + mocked_resource = Mock() + mocked_resource.query.return_value = [{"id": 1}] + + result = dg.materialize( + assets=[my_asset], + resources={"database": mocked_resource}, + ) + assert result.success +``` + +### Asset Checks + +```python +@dg.asset_check(asset=my_asset) +def validate_non_empty(my_asset): + return dg.AssetCheckResult( + passed=len(my_asset) > 0, + metadata={"row_count": len(my_asset)}, + ) +``` + +--- + +## dbt Integration + +For dbt integration, use the minimal pattern below. For comprehensive dbt patterns, see the `dbt-development` skill. + +### Basic dbt Assets + +```python +from dagster_dbt import DbtCliResource, dbt_assets +from pathlib import Path + +dbt_project_dir = Path(__file__).parent / "dbt_project" + +@dbt_assets(manifest=dbt_project_dir / "target" / "manifest.json") +def my_dbt_assets(context: dg.AssetExecutionContext, dbt: DbtCliResource): + yield from dbt.cli(["build"], context=context).stream() +``` + +### dbt Resource + +```python +dg.Definitions( + assets=[my_dbt_assets], + resources={"dbt": DbtCliResource(project_dir=dbt_project_dir)}, +) +``` + +**Full patterns**: See [Dagster dbt docs](https://docs.dagster.io/integrations/libraries/dbt) + +--- + +## When to Load References + +### Load `references/assets.md` when: +- Defining complex asset dependencies +- Adding metadata, groups, or key prefixes +- Working with asset factories +- Understanding asset materialization patterns + +### Load `references/resources.md` when: +- Creating custom `ConfigurableResource` classes +- Integrating with databases, APIs, or cloud services +- Understanding resource scoping and lifecycle + +### Load `references/automation.md` when: +- Creating schedules with complex cron patterns +- Building sensors with cursors and state management +- Implementing partitions and backfills +- Automating dbt or other integration runs + +### Load `references/testing.md` when: +- Writing unit tests for assets +- Mocking resources and dependencies +- Using `dg.materialize()` for integration tests +- Creating asset checks for data validation + +### Load `references/etl-patterns.md` when: +- Using dlt for embedded ETL +- Using Sling for database replication +- Loading data from files or APIs +- Integrating external ETL tools + +### Load `references/project-structure.md` when: +- Setting up a new Dagster project +- Configuring `Definitions` and code locations +- Using `dg` CLI for scaffolding +- Organizing large projects with Components + +--- + +## Project Structure + +### Recommended Layout + +``` +my_project/ +├── pyproject.toml +├── src/ +│ └── my_project/ +│ ├── definitions.py # Main Definitions +│ └── defs/ +│ ├── assets/ +│ │ ├── __init__.py +│ │ └── my_assets.py +│ ├── jobs.py +│ ├── schedules.py +│ ├── sensors.py +│ └── resources.py +└── tests/ + └── test_assets.py +``` + +### Definitions Pattern (Modern) + +```python +# src/my_project/definitions.py +from pathlib import Path +from dagster import definitions, load_from_defs_folder + +@definitions +def defs(): + return load_from_defs_folder(project_root=Path(__file__).parent.parent.parent) +``` + +### Scaffolding with dg CLI + +```bash +# Create new project +uvx create-dagster my_project + +# Scaffold new asset file +dg scaffold defs dagster.asset assets/new_asset.py + +# Scaffold schedule +dg scaffold defs dagster.schedule schedules.py + +# Scaffold sensor +dg scaffold defs dagster.sensor sensors.py + +# Validate definitions +dg check defs +``` + +--- + +## Common Patterns + +### Job Definition + +```python +trip_update_job = dg.define_asset_job( + name="trip_update_job", + selection=["taxi_trips", "taxi_zones"], +) +``` + +### Run Configuration + +```python +from dagster import Config + +class MyAssetConfig(Config): + filename: str + limit: int = 100 + +@dg.asset +def configurable_asset(config: MyAssetConfig) -> None: + print(f"Processing {config.filename} with limit {config.limit}") +``` + +### Asset Dependencies with External Sources + +```python +@dg.asset(deps=["external_table"]) +def derived_asset() -> None: + """Depends on external_table which isn't managed by Dagster.""" + pass +``` + +--- + +## Anti-Patterns to Avoid + +| Anti-Pattern | Better Approach | +| ------------ | --------------- | +| Hardcoding credentials in assets | Use `ConfigurableResource` with env vars | +| Giant assets that do everything | Split into focused, composable assets | +| Ignoring asset return types | Use type annotations for clarity | +| Skipping tests for assets | Test assets like regular Python functions | +| Not using partitions for time-series | Use `DailyPartitionsDefinition` etc. | +| Putting all assets in one file | Organize by domain in separate modules | + +--- + +## CLI Quick Reference + +```bash +# Development +dg dev # Start Dagster UI +dg check defs # Validate definitions + +# Scaffolding +dg scaffold defs dagster.asset assets/file.py +dg scaffold defs dagster.schedule schedules.py +dg scaffold defs dagster.sensor sensors.py + +# Production +dagster job execute -j my_job # Execute a job +dagster asset materialize -a my_asset # Materialize an asset +``` + +--- + +## References + +- **Assets**: `references/assets.md` - Detailed asset patterns +- **Resources**: `references/resources.md` - Resource configuration +- **Automation**: `references/automation.md` - Schedules, sensors, partitions +- **Testing**: `references/testing.md` - Testing patterns and asset checks +- **ETL Patterns**: `references/etl-patterns.md` - dlt, Sling, file/API ingestion +- **Project Structure**: `references/project-structure.md` - Definitions, Components +- **Official Docs**: https://docs.dagster.io +- **API Reference**: https://docs.dagster.io/api/dagster diff --git a/.agents/skills/dagster-development/references/assets.md b/.agents/skills/dagster-development/references/assets.md new file mode 100644 index 0000000..7e97753 --- /dev/null +++ b/.agents/skills/dagster-development/references/assets.md @@ -0,0 +1,358 @@ +# Asset Patterns Reference + +## Pattern Summary + +| Pattern | When to Use | +| ------- | ----------- | +| Basic `@dg.asset` | Simple computation with no dependencies | +| Parameter-based dependency | Asset depends on another managed asset | +| `deps=` dependency | Asset depends on external or non-Python asset | +| Asset with metadata | Track runtime metrics (row counts, timestamps) | +| Asset groups | Organize related assets visually | +| Asset key prefixes | Namespace assets for multi-tenant or layered data | +| Partitioned assets | Time-series or categorical data splits | + +--- + +## Basic Asset Definition + +```python +import dagster as dg + +@dg.asset +def my_asset() -> None: + """ + Docstring becomes the asset description in the UI. + """ + # Your computation logic + pass +``` + +**Key points**: +- Function name becomes the asset key +- Docstring becomes the description +- Return type annotation is optional but recommended + +--- + +## Asset Dependencies + +### Parameter-Based Dependencies + +When an asset depends on another Dagster-managed asset, use parameters: + +```python +@dg.asset +def upstream_asset() -> dict: + return {"data": [1, 2, 3]} + +@dg.asset +def downstream_asset(upstream_asset: dict) -> list: + """ + Receives the return value of upstream_asset automatically. + """ + return upstream_asset["data"] +``` + +**How it works**: +- Parameter name must match the upstream asset's function name +- Dagster automatically passes the materialized output +- Creates a visual dependency in the asset graph + +### External Dependencies with `deps=` + +When an asset depends on something not managed by Dagster or without a return value: + +```python +@dg.asset(deps=["external_table", "raw_file"]) +def processed_data() -> None: + """ + Depends on external_table and raw_file but doesn't receive their values. + """ + # Read from external sources directly + pass +``` + +**Use `deps=` when**: +- The upstream asset doesn't return a value (returns `None`) +- The asset is external (file created by another process) +- You need loose coupling between assets + +### Mixed Dependencies + +Combine both patterns when needed: + +```python +@dg.asset(deps=["raw_file"]) +def enriched_data(reference_table: dict) -> dict: + """ + Depends on: + - raw_file (via deps=, doesn't receive value) + - reference_table (via parameter, receives value) + """ + # Read raw_file from disk, enrich with reference_table + return {"enriched": reference_table} +``` + +--- + +## Asset Metadata + +### Definition Metadata (Static) + +Applied once when the asset is defined: + +```python +@dg.asset( + description="Detailed description for the UI", + group_name="analytics", + key_prefix=["warehouse", "staging"], + owners=["team:data-engineering", "user@example.com"], + tags={"priority": "high", "pii": "true"}, +) +def my_asset() -> None: + pass +``` + +### Materialization Metadata (Dynamic) + +Captured each time the asset materializes: + +```python +import dagster as dg + +@dg.asset +def my_asset() -> dg.MaterializeResult: + """Asset that reports metadata on each run.""" + data = fetch_data() + row_count = len(data) + + # Save data... + + return dg.MaterializeResult( + metadata={ + "row_count": dg.MetadataValue.int(row_count), + "last_updated": dg.MetadataValue.text(str(datetime.now())), + "sample_data": dg.MetadataValue.json(data[:5]), + } + ) +``` + +### MetadataValue Types + +| Type | Usage | +| ---- | ----- | +| `MetadataValue.int(n)` | Integer values (row counts) | +| `MetadataValue.float(n)` | Float values (percentages) | +| `MetadataValue.text(s)` | Short text values | +| `MetadataValue.json(obj)` | JSON-serializable objects | +| `MetadataValue.md(s)` | Markdown text | +| `MetadataValue.url(s)` | Clickable URLs | +| `MetadataValue.path(s)` | File paths | +| `MetadataValue.table(records)` | Tabular data | + +--- + +## Asset Groups + +Organize related assets visually in the UI: + +```python +@dg.asset(group_name="raw_data") +def raw_orders() -> None: + pass + +@dg.asset(group_name="raw_data") +def raw_customers() -> None: + pass + +@dg.asset(group_name="analytics") +def daily_revenue(raw_orders) -> None: + pass +``` + +**Best practices**: +- Group by data layer: `raw`, `staging`, `analytics`, `mart` +- Group by domain: `sales`, `marketing`, `finance` +- Group by source: `postgres`, `api`, `files` + +--- + +## Asset Key Prefixes + +Namespace assets for organization: + +```python +@dg.asset(key_prefix=["warehouse", "raw"]) +def orders() -> None: + """Asset key becomes: warehouse/raw/orders""" + pass + +@dg.asset(key_prefix=["warehouse", "staging"]) +def orders_cleaned() -> None: + """Asset key becomes: warehouse/staging/orders_cleaned""" + pass +``` + +**Use prefixes for**: +- Multi-tenant architectures +- Environment separation +- Data layer organization (bronze/silver/gold) + +--- + +## Asset with Execution Context + +Access runtime information during materialization: + +```python +@dg.asset +def context_aware_asset(context: dg.AssetExecutionContext) -> None: + context.log.info("Starting asset materialization") + + # Access asset key + asset_key = context.asset_key + + # Access partition key (if partitioned) + if context.has_partition_key: + partition = context.partition_key + context.log.info(f"Processing partition: {partition}") + + # Access run ID + run_id = context.run_id +``` + +--- + +## Asset with Configuration + +Make assets configurable at runtime: + +```python +from dagster import Config + +class MyAssetConfig(Config): + limit: int = 100 + include_archived: bool = False + source_path: str + +@dg.asset +def configurable_asset(config: MyAssetConfig) -> None: + """ + Run with specific configuration values. + """ + data = load_data( + path=config.source_path, + limit=config.limit, + include_archived=config.include_archived, + ) +``` + +--- + +## Asset Return Types + +### Returning Data Directly + +```python +@dg.asset +def returns_data() -> dict: + return {"key": "value"} +``` + +### Returning MaterializeResult + +```python +@dg.asset +def returns_result() -> dg.MaterializeResult: + # Do work... + return dg.MaterializeResult( + metadata={"rows": 100} + ) +``` + +### Returning Output with Type + +```python +from dagster import Output + +@dg.asset +def returns_output() -> Output[dict]: + data = {"key": "value"} + return Output( + value=data, + metadata={"size": len(data)}, + ) +``` + +--- + +## Multi-Asset Pattern + +Define multiple assets from one function: + +```python +from dagster import multi_asset, AssetOut + +@multi_asset( + outs={ + "users": AssetOut(), + "orders": AssetOut(), + } +) +def load_data(): + users_df = fetch_users() + orders_df = fetch_orders() + + yield Output(users_df, output_name="users") + yield Output(orders_df, output_name="orders") +``` + +**Use when**: +- One computation produces multiple logical assets +- Assets are always created together +- Shared setup is expensive + +--- + +## Asset Factories + +Generate similar assets programmatically: + +```python +def create_table_asset(table_name: str, schema: str): + @dg.asset( + name=f"{schema}_{table_name}", + group_name=schema, + ) + def _asset() -> None: + load_table(schema, table_name) + + return _asset + +# Generate assets +customers = create_table_asset("customers", "sales") +products = create_table_asset("products", "catalog") +orders = create_table_asset("orders", "sales") +``` + +--- + +## Common Anti-Patterns + +| Anti-Pattern | Better Approach | +| ------------ | --------------- | +| `load_customers` (verb-based name) | `customers` (noun describing output) | +| Giant asset doing everything | Split into focused, composable assets | +| No type annotations | Add return type: `-> dict`, `-> None` | +| No docstring | Add description in docstring or `description=` | +| Ignoring `MaterializeResult` | Return metadata for observability | +| Hardcoded paths | Use configuration or environment variables | + +--- + +## References + +- [Assets API](https://docs.dagster.io/api/dagster/assets) +- [Asset Metadata](https://docs.dagster.io/guides/build/assets/metadata-and-tags) +- [Multi-Assets](https://docs.dagster.io/guides/build/assets/multi-assets) diff --git a/.agents/skills/dagster-development/references/automation.md b/.agents/skills/dagster-development/references/automation.md new file mode 100644 index 0000000..7296237 --- /dev/null +++ b/.agents/skills/dagster-development/references/automation.md @@ -0,0 +1,461 @@ +# Automation Patterns Reference + +## Pattern Summary + +| Pattern | When to Use | +| ------- | ----------- | +| `ScheduleDefinition` | Fixed time intervals (daily, hourly, monthly) | +| `@dg.schedule` decorator | Custom schedule logic with dynamic job selection | +| `@dg.sensor` | Event-driven triggers (file changes, API updates) | +| `PartitionsDefinition` | Time-series or categorical data splits | +| Partitioned schedules | Automate partition materialization | + +--- + +## Jobs + +Jobs select which assets to materialize together: + +### Basic Job Definition + +```python +import dagster as dg + +trip_update_job = dg.define_asset_job( + name="trip_update_job", + selection=["taxi_trips", "taxi_zones"], +) +``` + +### Asset Selection Patterns + +```python +# Select specific assets +dg.AssetSelection.assets("asset_a", "asset_b") + +# Select all assets +dg.AssetSelection.all() + +# Select by group +dg.AssetSelection.groups("analytics") + +# Select with dependencies +dg.AssetSelection.assets("final_report").upstream() # Include upstream +dg.AssetSelection.assets("raw_data").downstream() # Include downstream + +# Combine selections +dg.AssetSelection.all() - dg.AssetSelection.assets("excluded_asset") +dg.AssetSelection.groups("a") | dg.AssetSelection.groups("b") # Union +dg.AssetSelection.groups("a") & dg.AssetSelection.assets("specific") # Intersection +``` + +### Job with Tags + +```python +daily_job = dg.define_asset_job( + name="daily_job", + selection=dg.AssetSelection.all(), + tags={"team": "data-eng", "priority": "high"}, +) +``` + +--- + +## Schedules + +### Basic Schedule + +```python +import dagster as dg +from my_project.defs.jobs import trip_update_job + +trip_update_schedule = dg.ScheduleDefinition( + job=trip_update_job, + cron_schedule="0 0 5 * *", # 5th of each month at midnight +) +``` + +### Common Cron Patterns + +| Pattern | Meaning | +| ------- | ------- | +| `* * * * *` | Every minute | +| `0 * * * *` | Every hour (at minute 0) | +| `0 0 * * *` | Daily at midnight | +| `0 6 * * *` | Daily at 6:00 AM | +| `0 0 * * 1` | Weekly on Monday at midnight | +| `0 0 * * 1-5` | Weekdays at midnight | +| `0 0 1 * *` | Monthly on the 1st at midnight | +| `0 0 5 * *` | Monthly on the 5th at midnight | +| `15 5 * * 1-5` | Weekdays at 5:15 AM | + +**Tip**: Use [Crontab Guru](https://crontab.guru/) to create and test cron expressions. + +### Schedule with Timezone + +```python +my_schedule = dg.ScheduleDefinition( + job=my_job, + cron_schedule="0 9 * * *", # 9:00 AM + execution_timezone="America/New_York", +) +``` + +### Custom Schedule with Decorator + +```python +@dg.schedule(cron_schedule="0 0 * * *", job=my_job) +def custom_schedule(context: dg.ScheduleEvaluationContext): + """Schedule with custom logic.""" + scheduled_date = context.scheduled_execution_time.strftime("%Y-%m-%d") + + return dg.RunRequest( + run_key=f"daily_{scheduled_date}", + run_config={ + "ops": { + "my_asset": { + "config": {"date": scheduled_date} + } + } + }, + ) +``` + +### Partitioned Schedule + +Automatically run for new partitions: + +```python +from my_project.defs.partitions import monthly_partition +from my_project.defs.jobs import partitioned_job + +monthly_schedule = dg.build_schedule_from_partitioned_job( + job=partitioned_job, + description="Materializes data for the previous month", +) +``` + +--- + +## Sensors + +### Sensor Anatomy + +Sensors follow this lifecycle: +1. Read cursor (previous state) +2. Observe current state +3. Compare states and create run requests for changes +4. Update cursor + +### Basic Sensor Pattern + +```python +import dagster as dg +import json + +@dg.sensor(job=my_job) +def file_sensor(context: dg.SensorEvaluationContext): + # 1. Read cursor (previous state) + previous_state = json.loads(context.cursor) if context.cursor else {} + current_state = {} + runs_to_request = [] + + # 2. Observe current state + for filepath in get_files_to_watch(): + last_modified = os.path.getmtime(filepath) + filename = os.path.basename(filepath) + current_state[filename] = last_modified + + # 3. Check for changes + if filename not in previous_state or previous_state[filename] != last_modified: + runs_to_request.append(dg.RunRequest( + run_key=f"file_{filename}_{last_modified}", + run_config={ + "ops": { + "process_file": { + "config": {"filename": filename} + } + } + } + )) + + # 4. Return result with updated cursor + return dg.SensorResult( + run_requests=runs_to_request, + cursor=json.dumps(current_state), + ) +``` + +### File Sensor + +```python +import os +import json + +@dg.sensor(job=adhoc_request_job) +def adhoc_request_sensor(context: dg.SensorEvaluationContext): + PATH_TO_REQUESTS = os.path.join( + os.path.dirname(__file__), + "../../../data/requests", + ) + + previous_state = json.loads(context.cursor) if context.cursor else {} + current_state = {} + runs_to_request = [] + + for filename in os.listdir(PATH_TO_REQUESTS): + file_path = os.path.join(PATH_TO_REQUESTS, filename) + if filename.endswith(".json") and os.path.isfile(file_path): + last_modified = os.path.getmtime(file_path) + current_state[filename] = last_modified + + if filename not in previous_state or previous_state[filename] != last_modified: + with open(file_path, "r") as f: + request_config = json.load(f) + + runs_to_request.append(dg.RunRequest( + run_key=f"request_{filename}_{last_modified}", + run_config={ + "ops": { + "adhoc_request": { + "config": { + "filename": filename, + **request_config + } + } + } + } + )) + + return dg.SensorResult( + run_requests=runs_to_request, + cursor=json.dumps(current_state), + ) +``` + +### Asset Sensor + +Trigger when another asset materializes: + +```python +@dg.asset_sensor(asset_key=dg.AssetKey("upstream_asset"), job=downstream_job) +def upstream_sensor(context: dg.SensorEvaluationContext, asset_event): + """Triggers when upstream_asset is materialized.""" + return dg.RunRequest( + run_key=f"downstream_{asset_event.dagster_event.event_specific_data.materialization.run_id}", + ) +``` + +### Sensor with Skip Reason + +```python +@dg.sensor(job=my_job, minimum_interval_seconds=60) +def conditional_sensor(context: dg.SensorEvaluationContext): + new_files = check_for_new_files() + + if not new_files: + return dg.SkipReason("No new files found") + + return dg.RunRequest(run_key=f"files_{len(new_files)}") +``` + +--- + +## Partitions + +### Time-Based Partitions + +```python +import dagster as dg + +# Daily partitions +daily_partition = dg.DailyPartitionsDefinition( + start_date="2023-01-01", + end_date="2024-12-31", # Optional +) + +# Weekly partitions +weekly_partition = dg.WeeklyPartitionsDefinition( + start_date="2023-01-01", +) + +# Monthly partitions +monthly_partition = dg.MonthlyPartitionsDefinition( + start_date="2023-01-01", + end_date="2023-12-31", +) + +# Hourly partitions +hourly_partition = dg.HourlyPartitionsDefinition( + start_date="2023-01-01-00:00", +) +``` + +### Static Partitions + +```python +# Fixed set of partitions +region_partition = dg.StaticPartitionsDefinition([ + "us-east", + "us-west", + "eu-west", + "ap-south", +]) + +# Category partitions +category_partition = dg.StaticPartitionsDefinition([ + "electronics", + "clothing", + "home", + "sports", +]) +``` + +### Multi-Dimensional Partitions + +```python +multi_partition = dg.MultiPartitionsDefinition({ + "date": dg.DailyPartitionsDefinition(start_date="2023-01-01"), + "region": dg.StaticPartitionsDefinition(["us", "eu", "ap"]), +}) + +@dg.asset(partitions_def=multi_partition) +def multi_partitioned_asset(context: dg.AssetExecutionContext) -> None: + partition_key = context.partition_key + # partition_key is a MultiPartitionKey with .keys_by_dimension + date = partition_key.keys_by_dimension["date"] + region = partition_key.keys_by_dimension["region"] +``` + +### Using Partitions in Assets + +```python +@dg.asset(partitions_def=monthly_partition) +def monthly_data(context: dg.AssetExecutionContext) -> None: + """Process data for a specific month.""" + partition_date_str = context.partition_key # "2023-01-01" + month = partition_date_str[:-3] # "2023-01" + + context.log.info(f"Processing partition: {month}") + + data = fetch_data_for_month(month) + save_data(data, month) +``` + +### Partition Window Access + +```python +@dg.asset(partitions_def=daily_partition) +def windowed_data(context: dg.AssetExecutionContext) -> None: + """Access partition time window.""" + time_window = context.partition_time_window + + start = time_window.start # datetime + end = time_window.end # datetime + + context.log.info(f"Processing from {start} to {end}") +``` + +--- + +## Partitioned Jobs + +### Define Partitioned Job + +```python +partitioned_job = dg.define_asset_job( + name="partitioned_job", + selection=["monthly_data"], + partitions_def=monthly_partition, +) +``` + +### Run Specific Partition + +```python +# In launchpad or programmatically +result = partitioned_job.execute_in_process( + partition_key="2023-06-01", +) +``` + +--- + +## Backfills + +Backfills materialize multiple partitions at once: + +### UI Backfill +1. Navigate to the asset in the Dagster UI +2. Click "Materialize" dropdown +3. Select "Backfill" +4. Choose partition range + +### Programmatic Backfill + +```python +from dagster import DagsterInstance + +instance = DagsterInstance.get() +instance.submit_run( + partition_key="2023-01-01", + job_name="partitioned_job", +) +``` + +--- + +## Combining Automation + +### Partitioned Schedule + +```python +# Create job with partition +monthly_job = dg.define_asset_job( + name="monthly_job", + selection=["monthly_data"], + partitions_def=monthly_partition, +) + +# Build schedule that runs for new partitions +monthly_schedule = dg.build_schedule_from_partitioned_job( + job=monthly_job, +) +``` + +### Sensor with Partitions + +```python +@dg.sensor(job=partitioned_job) +def partition_sensor(context: dg.SensorEvaluationContext): + missing_partitions = get_missing_partitions() + + return [ + dg.RunRequest( + run_key=f"backfill_{partition}", + partition_key=partition, + ) + for partition in missing_partitions + ] +``` + +--- + +## Anti-Patterns to Avoid + +| Anti-Pattern | Better Approach | +| ------------ | --------------- | +| No cursor in sensors | Always use cursor for state tracking | +| Polling too frequently | Set `minimum_interval_seconds` | +| Giant backfills at once | Use `max_runtime` on backfill policies | +| Hardcoded partition dates | Use dynamic start/end with constants | +| Ignoring timezone | Set `execution_timezone` on schedules | + +--- + +## References + +- [Schedules](https://docs.dagster.io/guides/automate/schedules) +- [Sensors](https://docs.dagster.io/guides/automate/sensors) +- [Partitions](https://docs.dagster.io/guides/build/partitions) +- [Asset Selection Syntax](https://docs.dagster.io/concepts/assets/asset-selection-syntax) diff --git a/.agents/skills/dagster-development/references/etl-patterns.md b/.agents/skills/dagster-development/references/etl-patterns.md new file mode 100644 index 0000000..79b1a26 --- /dev/null +++ b/.agents/skills/dagster-development/references/etl-patterns.md @@ -0,0 +1,547 @@ +# ETL Patterns Reference + +## Pattern Summary + +| Pattern | When to Use | +| ------- | ----------- | +| Cloud storage resources | Loading CSVs, Parquet, JSON from S3, GCS, Azure Blob (production) | +| Database resources | Direct database connections for data ingestion (preferred) | +| API resource + assets | Fetching data from REST APIs | +| dlt integration | Embedded ETL with schema inference and pagination | +| Sling integration | Database-to-database replication | +| Configurable ingestion | Dynamic file/endpoint selection at runtime | +| Local file storage | **Only for local development/testing** - use cloud storage or databases in production | + +--- + +## File Import Pattern + +**⚠️ Important**: Avoid saving files to local Dagster storage in production. Use cloud storage (S3, GCS, Azure Blob) or load directly into databases. Local file storage should only be used for local development/testing purposes. + +### Cloud Storage Pattern (Production) + +```python +import dagster as dg +from dagster_aws import S3Resource +import polars as pl +import io + +class S3FileConfig(dg.Config): + bucket: str + key: str # S3 object key (file path) + +@dg.asset +def raw_data_table( + context: dg.AssetExecutionContext, + s3: S3Resource, + database: DuckDBResource, + config: S3FileConfig, +) -> None: + """Load CSV file from S3 directly into database without local storage.""" + table_name = "raw_data" + + # Download file content directly into memory + s3_obj = s3.get_object(Bucket=config.bucket, Key=config.key) + file_content = s3_obj['Body'].read() + + # Load directly into Polars DataFrame + df = pl.read_csv(io.BytesIO(file_content)) + + # Write directly to database + with database.get_connection() as conn: + conn.register("temp_df", df) + conn.execute(f"CREATE OR REPLACE TABLE {table_name} AS SELECT * FROM temp_df") +``` + +### Google Cloud Storage Pattern + +```python +from dagster_gcp import GCSResource +import polars as pl +import io + +@dg.asset +def load_from_gcs( + context: dg.AssetExecutionContext, + gcs: GCSResource, + database: DuckDBResource, +) -> None: + """Load file from GCS directly into database.""" + bucket = gcs.get_bucket("my-bucket") + blob = bucket.blob("data/file.csv") + + # Read directly into memory + file_content = blob.download_as_bytes() + df = pl.read_csv(io.BytesIO(file_content)) + + # Write to database + with database.get_connection() as conn: + conn.register("temp_df", df) + conn.execute("CREATE OR REPLACE TABLE raw_data AS SELECT * FROM temp_df") +``` + +### Local File Pattern (Development Only) + +```python +import dagster as dg +from pathlib import Path +import os + +class IngestionFileConfig(dg.Config): + path: str + +@dg.asset +def import_file(config: IngestionFileConfig) -> str: + """ + Resolve file path from config. + + ⚠️ Only use for local development. In production, use cloud storage resources. + """ + # Check if we're in local dev environment + if os.getenv("ENVIRONMENT") == "dev": + file_path = Path(__file__).parent / f"../../../data/source/{config.path}" + return str(file_path.resolve()) + else: + raise ValueError( + "Local file paths should not be used in production. " + "Use S3Resource, GCSResource, or database connections instead." + ) +``` + +### Load File into Database (Direct) + +```python +from dagster_duckdb import DuckDBResource +import polars as pl + +@dg.asset(kinds={"duckdb"}) +def raw_data_table( + context: dg.AssetExecutionContext, + database: DuckDBResource, + file_content: bytes, # From cloud storage or API +) -> None: + """Load CSV file content directly into DuckDB table.""" + table_name = "raw_data" + + # Load directly from bytes into DataFrame + df = pl.read_csv(io.BytesIO(file_content)) + + with database.get_connection() as conn: + # Register DataFrame and create table directly + conn.register("temp_df", df) + conn.execute(f"CREATE OR REPLACE TABLE {table_name} AS SELECT * FROM temp_df") +``` + +### Running with Config + +**CLI:** +```bash +dg launch --assets import_file,raw_data_table \ + --config-json '{"ops": {"import_file": {"config": {"path": "2024-01-01.csv"}}}}' +``` + +**UI YAML:** +```yaml +ops: + import_file: + config: + path: 2024-01-01.csv +``` + +--- + +## API Integration Pattern + +### API Resource + +```python +import dagster as dg +import requests + +class NASAResource(dg.ConfigurableResource): + api_key: str + base_url: str = "https://api.nasa.gov" + + def get_near_earth_asteroids(self, start_date: str, end_date: str) -> list: + url = f"{self.base_url}/neo/rest/v1/feed" + params = { + "start_date": start_date, + "end_date": end_date, + "api_key": self.api_key, + } + + response = requests.get(url, params=params) + response.raise_for_status() + return response.json()["near_earth_objects"][start_date] +``` + +### API Asset + +```python +@dg.asset +def asteroid_data( + context: dg.AssetExecutionContext, + nasa: NASAResource, +) -> list[dict]: + """Fetch asteroid data from NASA API.""" + data = nasa.get_near_earth_asteroids("2024-01-01", "2024-01-07") + context.log.info(f"Retrieved {len(data)} asteroids") + return data +``` + +### Register API Resource + +```python +@dg.definitions +def resources(): + return dg.Definitions( + resources={ + "nasa": NASAResource( + api_key=dg.EnvVar("NASA_API_KEY"), + ), + }, + ) +``` + +--- + +## dlt Integration + +dlt (data load tool) handles schema inference, pagination, and loading automatically. + +### Basic dlt Pipeline + +```python +import dlt + +@dlt.source +def simple_source(): + @dlt.resource + def load_dict(): + data = [ + {"id": 1, "name": "Alice"}, + {"id": 2, "name": "Bob"}, + ] + yield data + + return load_dict + +# Standalone execution +pipeline = dlt.pipeline( + pipeline_name="simple_pipeline", + destination="duckdb", + dataset_name="mydata", +) +load_info = pipeline.run(simple_source()) +``` + +### dlt with Dagster + +```python +import dagster as dg +import dlt +from dagster_dlt import DagsterDltResource, dlt_assets + +@dlt.source +def simple_source(): + @dlt.resource + def load_dict(): + data = [ + {"id": 1, "name": "Alice"}, + {"id": 2, "name": "Bob"}, + ] + yield data + + return load_dict + +@dlt_assets( + dlt_source=simple_source(), + dlt_pipeline=dlt.pipeline( + pipeline_name="simple_pipeline", + dataset_name="simple", + destination="duckdb", + progress="log", + ), +) +def dlt_assets(context: dg.AssetExecutionContext, dlt: DagsterDltResource): + yield from dlt.run(context=context) +``` + +### dlt Resource Registration + +```python +from dagster_dlt import DagsterDltResource + +@dg.definitions +def resources(): + return dg.Definitions( + resources={ + "dlt": DagsterDltResource(), + }, + ) +``` + +### dlt Benefits + +- **Schema inference**: Automatically detects and creates table schemas +- **Type inference**: Maps Python types to database types +- **Pagination handling**: Built-in support for paginated APIs +- **Incremental loading**: Track state for incremental updates +- **Multiple destinations**: DuckDB, Snowflake, BigQuery, Postgres, etc. + +--- + +## Sling Integration + +Sling is declarative database-to-database replication. + +### Sling Connection Resources + +```python +from dagster_sling import SlingConnectionResource, SlingResource + +# Source database +source = SlingConnectionResource( + name="MY_POSTGRES", + type="postgres", + host="localhost", + port=5432, + database="source_db", + user="user", + password="password", +) + +# Destination database +destination = SlingConnectionResource( + name="MY_DUCKDB", + type="duckdb", + connection_string="duckdb:///data/staging/data.duckdb", +) + +# Combine into SlingResource +sling = SlingResource( + connections=[source, destination], +) +``` + +### Sling Replication Config + +```yaml +# sling_replication.yaml +source: MY_POSTGRES +target: MY_DUCKDB + +defaults: + mode: full-refresh + object: "{stream_schema}_{stream_table}" + +streams: + data.customers: + data.products: + data.orders: +``` + +### Sling Replication Modes + +| Mode | Description | +| ---- | ----------- | +| `full-refresh` | Drop and recreate table each run | +| `incremental` | Append new records | +| `truncate` | Truncate table before loading | +| `snapshot` | Full refresh with point-in-time snapshot | + +### Sling Assets + +```python +from dagster_sling import SlingResource, sling_assets + +replication_config = dg.file_relative_path(__file__, "sling_replication.yaml") + +@sling_assets(replication_config=replication_config) +def postgres_sling_assets(context: dg.AssetExecutionContext, sling: SlingResource): + yield from sling.replicate(context=context).fetch_column_metadata() +``` + +### Register Sling Resource + +```python +@dg.definitions +def resources(): + return dg.Definitions( + resources={ + "sling": sling, + }, + ) +``` + +--- + +## Choosing ETL Approach + +| Scenario | Recommended Approach | +| -------- | -------------------- | +| Load files from cloud storage (S3, GCS) | Cloud storage resource (S3Resource, GCSResource) + direct database load | +| Load files from local storage | **Only for local dev** - use cloud storage in production | +| Custom REST API | API resource + assets (load directly into database) | +| Standard API (Stripe, GitHub) | dlt verified sources | +| Schema-on-read ingestion | dlt | +| Database replication | Sling | +| Complex transformations | Custom assets or dbt | +| File downloads from APIs | Load directly into memory (BytesIO) then to database, avoid local saves | + +--- + +## Partitioned ETL + +### Partitioned File Import + +```python +daily_partition = dg.DailyPartitionsDefinition(start_date="2024-01-01") + +@dg.asset(partitions_def=daily_partition) +def daily_import(context: dg.AssetExecutionContext) -> str: + partition_date = context.partition_key # "2024-01-01" + file_path = f"data/source/{partition_date}.csv" + return file_path + +@dg.asset(partitions_def=daily_partition) +def daily_load( + context: dg.AssetExecutionContext, + database: DuckDBResource, + daily_import: str, +) -> None: + partition = context.partition_key + + with database.get_connection() as conn: + # Delete existing partition data + conn.execute(f"DELETE FROM raw_data WHERE date = '{partition}'") + # Load new data + conn.execute(f"COPY raw_data FROM '{daily_import}'") +``` + +### Partitioned API Fetch + +```python +@dg.asset(partitions_def=daily_partition) +def daily_api_data( + context: dg.AssetExecutionContext, + api: MyAPIResource, +) -> list[dict]: + partition_date = context.partition_key + return api.fetch_data_for_date(partition_date) +``` + +--- + +## Data Validation + +### Validate After Load + +```python +@dg.asset(deps=["raw_data_table"]) +def validated_data(database: DuckDBResource) -> dg.MaterializeResult: + with database.get_connection() as conn: + # Check row count + result = conn.execute("SELECT COUNT(*) FROM raw_data").fetchone() + row_count = result[0] + + # Check for nulls + null_check = conn.execute( + "SELECT COUNT(*) FROM raw_data WHERE value IS NULL" + ).fetchone() + null_count = null_check[0] + + return dg.MaterializeResult( + metadata={ + "row_count": row_count, + "null_values": null_count, + "null_percentage": round(null_count / row_count * 100, 2) if row_count > 0 else 0, + } + ) +``` + +### Asset Check for Data Quality + +```python +@dg.asset_check(asset=raw_data_table) +def no_duplicate_records(database: DuckDBResource) -> dg.AssetCheckResult: + with database.get_connection() as conn: + result = conn.execute(""" + SELECT COUNT(*) - COUNT(DISTINCT id) as duplicates + FROM raw_data + """).fetchone() + duplicates = result[0] + + return dg.AssetCheckResult( + passed=duplicates == 0, + metadata={"duplicate_count": duplicates}, + ) +``` + +--- + +## Error Handling + +### Retry Pattern + +```python +import time + +class RobustAPIResource(dg.ConfigurableResource): + api_key: str + max_retries: int = 3 + + def fetch_with_retry(self, endpoint: str) -> dict: + for attempt in range(self.max_retries): + try: + response = requests.get(endpoint, headers={"Authorization": self.api_key}) + response.raise_for_status() + return response.json() + except requests.RequestException as e: + if attempt == self.max_retries - 1: + raise + time.sleep(2 ** attempt) # Exponential backoff +``` + +### Graceful Failure + +```python +@dg.asset +def api_data_with_fallback( + context: dg.AssetExecutionContext, + api: MyAPIResource, +) -> dg.MaterializeResult: + try: + data = api.fetch_data() + return dg.MaterializeResult( + metadata={"status": "success", "row_count": len(data)} + ) + except Exception as e: + context.log.error(f"API fetch failed: {e}") + raise # Let Dagster handle the failure +``` + +--- + +## Anti-Patterns to Avoid + +| Anti-Pattern | Better Approach | +| ------------ | --------------- | +| Saving files to local storage in production | Use cloud storage (S3, GCS) or load directly into databases | +| Using `tempfile` or local file writes in assets | Load file content into memory (BytesIO) then directly to database | +| Hardcoded file paths | Use `Config` for dynamic paths, prefer cloud storage resources | +| No schema validation | Add asset checks for data quality | +| Ignoring pagination | Use dlt or implement proper pagination | +| Full refresh always | Consider incremental loading | +| No retry logic | Add retries with exponential backoff | +| Downloading files locally before processing | Stream directly from cloud storage or API into memory | + +--- + +## References + +- [dlt Documentation](https://dlthub.com/docs) +- [Sling Documentation](https://docs.slingdata.io/) +- [Dagster dlt Integration](https://docs.dagster.io/integrations/libraries/dlt) +- [Dagster Sling Integration](https://docs.dagster.io/integrations/libraries/sling) +- [ETL Pipeline Tutorial](https://docs.dagster.io/etl-pipeline-tutorial) diff --git a/.agents/skills/dagster-development/references/project-structure.md b/.agents/skills/dagster-development/references/project-structure.md new file mode 100644 index 0000000..3216e87 --- /dev/null +++ b/.agents/skills/dagster-development/references/project-structure.md @@ -0,0 +1,445 @@ +# Project Structure Reference + +## Pattern Summary + +| Pattern | When to Use | +| ------- | ----------- | +| Single code location | Small to medium projects, single team | +| Multiple code locations | Large organizations, isolated dependencies | +| Components | Reusable integrations, declarative config | +| Module-based organization | Domain-driven project structure | + +--- + +## Recommended Project Layout + +### Standard Layout + +``` +my_project/ +├── pyproject.toml # Dependencies and dg config +├── src/ +│ └── my_project/ +│ ├── __init__.py +│ ├── definitions.py # Main Definitions entry point +│ └── defs/ +│ ├── __init__.py +│ ├── assets/ +│ │ ├── __init__.py +│ │ ├── constants.py +│ │ ├── ingestion.py +│ │ └── analytics.py +│ ├── jobs.py +│ ├── schedules.py +│ ├── sensors.py +│ ├── partitions.py +│ └── resources.py +├── data/ +│ ├── source/ +│ ├── staging/ +│ └── outputs/ +└── tests/ + ├── __init__.py + ├── conftest.py + └── test_assets.py +``` + +### With Components + +``` +my_project/ +├── pyproject.toml +├── src/ +│ └── my_project/ +│ ├── definitions.py +│ └── defs/ +│ ├── dashboard/ # Component: Looker integration +│ │ └── defs.yaml +│ ├── ingest_files/ # Component: Sling replication +│ │ ├── defs.yaml +│ │ └── replication.yaml +│ └── transform/ # Component: dbt project +│ └── defs.yaml +└── tests/ +``` + +--- + +## Definitions Object + +The `Definitions` object is the entry point for all Dagster objects. + +### Modern Autoloading Pattern + +```python +# src/my_project/definitions.py +from pathlib import Path +from dagster import definitions, load_from_defs_folder + +@definitions +def defs(): + return load_from_defs_folder(project_root=Path(__file__).parent.parent.parent) +``` + +This pattern: +- Automatically loads all assets, jobs, schedules from `defs/` +- Uses `@definitions` decorator for discoverability +- Requires proper file placement in `defs/` directory + +### Explicit Definitions Pattern + +```python +# src/my_project/definitions.py +import dagster as dg + +from my_project.defs.assets import ingestion, analytics +from my_project.defs.jobs import daily_job, weekly_job +from my_project.defs.schedules import daily_schedule +from my_project.defs.resources import database_resource + +defs = dg.Definitions( + assets=[ + *ingestion.assets, + *analytics.assets, + ], + jobs=[daily_job, weekly_job], + schedules=[daily_schedule], + resources={"database": database_resource}, +) +``` + +### Resource Definitions Pattern + +```python +# src/my_project/defs/resources.py +import dagster as dg +from dagster_duckdb import DuckDBResource + +database_resource = DuckDBResource( + database=dg.EnvVar("DUCKDB_DATABASE") +) + +@dg.definitions +def resources(): + return dg.Definitions( + resources={ + "database": database_resource, + } + ) +``` + +--- + +## Code Locations + +### What is a Code Location? + +A code location is: +1. A Python module containing a `Definitions` object +2. A Python environment that can load that module + +### When to Use Multiple Code Locations + +| Use Case | Benefit | +| -------- | ------- | +| Different teams | Independent deployments, isolated failures | +| Different Python versions | Legacy code alongside modern code | +| Different dependency versions | PyTorch v1 vs v2, pandas versions | +| Compliance separation | HIPAA, PCI data isolation | +| Functional separation | ETL vs ML vs BI layers | + +### Code Location Configuration + +```yaml +# workspace.yaml +load_from: + - python_module: my_project.definitions + - python_module: ml_project.definitions + - python_module: etl_project.definitions +``` + +### Benefits of Code Locations + +- **Isolation**: Failures in one location don't affect others +- **Independent deployment**: Teams deploy without coordination +- **Dependency flexibility**: Different packages per location +- **Single pane of glass**: All assets visible in one UI +- **Cross-location dependencies**: Assets can depend across locations + +--- + +## dg CLI Scaffolding + +### Create New Project + +```bash +uvx create-dagster my_project +cd my_project +``` + +This creates: +- Virtual environment with uv +- Standard project layout +- pyproject.toml with Dagster dependencies +- definitions.py with autoloading + +### Scaffold Dagster Objects + +```bash +# Scaffold asset file +dg scaffold defs dagster.asset assets/new_asset.py + +# Scaffold job +dg scaffold defs dagster.job jobs.py + +# Scaffold schedule +dg scaffold defs dagster.schedule schedules.py + +# Scaffold sensor +dg scaffold defs dagster.sensor sensors.py + +# Scaffold resources +dg scaffold defs dagster.resources resources.py +``` + +### Validate Definitions + +```bash +dg check defs +# Output: All components validated successfully. +# Output: All definitions loaded successfully. +``` + +### Development Server + +```bash +dg dev +# Starts Dagster UI at http://localhost:3000 +``` + +--- + +## pyproject.toml Configuration + +### Basic Configuration + +```toml +[project] +name = "my_project" +version = "0.1.0" +requires-python = ">=3.10" +dependencies = [ + "dagster>=1.8.0", + "dagster-duckdb", +] + +[project.optional-dependencies] +dev = [ + "dagster-webserver", + "pytest", +] + +[tool.dg] +directory_type = "project" + +[tool.dg.project] +root_module = "my_project" +registry_modules = [ + "my_project.components.*", +] +``` + +### With Components + +```toml +[tool.dg] +directory_type = "project" + +[tool.dg.project] +root_module = "my_project" +registry_modules = [ + "dagster.components.*", # Built-in components + "my_project.components.*", # Custom components +] +``` + +--- + +## Components + +Components are reusable, declarative building blocks for integrations. + +### Component Structure + +``` +my_component/ +├── defs.yaml # Component configuration +└── (optional files) +``` + +### Sling Component Example + +```yaml +# defs/ingest_files/defs.yaml +component: dagster_sling.SlingReplicationComponent + +params: + replication_config: replication.yaml + sling: + connections: + - name: MY_POSTGRES + type: postgres + host: localhost + port: 5432 + database: source_db + - name: MY_DUCKDB + type: duckdb + connection_string: "duckdb:///data/staging/data.duckdb" +``` + +```yaml +# defs/ingest_files/replication.yaml +source: MY_POSTGRES +target: MY_DUCKDB + +defaults: + mode: full-refresh + +streams: + data.customers: + data.products: + data.orders: +``` + +### Component Best Practices + +1. **Isolation**: Each component is self-contained +2. **Typed config**: Use config models for clear interfaces +3. **Composability**: Small, focused components that combine +4. **Versioning**: Track component changes +5. **Testing**: Validate components independently + +--- + +## Module Organization + +### Domain-Based Organization + +``` +defs/ +├── assets/ +│ ├── ingestion/ # Raw data ingestion +│ │ ├── __init__.py +│ │ ├── files.py +│ │ └── apis.py +│ ├── staging/ # Data cleaning +│ │ ├── __init__.py +│ │ └── transformations.py +│ └── analytics/ # Business metrics +│ ├── __init__.py +│ └── reports.py +├── jobs.py +├── schedules.py +└── resources.py +``` + +### Layer-Based Organization + +``` +defs/ +├── bronze/ # Raw data layer +│ ├── __init__.py +│ └── ingestion.py +├── silver/ # Cleaned data layer +│ ├── __init__.py +│ └── cleaning.py +├── gold/ # Business layer +│ ├── __init__.py +│ └── metrics.py +└── automation/ + ├── jobs.py + ├── schedules.py + └── sensors.py +``` + +--- + +## Environment Configuration + +### Environment Variables + +```bash +# .env (not committed to git) +DUCKDB_DATABASE=data/staging/data.duckdb +SNOWFLAKE_ACCOUNT=myaccount +SNOWFLAKE_USER=myuser +SNOWFLAKE_PASSWORD=secret +``` + +### Loading in Dagster + +```python +import dagster as dg + +resource = MyResource( + database=dg.EnvVar("DUCKDB_DATABASE"), + api_key=dg.EnvVar("API_KEY"), +) +``` + +### Environment-Specific Config + +```python +import os + +if os.getenv("DAGSTER_ENV") == "production": + database_path = "/data/prod/data.duckdb" +else: + database_path = "data/staging/data.duckdb" +``` + +--- + +## Reloading Definitions + +### When to Reload + +Reload definitions when: +- Adding new assets, jobs, schedules, or sensors +- Modifying decorator arguments (`@dg.asset(...)`) +- Changing Definitions object + +**Not required** when: +- Editing asset function logic (with `-e` install) +- Updating SQL queries inside assets +- Changing resource method implementations + +### How to Reload + +**UI**: Click "Reload Definitions" button + +**CLI**: +```bash +dagster dev # Restart for full reload +``` + +--- + +## Anti-Patterns to Avoid + +| Anti-Pattern | Better Approach | +| ------------ | --------------- | +| All assets in one file | Organize by domain/layer | +| Hardcoded paths | Use constants or EnvVar | +| No definitions validation | Run `dg check defs` in CI | +| Mixing production/dev config | Use environment variables | +| Monolithic code location | Split by team/function as you grow | + +--- + +## References + +- [Project Structure](https://docs.dagster.io/guides/build/project-structure) +- [Code Locations](https://docs.dagster.io/guides/deploy/code-locations) +- [Components Guide](https://docs.dagster.io/guides/build/components) +- [dg CLI Reference](https://docs.dagster.io/api/clis/dg-cli/dg-cli-reference) diff --git a/.agents/skills/dagster-development/references/resources.md b/.agents/skills/dagster-development/references/resources.md new file mode 100644 index 0000000..c2b9967 --- /dev/null +++ b/.agents/skills/dagster-development/references/resources.md @@ -0,0 +1,407 @@ +# Resource Patterns Reference + +## Pattern Summary + +| Pattern | When to Use | +| ------- | ----------- | +| Built-in resource | Database/service with Dagster integration (DuckDB, Snowflake, etc.) | +| Custom `ConfigurableResource` | Custom API clients, services, or shared logic | +| `EnvVar` configuration | Secrets, environment-specific values | +| Resource with methods | Complex services with multiple operations | +| Nested resources | Resources that depend on other resources | + +--- + +## Using Built-in Resources + +Dagster provides pre-built resources for common services: + +### DuckDB Resource + +```python +from dagster_duckdb import DuckDBResource +import dagster as dg + +database_resource = DuckDBResource( + database=dg.EnvVar("DUCKDB_DATABASE") +) + +@dg.definitions +def resources(): + return dg.Definitions( + resources={"database": database_resource} + ) +``` + +**Using in assets:** + +```python +from dagster_duckdb import DuckDBResource + +@dg.asset(deps=["raw_data"]) +def processed_data(database: DuckDBResource) -> None: + query = """ + CREATE OR REPLACE TABLE processed AS + SELECT * FROM raw_data WHERE status = 'active' + """ + with database.get_connection() as conn: + conn.execute(query) +``` + +### Common Built-in Resources + +| Package | Resource | Use Case | +| ------- | -------- | -------- | +| `dagster_duckdb` | `DuckDBResource` | Local/embedded analytics | +| `dagster_snowflake` | `SnowflakeResource` | Snowflake data warehouse | +| `dagster_gcp` | `BigQueryResource` | Google BigQuery | +| `dagster_aws` | `S3Resource` | AWS S3 storage | +| `dagster_dbt` | `DbtCliResource` | dbt transformations | +| `dagster_dlt` | `DagsterDltResource` | dlt pipelines | +| `dagster_sling` | `SlingResource` | Database replication | + +--- + +## Custom ConfigurableResource + +Create custom resources for APIs and services: + +### Basic API Resource + +```python +import dagster as dg +import requests + +class NASAResource(dg.ConfigurableResource): + api_key: str + base_url: str = "https://api.nasa.gov" + + def get_near_earth_asteroids(self, start_date: str, end_date: str) -> list: + url = f"{self.base_url}/neo/rest/v1/feed" + params = { + "start_date": start_date, + "end_date": end_date, + "api_key": self.api_key, + } + response = requests.get(url, params=params) + response.raise_for_status() + return response.json()["near_earth_objects"][start_date] +``` + +### Resource with Multiple Methods + +```python +class DatabaseClient(dg.ConfigurableResource): + connection_string: str + schema: str = "public" + + def query(self, sql: str) -> list[dict]: + """Execute a query and return results.""" + # Implementation + pass + + def execute(self, sql: str) -> None: + """Execute a statement without returning results.""" + pass + + def table_exists(self, table_name: str) -> bool: + """Check if a table exists.""" + pass +``` + +--- + +## Configuration with EnvVar + +Use `EnvVar` for secrets and environment-specific values: + +```python +import dagster as dg + +class APIResource(dg.ConfigurableResource): + api_key: str + environment: str = "production" + +# In definitions +@dg.definitions +def resources(): + return dg.Definitions( + resources={ + "api": APIResource( + api_key=dg.EnvVar("API_KEY"), + environment=dg.EnvVar("ENVIRONMENT"), + ), + }, + ) +``` + +### EnvVar vs os.getenv + +| Feature | `dg.EnvVar` | `os.getenv` | +| ------- | ----------- | ----------- | +| When evaluated | At run start | At code location load | +| Dynamic updates | Yes | No (requires restart) | +| UI visibility | Shown in resource config | Not visible | +| Best for | Production secrets | Development defaults | + +--- + +## Registering Resources + +### Modern Pattern with @definitions + +```python +# src/my_project/defs/resources.py +import dagster as dg +from dagster_duckdb import DuckDBResource + +database = DuckDBResource(database=dg.EnvVar("DUCKDB_DATABASE")) + +@dg.definitions +def resources(): + return dg.Definitions( + resources={ + "database": database, + "api": MyAPIResource(api_key=dg.EnvVar("API_KEY")), + }, + ) +``` + +### Traditional Pattern + +```python +# src/my_project/definitions.py +import dagster as dg + +defs = dg.Definitions( + assets=[...], + resources={ + "database": DuckDBResource(...), + "api": MyAPIResource(...), + }, +) +``` + +--- + +## Using Resources in Assets + +### Type-Annotated Parameter + +```python +@dg.asset +def my_asset(database: DuckDBResource) -> None: + """ + The parameter name must match the resource key in Definitions. + The type annotation tells Dagster this is a resource, not an asset. + """ + with database.get_connection() as conn: + conn.execute("SELECT 1") +``` + +### Multiple Resources + +```python +@dg.asset +def my_asset( + database: DuckDBResource, + api: NASAResource, +) -> None: + """Asset using multiple resources.""" + data = api.get_near_earth_asteroids("2024-01-01", "2024-01-07") + with database.get_connection() as conn: + # Load data into database + pass +``` + +### Resource with Context + +```python +@dg.asset +def my_asset( + context: dg.AssetExecutionContext, + database: DuckDBResource, +) -> None: + """Combine context logging with resource usage.""" + context.log.info("Starting data load") + with database.get_connection() as conn: + result = conn.execute("SELECT COUNT(*) FROM table") + context.log.info(f"Processed {result.fetchone()[0]} rows") +``` + +--- + +## Resource Lifecycle + +### Context Manager Pattern + +Many resources support context managers for proper cleanup: + +```python +@dg.asset +def my_asset(database: DuckDBResource) -> None: + # Connection automatically closed when exiting 'with' block + with database.get_connection() as conn: + conn.execute("INSERT INTO table VALUES (1, 2, 3)") +``` + +### Manual Connection Management + +For resources that don't use context managers: + +```python +class MyResource(dg.ConfigurableResource): + connection_string: str + + def connect(self): + return create_connection(self.connection_string) + + def close(self, conn): + conn.close() + +@dg.asset +def my_asset(my_resource: MyResource) -> None: + conn = my_resource.connect() + try: + # Do work + pass + finally: + my_resource.close(conn) +``` + +--- + +## Nested Resources + +Resources that depend on other resources: + +```python +class AuthResource(dg.ConfigurableResource): + api_key: str + + def get_token(self) -> str: + # Authentication logic + return f"Bearer {self.api_key}" + +class DataAPIResource(dg.ConfigurableResource): + auth: AuthResource + base_url: str + + def fetch_data(self, endpoint: str) -> dict: + headers = {"Authorization": self.auth.get_token()} + response = requests.get(f"{self.base_url}/{endpoint}", headers=headers) + return response.json() + +# In definitions +@dg.definitions +def resources(): + auth = AuthResource(api_key=dg.EnvVar("API_KEY")) + return dg.Definitions( + resources={ + "auth": auth, + "data_api": DataAPIResource(auth=auth, base_url="https://api.example.com"), + }, + ) +``` + +--- + +## Testing Resources + +### Mock Resources + +```python +from unittest.mock import Mock + +def test_asset_with_mocked_resource(): + mocked_database = Mock() + mocked_database.get_connection.return_value.__enter__ = Mock( + return_value=Mock(execute=Mock(return_value=[{"id": 1}])) + ) + mocked_database.get_connection.return_value.__exit__ = Mock(return_value=None) + + result = dg.materialize( + assets=[my_asset], + resources={"database": mocked_database}, + ) + assert result.success +``` + +### Test Resource Implementation + +```python +class TestDatabaseResource(dg.ConfigurableResource): + """In-memory database for testing.""" + + def get_connection(self): + return sqlite3.connect(":memory:") + +# Use in tests +def test_with_test_resource(): + result = dg.materialize( + assets=[my_asset], + resources={"database": TestDatabaseResource()}, + ) + assert result.success +``` + +--- + +## Common Patterns + +### Retry Logic in Resources + +```python +import time + +class RobustAPIResource(dg.ConfigurableResource): + api_key: str + max_retries: int = 3 + retry_delay: float = 1.0 + + def _request_with_retry(self, url: str, params: dict) -> dict: + for attempt in range(self.max_retries): + try: + response = requests.get(url, params=params) + response.raise_for_status() + return response.json() + except requests.RequestException as e: + if attempt == self.max_retries - 1: + raise + time.sleep(self.retry_delay * (attempt + 1)) +``` + +### Caching in Resources + +```python +from functools import lru_cache + +class CachedAPIResource(dg.ConfigurableResource): + api_key: str + + @lru_cache(maxsize=100) + def get_reference_data(self, key: str) -> dict: + """Cached lookup for reference data.""" + response = requests.get(f"https://api.example.com/ref/{key}") + return response.json() +``` + +--- + +## Anti-Patterns to Avoid + +| Anti-Pattern | Better Approach | +| ------------ | --------------- | +| Hardcoded credentials | Use `dg.EnvVar("SECRET")` | +| Creating connections in assets | Use resources for connection management | +| No type annotation on resource params | Always type: `database: DuckDBResource` | +| Global connection objects | Pass resources as parameters | +| Ignoring cleanup | Use context managers or explicit close | + +--- + +## References + +- [Resources API](https://docs.dagster.io/api/dagster/resources) +- [ConfigurableResource](https://docs.dagster.io/guides/build/external-resources) +- [Built-in Integrations](https://docs.dagster.io/integrations) diff --git a/.agents/skills/dagster-development/references/testing.md b/.agents/skills/dagster-development/references/testing.md new file mode 100644 index 0000000..b2baf09 --- /dev/null +++ b/.agents/skills/dagster-development/references/testing.md @@ -0,0 +1,494 @@ +# Testing Patterns Reference + +## Pattern Summary + +| Pattern | When to Use | +| ------- | ----------- | +| Direct function test | Simple asset with no resources | +| Test with mock inputs | Asset with dependencies | +| `dg.materialize()` | Test asset graph execution | +| Mocked resources | Isolate from external services | +| Integration tests | Verify real service connections | +| Asset checks | Runtime data validation | + +--- + +## Unit Testing Assets + +### Direct Function Testing + +Assets are Python functions - test them directly: + +```python +# src/my_project/defs/assets/population.py +@dg.asset +def state_population_file() -> list[dict]: + file_path = Path(__file__).parent / "../data/ny.csv" + with open(file_path) as file: + reader = csv.DictReader(file) + return [row for row in reader] +``` + +```python +# tests/test_population.py +def test_state_population_file(): + result = population.state_population_file() + + assert len(result) == 3 + assert result[0] == { + "City": "New York", + "Population": "8804190", + } +``` + +### Testing with Dependencies + +Provide mock inputs for dependent assets: + +```python +# Asset with dependency +@dg.asset +def total_population(state_population_file: list[dict]) -> int: + return sum([int(x["Population"]) for x in state_population_file]) +``` + +```python +# Test with controlled input +def test_total_population(): + mock_input = [ + {"City": "New York", "Population": "8804190"}, + {"City": "Buffalo", "Population": "278349"}, + {"City": "Yonkers", "Population": "211569"}, + ] + + result = total_population(mock_input) + + assert result == 9294108 +``` + +--- + +## Testing with dg.materialize() + +### Basic Materialization Test + +```python +import dagster as dg +from my_project.defs.assets import population + +def test_asset_materialization(): + result = dg.materialize( + assets=[ + population.state_population_file, + population.total_population, + ] + ) + + assert result.success +``` + +### Accessing Asset Outputs + +```python +def test_asset_outputs(): + result = dg.materialize( + assets=[ + population.state_population_file, + population.total_population, + ] + ) + + assert result.success + + # Access individual outputs + file_output = result.output_for_node("state_population_file") + assert len(file_output) == 3 + + total = result.output_for_node("total_population") + assert total == 9294108 +``` + +### Materialization with Resources + +```python +def test_with_resources(): + result = dg.materialize( + assets=[my_asset], + resources={ + "database": DuckDBResource(database=":memory:"), + }, + ) + + assert result.success +``` + +### Materialization with Run Config + +```python +from my_project.defs.assets import configurable_asset, StateConfig + +def test_with_config(): + result = dg.materialize( + assets=[configurable_asset], + run_config=dg.RunConfig({ + "configurable_asset": StateConfig(name="ny", limit=100) + }), + ) + + assert result.success +``` + +--- + +## Mocking + +### Mocking External Services + +```python +from unittest.mock import Mock, patch + +@patch("requests.get") +def test_api_asset(mock_get): + # Configure mock response + mock_response = Mock() + mock_response.json.return_value = { + "cities": [ + {"city_name": "New York", "city_population": 8804190} + ] + } + mock_response.raise_for_status.return_value = None + mock_get.return_value = mock_response + + # Execute asset + result = my_api_asset() + + # Verify results + assert len(result) == 1 + assert result[0]["city"] == "New York" + + # Verify mock was called correctly + mock_get.assert_called_once() +``` + +### Mocking Resources + +Instead of mocking functions, mock the resource: + +```python +def test_with_mocked_resource(): + # Create mock resource + mocked_resource = Mock() + mocked_resource.get_cities.return_value = [ + {"city": "Fakestown", "population": 42} + ] + + # Pass mock to asset + result = state_population_api_resource(mocked_resource) + + assert len(result) == 1 + assert result[0]["city"] == "Fakestown" +``` + +### Mocked Resources with Materialization + +```python +def test_materialization_with_mocked_resource(): + mocked_resource = Mock() + mocked_resource.get_cities.return_value = [ + {"city": "Fakestown", "population": 42} + ] + + result = dg.materialize( + assets=[ + state_population_api_resource, + total_population_resource, + ], + resources={"state_population_resource": mocked_resource}, + run_config=dg.RunConfig({ + "state_population_api_resource": StateConfig(name="ny") + }), + ) + + assert result.success + assert result.output_for_node("state_population_api_resource") == [ + {"city": "Fakestown", "population": 42} + ] + assert result.output_for_node("total_population_resource") == 42 +``` + +### When to Mock Functions vs Resources + +| Mock Functions When | Mock Resources When | +| ------------------- | ------------------- | +| Testing resource implementation | Testing asset logic | +| Need to verify call parameters | Testing asset graph | +| Resource has simple interface | Resource has multiple methods | +| Testing error handling | Testing happy path | + +--- + +## Integration Tests + +### Test with Real Services + +```python +def test_database_integration(): + """Integration test with actual database.""" + postgres_resource = PostgresResource( + host="localhost", + port=5432, + database="test_db", + user="test_user", + password="test_pass", + ) + + result = state_population_database(postgres_resource) + + assert len(result) > 0 +``` + +### Test Resource with Different Config + +```python +def test_snowflake_staging(): + """Use staging credentials for integration test.""" + staging_resource = SnowflakeResource( + account=dg.EnvVar("SNOWFLAKE_ACCOUNT"), + user=dg.EnvVar("SNOWFLAKE_USERNAME"), + password=dg.EnvVar("SNOWFLAKE_PASSWORD"), + database="STAGING", + warehouse="STAGING_WAREHOUSE", + ) + + result = state_population_database(staging_resource) + assert result.success +``` + +### Docker-Based Integration Tests + +Use Docker Compose for isolated test environments: + +```yaml +# tests/docker-compose.yaml +version: "3.8" +services: + postgres: + image: postgres:15 + environment: + POSTGRES_DB: test_db + POSTGRES_USER: test_user + POSTGRES_PASSWORD: test_pass + ports: + - "5432:5432" +``` + +```python +# tests/conftest.py +import pytest + +@pytest.fixture(scope="session") +def postgres_resource(): + """Fixture that provides test database resource.""" + return PostgresResource( + host="localhost", + port=5432, + database="test_db", + user="test_user", + password="test_pass", + ) + +def test_with_postgres(postgres_resource): + result = my_database_asset(postgres_resource) + assert result.success +``` + +--- + +## Asset Checks + +### Defining Asset Checks + +```python +import dagster as dg + +@dg.asset +def total_population( + state_population_file: list[dict], + state_population_api: list[dict], +) -> int: + all_data = state_population_file + state_population_api + return sum([int(x["Population"]) for x in all_data]) + +@dg.asset_check(asset=total_population) +def non_negative(total_population: int) -> dg.AssetCheckResult: + """Verify population is never negative.""" + return dg.AssetCheckResult( + passed=total_population > 0, + metadata={"value": total_population}, + ) +``` + +### Asset Checks with Severity + +```python +@dg.asset_check(asset=my_data) +def row_count_check(my_data: list) -> dg.AssetCheckResult: + row_count = len(my_data) + + if row_count == 0: + return dg.AssetCheckResult( + passed=False, + severity=dg.AssetCheckSeverity.ERROR, + metadata={"row_count": 0}, + ) + elif row_count < 100: + return dg.AssetCheckResult( + passed=True, # Warning, not failure + severity=dg.AssetCheckSeverity.WARN, + metadata={"row_count": row_count, "message": "Low row count"}, + ) + else: + return dg.AssetCheckResult( + passed=True, + metadata={"row_count": row_count}, + ) +``` + +### Testing Asset Checks + +```python +def test_non_negative_check(): + # Test passing case + result_pass = non_negative(10) + assert result_pass.passed + + # Test failing case + result_fail = non_negative(-10) + assert not result_fail.passed +``` + +### Multiple Asset Checks + +```python +@dg.asset_check(asset=customer_data) +def unique_ids(customer_data: list[dict]) -> dg.AssetCheckResult: + ids = [row["id"] for row in customer_data] + unique_count = len(set(ids)) + total_count = len(ids) + + return dg.AssetCheckResult( + passed=unique_count == total_count, + metadata={ + "unique_count": unique_count, + "total_count": total_count, + "duplicates": total_count - unique_count, + }, + ) + +@dg.asset_check(asset=customer_data) +def no_null_emails(customer_data: list[dict]) -> dg.AssetCheckResult: + null_emails = sum(1 for row in customer_data if row["email"] is None) + + return dg.AssetCheckResult( + passed=null_emails == 0, + metadata={"null_count": null_emails}, + ) +``` + +--- + +## Pytest Fixtures + +### Common Fixtures + +```python +# tests/conftest.py +import pytest +import dagster as dg + +@pytest.fixture +def sample_population_data(): + return [ + {"City": "New York", "Population": "8804190"}, + {"City": "Buffalo", "Population": "278349"}, + {"City": "Yonkers", "Population": "211569"}, + ] + +@pytest.fixture +def mock_database_resource(): + from unittest.mock import Mock + + mock = Mock() + mock.query.return_value = [] + return mock + +@pytest.fixture +def test_resources(): + return { + "database": DuckDBResource(database=":memory:"), + } +``` + +### Using Fixtures in Tests + +```python +def test_with_fixtures(sample_population_data, mock_database_resource): + result = total_population(sample_population_data) + assert result == 9294108 +``` + +--- + +## Testing Definitions + +### Validate Definitions Load + +```python +def test_definitions_load(): + """Verify all definitions can be loaded without errors.""" + from my_project.definitions import defs + + # Check assets exist + assert len(defs.get_all_asset_keys()) > 0 + + # Check jobs exist + assert len(defs.get_all_job_defs()) > 0 +``` + +### Test Specific Assets Exist + +```python +def test_required_assets_exist(): + from my_project.definitions import defs + + required_assets = [ + dg.AssetKey("raw_data"), + dg.AssetKey("processed_data"), + dg.AssetKey("final_report"), + ] + + all_keys = defs.get_all_asset_keys() + + for asset in required_assets: + assert asset in all_keys, f"Missing required asset: {asset}" +``` + +--- + +## Anti-Patterns to Avoid + +| Anti-Pattern | Better Approach | +| ------------ | --------------- | +| Testing in production | Use staging or mock resources | +| No assertions beyond `success` | Use `output_for_node()` to verify outputs | +| Ignoring test isolation | Each test should be independent | +| Hardcoded test data paths | Use fixtures and relative paths | +| Skipping asset check tests | Test checks like any other function | + +--- + +## References + +- [Testing Assets](https://docs.dagster.io/guides/test/testing-assets) +- [Asset Checks](https://docs.dagster.io/guides/test/asset-checks) +- [Mocking Resources](https://docs.dagster.io/guides/test/mocking-resources) diff --git a/.agents/skills/dagster-expert/SKILL.md b/.agents/skills/dagster-expert/SKILL.md new file mode 100644 index 0000000..7906b15 --- /dev/null +++ b/.agents/skills/dagster-expert/SKILL.md @@ -0,0 +1,90 @@ +--- +name: dagster-expert +description: + Expert guidance for working with Dagster and the dg CLI. ALWAYS use before doing any task that requires + knowledge specific to Dagster, or that references assets, materialization, components, data tools or data pipelines. + Common tasks may include creating a new project, adding new definitions, understanding the current project structure, answering general questions about the codebase (finding asset, schedule, sensor, component or job definitions), debugging issues, or providing deep information about a specific Dagster concept. +--- + +## dg CLI + +The `dg` CLI is the recommended way to programmatically interact with Dagster (adding definitions, launching runs, exploring project structure, etc.). It is installed as part of the `dagster-dg-cli` package. If a relevant CLI command for a given task exists, always attempt to use it. + +ONLY explore the existing project structure if it is strictly necessary to accomplish the user's goal. In many cases, existing CLI tools will have sufficient understanding of the project structure, meaning listing and reading existing files is wasteful and unnecessary. + +Almost all `dg` commands that return information have a `--json` flag that can be used to get the information in a machine-readable format. This should be preferred over the default table output unless you are directly showing the information to the user. + +## UV Compatibility + +Projects typically use `uv` for dependency management, and it is recommended to use it for `dg` commands if possible: + +```bash +uv run dg list defs +uv run dg launch --assets my_asset +``` + +## Core Dagster Concepts + +Brief definitions only (see reference files for detailed examples): + +- **Asset**: Persistent object (table, file, model) produced by your pipeline +- **Component**: Reusable building block that generates definitions (assets, schedules, sensors, jobs, etc.) relevant to a particular domain. + +## CRITICAL: Always Read Reference Files Before Answering + +NEVER answer from memory or guess at CLI commands, APIs, or syntax. ALWAYS read the relevant reference file(s) from the Reference Index below before responding. + +For every question, identify which reference file(s) are relevant using the index descriptions, read them, then answer based on what you read. + +## Reference Index + + + +- [Asset Patterns](./references/assets.md) — defining assets, dependencies, metadata, partitions, or multi-asset definitions +- [Environment Variables](./references/env-vars.md) — configuring environment variables across different environments +- [Choosing an Automation Approach](./references/automation/choosing-automation.md) — deciding between schedules, sensors, and declarative automation +- [Schedules](./references/automation/schedules.md) — time-based automation with cron expressions +- [Declarative Automation](./references/automation/declarative-automation/INDEX.md) — asset-centric condition-based automation using AutomationCondition +- [Asset Sensors](./references/automation/sensors/asset-sensors.md) — triggering on asset materialization events +- [Basic Sensors](./references/automation/sensors/basic-sensors.md) — event-driven automation with file watching or custom polling +- [Run Status Sensors](./references/automation/sensors/run-status-sensors.md) — reacting to run success, failure, or other status changes +- [Asset Selection Syntax](./references/cli/asset-selection.md) — filtering assets by tag, group, kind, upstream, or downstream +- [dg check](./references/cli/check.md) — validating project configuration or definitions +- [create-dagster](./references/cli/create-dagster.md) — creating a new Dagster project from scratch +- [dg launch](./references/cli/launch.md) — materializing assets or executing jobs locally +- [dg api: General](./references/cli/api/general.md) — always read before using any dg api subcommand +- [dg api agent get](./references/cli/api/agent/get.md) — details about a specific Dagster Plus agent +- [dg api agent list](./references/cli/api/agent/list.md) — listing agents in Dagster Plus +- [dg api asset get-evaluations](./references/cli/api/asset/get-evaluations.md) — automation condition evaluation history for an asset +- [dg api asset get-events](./references/cli/api/asset/get-events.md) — materialization or observation event history for an asset +- [dg api asset get](./references/cli/api/asset/get.md) — details about a specific asset +- [dg api asset list](./references/cli/api/asset/list.md) — querying which assets exist in a deployment +- [dg api deployment get](./references/cli/api/deployment/get.md) — details about a specific deployment +- [dg api deployment list](./references/cli/api/deployment/list.md) — listing deployments in Dagster Plus +- [dg api run get-events](./references/cli/api/run/get-events.md) — debugging a run by reading its logs; filtering run events by level or step +- [dg api run get](./references/cli/api/run/get.md) — details about a specific run +- [dg api run list](./references/cli/api/run/list.md) — listing or filtering runs +- [dg api schedule get](./references/cli/api/schedule/get.md) — details about a specific schedule +- [dg api schedule list](./references/cli/api/schedule/list.md) — listing schedules in Dagster Plus +- [dg api secret get](./references/cli/api/secret/get.md) — details about a specific secret +- [dg api secret list](./references/cli/api/secret/list.md) — listing secrets in Dagster Plus +- [dg api sensor get](./references/cli/api/sensor/get.md) — details about a specific sensor +- [dg api sensor list](./references/cli/api/sensor/list.md) — listing sensors in Dagster Plus +- [dg list component-tree](./references/cli/list/component-tree.md) — viewing the component instance hierarchy +- [dg list components](./references/cli/list/components.md) — seeing available component types for scaffolding +- [dg list defs](./references/cli/list/defs.md) — listing or filtering registered definitions +- [dg list envs](./references/cli/list/envs.md) — seeing which environment variables the project requires +- [dg list projects](./references/cli/list/projects.md) — listing projects in the current workspace +- [dg plus login](./references/cli/plus/login.md) — authenticating with Dagster Plus +- [dg plus pull env](./references/cli/plus/pull/env.md) — pulling environment variables from Dagster Plus into a local .env file +- [dg scaffold component](./references/cli/scaffold/component.md) — creating a custom reusable component type +- [dg scaffold defs](./references/cli/scaffold/defs.md) — adding new definitions (assets, schedules, sensors, components) to a project +- [Creating Components](./references/components/creating-components.md) — building a new custom component from scratch +- [Designing Component Integrations](./references/components/designing-component-integrations.md) — designing a component that wraps an external service or tool +- [Resolved Framework](./references/components/resolved-framework.md) — defining custom YAML schema types using Resolver, Model, or Resolvable +- [Subclassing Components](./references/components/subclassing-components.md) — extending an existing component via subclassing +- [Template Variables](./references/components/template-variables.md) — using Jinja2 template variables in component YAML (env, dg, context, or custom scopes) +- [Creating State-Backed Components](./references/components/state-backed/creating.md) — building a component that fetches and caches external state +- [Using State-Backed Components](./references/components/state-backed/using.md) — managing state-backed components in production, CI/CD, or refreshing state +- [Integrations](./references/integrations/INDEX.md) — needs an integration library for an external tool or technology + diff --git a/.agents/skills/dagster-expert/references/assets.md b/.agents/skills/dagster-expert/references/assets.md new file mode 100644 index 0000000..d54cd69 --- /dev/null +++ b/.agents/skills/dagster-expert/references/assets.md @@ -0,0 +1,530 @@ +--- +title: Asset Patterns +triggers: + - "defining assets, dependencies, metadata, partitions, or multi-asset definitions" +--- + +# Asset Patterns Reference + +## Pattern Summary + +| Pattern | When to Use | +| -------------------------- | ------------------------------------------------- | +| Basic `@dg.asset` | Simple one-to-one transformation | +| `@multi_asset` | Single operation produces multiple related assets | +| `@graph_asset` | Multiple steps needed to produce one asset | +| `@graph_multi_asset` | Complex pipeline producing multiple assets | +| Parameter-based dependency | Asset depends on another managed asset | +| `deps=` dependency | Asset depends on external or non-Python asset | +| Asset with metadata | Track runtime metrics (row counts, timestamps) | +| Asset groups | Organize related assets visually | +| Asset key prefixes | Namespace assets for multi-tenant or layered data | +| Partitioned assets | Time-series or categorical data splits | +| Asset with `code_version` | Track when asset logic changes | + +--- + +## Launching Assets + +Once you've defined assets, use the `dg launch` command to materialize them. For comprehensive documentation on launching assets, see the [CLI launch reference](./cli/launch.md). + +--- + +## Basic Asset Definition + +```python +import dagster as dg + +@dg.asset +def my_asset() -> None: + """ + Docstring becomes the asset description in the UI. + """ + # Your computation logic + pass +``` + +**Key points**: + +- Function name becomes the asset key +- Docstring becomes the description +- Return type annotation is optional but recommended + +--- + +## Asset Dependencies + +### Parameter-Based Dependencies + +When an asset depends on another Dagster-managed asset, use parameters: + +```python +@dg.asset +def upstream_asset() -> dict: + return {"data": [1, 2, 3]} + +@dg.asset +def downstream_asset(upstream_asset: dict) -> list: + """ + Receives the return value of upstream_asset automatically. + """ + return upstream_asset["data"] +``` + +**How it works**: + +- Parameter name must match the upstream asset's function name +- Dagster automatically passes the materialized output +- Creates a visual dependency in the asset graph + +### External Dependencies with `deps=` + +When an asset depends on something not managed by Dagster or without a return value: + +```python +@dg.asset(deps=["external_table", "raw_file"]) +def processed_data() -> None: + """ + Depends on external_table and raw_file but doesn't receive their values. + """ + # Read from external sources directly + pass +``` + +**Use `deps=` when**: + +- The upstream asset doesn't return a value (returns `None`) +- The asset is external (file created by another process) +- You need loose coupling between assets + +### Mixed Dependencies + +Combine both patterns when needed: + +```python +@dg.asset(deps=["raw_file"]) +def enriched_data(reference_table: dict) -> dict: + """ + Depends on: + - raw_file (via deps=, doesn't receive value) + - reference_table (via parameter, receives value) + """ + # Read raw_file from disk, enrich with reference_table + return {"enriched": reference_table} +``` + +--- + +## Asset Metadata + +### Definition Metadata (Static) + +Applied once when the asset is defined: + +```python +@dg.asset( + description="Detailed description for the UI", + group_name="analytics", + key_prefix=["warehouse", "staging"], + owners=["team:data-engineering", "user@example.com"], + tags={"priority": "high", "pii": "true", "domain": "sales"}, + code_version="1.2.0", +) +def my_asset() -> None: + pass +``` + +**Best Practices for Metadata**: + +- **owners**: Specify team (`team:name`) or individuals for accountability +- **tags**: Primary organizational mechanism—use liberally for filtering and grouping +- **code_version**: Track when asset logic changes for lineage and debugging +- **description**: Explain what the asset represents and its business purpose +- **group_name**: Visual organization in UI (use for layers or domains) +- **key_prefix**: Hierarchical namespacing for multi-tenant or layered architectures + +### Materialization Metadata (Dynamic) + +Captured each time the asset materializes: + +```python +import dagster as dg +from datetime import datetime + +@dg.asset +def my_asset() -> dg.MaterializeResult: + """Asset that reports metadata on each run.""" + data = [...] + row_count = len(data) + + # Save data... + + return dg.MaterializeResult( + metadata={ + "row_count": dg.MetadataValue.int(row_count), + "last_updated": dg.MetadataValue.text(str(datetime.now())), + "sample_data": dg.MetadataValue.json(data[:5]), + } + ) +``` + +### MetadataValue Types + +| Type | Usage | +| ------------------------------ | --------------------------- | +| `MetadataValue.int(n)` | Integer values (row counts) | +| `MetadataValue.float(n)` | Float values (percentages) | +| `MetadataValue.text(s)` | Short text values | +| `MetadataValue.json(obj)` | JSON-serializable objects | +| `MetadataValue.md(s)` | Markdown text | +| `MetadataValue.url(s)` | Clickable URLs | +| `MetadataValue.path(s)` | File paths | +| `MetadataValue.table(records)` | Tabular data | + +--- + +## Asset Groups + +Organize related assets visually in the UI: + +```python +@dg.asset(group_name="raw_data") +def raw_orders() -> None: + pass + +@dg.asset(group_name="raw_data") +def raw_customers() -> None: + pass + +@dg.asset(group_name="analytics") +def daily_revenue(raw_orders) -> None: + pass +``` + +**Best practices**: + +- Group by data layer: `raw`, `staging`, `analytics`, `mart` +- Group by domain: `sales`, `marketing`, `finance` +- Group by source: `postgres`, `api`, `files` + +--- + +## Asset Key Prefixes + +Namespace assets for organization: + +```python +@dg.asset(key_prefix=["warehouse", "raw"]) +def orders() -> None: + """Asset key becomes: warehouse/raw/orders""" + pass + +@dg.asset(key_prefix=["warehouse", "staging"]) +def orders_cleaned() -> None: + """Asset key becomes: warehouse/staging/orders_cleaned""" + pass +``` + +**Use prefixes for**: + +- Multi-tenant architectures +- Environment separation +- Data layer organization (bronze/silver/gold) + +--- + +## Asset with Execution Context + +Access runtime information during materialization: + +```python +@dg.asset +def context_aware_asset(context: dg.AssetExecutionContext) -> None: + context.log.info("Starting asset materialization") + + # Access asset key + asset_key = context.asset_key + + # Access partition key (if partitioned) + if context.has_partition_key: + partition = context.partition_key + context.log.info(f"Processing partition: {partition}") + + # Access run ID + run_id = context.run_id +``` + +--- + +## Asset with Configuration + +Make assets configurable at runtime: + +```python nocheckundefined +class MyAssetConfig(dg.Config): + limit: int = 100 + include_archived: bool = False + source_path: str + +@dg.asset +def configurable_asset(config: MyAssetConfig) -> None: + """ + Run with specific configuration values. + """ + data = load_data( + path=config.source_path, + limit=config.limit, + include_archived=config.include_archived, + ) +``` + +--- + +## Asset Return Types + +### Returning Data Directly + +```python +@dg.asset +def returns_data() -> dict: + return {"key": "value"} +``` + +### Returning MaterializeResult + +```python +@dg.asset +def returns_result() -> dg.MaterializeResult: + # Do work... + return dg.MaterializeResult( + metadata={"rows": 100} + ) +``` + +### Returning Output with Type + +```python +from dagster import Output + +@dg.asset +def returns_output() -> Output[dict]: + data = {"key": "value"} + return Output( + value=data, + metadata={"size": len(data)}, + ) +``` + +--- + +## Multi-Asset Pattern + +Define multiple assets from one function: + +```python nocheckundefined +@dg.multi_asset( + outs={ + "users": dg.AssetOut(), + "orders": dg.AssetOut(), + } +) +def load_data(): + users_df = fetch_users() + orders_df = fetch_orders() + + yield dg.Output(users_df, output_name="users") + yield dg.Output(orders_df, output_name="orders") +``` + +**Use when**: + +- One computation produces multiple logical assets +- Assets are always created together +- Shared setup is expensive + +--- + +## Graph Assets + +### @graph_asset Pattern + +Use when multiple steps are needed to produce a single asset: + +```python +@dg.op +def fetch_data() -> dict: + return {"raw": [1, 2, 3]} + +@dg.op +def transform_data(data: dict) -> dict: + return {"processed": [x * 2 for x in data["raw"]]} + +@dg.op +def validate_data(data: dict) -> dict: + assert len(data["processed"]) > 0 + return data + +@dg.graph_asset +def complex_asset(): + """Encapsulates multi-step logic into a single asset.""" + raw = fetch_data() + processed = transform_data(raw) + return validate_data(processed) +``` + +**Use When**: + +- Single asset requires multiple distinct steps +- You want to encapsulate complexity +- Testing individual steps is important +- Steps are reusable across multiple assets + +### @graph_multi_asset Pattern + +Use for complex pipelines producing multiple assets: + +```python nocheckundefined +@dg.graph_multi_asset( + outs={ + "users": dg.AssetOut(), + "orders": dg.AssetOut(), + } +) +def etl_pipeline(): + """Multi-step pipeline producing multiple assets.""" + raw_data = extract_from_api() + cleaned = clean_data(raw_data) + users_out = extract_users(cleaned) + orders_out = extract_orders(cleaned) + + return {"users": users_out, "orders": orders_out} +``` + +**Use When**: + +- Multiple assets require shared complex logic +- Steps are expensive and should be shared +- Better encapsulation than separate assets with deps + +--- + +## Asset Factories + +Generate similar assets programmatically: + +```python nocheckundefined +def create_table_asset(table_name: str, schema: str): + @dg.asset( + name=f"{schema}_{table_name}", + group_name=schema, + ) + def _asset() -> None: + load_table(schema, table_name) + + return _asset + +# Generate assets +customers = create_table_asset("customers", "sales") +products = create_table_asset("products", "catalog") +orders = create_table_asset("orders", "sales") +``` + +--- + +## Asset Selection Syntax + +Select subsets of assets for materialization or job definition: + +### Basic Selection + +```python +# Select specific assets +dg.AssetSelection.assets("asset_a", "asset_b", "asset_c") + +# Select all assets +dg.AssetSelection.all() + +# Select by group +dg.AssetSelection.groups("analytics", "raw_data") + +# Select by key prefix +dg.AssetSelection.key_prefixes(["warehouse", "staging"]) + +# Select by tag +dg.AssetSelection.tag("priority", "high") +``` + +### Dependency-Based Selection + +```python +# Select asset and all upstream dependencies +dg.AssetSelection.assets("final_report").upstream() + +# Select asset and all downstream dependencies +dg.AssetSelection.assets("raw_data").downstream() + +# Select asset and immediate upstream only +dg.AssetSelection.assets("final_report").upstream(depth=1) +``` + +### Combining Selections + +```python +selection_a = dg.AssetSelection.assets("a") +selection_b = dg.AssetSelection.assets("b") + +# Union: assets in A OR B +selection_a | selection_b + +# Intersection: assets in A AND B +selection_a & selection_b + +# Difference: assets in A but not in B +selection_a - selection_b + +# Example: All analytics assets except one +dg.AssetSelection.groups("analytics") - dg.AssetSelection.assets("excluded_asset") +``` + +### Using in Jobs + +```python +analytics_job = dg.define_asset_job( + name="analytics_job", + selection=dg.AssetSelection.groups("analytics").downstream(), +) +``` + +### Using in CLI + +```bash +# Materialize specific assets +dg launch --assets asset_a,asset_b + +# Materialize all assets +dg launch --assets "*" + +# Materialize by group (requires selection in job) +dg launch --job analytics_job +``` + +--- + +## Common Anti-Patterns + +| Anti-Pattern | Better Approach | +| ---------------------------------- | ---------------------------------------------- | +| `load_customers` (verb-based name) | `customers` (noun describing output) | +| Giant asset doing everything | Split into focused, composable assets | +| No type annotations | Add return type: `-> dict`, `-> None` | +| No docstring | Add description in docstring or `description=` | +| Ignoring `MaterializeResult` | Return metadata for observability | +| Hardcoded paths | Use configuration or environment variables | + +--- + +## References + +- [Assets API](https://docs.dagster.io/api/dagster/assets) +- [Asset Metadata](https://docs.dagster.io/guides/build/assets/metadata-and-tags) +- [Multi-Assets](https://docs.dagster.io/guides/build/assets/multi-assets) diff --git a/.agents/skills/dagster-expert/references/automation/choosing-automation.md b/.agents/skills/dagster-expert/references/automation/choosing-automation.md new file mode 100644 index 0000000..93dd444 --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/choosing-automation.md @@ -0,0 +1,59 @@ +--- +title: Choosing an Automation Approach +triggers: + - "deciding between schedules, sensors, and declarative automation" +--- + +# Choosing an Automation Approach + +Dagster provides three main approaches to automation: schedules for time-based execution, sensors for event-driven triggers, and declarative automation for asset-centric condition-based orchestration. + +## Workflow Decision Tree + +Choose your automation approach based on your use case: + +- Simple, fixed time-based execution → **Schedules** +- Custom polling logic → **Basic Sensors** +- Launching jobs in response to asset materialization events → **Asset Sensors** +- Triggering compute based on run success/failure → **Run Status Sensors** +- Partition-aware scheduling, declarative/asset-based scheduling, scheduling depending on asset graph state and materialization events → **Declarative Automation** + +## Core Concepts + +### Jobs + +A **job** is a selection of assets to execute together. Jobs are the unit of execution that schedules and sensors trigger. + +```python +import dagster as dg + +# Define a job that selects specific assets +analytics_job = dg.define_asset_job( + name="analytics_job", + selection=["sales_data", "customer_metrics"] +) +``` + +Jobs can also select assets by tags, groups, or patterns: + +```python +# Select all assets with a specific tag +tagged_job = dg.define_asset_job( + name="daily_job", + selection=dg.AssetSelection.tag("priority", "high") +) + +# Select all assets in a group +group_job = dg.define_asset_job( + name="etl_job", + selection=dg.AssetSelection.groups("etl") +) +``` + +### Automation Approaches + +**Schedules**: Time-based execution with cron expressions. Best for predictable, recurring tasks. + +**Sensors**: Poll for external events and trigger runs. Best for file arrivals, API events, or custom conditions. + +**Declarative Automation**: Set conditions directly on assets. Best for complex dependency logic and asset-centric workflows. Automatic handling of asset and partition state and dependencies. diff --git a/.agents/skills/dagster-expert/references/automation/declarative-automation/INDEX.md b/.agents/skills/dagster-expert/references/automation/declarative-automation/INDEX.md new file mode 100644 index 0000000..0e2a578 --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/declarative-automation/INDEX.md @@ -0,0 +1,64 @@ +--- +title: Declarative Automation +type: index +triggers: + - "asset-centric condition-based automation using AutomationCondition" +--- + +# Declarative Automation Reference + +Declarative automation uses `AutomationCondition` objects to describe when assets should execute. Instead of scheduling jobs, you define conditions on assets that the system evaluates automatically. + +## Overview + +**Modern automation pattern**: Set conditions directly on assets rather than creating separate schedules or sensors. The system evaluates conditions every 30 seconds and launches runs when conditions are met. + +**Benefits**: + +- Asset-native: No separate job definitions needed +- Dependency-aware: Automatically considers upstream state +- Composable: Build complex conditions from simple building blocks +- Declarative: Easier to reason about than imperative sensors + +**Basic examples**: See the main SKILL.md Quick Reference for `eager()`, `on_cron()`, and `on_missing()` examples. + +## Requirements + +- **Assets only**: Declarative automation does not work with ops or graphs +- **Sensor must be enabled**: The `default_automation_condition_sensor` must be toggled on in the Dagster UI under **Automation → Sensors** + +## Core Concepts + +### The Three Main Conditions + +Start with one of these three conditions rather than building conditions from scratch: + +- **`eager()`**: Execute immediately when dependencies update +- **`on_cron()`**: Execute on a schedule after dependencies update +- **`on_missing()`**: Execute missing partitions when dependencies are ready + +### Customization + +All three main conditions can be customized: + +- Remove sub-conditions with `.without()` +- Replace sub-conditions with `.replace()` +- Filter dependencies with `.allow()` and `.ignore()` +- Combine with boolean operators: `&` (AND), `|` (OR), `~` (NOT) + +### Advanced Concepts + +- **Status vs Events**: Conditions can be persistent states or transient moments +- **Operands**: Base building blocks like `missing()`, `newly_updated()` +- **Operators**: Tools for composition like `since()`, `any_deps_match()` + +## Reference Files Index + + + +- [Advanced](./advanced.md) — status vs events, run grouping, or filtering in declarative automation +- [Core Concepts](./core-concepts.md) — using eager(), on_cron(), or on_missing() conditions +- [Customization](./customization.md) — customizing conditions with without(), replace(), allow(), or ignore() +- [Operands](./operands.md) — base condition building blocks like missing() or newly_updated() +- [Operators](./operators.md) — combining conditions using since, any_deps_match, or boolean operators + diff --git a/.agents/skills/dagster-expert/references/automation/declarative-automation/advanced.md b/.agents/skills/dagster-expert/references/automation/declarative-automation/advanced.md new file mode 100644 index 0000000..5a444d2 --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/declarative-automation/advanced.md @@ -0,0 +1,225 @@ +--- +title: Advanced +triggers: + - "status vs events, run grouping, or filtering in declarative automation" +--- + +# Declarative Automation: Advanced Concepts + +This document covers advanced topics for deep understanding of the declarative automation system. + +## Status vs Events + +Understanding the distinction between statuses and events is fundamental to building correct automation conditions. + +### Statuses + +**Statuses** are persistent conditions that remain true for multiple evaluation ticks. + +Examples: + +- `AutomationCondition.missing()` - Stays true until the partition is materialized +- `AutomationCondition.in_progress()` - True while a run is executing +- `AutomationCondition.in_latest_time_window()` - True for the latest time partition(s) + +**Characteristic**: If the underlying state doesn't change, the status will be true for consecutive evaluations. + +### Events + +**Events** are transient conditions that are true only on a single evaluation tick. + +Examples: + +- `AutomationCondition.newly_updated()` - True only on the tick when materialization occurs +- `AutomationCondition.code_version_changed()` - True only on the first tick after code changes +- `AutomationCondition.cron_tick_passed()` - True only on the first tick after the cron tick + +**Characteristic**: Even if evaluated immediately again, the event would be false (assuming no new change). + +### Converting Between Status and Event + +**Status → Event with `newly_true()`**: + +```python +# missing() is a status (stays true for many ticks) +# newly_true() converts it to an event (true only when becoming missing) +condition = dg.AutomationCondition.missing().newly_true() +``` + +**Use case**: Prevent repeated requests during persistent states. A partition stays missing while a run is in progress. Using `newly_true()` ensures you only request it once. + +**Two Events → Status with `since()`**: + +```python +# Both newly_updated() and newly_requested() are events +# since() converts them to a status: "updated more recently than requested" +condition = dg.AutomationCondition.newly_updated().since( + dg.AutomationCondition.newly_requested() +) +``` + +**Use case**: Create persistent states from transient events. This condition becomes true when an update occurs and stays true until a request is made. + +### Example: Preventing Duplicate Requests + +The default `eager()` condition uses this pattern: + +```python +( + dg.AutomationCondition.newly_missing() + | dg.AutomationCondition.any_deps_updated() +).since_last_handled() +``` + +- `newly_missing()` and `any_deps_updated()` are events +- `since_last_handled()` converts them to a status that persists until the asset is requested or updated +- Without this conversion, the condition would only be true for a single tick, potentially missing the opportunity to launch a run + +## Run Grouping + +Run grouping allows multiple assets to execute in a single run even though downstream assets' dependencies haven't been materialized yet. + +### The Problem + +Consider assets A → B → C, all with `eager()` conditions: + +1. A's upstream updates, triggering A +2. A is requested and begins executing +3. On the next tick, B sees that A hasn't finished materializing +4. Without run grouping, B would wait for A to complete +5. This results in three separate runs instead of one + +### The Solution: will_be_requested() + +The `will_be_requested()` operand is true for assets that will be requested in the current tick. Dependency conditions use this to group assets: + +```python +# From any_deps_updated() definition: +dg.AutomationCondition.any_deps_match( + ( + dg.AutomationCondition.newly_updated() + & ~dg.AutomationCondition.executed_with_root_target() + ) + | dg.AutomationCondition.will_be_requested() # Enables run grouping +) +``` + +When evaluating B: + +1. B checks if any dependencies are updated OR will be requested this tick +2. A is marked as "will be requested" this tick +3. B treats A as if it were already updated +4. B is also marked for execution in the same run as A + +### Requirements for Same-Run Execution + +Two assets can execute in the same run if: + +1. **Same repository**: They must be in the same code location +2. **Compatible partitions**: They must have matching `PartitionsDefinition` objects +3. **Compatible partition mapping**: Must use `TimeWindowPartitionMapping` or `IdentityPartitionMapping` + +If these requirements aren't met, assets execute in separate runs even with run grouping logic. + +## Dependency Filtering with allow() and ignore() + +Dependency operators (`any_deps_match()`, `all_deps_match()`) check conditions on upstream assets. Filtering controls which upstreams are checked. + +### allow() Creates Intersection + +Only dependencies in the selection are checked: + +```python +condition = dg.AutomationCondition.any_deps_match( + dg.AutomationCondition.missing() +).allow(dg.AssetSelection.groups("critical")) +``` + +If the asset has 10 upstreams but only 2 are in the "critical" group, only those 2 are checked. + +### ignore() Creates Subtraction + +Dependencies in the selection are excluded: + +```python +condition = dg.AutomationCondition.any_deps_updated().ignore( + dg.AssetSelection.assets("test_data", "staging_data") +) +``` + +Updates to "test_data" and "staging_data" won't trigger the condition. + +### Propagation Through Operators + +When applied to composite conditions (AND/OR), filtering propagates to all sub-conditions: + +```python +# Applies to both any_deps_missing() and any_deps_in_progress() within eager() +condition = dg.AutomationCondition.eager().allow( + dg.AssetSelection.groups("production") +) +``` + +**What gets filtered**: All `any_deps_match()` and `all_deps_match()` calls + +**What doesn't get filtered**: Direct operands like `missing()` on the asset itself + +## Understanding since_last_handled() + +`since_last_handled()` is a convenience method that converts events to a status: + +```python +condition = dg.AutomationCondition.newly_missing() + +# These are equivalent: +condition.since_last_handled() + +condition.since( + dg.AutomationCondition.newly_requested() + | dg.AutomationCondition.newly_updated() + | dg.AutomationCondition.initial_evaluation() +) +``` + +**Behavior**: + +- Becomes true when `condition` becomes true +- Stays true until the asset is requested, updated, or the condition is first applied +- Resets on initial evaluation to handle condition changes + +**Use case**: Persist an event until it's "handled" by either requesting or materializing the asset. This prevents duplicate requests while ensuring the event isn't lost. + +## Composite Conditions Deep Dive + +### any_deps_updated() + +```python +dg.AutomationCondition.any_deps_match( + (dg.AutomationCondition.newly_updated() & ~dg.AutomationCondition.executed_with_root_target()) + | dg.AutomationCondition.will_be_requested() +) +``` + +Checks if any dependency has newly updated (excluding same-run updates) OR will be requested this tick. + +### any_deps_missing() + +```python +dg.AutomationCondition.any_deps_match( + dg.AutomationCondition.missing() & ~dg.AutomationCondition.will_be_requested() +) +``` + +Checks if any dependency is missing AND will NOT be requested this tick. Dependencies that will be requested aren't considered blocking. + +### all_deps_updated_since_cron() + +```python nocheckundefined +dg.AutomationCondition.all_deps_match( + dg.AutomationCondition.newly_updated().since( + dg.AutomationCondition.cron_tick_passed(cron_schedule, cron_timezone) + ) +) +``` + +For each dependency, checks if it has been updated since the last cron tick. All dependencies must have at least one partition updated since the tick. diff --git a/.agents/skills/dagster-expert/references/automation/declarative-automation/core-concepts.md b/.agents/skills/dagster-expert/references/automation/declarative-automation/core-concepts.md new file mode 100644 index 0000000..1bad241 --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/declarative-automation/core-concepts.md @@ -0,0 +1,199 @@ +--- +title: Core Concepts +triggers: + - "using eager(), on_cron(), or on_missing() conditions" +--- + +# Declarative Automation: Core Concepts + +For basic examples, see the main SKILL.md Quick Reference section on Declarative Automation. + +## The Three Main Conditions + +Dagster provides three primary conditions optimized for common use cases. Start with one of these rather than building conditions from scratch. + +### eager() + +Executes an asset whenever any dependency updates. Also materializes partitions that become missing after the condition is applied. + +```python +import dagster as dg + +@dg.asset(automation_condition=dg.AutomationCondition.eager()) +def downstream_asset(upstream_asset): + # Executes immediately when upstream_asset materializes + ... +``` + +**Behavior**: + +- Triggers immediately when any upstream updates +- Waits for all upstreams to be materialized or in-progress +- Does not execute if any dependencies are missing +- **Does not execute if any dependencies are currently in-progress** (waits for all deps to finish first) +- Does not execute if the asset is already in-progress +- For time-partitioned assets, only considers the latest partition +- For static/dynamic-partitioned assets, considers all partitions + +**Full expanded form**: + +```python +( + dg.AutomationCondition.in_latest_time_window() # latest partition only (time-partitioned) + & ( + dg.AutomationCondition.newly_missing() + | dg.AutomationCondition.any_deps_updated() + ).since_last_handled() # trigger event, persisted until handled + & ~dg.AutomationCondition.any_deps_missing() # no deps missing + & ~dg.AutomationCondition.any_deps_in_progress() # no deps currently running + & ~dg.AutomationCondition.in_progress() # asset itself not running +).with_label("eager") +``` + +The `~any_deps_in_progress()` guard is critical: it prevents the asset from firing until ALL upstream deps have finished materializing. Without it, the asset would fire each time an individual dep completes, causing redundant executions when multiple deps update in quick succession (e.g., from the same scheduled job). + +**Use when**: You want updates to propagate downstream immediately without waiting for a schedule. + +### on_cron() + +Executes an asset on a cron schedule after all dependencies have updated since the latest cron tick. + +```python +@dg.asset( + automation_condition=dg.AutomationCondition.on_cron("0 9 * * *", "America/Los_Angeles") +) +def daily_summary(hourly_data): + # Executes at 9 AM only if hourly_data has updated since the previous 9 AM tick + ... +``` + +**Behavior**: + +- Waits for a cron tick to occur +- After the tick, waits for all dependencies to update since that tick +- Once all dependencies are updated, executes immediately +- For time-partitioned assets, only considers the latest partition + **Full expanded form**: + +```python +cron_schedule = "0 1 * * *" +cron_timezone = "US/Eastern" +( + dg.AutomationCondition.in_latest_time_window() + & dg.AutomationCondition.cron_tick_passed( + cron_schedule, cron_timezone + ).since_last_handled() + & dg.AutomationCondition.all_deps_updated_since_cron(cron_schedule, cron_timezone) +).with_label(f"on_cron({cron_schedule}, {cron_timezone})") +``` + +**Use when**: You want scheduled execution but only after upstream data is ready. More intelligent than simple schedules. + +### on_missing() + +Executes missing asset partitions when all upstream partitions are available. + +```python +@dg.asset(automation_condition=dg.AutomationCondition.on_missing()) +def backfill_asset(upstream): + # Executes for any missing partitions when upstream is ready + ... +``` + +**Behavior**: + +- Only materializes partitions that are missing +- Only considers partitions added after the condition was applied (not historical) +- Waits for all upstream dependencies to be available +- For time-partitioned assets, only considers the latest partition + **Full expanded form**: + +```python +( + dg.AutomationCondition.in_latest_time_window() + & ( + dg.AutomationCondition.missing() + .newly_true() + .since_last_handled() + .with_label("missing_since_last_handled") + ) + & ~dg.AutomationCondition.any_deps_missing() +).with_label("on_missing") +``` + +**Use when**: You want to fill in missing partitions as upstream data becomes available. Good for backfilling or catching up. + +## Identifying Built-in vs Custom Conditions from the API + +When debugging DA behavior via `dg api asset get`, the `automation_condition.expanded_label` field shows the condition tree as a list of strings. Compare this against the full expanded forms above to determine if the asset is using a built-in condition or a custom one with missing guards. When you see a condition that looks similar to but doesn't match a built-in, always identify the missing sub-conditions and explain how their absence changes behavior. + +## Evaluation by Sensor + +The `AutomationConditionSensorDefinition` evaluates conditions at regular intervals. + +**Default sensor**: A sensor named `default_automation_condition_sensor` is created automatically in code locations with automation conditions. + +**Configuration**: + +- Evaluates all conditions every 30 seconds +- Must be toggled on in the UI under **Automation → Sensors** +- Launches runs when conditions evaluate to true + +**Important**: If the sensor is not enabled, conditions will not be evaluated and no runs will launch. + +## Basic Customization + +All three main conditions are built from smaller components and can be customized. + +### Modifying conditions + +```python +# Remove sub-conditions +condition = dg.AutomationCondition.eager().without( + ~dg.AutomationCondition.any_deps_missing() +) + +# Replace sub-conditions +condition = dg.AutomationCondition.on_cron("0 9 * * *").replace( + old=dg.AutomationCondition.all_deps_updated_since_cron("0 9 * * *"), + new=dg.AutomationCondition.all_deps_updated_since_cron("0 0 * * *"), +) +``` + +### Boolean composition + +```python +# AND: Both conditions must be true +condition = ( + dg.AutomationCondition.eager() + & ~dg.AutomationCondition.in_progress() +) + +# OR: Either condition can be true +condition = ( + dg.AutomationCondition.on_cron("0 9 * * *") + | dg.AutomationCondition.any_deps_updated() +) +``` + +See [customization.md](customization.md) for detailed patterns and examples. + +## When to Use Declarative Automation + +**Use declarative automation when**: + +- Asset-centric pipelines with complex update logic +- Condition-based triggers (data availability, freshness) +- Dependency-aware execution is needed +- You prefer declarative over imperative + +**Use schedules when**: + +- Simple time-based execution without dependency logic +- Predictable, fixed-time execution is sufficient + +**Use sensors when**: + +- Custom polling logic for external systems +- Imperative actions beyond asset execution +- File watching or API event monitoring diff --git a/.agents/skills/dagster-expert/references/automation/declarative-automation/customization.md b/.agents/skills/dagster-expert/references/automation/declarative-automation/customization.md new file mode 100644 index 0000000..60862f2 --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/declarative-automation/customization.md @@ -0,0 +1,162 @@ +--- +title: Customization +triggers: + - "customizing conditions with without(), replace(), allow(), or ignore()" +--- + +# Declarative Automation: Customization + +Start with one of the three main conditions (`eager()`, `on_cron()`, `on_missing()`) and customize them using these patterns. + +## Pattern 1: Removing Sub-conditions with without() + +Remove unwanted sub-conditions from composite conditions. + +**Allow missing upstreams**: By default, `eager()` waits for all dependencies. Remove this requirement: + +```python +import dagster as dg + +condition = ( + dg.AutomationCondition.eager() + .without(~dg.AutomationCondition.any_deps_missing()) + .with_label("eager_allow_missing") +) +``` + +**Update all time partitions**: By default, `eager()` only updates the latest time partition. Remove this restriction: + +```python +condition = ( + dg.AutomationCondition.eager() + .without(dg.AutomationCondition.in_latest_time_window()) + .with_label("eager_all_partitions") +) +``` + +## Pattern 2: Replacing Sub-conditions with replace() + +Swap one sub-condition for another with different parameters. + +**Multiple cron schedules**: Execute at 9 AM but wait for dependencies to update since midnight: + +```python +NINE_AM_CRON = "0 9 * * *" +MIDNIGHT_CRON = "0 0 * * *" + +condition = dg.AutomationCondition.on_cron(NINE_AM_CRON).replace( + old=dg.AutomationCondition.all_deps_updated_since_cron(NINE_AM_CRON), + new=dg.AutomationCondition.all_deps_updated_since_cron(MIDNIGHT_CRON), +) +``` + +**Partition lookback window**: Expand `on_missing()` to consider the last 24 hours of partitions: + +```python +import datetime + +condition = dg.AutomationCondition.on_missing().replace( + old=dg.AutomationCondition.in_latest_time_window(), + new=dg.AutomationCondition.in_latest_time_window( + lookback_delta=datetime.timedelta(hours=24) + ), +) +``` + +## Pattern 3: Filtering Dependencies with allow() and ignore() + +Control which dependencies are considered. + +**Only specific dependencies**: Only trigger on updates from assets in the "abc" group: + +```python +condition = dg.AutomationCondition.eager().allow( + dg.AssetSelection.groups("abc") +) +``` + +**Exclude specific dependencies**: Ignore updates from the "foo" asset: + +```python +condition = dg.AutomationCondition.eager().ignore( + dg.AssetSelection.assets("foo") +) +``` + +## Pattern 4: Boolean Composition + +Combine multiple conditions with AND (`&`), OR (`|`), NOT (`~`). + +**Scheduled with dependency-driven fallback**: Run every 5 minutes or when dependencies update (if updated today): + +```python +daily_success_condition = dg.AutomationCondition.newly_updated().since( + dg.AutomationCondition.cron_tick_passed("0 0 * * *") +) + +condition = ( + dg.AutomationCondition.cron_tick_passed("*/5 * * * *") + | ( + dg.AutomationCondition.any_deps_updated() + & daily_success_condition + & ~dg.AutomationCondition.any_deps_missing() + & ~dg.AutomationCondition.any_deps_in_progress() + ) +) +``` + +**Only execute when checks pass**: Ensure all blocking checks on dependencies pass: + +```python +condition = ( + dg.AutomationCondition.eager() + & dg.AutomationCondition.all_deps_match( + dg.AutomationCondition.all_checks_match( + dg.AutomationCondition.check_passed(), + blocking_only=True, + ) + ) +) +``` + +## Pattern 5: Custom Event-Based Conditions + +Build conditions from operands and operators for specific scenarios. + +**On code version change**: Execute when code version changes: + +```python +condition = ( + dg.AutomationCondition.code_version_changed().since_last_handled() + & ~dg.AutomationCondition.any_deps_missing() +) +``` + +**After upstream success**: Execute only after a specific upstream asset updates: + +```python +condition = ( + dg.AutomationCondition.any_deps_match( + dg.AutomationCondition.newly_updated() + ).allow(dg.AssetSelection.assets("critical_upstream")) + .since_last_handled() +) +``` + +## Combining Patterns + +Multiple patterns can be combined for complex requirements: + +```python +condition = ( + dg.AutomationCondition.eager() + .without(dg.AutomationCondition.in_latest_time_window()) # Pattern 1 + .ignore(dg.AssetSelection.assets("staging_data")) # Pattern 3 + & dg.AutomationCondition.all_checks_match( # Pattern 4 + dg.AutomationCondition.check_passed(), + blocking_only=True, + ) +).with_label("custom_backfill_with_checks") +``` + +This condition uses `eager()` as the base, removes the latest partition restriction, ignores a specific dependency, and adds a check requirement. diff --git a/.agents/skills/dagster-expert/references/automation/declarative-automation/operands.md b/.agents/skills/dagster-expert/references/automation/declarative-automation/operands.md new file mode 100644 index 0000000..02e5403 --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/declarative-automation/operands.md @@ -0,0 +1,98 @@ +--- +title: Operands +triggers: + - "base condition building blocks like missing() or newly_updated()" +--- + +# Declarative Automation: Operands + +Operands are base conditions that evaluate to true or false for a given asset or asset partition. They represent fundamental states and events. + +## Complete List of Operands + +| Operand | Description | Type | +| ----------------------------------------------------------- | ---------------------------------------------------------------------- | ------ | +| `AutomationCondition.missing()` | Target has not been executed | Status | +| `AutomationCondition.in_progress()` | Target is part of an in-progress run or backfill | Status | +| `AutomationCondition.execution_failed()` | Target failed in its latest run | Status | +| `AutomationCondition.newly_updated()` | Target was updated since the previous evaluation | Event | +| `AutomationCondition.newly_requested()` | Target was requested on the previous evaluation | Event | +| `AutomationCondition.code_version_changed()` | Target has a new code version since the previous evaluation | Event | +| `AutomationCondition.cron_tick_passed(schedule, timezone)` | A new tick of the cron schedule occurred since previous evaluation | Event | +| `AutomationCondition.in_latest_time_window(lookback_delta)` | Target falls within the latest time window of the PartitionsDefinition | Status | +| `AutomationCondition.will_be_requested()` | Target will be requested in this tick | Status | +| `AutomationCondition.initial_evaluation()` | This is the first evaluation of this condition | Event | + +## Status vs Event Operands + +**Status operands** are persistent and remain true for multiple evaluations: + +- `missing()` - Stays true until the asset is materialized +- `in_progress()` - True while a run is executing +- `execution_failed()` - True until the asset succeeds or is re-requested +- `in_latest_time_window()` - True for the latest time partition(s) +- `will_be_requested()` - True during the tick when a request will be made + +**Event operands** are transient and true for only one evaluation: + +- `newly_updated()` - True only on the tick when the update occurs +- `newly_requested()` - True only on the tick when the request is made +- `code_version_changed()` - True only on the first tick after the change +- `cron_tick_passed()` - True only on the first tick after the cron tick +- `initial_evaluation()` - True only on the very first evaluation + +## Detailed Descriptions + +**`missing()`**: True if the asset partition has never been materialized or observed. + +**`in_progress()`**: True if the asset partition is part of an in-progress run or backfill. Combines `run_in_progress()` and `backfill_in_progress()`. + +**`execution_failed()`**: True if the latest execution of the asset partition failed. + +**`newly_updated()`**: True if the asset partition was materialized or observed since the previous evaluation. For observations, only true if the data version changed. + +**`newly_requested()`**: True if the asset partition was requested on the previous evaluation tick. + +**`code_version_changed()`**: True if the asset's code version has changed since the previous evaluation. + +**`cron_tick_passed(cron_schedule, cron_timezone)`**: True on the first evaluation after a tick of the specified cron schedule occurs. + +Parameters: + +- `cron_schedule` (str): Cron expression +- `cron_timezone` (str): Timezone string (default: "UTC") + +**`in_latest_time_window(lookback_delta)`**: True for time partitions within the latest time window. For unpartitioned or non-time-partitioned assets, always true. + +Parameter: + +- `lookback_delta` (Optional[timedelta]): If provided, returns partitions within this delta of the latest window end. For daily partitions with `lookback_delta=timedelta(hours=48)`, returns the latest 2 partitions. + +**`will_be_requested()`**: True if the asset partition will be requested in the current tick. Used internally for run grouping (see [advanced.md](advanced.md)). + +**`initial_evaluation()`**: True only on the first evaluation after the condition is applied or modified. + +## Composite Conditions + +Built from base operands for convenience: + +| Composite Condition | Expansion | +| --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | +| `AutomationCondition.any_deps_updated()` | `any_deps_match((newly_updated() & ~executed_with_root_target()) \| will_be_requested())` | +| `AutomationCondition.any_deps_missing()` | `any_deps_match(missing() & ~will_be_requested())` | +| `AutomationCondition.any_deps_in_progress()` | `any_deps_match(in_progress())` | +| `AutomationCondition.all_deps_updated_since_cron(schedule, timezone)` | `all_deps_match(newly_updated().since(cron_tick_passed(schedule, timezone)))` | + +## Usage + +Operands are composed using operators (see [operators.md](operators.md)) to build complex conditions: + +```python +import dagster as dg + +# Using operands directly +condition = dg.AutomationCondition.missing() & ~dg.AutomationCondition.in_progress() + +# Using composite conditions +condition = dg.AutomationCondition.any_deps_updated() +``` diff --git a/.agents/skills/dagster-expert/references/automation/declarative-automation/operators.md b/.agents/skills/dagster-expert/references/automation/declarative-automation/operators.md new file mode 100644 index 0000000..01b4bfd --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/declarative-automation/operators.md @@ -0,0 +1,184 @@ +--- +title: Operators +triggers: + - "combining conditions using since, any_deps_match, or boolean operators" +--- + +# Declarative Automation: Operators + +Operators combine operands and other conditions into complex expressions using boolean logic and transformations. + +## Boolean Operators + +**`&` (AND)**: Both conditions must be true: + +```python +import dagster as dg + +condition = ( + dg.AutomationCondition.newly_updated() + & ~dg.AutomationCondition.in_progress() +) +``` + +**`|` (OR)**: Either condition must be true: + +```python +condition = ( + dg.AutomationCondition.missing() + | dg.AutomationCondition.newly_updated() +) +``` + +**`~` (NOT)**: Negates the condition: + +```python +condition = ~dg.AutomationCondition.any_deps_missing() +``` + +## Transformation Operators + +### since(reset_condition) + +Converts events into status. Becomes true when the operand becomes true and remains true until the reset condition becomes true. + +```python +# True from when dependency updates until asset is requested +condition = dg.AutomationCondition.any_deps_updated().since( + dg.AutomationCondition.newly_requested() +) +``` + +**Pattern**: `A.since(B)` means "A has occurred more recently than B" + +**Use case**: Create persistent states from transient events. "Upstream updated" is an event, but "upstream updated since I was last requested" is a status. + +### newly_true() + +Converts status into an event. True only on the tick when the operand transitions from false to true. + +```python +# True only on the tick when the asset becomes missing +condition = dg.AutomationCondition.missing().newly_true() +``` + +**Use case**: Prevent repeated actions during persistent states. `missing()` stays true for many ticks, but `missing().newly_true()` is only true once. + +### since_last_handled() + +Convenience method equivalent to `.since(newly_requested() | newly_updated() | initial_evaluation())`. + +```python +condition = dg.AutomationCondition.any_deps_updated().since_last_handled() +``` + +True from when the condition becomes true until the asset is requested, updated, or the condition is first applied. + +## Dependency Operators + +### any_deps_match(condition) + +True if the condition is true for at least one partition of any upstream dependency. + +```python +condition = dg.AutomationCondition.any_deps_match( + dg.AutomationCondition.missing() +) +``` + +Supports filtering with `.allow()` and `.ignore()`. + +### all_deps_match(condition) + +True if the condition is true for at least one partition of all upstream dependencies. + +```python +condition = dg.AutomationCondition.all_deps_match( + dg.AutomationCondition.newly_updated() +) +``` + +Requires every upstream asset to have at least one partition matching the condition. + +## Dependency Filtering + +### allow(selection) + +Restricts which dependencies are checked to only those in the `AssetSelection`: + +```python +# Only consider dependencies in the "important" group +condition = dg.AutomationCondition.any_deps_match( + dg.AutomationCondition.missing() +).allow(dg.AssetSelection.groups("important")) +``` + +Creates an intersection: `dep_keys & allowed_selection` + +### ignore(selection) + +Excludes dependencies in the `AssetSelection` from being checked: + +```python +# Ignore the "foo" asset when checking for updates +condition = dg.AutomationCondition.any_deps_updated().ignore( + dg.AssetSelection.assets("foo") +) +``` + +Creates a subtraction: `dep_keys - ignored_selection` + +### Propagation Through Boolean Operators + +When applied to `AND`/`OR` conditions, `.allow()` and `.ignore()` propagate to all sub-conditions: + +```python +# Applies allow() to all dependency checks within eager() +condition = dg.AutomationCondition.eager().allow( + dg.AssetSelection.groups("critical") +) +``` + +## Check Operators + +### any_checks_match(condition, blocking_only) + +True if any of the asset's checks match the condition. + +```python +condition = dg.AutomationCondition.any_checks_match( + dg.AutomationCondition.check_failed(), + blocking_only=True, +) +``` + +Parameters: + +- `condition`: Condition to evaluate against checks +- `blocking_only` (bool): If True, only considers blocking checks (default: False) + +### all_checks_match(condition, blocking_only) + +True if all of the asset's checks match the condition. + +```python +condition = dg.AutomationCondition.all_checks_match( + dg.AutomationCondition.check_passed(), + blocking_only=True, +) +``` + +## Labeling + +### with_label(label) + +Adds a human-readable label to a condition for debugging and UI display: + +```python +condition = ( + dg.AutomationCondition.any_deps_updated() + .since(dg.AutomationCondition.newly_requested()) +).with_label("updated_since_requested") +``` + +Labels appear in condition evaluation traces in the Dagster UI, making complex conditions easier to understand. diff --git a/.agents/skills/dagster-expert/references/automation/schedules.md b/.agents/skills/dagster-expert/references/automation/schedules.md new file mode 100644 index 0000000..b9448a4 --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/schedules.md @@ -0,0 +1,120 @@ +--- +title: Schedules +triggers: + - "time-based automation with cron expressions" +--- + +# Schedules + +Basic schedule patterns are covered in the main SKILL.md Quick Reference. This reference covers advanced schedule configuration and partitioned job automation. + +## Basic Schedule Review + +A schedule executes a job at specified times using cron expressions. See SKILL.md for the basic pattern. + +```python +import dagster as dg + +daily_job = dg.define_asset_job("daily_job", selection="*") + +daily_schedule = dg.ScheduleDefinition( + job=daily_job, + cron_schedule="0 0 * * *", # Midnight UTC +) +``` + +## Execution Timezone + +Schedules default to UTC. Specify a different timezone with `execution_timezone`: + +```python nocheckundefined +daily_schedule = dg.ScheduleDefinition( + job=daily_refresh_job, + cron_schedule="0 9 * * *", # 9 AM + execution_timezone="America/Los_Angeles", +) +``` + +**Timezone string format**: Use IANA timezone database names like `"America/New_York"`, `"Europe/London"`, or `"Asia/Tokyo"`. + +**Daylight saving time**: Dagster handles DST transitions automatically based on the specified timezone. + +## Schedules from Partitioned Assets + +For partitioned assets or jobs, use `build_schedule_from_partitioned_job` to automatically create a schedule matching the partition cadence: + +```python + +@dg.asset(partitions_def=dg.DailyPartitionsDefinition(start_date="2024-01-01")) +def daily_asset(context: dg.AssetExecutionContext): + partition_date = context.partition_key + # Process data for this partition + ... + +partitioned_job = dg.define_asset_job( + name="daily_partitioned_job", + selection=[daily_asset] +) + +# Schedule automatically inherits daily cadence and timezone from partition definition +schedule = dg.build_schedule_from_partitioned_job(partitioned_job) +``` + +**How it works**: The schedule's cron expression is derived from the `PartitionsDefinition`: + +- `DailyPartitionsDefinition` → daily cron schedule +- `WeeklyPartitionsDefinition` → weekly cron schedule +- `MonthlyPartitionsDefinition` → monthly cron schedule +- `HourlyPartitionsDefinition` → hourly cron schedule + +Each schedule run materializes the partition corresponding to the schedule time. + +## Cron Expression Reference + +Common cron patterns for schedules: + +| Cron Expression | Description | +| ---------------- | --------------------------- | +| `0 * * * *` | Every hour | +| `0 0 * * *` | Daily at midnight | +| `0 9 * * *` | Daily at 9 AM | +| `0 0 * * 1` | Weekly on Monday | +| `0 0 1 * *` | Monthly on the 1st | +| `0 0 1 1 *` | Yearly on January 1st | +| `*/15 * * * *` | Every 15 minutes | +| `0 9-17 * * 1-5` | Hourly, 9 AM-5 PM, weekdays | + +**Cron format**: `minute hour day_of_month month day_of_week` + +## Configuration Options + +```python nocheckundefined +schedule = dg.ScheduleDefinition( + job=my_job, + cron_schedule="0 0 * * *", + execution_timezone="UTC", + default_status=dg.DefaultScheduleStatus.RUNNING, # Start enabled + description="Daily data refresh for analytics", + tags={"team": "data-eng", "priority": "high"}, +) +``` + +**Key parameters**: + +- `default_status`: Set to `DefaultScheduleStatus.RUNNING` to enable schedule automatically when deployed (default is `STOPPED`) +- `description`: Human-readable description shown in the Dagster UI +- `tags`: Metadata tags for organization and filtering + +## When to Use Schedules + +**Use schedules when**: + +- Execution time is predictable and fixed +- No dependency logic is needed (runs regardless of upstream status) +- Simple time-based triggers are sufficient + +**Prefer declarative automation when**: + +- You need dependency-aware execution +- Conditions involve asset freshness or upstream state +- Complex logic determines when to execute diff --git a/.agents/skills/dagster-expert/references/automation/sensors/asset-sensors.md b/.agents/skills/dagster-expert/references/automation/sensors/asset-sensors.md new file mode 100644 index 0000000..9b797a4 --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/sensors/asset-sensors.md @@ -0,0 +1,112 @@ +--- +title: Asset Sensors +triggers: + - "triggering on asset materialization events" +--- + +# Asset Sensors + +Asset sensors monitor asset materializations and trigger jobs when specific assets are materialized. + +## Basic Asset Sensor + +Use `@asset_sensor` to monitor a specific asset: + +```python nocheckundefined +@dg.asset_sensor(asset_key=dg.AssetKey("daily_sales_data"), job=downstream_job) +def sales_data_sensor(context: dg.SensorEvaluationContext, asset_event: dg.EventLogEntry): + # Triggered whenever daily_sales_data is materialized + yield dg.RunRequest(run_key=context.cursor) +``` + +The sensor is called once per materialization event with the event details in `asset_event`. + +## Cross-Job Dependencies + +Asset sensors enable dependencies across different jobs: + +```python nocheckundefined +# Job A contains upstream_asset +@dg.asset +def upstream_asset(): + ... + +job_a = dg.define_asset_job("job_a", selection=[upstream_asset]) + +# Job B is triggered when upstream_asset materializes +@dg.asset_sensor(asset_key=dg.AssetKey("upstream_asset"), job=job_b) +def cross_job_sensor(context, asset_event): + return dg.RunRequest() +``` + +This pattern is useful when you have logically separate jobs that need coordination. + +## Cross-Code Location Dependencies + +Asset sensors can monitor assets in different code locations: + +```python nocheckundefined +@dg.asset_sensor( + asset_key=dg.AssetKey("other_location_asset"), + job=my_job, +) +def cross_location_sensor(context, asset_event): + return dg.RunRequest() +``` + +The sensor can be in a different code location than the monitored asset. + +## Custom Evaluation Logic + +Add conditional logic to control when to trigger: + +```python nocheckundefined +@dg.asset_sensor(asset_key=dg.AssetKey("daily_sales_data"), job=downstream_job) +def conditional_sensor(context, asset_event): + # Access materialization metadata + metadata = asset_event.dagster_event.event_specific_data.materialization.metadata + + row_count = metadata.get("row_count").value if "row_count" in metadata else 0 + + if row_count > 1000: + return dg.RunRequest() + else: + return dg.SkipReason(f"Row count {row_count} too low, threshold is 1000") +``` + +**Use cases for conditional logic**: Trigger downstream processing only when data volume is sufficient, quality checks pass, or specific metadata conditions are met. + +## Triggering with Custom Configuration + +Pass runtime configuration to the triggered job: + +```python nocheckundefined +@dg.asset_sensor(asset_key=dg.AssetKey("source_data"), job=processing_job) +def config_sensor(context, asset_event): + # Extract partition key from the materialization + partition_key = asset_event.dagster_event.logging_tags.get("dagster/partition") + + return dg.RunRequest( + run_key=partition_key, + tags={"dagster/partition": partition_key}, + ) +``` + +This allows you to propagate partition information or other metadata from the upstream asset to the triggered job. + +## Asset Sensors vs Declarative Automation + +**Use asset sensors when**: + +- Triggering imperative side effects (notifications, external API calls) +- Launching jobs with complex custom logic +- Cross-code location dependencies with conditional triggers +- Need to inspect materialization metadata before deciding to trigger + +**Use declarative automation when**: + +- Automating asset-to-asset execution within the same code location +- Defining dependencies based on asset freshness or missing status +- Requiring sophisticated dependency logic with composable conditions + +Declarative automation is the recommended approach for asset-centric workflows. Asset sensors remain valuable for triggering actions outside the asset graph or when you need imperative control. diff --git a/.agents/skills/dagster-expert/references/automation/sensors/basic-sensors.md b/.agents/skills/dagster-expert/references/automation/sensors/basic-sensors.md new file mode 100644 index 0000000..0755430 --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/sensors/basic-sensors.md @@ -0,0 +1,110 @@ +--- +title: Basic Sensors +triggers: + - "event-driven automation with file watching or custom polling" +--- + +# Basic Sensors + +For the basic sensor pattern with cursors, see the main SKILL.md Quick Reference section. + +## File Watching Sensor + +A canonical file sensor that monitors a directory for new files and triggers runs: + +```python nocheckundefined +import os +import json +import dagster as dg + +@dg.sensor(job=my_job, minimum_interval_seconds=30) +def file_sensor(context: dg.SensorEvaluationContext): + # Load cursor (tracks files we've already processed) + processed_files = json.loads(context.cursor) if context.cursor else {} + + # Check directory for files + directory = "/data/incoming" + current_files = {} + runs_to_request = [] + + for filename in os.listdir(directory): + filepath = os.path.join(directory, filename) + mtime = os.path.getmtime(filepath) + current_files[filename] = mtime + + # File is new or modified + if filename not in processed_files or processed_files[filename] != mtime: + runs_to_request.append( + dg.RunRequest( + run_key=f"{filename}_{mtime}", + run_config={"ops": {"my_op": {"config": {"filepath": filepath}}}}, + ) + ) + + # Update cursor to track current state + return dg.SensorResult( + run_requests=runs_to_request, + cursor=json.dumps(current_files), + ) +``` + +**Key pattern**: Store file names and modification times in the cursor to track which files have been processed. + +## Cursor State Management + +**Best practices for cursors**: + +- **Use JSON for structured state**: `json.dumps()` and `json.loads()` make it easy to store dictionaries or lists +- **Handle missing cursor**: Always check if `context.cursor` is None on first evaluation +- **Update cursor atomically**: Return the new cursor value in `SensorResult` or call `context.update_cursor()` +- **Keep cursors small**: Cursors are stored in the database; avoid storing large data structures + +**Two ways to update cursors**: + +```python nocheck +# Option 1: Return SensorResult +return dg.SensorResult( + run_requests=[...], + cursor=json.dumps(new_state) +) + +# Option 2: Call update_cursor() directly +context.update_cursor(json.dumps(new_state)) +yield dg.RunRequest(...) +``` + +## Evaluation Configuration + +**Control evaluation frequency**: + +```python nocheckundefined +@dg.sensor( + job=my_job, + minimum_interval_seconds=60, # Minimum 60 seconds between evaluations + default_status=dg.DefaultSensorStatus.RUNNING, # Auto-enable when deployed +) +def my_sensor(context): + ... +``` + +**Important**: `minimum_interval_seconds` is a minimum, not exact. If sensor evaluation takes 10 seconds and the interval is 30 seconds, the next evaluation happens 30 seconds after the previous evaluation started (20 seconds after it completed). + +## SensorEvaluationContext + +Properties available in sensor context: + +- `cursor`: String cursor from the previous evaluation (None if first evaluation) +- `update_cursor(str)`: Update the cursor for the next evaluation +- `instance`: DagsterInstance for querying the event log or other instance data +- `log`: Logger for recording sensor evaluation details +- `repository_def`: Repository containing the sensor +- `resources`: Access configured resources (if defined) + +**Example using context.log**: + +```python nocheckundefined +@dg.sensor(job=my_job) +def logging_sensor(context): + context.log.info(f"Evaluating sensor, cursor: {context.cursor}") + # ... sensor logic +``` diff --git a/.agents/skills/dagster-expert/references/automation/sensors/run-status-sensors.md b/.agents/skills/dagster-expert/references/automation/sensors/run-status-sensors.md new file mode 100644 index 0000000..379a714 --- /dev/null +++ b/.agents/skills/dagster-expert/references/automation/sensors/run-status-sensors.md @@ -0,0 +1,134 @@ +--- +title: Run Status Sensors +triggers: + - "reacting to run success, failure, or other status changes" +--- + +# Run Status Sensors + +Run status sensors monitor runs for specific status changes and trigger actions when those statuses occur. + +## Run Failure Sensor + +Use `@run_failure_sensor` to monitor run failures across all jobs: + +```python nocheckundefined +import dagster as dg + +@dg.run_failure_sensor +def failure_alert_sensor(context: dg.RunFailureSensorContext): + slack_client.chat_postMessage( + channel="#alerts", + text=f'Job "{context.dagster_run.job_name}" failed: {context.failure_event.message}', + ) +``` + +Run failure sensors are commonly used for alerting and error notification. + +## Run Status Sensor + +Use `@run_status_sensor` to monitor any run status: + +```python nocheckundefined +@dg.run_status_sensor( + run_status=dg.DagsterRunStatus.SUCCESS, + request_job=downstream_job, +) +def success_sensor(context: dg.RunStatusSensorContext): + if context.dagster_run.job_name == "upstream_job": + return dg.RunRequest(run_key=context.dagster_run.run_id) +``` + +This pattern enables job-to-job dependencies based on run completion. + +## Available Run Statuses + +Common run statuses for monitoring: + +- `DagsterRunStatus.SUCCESS` - Run completed successfully +- `DagsterRunStatus.FAILURE` - Run failed +- `DagsterRunStatus.STARTED` - Run execution started +- `DagsterRunStatus.CANCELED` - Run was canceled +- `DagsterRunStatus.CANCELING` - Run is being canceled + +Additional statuses: `QUEUED`, `NOT_STARTED`, `MANAGED`, `STARTING` + +## Monitoring Specific Jobs + +Use `monitored_jobs` to filter which jobs trigger the sensor: + +```python nocheckundefined +@dg.run_status_sensor( + run_status=dg.DagsterRunStatus.SUCCESS, + monitored_jobs=[job1, job2], +) +def job_specific_sensor(context): + # Only triggered when job1 or job2 succeeds + ... +``` + +Without `monitored_jobs`, the sensor triggers for all runs with the specified status. + +## Cross-Code Location Monitoring + +Monitor runs across all code locations: + +```python +@dg.run_status_sensor( + run_status=dg.DagsterRunStatus.FAILURE, + monitor_all_code_locations=True, +) +def global_failure_sensor(context): + # Monitors failures across the entire deployment + ... +``` + +Set `monitor_all_code_locations=True` to enable deployment-wide monitoring. + +## Context Properties + +**RunStatusSensorContext** provides: + +- `dagster_run`: The run that triggered the sensor +- `dagster_event`: The event associated with the status change +- `partition_key`: Partition key from run tags (if partitioned) +- `instance`: DagsterInstance +- `log`: Logger + +**RunFailureSensorContext** adds: + +- `failure_event`: The run failure event with error details +- `get_step_failure_events()`: List of step-level failures with stack traces + +## Common Use Cases + +**Alerting**: Send notifications on run failures: + +```python nocheckundefined +@dg.run_failure_sensor +def slack_alert(context: dg.RunFailureSensorContext): + slack_client.chat_postMessage( + channel="#alerts", + text=f"Job {context.dagster_run.job_name} failed" + ) +``` + +**Job coordination**: Trigger downstream jobs after success: + +```python nocheckundefined +@dg.run_status_sensor( + run_status=dg.DagsterRunStatus.SUCCESS, + monitored_jobs=[upstream_job], + request_job=downstream_job, +) +def chain_jobs(context): + return dg.RunRequest() +``` + +**Error handling**: Trigger cleanup on failure: + +```python nocheckundefined +@dg.run_failure_sensor(monitored_jobs=[data_job]) +def cleanup_on_failure(context): + cleanup_partial_data() +``` diff --git a/.agents/skills/dagster-expert/references/cli/api/agent/get.md b/.agents/skills/dagster-expert/references/cli/api/agent/get.md new file mode 100644 index 0000000..a13ffc4 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/agent/get.md @@ -0,0 +1,9 @@ +--- +title: dg api agent get +triggers: + - "details about a specific Dagster Plus agent" +--- + +```bash +dg api agent get +``` diff --git a/.agents/skills/dagster-expert/references/cli/api/agent/list.md b/.agents/skills/dagster-expert/references/cli/api/agent/list.md new file mode 100644 index 0000000..39cf5cc --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/agent/list.md @@ -0,0 +1,9 @@ +--- +title: dg api agent list +triggers: + - "listing agents in Dagster Plus" +--- + +```bash +dg api agent list +``` diff --git a/.agents/skills/dagster-expert/references/cli/api/asset/get-evaluations.md b/.agents/skills/dagster-expert/references/cli/api/asset/get-evaluations.md new file mode 100644 index 0000000..fab49e0 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/asset/get-evaluations.md @@ -0,0 +1,11 @@ +--- +title: dg api asset get-evaluations +triggers: + - "automation condition evaluation history for an asset" +--- + +```bash +dg api asset get-evaluations +``` + +`--include-nodes` — includes individual evaluation nodes in the response. Warning: this produces dense output with the full tree of conditions evaluated for each record. Only use when processing a small number of records. diff --git a/.agents/skills/dagster-expert/references/cli/api/asset/get-events.md b/.agents/skills/dagster-expert/references/cli/api/asset/get-events.md new file mode 100644 index 0000000..39ce16f --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/asset/get-events.md @@ -0,0 +1,13 @@ +--- +title: dg api asset get-events +triggers: + - "materialization or observation event history for an asset" +--- + +```bash +dg api asset get-events +``` + +- `--event-type` — filter by event type (e.g. `ASSET_MATERIALIZATION`, `ASSET_OBSERVATION`) +- `--partition` — filter events by partition key +- `--before` — return events before this timestamp; use with `--limit` to paginate chronologically diff --git a/.agents/skills/dagster-expert/references/cli/api/asset/get.md b/.agents/skills/dagster-expert/references/cli/api/asset/get.md new file mode 100644 index 0000000..dc6863e --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/asset/get.md @@ -0,0 +1,13 @@ +--- +title: dg api asset get +triggers: + - "details about a specific asset" +--- + +```bash +dg api asset get +``` + +For prefixed asset keys, use slash-separated syntax: `dg api asset get my_prefix/my_asset`. + +`--status` — includes materialization status information. Not included by default as it requires additional API calls. diff --git a/.agents/skills/dagster-expert/references/cli/api/asset/list.md b/.agents/skills/dagster-expert/references/cli/api/asset/list.md new file mode 100644 index 0000000..2e30a54 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/asset/list.md @@ -0,0 +1,12 @@ +--- +title: dg api asset list +triggers: + - "querying which assets exist in a deployment" +--- + +```bash +dg api asset list +``` + +- `--status` — includes detailed status information in the response +- `--limit` / `--cursor` — pagination support diff --git a/.agents/skills/dagster-expert/references/cli/api/deployment/get.md b/.agents/skills/dagster-expert/references/cli/api/deployment/get.md new file mode 100644 index 0000000..3c6c86a --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/deployment/get.md @@ -0,0 +1,9 @@ +--- +title: dg api deployment get +triggers: + - "details about a specific deployment" +--- + +```bash +dg api deployment get +``` diff --git a/.agents/skills/dagster-expert/references/cli/api/deployment/list.md b/.agents/skills/dagster-expert/references/cli/api/deployment/list.md new file mode 100644 index 0000000..365e015 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/deployment/list.md @@ -0,0 +1,9 @@ +--- +title: dg api deployment list +triggers: + - "listing deployments in Dagster Plus" +--- + +```bash +dg api deployment list +``` diff --git a/.agents/skills/dagster-expert/references/cli/api/general.md b/.agents/skills/dagster-expert/references/cli/api/general.md new file mode 100644 index 0000000..3246bda --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/general.md @@ -0,0 +1,16 @@ +--- +title: "dg api: General" +triggers: + - "always read before using any dg api subcommand" +--- + +All `dg api` subcommands support `--json`, `--response-schema`, `--deployment`, `--organization`, `--api-token`, and `--view-graphql`. + +- `--response-schema` — prints the JSON schema for the command's response and exits. Run this before writing any parsing logic to get exact field names, types, and valid enum values. +- `--view-graphql` — prints GraphQL queries and responses to stderr, useful for debugging. + +## Tips + +For complex debugging/analysis workflows, ALWAYS use `--json` to get machine-readable output. Pipe into `jq` (recommended) or other tools for further processing. + +Flags like `--deployment`/`--organization`/`--api-token` are typically not needed when authenticated via `dg plus login`. diff --git a/.agents/skills/dagster-expert/references/cli/api/run/get-events.md b/.agents/skills/dagster-expert/references/cli/api/run/get-events.md new file mode 100644 index 0000000..c667d06 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/run/get-events.md @@ -0,0 +1,15 @@ +--- +title: dg api run get-events +triggers: + - "debugging a run by reading its logs; filtering run events by level or step" +--- + +```bash +dg api run get-events +``` + +- `--level` — filter by log level: DEBUG, INFO, WARNING, ERROR, CRITICAL. Repeatable. +- `--event-type` — filter by event type (e.g. `STEP_FAILURE`, `RUN_START`). Repeatable. +- `--step` — filter by step key. Supports partial matching — `--step my_asset` will match step keys containing that substring. Repeatable. +- `--limit` — maximum number of events to return. +- `--cursor` — pagination cursor for retrieving more events. diff --git a/.agents/skills/dagster-expert/references/cli/api/run/get.md b/.agents/skills/dagster-expert/references/cli/api/run/get.md new file mode 100644 index 0000000..0c86f7c --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/run/get.md @@ -0,0 +1,9 @@ +--- +title: dg api run get +triggers: + - "details about a specific run" +--- + +```bash +dg api run get +``` diff --git a/.agents/skills/dagster-expert/references/cli/api/run/list.md b/.agents/skills/dagster-expert/references/cli/api/run/list.md new file mode 100644 index 0000000..344e282 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/run/list.md @@ -0,0 +1,12 @@ +--- +title: dg api run list +triggers: + - "listing or filtering runs" +--- + +```bash +dg api run list +``` + +- `--status` — filter by run status: QUEUED, STARTING, STARTED, SUCCESS, FAILURE, CANCELING, CANCELED. Repeatable (e.g. `--status FAILURE --status CANCELED`). +- `--job` — filter by job name diff --git a/.agents/skills/dagster-expert/references/cli/api/schedule/get.md b/.agents/skills/dagster-expert/references/cli/api/schedule/get.md new file mode 100644 index 0000000..be97bd4 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/schedule/get.md @@ -0,0 +1,9 @@ +--- +title: dg api schedule get +triggers: + - "details about a specific schedule" +--- + +```bash +dg api schedule get +``` diff --git a/.agents/skills/dagster-expert/references/cli/api/schedule/list.md b/.agents/skills/dagster-expert/references/cli/api/schedule/list.md new file mode 100644 index 0000000..3fc68fb --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/schedule/list.md @@ -0,0 +1,11 @@ +--- +title: dg api schedule list +triggers: + - "listing schedules in Dagster Plus" +--- + +```bash +dg api schedule list +``` + +`--status` — filter schedules by status (e.g. RUNNING, STOPPED). diff --git a/.agents/skills/dagster-expert/references/cli/api/secret/get.md b/.agents/skills/dagster-expert/references/cli/api/secret/get.md new file mode 100644 index 0000000..7be6ca7 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/secret/get.md @@ -0,0 +1,12 @@ +--- +title: dg api secret get +triggers: + - "details about a specific secret" +--- + +```bash +dg api secret get +``` + +- `--show-value` — includes the secret value in the response +- `--location` — filter by code location diff --git a/.agents/skills/dagster-expert/references/cli/api/secret/list.md b/.agents/skills/dagster-expert/references/cli/api/secret/list.md new file mode 100644 index 0000000..9f64675 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/secret/list.md @@ -0,0 +1,12 @@ +--- +title: dg api secret list +triggers: + - "listing secrets in Dagster Plus" +--- + +```bash +dg api secret list +``` + +- `--location` — filter by code location +- `--scope` — filter by secret scope diff --git a/.agents/skills/dagster-expert/references/cli/api/sensor/get.md b/.agents/skills/dagster-expert/references/cli/api/sensor/get.md new file mode 100644 index 0000000..d0dd88b --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/sensor/get.md @@ -0,0 +1,9 @@ +--- +title: dg api sensor get +triggers: + - "details about a specific sensor" +--- + +```bash +dg api sensor get +``` diff --git a/.agents/skills/dagster-expert/references/cli/api/sensor/list.md b/.agents/skills/dagster-expert/references/cli/api/sensor/list.md new file mode 100644 index 0000000..6776c38 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/api/sensor/list.md @@ -0,0 +1,11 @@ +--- +title: dg api sensor list +triggers: + - "listing sensors in Dagster Plus" +--- + +```bash +dg api sensor list +``` + +`--status` — filter sensors by status (e.g. RUNNING, STOPPED). diff --git a/.agents/skills/dagster-expert/references/cli/asset-selection.md b/.agents/skills/dagster-expert/references/cli/asset-selection.md new file mode 100644 index 0000000..27cbe3f --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/asset-selection.md @@ -0,0 +1,66 @@ +--- +title: Asset Selection Syntax +triggers: + - "filtering assets by tag, group, kind, upstream, or downstream" +--- + +Reference for the asset selection syntax used by `dg list defs --assets` and `dg launch --assets`. + +## Attributes + +- `key:` or just `` — select by asset key (e.g. `customers`) +- `tag:=` or `tag:` — select by tag (e.g. `tag:priority=high`) +- `owner:` — select by owner (e.g. `owner:team@company.com`) +- `group:` — select by group (e.g. `group:sales_analytics`) +- `kind:` — select by kind (e.g. `kind:dbt`) + +**Wildcards:** `key:customer*`, `key:*_raw`, `*` (all assets) + +## Operators + +- `and` / `AND` — e.g. `tag:priority=high and kind:dbt` +- `or` / `OR` — e.g. `group:sales or group:marketing` +- `not` / `NOT` — e.g. `not kind:dbt` +- `(expr)` — grouping, e.g. `tag:priority=high and (kind:dbt or kind:python)` + +## Functions + +- `sinks(expr)` — assets with no downstream dependents (e.g. `sinks(group:analytics)`) +- `roots(expr)` — assets with no upstream dependencies (e.g. `roots(kind:dbt)`) + +## Traversals + +- `+expr` — all upstream dependencies (e.g. `+customers`) +- `expr+` — all downstream dependents (e.g. `customers+`) +- `N+expr` — N levels upstream (e.g. `2+kind:dbt`) +- `expr+N` — N levels downstream (e.g. `group:sales+1`) +- `N+expr+M` — N up, M down (e.g. `1+key:customers+2`) + +## Examples + +```bash +# Select by name +dg launch --assets customers +dg launch --assets "customer*" + +# Select by metadata +dg launch --assets "tag:priority=high" +dg launch --assets "group:sales_analytics" +dg launch --assets "kind:dbt" +dg launch --assets "owner:team@company.com" + +# Combine with operators +dg launch --assets "tag:priority=high and kind:dbt" +dg launch --assets "group:sales or group:marketing" +dg launch --assets "not kind:dbt" + +# With traversals +dg launch --assets "+kind:dbt" # all upstream of dbt assets +dg launch --assets "group:sales+" # group:sales + all downstream +dg launch --assets "2+key:customers" # customers + 2 levels upstream +dg launch --assets "kind:python+1" # kind:python + 1 level downstream + +# With functions +dg launch --assets "sinks(group:analytics)" # terminal assets in group +dg launch --assets "roots(kind:dbt)" # source dbt assets +``` diff --git a/.agents/skills/dagster-expert/references/cli/check.md b/.agents/skills/dagster-expert/references/cli/check.md new file mode 100644 index 0000000..6061430 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/check.md @@ -0,0 +1,30 @@ +--- +title: dg check +triggers: + - "validating project configuration or definitions" +--- + +## dg check defs + +Verify all definitions load without errors. + +```bash +dg check defs +dg check defs --verbose # Detailed output +``` + +## dg check yaml + +Validate `defs.yaml` files for syntax errors and valid component configuration. + +```bash +dg check yaml +``` + +## dg check toml + +Validate `pyproject.toml` and `dg.toml` for syntax errors. + +```bash +dg check toml +``` diff --git a/.agents/skills/dagster-expert/references/cli/create-dagster.md b/.agents/skills/dagster-expert/references/cli/create-dagster.md new file mode 100644 index 0000000..2fd632b --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/create-dagster.md @@ -0,0 +1,24 @@ +--- +title: create-dagster +triggers: + - "creating a new Dagster project from scratch" +--- + +The `create-dagster` command scaffolds a new Dagster project with the proper Python package structure and Dagster-specific configuration. + +Two structures are available: + +- `project` — a single Dagster project (default choice unless user needs multiple independent packages) +- `workspace` — a collection of related Dagster projects with independent dependencies + +## Project Creation + +```bash +uvx create-dagster project --uv-sync # --uv-sync creates venv and installs deps (recommended) +``` + +## Workspace Creation + +```bash +uvx create-dagster workspace # For multiple related projects +``` diff --git a/.agents/skills/dagster-expert/references/cli/launch.md b/.agents/skills/dagster-expert/references/cli/launch.md new file mode 100644 index 0000000..5b4c741 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/launch.md @@ -0,0 +1,33 @@ +--- +title: dg launch +triggers: + - "materializing assets or executing jobs locally" +--- + +`dg launch` executes runs of assets or jobs locally and in-process. Useful for development but will NOT execute runs on a remote Dagster deployment. + +```bash +dg launch --assets +dg launch --job +``` + +See [asset-selection.md](./asset-selection.md) for complete selection syntax. + +## Partitions + +```bash +dg launch --assets my_asset --partition 2024-01-15 +dg launch --assets my_asset --partition-range "2024-01-01...2024-01-31" +``` + +**Note:** Use three dots (`...`) for inclusive ranges, not two dots. + +## Configuration + +```bash +# Inline JSON +dg launch --assets my_asset --config '{"limit": 100}' + +# From file +dg launch --assets my_asset --config-file config.yaml +``` diff --git a/.agents/skills/dagster-expert/references/cli/list/component-tree.md b/.agents/skills/dagster-expert/references/cli/list/component-tree.md new file mode 100644 index 0000000..02166a2 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/list/component-tree.md @@ -0,0 +1,9 @@ +--- +title: dg list component-tree +triggers: + - "viewing the component instance hierarchy" +--- + +```bash +dg list component-tree +``` diff --git a/.agents/skills/dagster-expert/references/cli/list/components.md b/.agents/skills/dagster-expert/references/cli/list/components.md new file mode 100644 index 0000000..cfa7d7e --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/list/components.md @@ -0,0 +1,11 @@ +--- +title: dg list components +triggers: + - "seeing available component types for scaffolding" +--- + +List all available Dagster component types in the current Python environment. + +```bash +dg list components +``` diff --git a/.agents/skills/dagster-expert/references/cli/list/defs.md b/.agents/skills/dagster-expert/references/cli/list/defs.md new file mode 100644 index 0000000..34ddd85 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/list/defs.md @@ -0,0 +1,18 @@ +--- +title: dg list defs +triggers: + - "listing or filtering registered definitions" +--- + +List all registered Dagster definitions (assets, jobs, schedules, sensors, resources) in the current project. + +```bash +dg list defs +``` + +- `--assets ` — filter by asset selection syntax +- `--columns ` — columns to display (comma-separated or repeated flag) +- `--json` — output as JSON instead of a table +- `--response-schema` — print the JSON schema of the response and exit. Use before writing any parsing logic. + +**Available columns:** `key`, `group`, `deps`, `kinds`, `description`, `tags`, `cron`, `is_executable` diff --git a/.agents/skills/dagster-expert/references/cli/list/envs.md b/.agents/skills/dagster-expert/references/cli/list/envs.md new file mode 100644 index 0000000..940cfe9 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/list/envs.md @@ -0,0 +1,13 @@ +--- +title: dg list envs +triggers: + - "seeing which environment variables the project requires" +--- + +List environment variables from the `.env` file of the current project. Shows variable name, whether it is set locally, and which components use it. + +```bash +dg list envs +``` + +With Dagster Plus authentication (`dg plus login`), also shows deployment scope status (Dev/Branch/Full). diff --git a/.agents/skills/dagster-expert/references/cli/list/projects.md b/.agents/skills/dagster-expert/references/cli/list/projects.md new file mode 100644 index 0000000..08dca8b --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/list/projects.md @@ -0,0 +1,9 @@ +--- +title: dg list projects +triggers: + - "listing projects in the current workspace" +--- + +```bash +dg list projects +``` diff --git a/.agents/skills/dagster-expert/references/cli/plus/login.md b/.agents/skills/dagster-expert/references/cli/plus/login.md new file mode 100644 index 0000000..ec35722 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/plus/login.md @@ -0,0 +1,13 @@ +--- +title: dg plus login +triggers: + - "authenticating with Dagster Plus" +--- + +Authenticate with Dagster Plus. Opens a browser for interactive login. + +```bash +dg plus login +``` + +`--region eu` — login to the European region (default is `us`). diff --git a/.agents/skills/dagster-expert/references/cli/plus/pull/env.md b/.agents/skills/dagster-expert/references/cli/plus/pull/env.md new file mode 100644 index 0000000..a4128dd --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/plus/pull/env.md @@ -0,0 +1,11 @@ +--- +title: dg plus pull env +triggers: + - "pulling environment variables from Dagster Plus into a local .env file" +--- + +Pull environment variables from Dagster Plus and save them to a local `.env` file. Requires authentication via `dg plus login`. + +```bash +dg plus pull env +``` diff --git a/.agents/skills/dagster-expert/references/cli/scaffold/component.md b/.agents/skills/dagster-expert/references/cli/scaffold/component.md new file mode 100644 index 0000000..b2ff616 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/scaffold/component.md @@ -0,0 +1,15 @@ +--- +title: dg scaffold component +triggers: + - "creating a custom reusable component type" +--- + +Scaffold a new custom Dagster component type class. Must be run inside a Dagster project directory. The scaffold is placed in `.lib.`. + +Use `dg scaffold component` when the component will be used multiple times. For one-off components, use `dg scaffold defs inline-component` instead, which places the definition directly under `defs/`. + +```bash +dg scaffold component +``` + +`--model / --no-model` — whether the generated class inherits from `dagster.components.Model` (default: `--model`). diff --git a/.agents/skills/dagster-expert/references/cli/scaffold/defs.md b/.agents/skills/dagster-expert/references/cli/scaffold/defs.md new file mode 100644 index 0000000..c719473 --- /dev/null +++ b/.agents/skills/dagster-expert/references/cli/scaffold/defs.md @@ -0,0 +1,43 @@ +--- +title: dg scaffold defs +triggers: + - "adding new definitions (assets, schedules, sensors, components) to a project" +--- + +`dg scaffold defs` is the preferred way to add new definitions to the project. It automatically ensures new code is added to the correct location. + +## Python Definition Objects + +Scaffolds a single `.py` file at the specified path (relative to `defs/`). ALWAYS include the `.py` extension. + +```bash +dg scaffold defs dagster.asset assets/my_asset.py +dg scaffold defs dagster.schedule schedules/daily.py +dg scaffold defs dagster.sensor sensors/watcher.py +``` + +## Component Types + +Scaffold a component directory with `defs.yaml`. Additional arguments can be provided via flags or `--json-params`. + +**Important**: After scaffolding a custom component with `dg scaffold component`, run `dg list components` to get the exact registered type path. The path includes the file module name — e.g. `my_project.lib.my_component.MyComponent`, not `my_project.lib.MyComponent`. + +```bash +dg scaffold defs some_lib.SomeComponent my_component + +# With flags +dg scaffold defs dagster_dbt.DbtProjectComponent my_dbt --project-dir dbt_project + +# With JSON params +dg scaffold defs dagster_dbt.DbtProjectComponent my_dbt --json-params '{"project_dir": "dbt_project"}' +``` + +## Inline Components + +For one-off components, use `inline-component` to place the component class definition directly under `defs/` alongside its `defs.yaml`. Use `dg scaffold component` instead if you expect to reuse it. + +```bash +dg scaffold defs inline-component +``` + +## Important: Always run `dg list defs` to confirm the definitions were scaffolded correctly \ No newline at end of file diff --git a/.agents/skills/dagster-expert/references/components/creating-components.md b/.agents/skills/dagster-expert/references/components/creating-components.md new file mode 100644 index 0000000..78ebbb1 --- /dev/null +++ b/.agents/skills/dagster-expert/references/components/creating-components.md @@ -0,0 +1,96 @@ +--- +title: Creating Components +triggers: + - "building a new custom component from scratch" +--- + +# Creating Custom Components + +Components are the primary unit of reuse in Dagster projects. A component is a Python class that maps YAML configuration to Dagster definitions via the [Resolved framework](./resolved-framework.md). The core method is `build_defs()`, which returns a `dg.Definitions` object. + +## Scaffolding + +Use the CLI to generate boilerplate for a new component: + +```bash +dg scaffold component MyComponent +``` + +This creates the class file and registers it. Verify it appears in the component list, and note its full path (e.g. `my_project.lib.my_component.MyComponent`) for future scaffolding: + +```bash +dg list components +``` + +## Component Structure + +A component inherits from `dg.Component` and `dg.Resolvable`, plus a base class for field definitions. + +See [Resolved Framework](./resolved-framework.md#nested-resolution) for details on how to structure your component fields. + +ALWAYS use the built-in resolved types for asset-related fields instead of raw strings or dicts: + +- **`dg.ResolvedAssetKey`** — for a single asset key (accepts `"a/b/c"` string in YAML) +- **`dg.ResolvedAssetSpec`** — for a full asset spec (accepts structured mapping in YAML) +- **`dg.ResolvedAssetCheckSpec`** — for asset check specs + +These handle YAML-to-Python resolution automatically. + +## Building Definitions + +`build_defs()` returns `dg.Definitions` — this is the primary concern of a component. + +Prefer `@dg.multi_asset(specs=[...])` even for a single asset. This lets you pass `AssetSpec` objects directly via `specs=` instead of mapping all spec subfields to individual `@dg.asset()` kwargs: + +```python +import dagster as dg + + +class MyComponent(dg.Component, dg.Resolvable, dg.Model): + spec: dg.ResolvedAssetSpec + query: str + + def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions: + spec = self.spec + + @dg.multi_asset(specs=[spec]) + def my_asset(ctx: dg.AssetExecutionContext): + ctx.log.info(f"Running query: {self.query}") + # ... materialize the asset ... + + return dg.Definitions(assets=[my_asset]) +``` + +Corresponding YAML: + +```yaml +type: my_project.components.MyComponent + +attributes: + spec: + key: my_database/my_schema/orders + group_name: ingestion + kinds: + - sql + query: "SELECT * FROM orders" +``` + +## Expensive Operations + +If building definitions requires expensive work — querying a database, hitting an API, cloning a repo, compiling artifacts — ALWAYS use [StateBackedComponent](./state-backed/creating.md). It separates state-fetching from definition-building so that code server loads remain efficient. + +```python +# Use StateBackedComponent instead of Component when external state is involved +class MyApiComponent(dg.StateBackedComponent, dg.Model, dg.Resolvable): + ... +``` + +See [State-Backed Components](./state-backed/creating.md) for full implementation details. + +## References + +- [Resolved Framework](./resolved-framework.md) +- [Template Variables](./template-variables.md) +- [State-Backed Components](./state-backed/creating.md) +- [`dg scaffold component`](../cli/scaffold/component.md) +- [`dg list components`](../cli/list/components.md) diff --git a/.agents/skills/dagster-expert/references/components/designing-component-integrations.md b/.agents/skills/dagster-expert/references/components/designing-component-integrations.md new file mode 100644 index 0000000..b3dda76 --- /dev/null +++ b/.agents/skills/dagster-expert/references/components/designing-component-integrations.md @@ -0,0 +1,142 @@ +--- +title: Designing Component Integrations +triggers: + - "designing a component that wraps an external service or tool" +--- + +# Designing Component Integrations + +## Three Levels of Integration + +When designing a component integration, first determine how Dagster should interact with the external tool. There are three levels: + +- **Definition-only**: Dagster understands assets defined in an external tool and their dependencies. Example: `OmniComponent` maps Omni dashboards to Dagster assets. +- **Observing**: Definition-only plus Dagster monitors for events via sensors, emitting `AssetObservation` or `AssetMaterialization` events. (Aspirational — no clean component example yet.) +- **Orchestrating**: Definition-only plus Dagster can trigger execution. Example: `FivetranAccountComponent` kicks off Fivetran syncs. + +If it is necessary to fetch data from APIs in order to understand the _definitions_ of the assets, then [State-Backed Components](./state-backed/creating.md) should always be used. If creating the definitions does NOT require fetching data (e.g. tool configuration is checked into the git repository), then a regular `Component` should be used, regardless of if executing the asset requires external API calls or not. + +## Pattern: External Data Class + +Create a data class representing the raw data (for a specific asset) from the external tool's API. This is the "props" type that flows through translation and into `get_asset_spec`. + +```python +from dagster_shared.record import record + +@record +class MyConnectorTableProps: + """Raw data from the external tool for a single asset.""" + connector_id: str + table_name: str + schema_name: str + sync_enabled: bool +``` + +## Pattern: Translation Field + +Translation allows YAML users to customize asset properties without subclassing. Use `TranslationFnResolver` with an `Annotated` type: + +```python nocheck +from typing import Annotated +from dagster.components.utils.translation import TranslationFn, TranslationFnResolver + +class MyServiceComponent(dg.Component, dg.Model, dg.Resolvable): + translation: ( + Annotated[ + TranslationFn[MyConnectorTableProps], + TranslationFnResolver( + template_vars_for_translation_fn=lambda data: { + "table_name": data.table_name, + "schema_name": data.schema_name, + } + ), + ] + | None + ) = None +``` + +`TranslationFn` is a type alias for `Callable[[AssetSpec, T], AssetSpec]`. The `template_vars_for_translation_fn` callback exposes fields as Jinja template variables for YAML users. The variable `spec` is always available automatically. + +Corresponding YAML usage: + +```yaml +component_type: my_service +params: + translation: + key: "my_prefix/{{ schema_name }}/{{ table_name }}" + group: "{{ schema_name }}" +``` + +## Pattern: `get_asset_spec` Method + +A public method that converts external data into a Dagster `AssetSpec`. It should provide sensible defaults (name → key, extract tags, set `kinds`, add metadata) and be designed for subclass override. + +```python nocheckundefined +import dagster as dg + +class MyServiceComponent(dg.Component, dg.Resolvable, dg.Model): + translation: ... + + def get_asset_spec(self, data: MyConnectorTableProps) -> dg.AssetSpec: + """Generates an AssetSpec for a given connector table.""" + base_spec = dg.AssetSpec( + key=dg.AssetKey([data.schema_name, data.table_name]), + metadata={"connector_id": data.connector_id}, + kinds={"myservice"}, + ) + if self.translation: + return self.translation(base_spec, data) + return base_spec +``` + +Subclasses can override to customize defaults: + +```python nocheckundefined +class CustomComponent(MyServiceComponent): + def get_asset_spec(self, data: MyConnectorTableProps) -> dg.AssetSpec: + spec = super().get_asset_spec(data) + return spec.replace_attributes(group_name="my_group") +``` + +## Pattern: `execute()` Method + +A public method for triggering external tool execution. Designed for subclass override. + +```python nocheckundefined +def execute( + self, context: dg.AssetExecutionContext, resource: MyServiceWorkspace +) -> Iterable[dg.AssetMaterialization | dg.MaterializeResult]: + """Executes a sync for the selected connector.""" + yield from resource.sync_and_poll(context=context) +``` + +The component wires `execute` into a `@multi_asset`: + +```python +def _build_multi_asset(self, connector_id, asset_specs, resource): + @dg.multi_asset(name=connector_id, specs=asset_specs) + def _assets(context: dg.AssetExecutionContext): + yield from self.execute(context=context, resource=resource) + + return _assets +``` + +## Pattern: Observation via Sensors + +For components that need to monitor external tool events without orchestrating them, a component's `build_defs` would include a sensor definition (adding this sensor could be controlled via a boolean flag on the component): + +```python +def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions: + asset_specs = [self.get_asset_spec(d) for d in []] + + @dg.sensor(name="my_service_sensor") + def _sensor(context: dg.SensorEvaluationContext): + new_events = self._poll_for_events(context) + for event in new_events: + context.instance.report_runless_asset_event( + dg.AssetObservation(asset_key=self.get_asset_spec(event).key) + ) + + + return dg.Definitions(assets=asset_specs, sensors=[_sensor] if self.enable_sensor else None) +``` diff --git a/.agents/skills/dagster-expert/references/components/resolved-framework.md b/.agents/skills/dagster-expert/references/components/resolved-framework.md new file mode 100644 index 0000000..926ceab --- /dev/null +++ b/.agents/skills/dagster-expert/references/components/resolved-framework.md @@ -0,0 +1,171 @@ +--- +title: Resolved Framework +triggers: + - "defining custom YAML schema types using Resolver, Model, or Resolvable" +--- + +# Resolved Framework + +## Overview + +The Resolved framework lets you write Pythonic classes that auto-generate YAML schemas with Jinja2 templating support. When a component field needs a non-primitive type (like `AssetKey`, `datetime`, or a custom object), you use `Annotated` with a `Resolver` to define how raw YAML values become Python objects. + +Key classes (all importable via `import dagster as dg`): + +- **Resolvable** — mixin that enables YAML-to-Python resolution on any class +- **Model** — pydantic `BaseModel` with `extra="forbid"`, recommended for component attributes +- **Resolver** — metadata annotation that defines how a field is resolved from YAML + +## Choosing a Base Class + +Both `dg.Model` and `@dataclass` are fully supported. Either works well for most components. + +**Model (pydantic)** extends pydantic's `BaseModel` with `extra="forbid"`. Benefits: + +- Catches typos in YAML early (unknown fields are rejected) +- `Field()` metadata (descriptions, examples, defaults) propagates into the generated YAML schema, improving IDE autocompletion +- More powerful subclassing capabilities (subclasses can add new required fields) + +**Dataclass** (`@dataclass`) is a lighter-weight alternative. Use `field(default_factory=...)` for mutable defaults. Dataclasses don't support `Field()` metadata propagation and have the standard Python limitation where subclasses cannot add required fields after optional ones. + +**Plain class with annotated `__init__`** is supported for highly specific use cases, but is not recommended for standard use. The framework inspects the `__init__` signature to derive fields. + +## Nested Resolution + +The Resolved framework supports nesed resolution of classes, making it possible to simultaneously maintain complex python-native classes alongside automatically-generated YAML schemas. + +There are a few options for nesting resoltion, depending on the specific use case. + +## Nested Resolvable Classes + +Any class — not just components — can inherit `dg.Resolvable` + `dg.Model` to get automatic YAML resolution including Jinja2 template support. This is the **preferred approach for structured config objects** like connection configs, database configs, etc. + +```python +import dagster as dg + + +class ConnectionConfig(dg.Model, dg.Resolvable): + token: str + hostname: str + + +class MyComponent(dg.Component, dg.Resolvable, dg.Model): + connection: ConnectionConfig # auto-resolved, templates supported + assets: list[dg.ResolvedAssetSpec] + + def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions: ... +``` + +Corresponding YAML — Jinja2 templates work inside nested Resolvable objects: + +```yaml +type: my_project.components.my_component.MyComponent + +attributes: + connection: + token: "{{ env.TOKEN }}" + hostname: "api.example.com" + assets: + - key: my_data/api_results +``` + +### `Annotated` + `Resolver` + +If your value contains a type that is not possible to natively represent in YAML, you can define a custom `Resolver` to handle this translation, and then use `Annotated` to create a new type alias that associates the custom resolver with the target python type. + +```python +from typing import Annotated, TypeAlias +from datetime import datetime +import dagster as dg + + +def resolve_datetime(context: dg.ResolutionContext, raw: str) -> datetime: + resolved = context.resolve_value(raw, as_type=str) # process templates first + return datetime.fromisoformat(resolved) + +ResolvedDatetime: TypeAlias = Annotated[datetime, dg.Resolver(resolve_datetime, model_field_type=str)] + +class MyComponent(dg.Component, dg.Resolvable, dg.Model): + # In YAML this field accepts a string; at load time it becomes a datetime + start_date: ResolvedDatetime + + def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions: ... +``` + +### `*Args` Classes + +Sometimes, the target python type is a class that you cannot change the base class of (e.g. a third-party library class), but you still want to be able to resolve it from YAML. + +In these cases, you can create a separate class that inherits from `dg.Resolvable` + `dg.Model` and mirrors the target python class's constructor signature. + +```python +from typing import Annotated, TypeAlias + +# ... defined elsewhere ... +class SomeLibraryClass: + + def __init__(self, name: str, age: int): ... + +class SomeLibraryClassArgs(dg.Resolvable, dg.Model): + name: str + age: int + +def resolve_some_library_class(context: dg.ResolutionContext, model) -> SomeLibraryClass: + # `model` will be an instance of SomeLibraryClassArgs.model() + # this step will ensure all jinja templates are resolved (and handle any nested resolution) + args = SomeLibraryClassArgs.resolve_from_model(context, model) + # once the arguments are fully resolved, we can instantiate the target class using the resolved arguments + return SomeLibraryClass(**args.model_dump()) + +ResolvedSomeLibraryClass: TypeAlias = Annotated[SomeLibraryClass, dg.Resolver(resolve_some_library_class, model_field_type=SomeLibraryClassArgs.model())] + +class MyComponent(dg.Component, dg.Resolvable, dg.Model): + some_library_class: ResolvedSomeLibraryClass + + def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions: ... +``` + +## Resolver Types + +**`Resolver(fn)`** — custom resolution function. The function receives `(context: ResolutionContext, raw_value)` and returns the resolved Python object. String values are template-resolved before the function is called (controlled by `inject_before_resolve`, default `True`). + +**`Resolver.default()`** — standard recursive resolution. Resolves templates in the value and any nested Resolvable objects. Use this when the default behavior is sufficient but you want to add `description` or `examples` metadata: + +```python +from typing import Annotated + +name: Annotated[ + str | None, + dg.Resolver.default( + description="Human-readable name of the asset.", + examples=["my_asset"], + ), +] = None +``` + +**`Resolver.passthrough()`** — returns the raw value without template processing or nested resolution. Use for fields that should receive the literal YAML value. This can be useful in cases where you intend to process jinja templates _after_ the component has been loaded. + +**`Resolver.from_model(fn)`** — the function receives the entire parent model instead of just the field value. Use when resolution depends on multiple fields together. + +`Injected[T]` is a shorthand for `Annotated[T, Resolver.default(model_field_type=str)]` — the field accepts a string in YAML (for template injection) and resolves to type `T`. + +## Built-in Type Aliases + +**`dg.ResolvedAssetKey`** accepts a string like `"my_database/my_schema/my_table"` in YAML and resolves to an `AssetKey`. Template strings are resolved before parsing. + +**`dg.ResolvedAssetSpec`** accepts a structured mapping in YAML (with fields like `key`, `deps`, `group_name`, `tags`, `kinds`, `automation_condition`, `partitions_def`) and resolves to an `AssetSpec`. + +**`dg.ResolvedAssetCheckSpec`** accepts a structured mapping and resolves to an `AssetCheckSpec`. + +## How Schema Generation Works + +When you define a Resolvable class, the framework auto-generates a pydantic model for YAML validation: + +1. Each field's type and `Resolver` metadata are inspected +2. If `model_field_type` is set on the Resolver, that type is used in the schema instead of the Python type +3. All non-`str` fields become `field_type | str` in the schema, allowing any field to accept a Jinja2 template string +4. `description` and `examples` from the Resolver propagate into the generated schema, appearing in IDE autocompletion during YAML editing + +## References + +- [Template Variables](./template-variables.md) diff --git a/.agents/skills/dagster-expert/references/components/state-backed/creating.md b/.agents/skills/dagster-expert/references/components/state-backed/creating.md new file mode 100644 index 0000000..fd7df11 --- /dev/null +++ b/.agents/skills/dagster-expert/references/components/state-backed/creating.md @@ -0,0 +1,188 @@ +--- +title: Creating State-Backed Components +triggers: + - "building a component that fetches and caches external state" +--- + +# Creating State-Backed Components + +## Overview + +State-backed components are for integrations where Dagster definitions depend on external systems (APIs, compiled artifacts) rather than just code and config files. The framework manages fetching, storing, and loading this state so that expensive operations (API calls, compilation) don't run on every code server load. + +Subclasses implement three abstract members: + +- **`defs_state_config`** — property returning state management configuration +- **`write_state_to_path`** — fetches external state and writes it to a local path +- **`build_defs_from_state`** — builds definitions from the cached state + +## Basic Structure + +```python nocheckundefined +from pathlib import Path +from typing import Optional +import dagster as dg +from dagster.components.utils.defs_state import ( + DefsStateConfig, + DefsStateConfigArgs, + ResolvedDefsStateConfig, +) + + +class MyApiComponent(dg.StateBackedComponent, dg.Model, dg.Resolvable): + """Pulls assets from an external API.""" + + api_url: str + defs_state: ResolvedDefsStateConfig = DefsStateConfigArgs.local_filesystem() + + @property + def defs_state_config(self) -> DefsStateConfig: + return DefsStateConfig.from_args(self.defs_state, default_key=self.__class__.__name__) + + def write_state_to_path(self, state_path: Path) -> None: + # Fetch from external system and persist + data = fetch_from_api(self.api_url) + state_path.write_text(dg.serialize_value(data)) + + def build_defs_from_state( + self, context: dg.ComponentLoadContext, state_path: Optional[Path] + ) -> dg.Definitions: + if state_path is None: + return dg.Definitions() + data = dg.deserialize_value(state_path.read_text(), MyApiData) + # Build assets from data... + return dg.Definitions(assets=[]) +``` + +Corresponding YAML: + +```yaml +type: my_project.MyApiComponent + +attributes: + api_url: "https://api.example.com/v1" + defs_state: + management_type: LOCAL_FILESYSTEM +``` + +## The `defs_state_config` Property + +### User-configurable (recommended) + +Expose a `defs_state` field so users can choose the management strategy in YAML. This is the OmniComponent pattern: + +```python +from dagster.components.utils.defs_state import ( + DefsStateConfig, + DefsStateConfigArgs, + ResolvedDefsStateConfig, +) + + +class MyComponent(dg.StateBackedComponent, dg.Model, dg.Resolvable): + defs_state: ResolvedDefsStateConfig = DefsStateConfigArgs.versioned_state_storage() + + @property + def defs_state_config(self) -> DefsStateConfig: + return DefsStateConfig.from_args(self.defs_state, default_key=self.__class__.__name__) +``` + +The `default_key` is used when the user doesn't specify a `key` in YAML. Convention: `ClassName` or `ClassName[discriminator]`. + +### Hardcoded + +Construct `DefsStateConfig` directly when the strategy should not be user-configurable. This is the DbtProjectComponent pattern: + +```python +from dagster.components.utils.defs_state import DefsStateConfig +from dagster_shared.serdes.objects.models.defs_state_info import DefsStateManagementType + + +class MyToolComponent(dg.StateBackedComponent, dg.Resolvable): + project_name: str + + @property + def defs_state_config(self) -> DefsStateConfig: + return DefsStateConfig( + key=f"MyToolComponent[{self.project_name}]", + management_type=DefsStateManagementType.LOCAL_FILESYSTEM, + refresh_if_dev=True, + ) +``` + +**Key format convention:** `ClassName[discriminator]` — the discriminator distinguishes multiple instances of the same component type (e.g., different dbt projects). + +## State Serialization Strategies + +### Dagster serdes (for API responses) + +Define typed state objects with `@whitelist_for_serdes` and `@record`, then serialize with `dg.serialize_value()` / `dg.deserialize_value()`. Best when state comes from API responses. Used by OmniComponent. + +```python +from dagster_shared.record import record +from dagster_shared.serdes import whitelist_for_serdes + + +@whitelist_for_serdes +@record +class WorkspaceItem: + id: str + name: str + item_type: str + + +@whitelist_for_serdes +@record +class WorkspaceData: + items: list[WorkspaceItem] +``` + +Write and read state: + +```python nocheckundefined +class MyComponent(dg.StateBackedComponent, dg.Model, dg.Resolvable): + client: MyApiClient + + def write_state_to_path(self, state_path: Path) -> None: + items = self.client.list_items() + state = WorkspaceData(items=items) + state_path.write_text(dg.serialize_value(state)) + + def build_defs_from_state( + self, context: dg.ComponentLoadContext, state_path: Optional[Path] + ) -> dg.Definitions: + if state_path is None: + return dg.Definitions() + state = dg.deserialize_value(state_path.read_text(), WorkspaceData) + # Build assets from state.items... + return dg.Definitions(assets=[]) +``` + +### Tool's native format (for tools with existing artifacts) + +Write tool artifacts directly and read them with the tool's own parser. Best when the tool already produces a well-defined artifact format. Used by DbtProjectComponent (writes dbt manifest, reads it with dbt's manifest parser). + +```python nocheckundefined +class MyToolComponent(dg.StateBackedComponent, dg.Resolvable): + tool: MyTool + + def write_state_to_path(self, state_path: Path) -> None: + # Run the tool's build/compile step, outputting to state_path + self.tool.compile(output_dir=state_path) + + def build_defs_from_state( + self, context: dg.ComponentLoadContext, state_path: Optional[Path] + ) -> dg.Definitions: + if state_path is None: + return dg.Definitions() + manifest = self.tool.parse_manifest(state_path) + # Build assets from manifest... + return dg.Definitions(assets=[]) +``` + +## Notes + +- **Async support:** `write_state_to_path` can be `async`. The framework detects this automatically. Use async when calling async APIs (e.g., `await client.fetch(...)`). +- **Handling `state_path=None`:** `build_defs_from_state` receives `state_path=None` when no state has been written yet. Return `dg.Definitions()` (empty) in this case. +- **Do not override `build_defs`:** The base class `build_defs` manages the state lifecycle (refresh timing, storage). Override `build_defs_from_state` instead. +- **MRO:** Use `dg.StateBackedComponent, dg.Model, dg.Resolvable` for pydantic-based components, or `dg.StateBackedComponent, dg.Resolvable` with `@dataclass` for dataclass-based components. diff --git a/.agents/skills/dagster-expert/references/components/state-backed/using.md b/.agents/skills/dagster-expert/references/components/state-backed/using.md new file mode 100644 index 0000000..1205a0f --- /dev/null +++ b/.agents/skills/dagster-expert/references/components/state-backed/using.md @@ -0,0 +1,144 @@ +--- +title: Using State-Backed Components +triggers: + - "managing state-backed components in production, CI/CD, or refreshing state" +--- + +# Using State-Backed Components + +## What Are State-Backed Components + +State-backed components produce Dagster definitions that depend on external systems (BI tools, ETL platforms, compiled artifacts). Instead of calling external APIs on every code server load, they split definition-building into two steps: + +1. **`write_state_to_path`** — fetches external state and persists it locally (expensive, runs on refresh) +2. **`build_defs_from_state`** — builds definitions from persisted state (cheap, runs on every load) + +The framework controls when refresh happens based on the configured management strategy. + +## State Management Strategies + +| Strategy | Storage | Refresh Mechanism | Best For | +| ------------------------------ | ----------------------------------- | ------------------------------------------------------------ | ------------------------------------------ | +| `LOCAL_FILESYSTEM` | `.local_defs_state/` in project dir | `dg utils refresh-defs-state` in CI before building artifact | Most deployments | +| `VERSIONED_STATE_STORAGE` | Cloud storage (S3/GCS/Azure) | `dg utils refresh-defs-state` or independent of deploys | Decoupling state updates from image builds | +| `LEGACY_CODE_SERVER_SNAPSHOTS` | In-memory | Auto-refresh on every code server load | Not recommended; changing in 1.13.0 | + +### LOCAL_FILESYSTEM + +State is written to `.local_defs_state/` inside your project directory. This directory is auto-gitignored but must be included in your deployment artifact (Docker image, PEX). + +``` +my_project/ +├── my_project/ +│ ├── defs/ +│ └── ... +├── .local_defs_state/ +│ ├── DbtProjectComponent[my_dbt_project]/ +│ └── FivetranAccountComponent/ +├── pyproject.toml +└── .gitignore # includes .local_defs_state/ +``` + +### VERSIONED_STATE_STORAGE + +State is stored in cloud storage with UUID-based versioning. Enables state updates without rebuilding deployment artifacts. Requires `dagster.yaml` configuration: + +```yaml +# dagster.yaml +defs_state_storage: + module: dagster._core.storage.defs_state.blob_storage_state_storage + class: UPathDefsStateStorage + config: + base_path: s3://my-bucket/dagster/defs-state # or gs:// or az:// +``` + +### LEGACY_CODE_SERVER_SNAPSHOTS + +In-memory state that auto-refreshes on every code server load. This is the current default for backwards compatibility but is **not recommended**. The default will change in version 1.13.0. Explicitly set `management_type` to `LOCAL_FILESYSTEM` or `VERSIONED_STATE_STORAGE` instead. + +## Configuring State-Backed Components + +Each state-backed component accepts a `defs_state` field in its YAML configuration: + +```yaml +type: dagster_fivetran.FivetranAccountComponent + +attributes: + account: ... + defs_state: + management_type: LOCAL_FILESYSTEM + refresh_if_dev: false +``` + +| Field | Type | Default | Description | +| ----------------- | --------------------------------------------------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------- | +| `management_type` | `LOCAL_FILESYSTEM` \| `VERSIONED_STATE_STORAGE` \| `LEGACY_CODE_SERVER_SNAPSHOTS` | Component-specific | Storage strategy for state | +| `key` | string | Auto-generated from class name | Unique identifier for this component's state | +| `refresh_if_dev` | boolean | `true` | Auto-refresh state during `dagster dev`. Set `false` to reduce API calls during local dev | + +### Dev mode auto-refresh + +When `refresh_if_dev: true` (the default), running `dagster dev` or using the `dg` CLI automatically refreshes state on startup. Set to `false` when: + +- External API rate limits are a concern +- You want faster local iteration with previously-cached state +- The external system is unavailable in your dev environment + +### Viewing state in the Dagster UI + +Navigate to **Deployment → [your code location] → Defs state** to see registered state keys, last update timestamps, and version identifiers. + +## Defs State Keys + +Keys uniquely identify a component's state within a deployment. Default format: `ClassName` or `ClassName[discriminator]`. + +- **Auto-generated:** Most components use `self.__class__.__name__` as the default key +- **Discriminator:** Used when multiple instances of the same component exist (e.g., `DbtProjectComponent[analytics]`, `DbtProjectComponent[marketing]`) +- **Custom override:** Set `key` in the YAML `defs_state` block to use a custom key +- **Shared keys:** Two components sharing a key will share the same state — use this intentionally or not at all + +## CI/CD State Refresh + +Before deploying, refresh state for all state-backed components: + +```bash +uv run dg utils refresh-defs-state +``` + +This fetches current state from all external systems and persists it according to each component's configured strategy. + +### LOCAL_FILESYSTEM deployments + +Run `dg utils refresh-defs-state` **before** building your Docker image or PEX artifact. The `.local_defs_state/` directory must be included when copying project files into the image. + +### VERSIONED_STATE_STORAGE deployments + +Ensure the CI environment has: + +- `DAGSTER_HOME` pointing to a directory containing `dagster.yaml` with `defs_state_storage` configured +- Cloud storage credentials (AWS, GCS, or Azure) available in the environment + +### Dagster+ deployments + +Use the Dagster+ variant of the refresh command: + +```bash +uv run dg plus deploy refresh-defs-state +``` + +Or scaffold a complete CI/CD workflow with state refresh included: + +```bash +uv run dg scaffold github-actions +``` + +## Common State-Backed Components + +- **Tableau** — `dagster_tableau.TableauComponent` +- **Looker** — `dagster_looker.LookerComponent` +- **Sigma** — `dagster_sigma.SigmaComponent` +- **Power BI** — `dagster_powerbi.PowerBIWorkspaceComponent` +- **Fivetran** — `dagster_fivetran.FivetranAccountComponent` +- **Airbyte** — `dagster_airbyte.AirbyteWorkspaceComponent` +- **dbt** — `dagster_dbt.DbtProjectComponent` +- **Omni** — `dagster_omni.OmniComponent` diff --git a/.agents/skills/dagster-expert/references/components/subclassing-components.md b/.agents/skills/dagster-expert/references/components/subclassing-components.md new file mode 100644 index 0000000..4d118bb --- /dev/null +++ b/.agents/skills/dagster-expert/references/components/subclassing-components.md @@ -0,0 +1,7 @@ +--- +title: Subclassing Components +triggers: + - "extending an existing component via subclassing" +--- + +TODO diff --git a/.agents/skills/dagster-expert/references/components/template-variables.md b/.agents/skills/dagster-expert/references/components/template-variables.md new file mode 100644 index 0000000..256928e --- /dev/null +++ b/.agents/skills/dagster-expert/references/components/template-variables.md @@ -0,0 +1,159 @@ +--- +title: Template Variables +triggers: + - "using Jinja2 template variables in component YAML (env, dg, context, or custom scopes)" +--- + +# Template Variables + +## Overview + +Dagster component YAML files support Jinja2 expressions with `{{ expression }}` syntax. Any Resolvable field can accept template strings because the generated schema unions every non-`str` field type with `str`. + +Templates are often resolved at component load time, but some contexts resolve independently. For example, post-processing templates (`post_processing.assets[].attributes`) are resolved once per asset, with the current asset available as a template variable. + +## Built-in Scopes + +### env — Environment Variables + +Access environment variables directly or with a fallback default: + +```yaml +attributes: + connection_string: "{{ env.DATABASE_URL }}" + api_key: "{{ env('API_KEY', 'dev-key-fallback') }}" +``` + +`{{ env.MY_VAR }}` returns the value or `None` if unset. `{{ env('MY_VAR', 'default') }}` raises an error if unset and no default is provided. + +### dg — Dagster Objects + +Instantiate Dagster objects directly in YAML: + +```yaml +attributes: + automation_condition: "{{ dg.AutomationCondition.eager() }}" + partitions_def: "{{ dg.DailyPartitionsDefinition(start_date='2024-01-01') }}" +``` + +Available objects: `AutomationCondition`, `FreshnessPolicy`, `DailyPartitionsDefinition`, `WeeklyPartitionsDefinition`, `MonthlyPartitionsDefinition`, `HourlyPartitionsDefinition`, `StaticPartitionsDefinition`, `TimeWindowPartitionsDefinition`. + +### datetime — Python datetime + +```yaml +attributes: + lookback: "{{ datetime.timedelta(days=7) }}" + start: "{{ datetime.datetime(2024, 1, 1) }}" +``` + +Available: `datetime` and `timedelta`. + +### context — Component Load Context + +Access project information and load other components: + +```yaml +attributes: + root: "{{ context.project_root }}" + other_component: "{{ context.load_component('other/path') }}" + sub_defs: "{{ context.build_defs('submodule') }}" +``` + +### asset — Current Asset (Post-Processing Only) + +Available inside `post_processing.assets[].attributes` blocks. The template is resolved independently per matching asset, with `asset` bound to the current `AssetSpec`: + +```yaml +post_processing: + assets: + - target: "*" + attributes: + tags: + source_group: "{{ asset.group_name }}" + metadata: + key_path: "{{ asset.key }}" +``` + +Available attributes include `key`, `tags`, `group_name`, `metadata`, `kinds`, and all other `AssetSpec` fields. + +## Custom Template Variables + +### Module-Level Template Vars + +Create a `template_vars.py` file and reference it in `defs.yaml`: + +```yaml +# defs.yaml +template_vars_module: template_vars.py +``` + +```python +# template_vars.py +import dagster as dg + + +@dg.template_var +def team_prefix(): + """Static value — no parameters.""" + return "analytics" + + +@dg.template_var +def project_name(context: dg.ComponentLoadContext): + """Context-aware — receives the load context.""" + return context.path.parent.name +``` + +Template var functions have one of two signatures: + +- `() -> Any` — zero parameters, returns a static value +- `(context: ComponentLoadContext) -> Any` — receives the component load context + +### Component Static Methods + +Define `@template_var` as static methods on a Component class. These are auto-discovered without needing `template_vars_module`: + +```python +class MyComponent(dg.Component, dg.Resolvable, dg.Model): + asset_key: dg.ResolvedAssetKey + + @staticmethod + @dg.template_var + def default_group(): + return "ingestion" + + def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions: ... +``` + +```yaml +type: my_project.components.MyComponent + +attributes: + asset_key: "{{ default_group }}/my_asset" +``` + +### User-Defined Functions (UDFs) + +Template vars can return callables, which are then invoked with arguments in templates: + +```python +# template_vars.py +import dagster as dg + + +@dg.template_var +def prefixed_key(): + def _make_key(table_name: str) -> str: + return f"analytics/raw/{table_name}" + + return _make_key +``` + +```yaml +attributes: + asset_key: "{{ prefixed_key('orders') }}" +``` + +## References + +- [Resolved Framework](./resolved-framework.md) diff --git a/.agents/skills/dagster-expert/references/env-vars.md b/.agents/skills/dagster-expert/references/env-vars.md new file mode 100644 index 0000000..8a386ec --- /dev/null +++ b/.agents/skills/dagster-expert/references/env-vars.md @@ -0,0 +1,124 @@ +--- +title: Environment Variables +triggers: + - "configuring environment variables across different environments" +--- + +# Environment Variables Reference + +## Overview + +Environment variables provide a clean separation between configuration and code, allowing the same Dagster project to work across different environments (development, staging, production) without code changes. + +## How `dg` Commands Handle Environment Variables + +The `dg` CLI automatically loads environment variables from `.env` files in your project root before executing commands. This means: + +- No manual `export` or `source` commands needed +- Commands like `dg launch`, `dg dev`, `dg check defs` automatically use `.env` values +- Works consistently across local development and CI/CD environments + +## Listing Available Environment Variables + +Use the `dg list envs` command to see what environment variables your Dagster definitions reference: + +```bash +dg list envs +``` + +This shows all `EnvVar` references in your code, helping you identify what needs to be configured. + +## Basic .env File Pattern + +```bash +# .env (at project root, never commit to git) +DATABASE_URL=postgresql://localhost:5432/mydb +API_KEY=secret_key_here +SNOWFLAKE_ACCOUNT=myaccount +SNOWFLAKE_USER=myuser +SNOWFLAKE_PASSWORD=secret +``` + +**Important:** Add `.env` to your `.gitignore` to prevent committing secrets. + +## Using Environment Variables in Dagster + +### In Resources + +```python +import dagster as dg + +class DatabaseResource(dg.ConfigurableResource): + connection_string: str = dg.EnvVar("DATABASE_URL") + timeout: int = dg.EnvVar.int("DB_TIMEOUT") + +# Dagster automatically loads DATABASE_URL and DB_TIMEOUT from .env +``` + +### In Component YAML Files + +```yaml +# defs/my_component/defs.yaml +component: dagster_sling.SlingReplicationComponent + +params: + sling: + connections: + - name: MY_POSTGRES + type: postgres + host: "{{ env.DB_HOST }}" + port: "{{ env.DB_PORT }}" + database: "{{ env.DB_NAME }}" +``` + +## Environment-Specific Configuration + +For multiple environments, use separate `.env` files: + +``` +my_project/ +├── .env # Local development (gitignored) +├── .env.example # Template (committed to git) +├── .env.staging # Staging config (gitignored) +└── .env.prod # Production config (gitignored) +``` + +### .env.example Template + +```bash +# .env.example (committed to git) +DATABASE_URL=postgresql://localhost:5432/mydb +API_KEY=your_api_key_here +SNOWFLAKE_ACCOUNT=your_account +SNOWFLAKE_USER=your_user +SNOWFLAKE_PASSWORD=your_password +``` + +Developers copy `.env.example` to `.env` and fill in real values. + +### Loading Environment-Specific Files + +```bash +# Development (uses .env by default) +dg dev + +# Staging +dg launch --env-file .env.staging --assets my_asset + +# Production +dg launch --env-file .env.prod --assets my_asset +``` + +## Best Practices + +1. **Never commit `.env` files** - Always add to `.gitignore` +2. **Provide `.env.example`** - Template for required variables +3. **Use descriptive names** - `SNOWFLAKE_ACCOUNT` not `ACCT` +4. **Group related vars** - Prefix related vars (e.g., `SNOWFLAKE_*`, `S3_*`) +5. **Document required vars** - List required environment variables in README +6. **Use `dg list envs`** - Verify all environment variables are configured + +## References + +- [CLI list envs reference](./cli/list/envs.md) - `dg list envs` command +- [Dagster EnvVar API](https://docs.dagster.io/api/dagster/dagster.EnvVar) diff --git a/.agents/skills/dagster-expert/references/integrations/INDEX.md b/.agents/skills/dagster-expert/references/integrations/INDEX.md new file mode 100644 index 0000000..67ef67f --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/INDEX.md @@ -0,0 +1,66 @@ +--- +title: Integrations +type: index +triggers: + - "needs an integration library for an external tool or technology" +--- + +# Integrations Reference + +Dagster provides integration libraries for a range of tools and technologies. This reference directory contains detailed information about specific integrations. + +Integration libraries are typically named `dagster-`, where `` is the name of the tool or technology being integrated. Integrations marked as _(community)_ are maintained by the community rather than the Dagster team. + +All integration reference files contain a link to the official documentation for the integration library, which can be referenced in cases where the local documentation does not provide sufficient information. + +## Reference Files Index + + + +- [dagster-airbyte](./dagster-airbyte/INDEX.md) — Airbyte extract-load syncs as Dagster assets +- [dagster-airlift](./dagster-airlift/INDEX.md) — migrating or co-orchestrating Airflow DAGs with Dagster +- [dagster-aws](./dagster-aws/INDEX.md) — AWS services (S3, ECS, Lambda) from Dagster +- [dagster-azure](./dagster-azure/INDEX.md) — Azure services (ADLS, Blob Storage) from Dagster +- [dagster-celery](./dagster-celery/INDEX.md) — distributed task execution with Celery +- [dagster-census](./dagster-census/INDEX.md) — reverse ETL syncs with Census +- [dagster-dask](./dagster-dask/INDEX.md) — parallel and distributed computing with Dask +- [dagster-databricks](./dagster-databricks/INDEX.md) — Spark-based data processing on Databricks +- [dagster-datadog](./dagster-datadog/INDEX.md) — monitoring and observability with Datadog +- [dagster-datahub](./dagster-datahub/INDEX.md) — metadata management and data cataloging with DataHub +- [dagster-dbt](./dagster-dbt/INDEX.md) — integrating dbt Core or dbt Cloud with Dagster +- [dagster-deltalake](./dagster-deltalake/INDEX.md) — lakehouse storage with Delta Lake +- [dagster-docker](./dagster-docker/INDEX.md) — containerized execution with Docker +- [dagster-duckdb](./dagster-duckdb/INDEX.md) — in-process analytical queries with DuckDB +- [dagster-fivetran](./dagster-fivetran/INDEX.md) — managed extract-load connectors with Fivetran +- [dagster-gcp](./dagster-gcp/INDEX.md) — Google Cloud Platform (BigQuery, GCS) from Dagster +- [dagster-github](./dagster-github/INDEX.md) — GitHub repository event handling from Dagster +- [dagster-great-expectations](./dagster-great-expectations/INDEX.md) — data validation and testing with Great Expectations +- [dagster-hightouch](./dagster-hightouch/INDEX.md) — reverse ETL and data activation with Hightouch +- [dagster-iceberg](./dagster-iceberg/INDEX.md) — Apache Iceberg table format management +- [dagster-jupyter](./dagster-jupyter/INDEX.md) — notebook-based assets with Jupyter +- [dagster-k8s](./dagster-k8s/INDEX.md) — Kubernetes container orchestration and execution +- [dagster-looker](./dagster-looker/INDEX.md) — Looker BI dashboard assets +- [dagster-mlflow](./dagster-mlflow/INDEX.md) — ML experiment tracking and model management with MLflow +- [dagster-msteams](./dagster-msteams/INDEX.md) — Microsoft Teams notifications and alerts from Dagster +- [dagster-mysql](./dagster-mysql/INDEX.md) — MySQL as a Dagster storage backend +- [dagster-omni](./dagster-omni/INDEX.md) — analytics and BI with Omni +- [dagster-openai](./dagster-openai/INDEX.md) — LLM-powered assets with OpenAI +- [dagster-pagerduty](./dagster-pagerduty/INDEX.md) — incident management alerts with PagerDuty +- [dagster-pandas](./dagster-pandas/INDEX.md) — Pandas DataFrame type checking and validation +- [dagster-pandera](./dagster-pandera/INDEX.md) — DataFrame schema validation with Pandera +- [dagster-papertrail](./dagster-papertrail/INDEX.md) — log management with Papertrail +- [dagster-polars](./dagster-polars/INDEX.md) — fast DataFrame processing with Polars +- [dagster-postgres](./dagster-postgres/INDEX.md) — PostgreSQL as a Dagster storage backend +- [dagster-powerbi](./dagster-powerbi/INDEX.md) — Power BI dashboard assets +- [dagster-prometheus](./dagster-prometheus/INDEX.md) — metrics collection with Prometheus +- [dagster-pyspark](./dagster-pyspark/INDEX.md) — distributed data processing with PySpark +- [dagster-sigma](./dagster-sigma/INDEX.md) — BI and analytics assets with Sigma +- [dagster-slack](./dagster-slack/INDEX.md) — Slack notifications or alerts from Dagster +- [dagster-sling](./dagster-sling/INDEX.md) — EL data replication with Sling +- [dagster-snowflake](./dagster-snowflake/INDEX.md) — interacting with Snowflake from Dagster +- [dagster-spark](./dagster-spark/INDEX.md) — distributed data processing with Apache Spark +- [dagster-ssh](./dagster-ssh/INDEX.md) — remote command execution via SSH +- [dagster-tableau](./dagster-tableau/INDEX.md) — Tableau BI dashboard assets +- [dagster-twilio](./dagster-twilio/INDEX.md) — SMS and communication with Twilio +- [dagster-wandb](./dagster-wandb/INDEX.md) — ML experiment tracking with Weights & Biases + diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-airbyte/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-airbyte/INDEX.md new file mode 100644 index 0000000..676082c --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-airbyte/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-airbyte +triggers: + - "Airbyte extract-load syncs as Dagster assets" +--- + +# dagster-airbyte + +Docs: https://docs.dagster.io/integrations/libraries/airbyte diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-airlift/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-airlift/INDEX.md new file mode 100644 index 0000000..1d49cf0 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-airlift/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-airlift +triggers: + - "migrating or co-orchestrating Airflow DAGs with Dagster" +--- + +# dagster-airlift + +Docs: https://docs.dagster.io/integrations/libraries/airlift diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-aws/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-aws/INDEX.md new file mode 100644 index 0000000..0da4dc9 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-aws/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-aws +triggers: + - "AWS services (S3, ECS, Lambda) from Dagster" +--- + +# dagster-aws + +Docs: https://docs.dagster.io/integrations/libraries/aws diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-azure/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-azure/INDEX.md new file mode 100644 index 0000000..a7b9374 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-azure/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-azure +triggers: + - "Azure services (ADLS, Blob Storage) from Dagster" +--- + +# dagster-azure + +Docs: https://docs.dagster.io/integrations/libraries/azure diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-celery/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-celery/INDEX.md new file mode 100644 index 0000000..fd0c9c3 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-celery/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-celery +triggers: + - "distributed task execution with Celery" +--- + +# dagster-celery + +Docs: https://docs.dagster.io/integrations/libraries/celery diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-census/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-census/INDEX.md new file mode 100644 index 0000000..ded773d --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-census/INDEX.md @@ -0,0 +1,11 @@ +--- +title: dagster-census +triggers: + - "reverse ETL syncs with Census" +--- + +# dagster-census + +> **Community-supported integration.** + +Docs: https://docs.dagster.io/integrations/libraries/census diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-dask/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-dask/INDEX.md new file mode 100644 index 0000000..8709626 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-dask/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-dask +triggers: + - "parallel and distributed computing with Dask" +--- + +# dagster-dask + +Docs: https://docs.dagster.io/integrations/libraries/dask diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-databricks/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-databricks/INDEX.md new file mode 100644 index 0000000..a01e03f --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-databricks/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-databricks +triggers: + - "Spark-based data processing on Databricks" +--- + +# dagster-databricks + +Docs: https://docs.dagster.io/integrations/libraries/databricks diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-datadog/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-datadog/INDEX.md new file mode 100644 index 0000000..6d1e5a4 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-datadog/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-datadog +triggers: + - "monitoring and observability with Datadog" +--- + +# dagster-datadog + +Docs: https://docs.dagster.io/integrations/libraries/datadog diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-datahub/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-datahub/INDEX.md new file mode 100644 index 0000000..d17d7e9 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-datahub/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-datahub +triggers: + - "metadata management and data cataloging with DataHub" +--- + +# dagster-datahub + +Docs: https://docs.dagster.io/integrations/libraries/datahub diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-dbt/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/INDEX.md new file mode 100644 index 0000000..60a10dd --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/INDEX.md @@ -0,0 +1,35 @@ +--- +title: dagster-dbt +type: index +triggers: + - "integrating dbt Core or dbt Cloud with Dagster" +--- + +# dagster-dbt Integration Reference + +Docs: https://docs.dagster.io/integrations/libraries/dbt + +The dagster-dbt integration represents each dbt models as Dagster assets, enabling granular orchestration +at the individual model level. + +### Workflow Decision Tree + +Depending on the user's request, choose the appropriate reference file: + +- Creating/scaffolding a new dbt component? → [Scaffolding](scaffolding.md) +- Configuring or customizing an existing dbt component? → [Component-Based Integration](component-based-integration.md) +- Using dbt Cloud? → [dbt Cloud Integration](dbt-cloud.md) +- General questions about dbt and Dagster? + - Determine which reference file from the [Reference Files Index](#reference-files-index) below is most relevant to the user's request. + +## Reference Files Index + + + +- [dbt: Asset Checks](./asset-checks.md) — how dbt tests map to Dagster asset checks +- [dbt: Component-Based Integration](./component-based-integration.md) — configuring or customizing DbtProjectComponent +- [dbt: Cloud Integration](./dbt-cloud.md) — integrating dbt Cloud with Dagster +- [dbt: Dependencies](./dependencies.md) — understanding or defining upstream dependencies for dbt models +- [dbt: Pythonic Integration](./pythonic-integration.md) — using @dbt_assets decorator for programmatic dbt integration +- [dbt: Scaffolding](./scaffolding.md) — scaffolding a new dbt component in a Dagster project + diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-dbt/asset-checks.md b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/asset-checks.md new file mode 100644 index 0000000..ee64fb5 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/asset-checks.md @@ -0,0 +1,67 @@ +--- +title: "dbt: Asset Checks" +triggers: + - "how dbt tests map to Dagster asset checks" +--- + +# Asset Checks + +dbt tests are loaded as Dagster asset checks by default (enabled in dagster-dbt 0.23.0+). + +## Enabling Source Tests + +By default, only tests on dbt models are loaded as checks. To include tests on sources: + +**Component approach:** + +```yaml +attributes: + translator_settings: + enable_source_tests_as_checks: true +``` + +**Pythonic approach:** + +```python nocheck +from dagster_dbt import DagsterDbtTranslator, DagsterDbtTranslatorSettings + +translator = DagsterDbtTranslator( + settings=DagsterDbtTranslatorSettings( + enable_source_tests_as_checks=True + ) +) + +@dbt_assets( + manifest=my_dbt_project.manifest_path, + dagster_dbt_translator=translator +) +def my_dbt_assets(context, dbt: DbtCliResource): + yield from dbt.cli(["build"], context=context).stream() +``` + +## Singular Tests with Multiple Dependencies + +For singular tests that depend on multiple models, specify the target model in the test's config +block: + +```sql +{{ + config( + meta={ + 'dagster': { + 'ref': { + 'name': 'customers', + 'package': 'my_dbt_assets', + 'version': 1, + }, + } + } + ) +}} + +SELECT ... +``` + +The `ref` structure mirrors dbt's +[ref function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref) parameters. Without this +metadata, the test still runs but emits an AssetObservation instead of an asset check result. diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-dbt/component-based-integration.md b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/component-based-integration.md new file mode 100644 index 0000000..586a8ae --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/component-based-integration.md @@ -0,0 +1,242 @@ +--- +title: "dbt: Component-Based Integration" +triggers: + - "configuring or customizing DbtProjectComponent" +--- + +# Component-Based Integration + +The Component-based approach uses `DbtProjectComponent` to define dbt assets via YAML configuration. +This is the recommended approach for new projects. + +> For initial setup and scaffolding, see [Scaffolding](scaffolding.md). + +## Overview + +`DbtProjectComponent` is a StateBackedComponent that automatically compiles and caches your dbt +project's manifest. See +[StateBackedComponents](../../components/state-backed/using.md) for general +state management patterns. + +### dbt-Specific State Management + +**What state is managed**: The dbt `manifest.json` file, which contains all dbt models, tests, +sources, and their relationships. + +**How it's compiled**: Runs `dbt parse` (and `dbt deps` if needed) to generate the manifest. + +**Configuration**: Use `prepare_if_dev` setting to control whether manifest is recompiled during +local development (defaults to `true`). + +In CI/CD, use `dg utils refresh-defs-state` or `dg plus deploy refresh-defs-state` to compile the +manifest before deployment. + +## Basic Configuration + +### Selecting Models + +Use dbt selection syntax to filter which models are included: + +```yaml +attributes: + select: "tag:daily" + exclude: "tag:deprecated" +``` + +### CLI Arguments + +Customize the dbt command executed for asset materialization: + +```yaml +attributes: + cli_args: + - build + - --full-refresh +``` + +CLI args support template variables for dynamic values based on execution context. + +## Translation and Customization + +### Simple Customization: YAML Translation Block + +For basic metadata customization, use the `translation` block: + +```yaml +attributes: + translation: + group_name: analytics + description: "dbt model {{ node.name }}" +``` + +Template variables have access to the dbt node properties. + +### Advanced Customization: Subclassing + +For complex customization, create a subclass and override `get_asset_spec()`: + +```python +from collections.abc import Mapping +from typing import Any, Optional + +import dagster as dg +from dagster_dbt import DbtProject, DbtProjectComponent + + +class CustomDbtComponent(DbtProjectComponent): + def get_asset_spec( + self, + manifest: Mapping[str, Any], + unique_id: str, + project: Optional[DbtProject], + ) -> dg.AssetSpec: + base_spec = super().get_asset_spec(manifest, unique_id, project) + dbt_props = self.get_resource_props(manifest, unique_id) + + return base_spec.merge_attributes(metadata={"model_name": dbt_props["name"]}) +``` + +Then use your custom component in `defs.yaml`: + +```yaml +type: my_project.lib.custom_dbt_component.CustomDbtComponent +``` + +### dbt Meta Config + +Define custom metadata in your dbt project files that can be consumed by your custom +`get_asset_spec()` implementation: + +```yaml +models: + - name: customers + meta: + custom_field: custom_value +``` + +This metadata is accessible via the manifest in your translator code. + +## Incremental Models and Partitioning + +To partition incremental dbt models: + +1. Define a partitions definition as a template variable +2. Apply it to assets via `post_processing` +3. Pass partition-specific vars through `cli_args` + +```yaml +template_vars_module: .template_vars + +attributes: + cli_args: + - build + - --vars: + min_date: "{{ context.partition_time_window.start.strftime('%Y-%m-%d') }}" + max_date: "{{ context.partition_time_window.end.strftime('%Y-%m-%d') }}" + +post_processing: + assets: + - target: "*" + attributes: + partitions_def: "{{ my_partitions_def }}" +``` + +The `context.partition_time_window` variable is available in `cli_args` during execution. Dagster +automatically converts the vars dict to JSON format for dbt CLI. + +For multiple partitions definitions, create separate `DbtProjectComponent` instances and use +`select` to filter models for each. + +## Metadata + +Enable automatic metadata fetching during materialization: + +```yaml +attributes: + include_metadata: + - row_count + - column_metadata +``` + +- `row_count`: Fetch row counts for tables +- `column_metadata`: Fetch column schema and lineage (extracted via sqlglot parsing) + +Both metadata types are fetched in parallel during dbt execution. + +## Asset Checks + +See [Asset Checks](asset-checks.md) for details on how dbt tests are loaded as Dagster asset checks. + +## Dependencies + +See [Dependencies](dependencies.md) for details on how Dagster parses dbt project dependencies and +patterns for defining additional dependencies. + +### Referencing dbt Models in Other Components + +Use the `asset_key_for_model` method to reference dbt models from other components: + +```yaml +type: dagster.PythonScriptComponent +attributes: + script_path: export_customers.py + dependencies: + - "{{ context.load_component('dbt_component').asset_key_for_model('customers') }}" +``` + +## Scheduling + +Use standard Dagster scheduling approaches with asset selections: + +```python +import dagster as dg + +daily_dbt_job = dg.define_asset_job( + name="daily_dbt_models", + selection=dg.AssetSelection.groups("analytics"), +) + +daily_schedule = dg.ScheduleDefinition( + job=daily_dbt_job, + cron_schedule="0 0 * * *", +) +``` + +For declarative automation, configure AutomationConditions via custom `get_asset_spec()` +implementation. + +## Runtime Configuration + +Override the `execute()` method to customize execution behavior: + +```python +from collections.abc import Iterator + +import dagster as dg +from dagster_dbt import DbtCliResource, DbtProjectComponent + + +class ConfigurableDbtComponent(DbtProjectComponent): + @property + def op_config_schema(self) -> type[dg.Config]: + class DbtConfig(dg.Config): + full_refresh: bool = False + + return DbtConfig + + def execute( + self, context: dg.AssetExecutionContext, dbt: DbtCliResource + ) -> Iterator: + if context.op_config.get("full_refresh"): + args = ["build", "--full-refresh"] + else: + args = self.get_cli_args(context) + + yield from dbt.cli(args, context=context).stream() +``` + +Define `op_config_schema` property to specify available config options. + +## dbt Cloud + +For dbt Cloud projects, see [dbt Cloud Integration](dbt-cloud.md). diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-dbt/dbt-cloud.md b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/dbt-cloud.md new file mode 100644 index 0000000..7cc0da9 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/dbt-cloud.md @@ -0,0 +1,74 @@ +--- +title: "dbt: Cloud Integration" +triggers: + - "integrating dbt Cloud with Dagster" +--- + +# dbt Cloud Integration + +The dagster-dbt library provides integration with dbt Cloud via the v2 API (recommended) for +projects hosted on dbt Cloud. + +## When to Use + +Use the dbt Cloud integration when: + +- Your dbt project is hosted and run in dbt Cloud +- You want to trigger dbt Cloud jobs from Dagster +- You want to import dbt Cloud job metadata into Dagster as assets + +For dbt Core projects (self-hosted), use the [Component-Based](component-based-integration.md) or +[Pythonic](pythonic-integration.md) integration instead. + +## Key Components + +- **`DbtCloudWorkspace`**: Resource for accessing a dbt Cloud workspace +- **`DbtCloudCredentials`**: Authentication credentials (account_id, token, access_url) +- **`dbt_cloud_assets`**: Decorator for defining assets from a dbt Cloud job +- **`load_dbt_cloud_asset_specs()`**: Load asset specs from dbt Cloud job +- **`build_dbt_cloud_polling_sensor()`**: Create a sensor that polls for dbt Cloud job completion + +## Basic Setup + +```python +import dagster as dg +from dagster_dbt import DbtCloudCredentials, DbtCloudWorkspace, dbt_cloud_assets + +# Define credentials +dbt_cloud_credentials = DbtCloudCredentials( + account_id=12345, + token=dg.EnvVar("DBT_CLOUD_API_TOKEN"), + access_url="https://cloud.getdbt.com", +) + +# Define workspace resource +dbt_cloud = DbtCloudWorkspace( + credentials=dbt_cloud_credentials, + project_id=67890, + environment_id=1, +) + + +# Define assets from dbt Cloud job +@dbt_cloud_assets(workspace=dbt_cloud) +def my_dbt_cloud_assets(context: dg.AssetExecutionContext): + # Trigger the dbt Cloud job and wait for completion + ... + + +defs = dg.Definitions( + assets=[my_dbt_cloud_assets], + resources={"dbt_cloud": dbt_cloud}, +) +``` + +## Features + +The dbt Cloud integration: + +- Triggers dbt Cloud jobs from Dagster +- Imports dbt models as Dagster assets with proper lineage +- Syncs metadata from dbt Cloud job runs +- Supports polling sensors for external job triggers + +For detailed usage, refer to the dagster-dbt dbt Cloud documentation. diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-dbt/dependencies.md b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/dependencies.md new file mode 100644 index 0000000..3cdadde --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/dependencies.md @@ -0,0 +1,48 @@ +--- +title: "dbt: Dependencies" +triggers: + - "understanding or defining upstream dependencies for dbt models" +--- + +# Dependencies + +## How Dependencies Work + +Dagster parses the dependencies already present in your dbt project: + +- dbt `ref()` calls create dependencies between models +- dbt `source()` calls create dependencies on upstream assets + +## Defining Upstream Dagster Assets + +Define a Dagster asset as a dbt source in `sources.yml`: + +```yaml +sources: + - name: dagster + tables: + - name: my_upstream_asset +``` + +Then reference it in your dbt model: + +```sql +select * from {{ source('dagster', 'my_upstream_asset') }} +``` + +This creates a data dependency where the dbt model reads from the Dagster asset. + +## Adding Dependencies via Jinja Comments + +To add dependencies not encoded in dbt's data lineage (e.g., scheduling constraints without data +reading), use a Jinja comment in your model: + +```sql +-- depends_on: {{ source('dagster', 'upstream') }} + +SELECT ... +``` + +When dbt compiles the project, it evaluates this Jinja template and adds the source to the model's +dependency list in the manifest. Dagster parses the manifest and creates the dependency edge. This +creates a scheduling dependency without requiring the model to SELECT from that source. diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-dbt/pythonic-integration.md b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/pythonic-integration.md new file mode 100644 index 0000000..cccc7e1 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/pythonic-integration.md @@ -0,0 +1,293 @@ +--- +title: "dbt: Pythonic Integration" +triggers: + - "using @dbt_assets decorator for programmatic dbt integration" +--- + +# Pythonic Integration + +The Pythonic approach uses the `@dbt_assets` decorator to define dbt assets programmatically. This +provides maximum flexibility for complex customization scenarios. + +## Overview + +Key classes: + +- **`@dbt_assets`**: Decorator that loads dbt models as Dagster assets from a manifest +- **`DbtCliResource`**: Resource for executing dbt CLI commands +- **`DbtProject`**: Represents a dbt project and manages manifest compilation +- **`DagsterDbtTranslator`**: Customizes how dbt nodes map to Dagster assets + +## Basic Setup + +```python +import dagster as dg +from dagster_dbt import DbtCliResource, DbtProject, dbt_assets + +# Define the dbt project +my_dbt_project = DbtProject(project_dir="path/to/dbt_project", target="dev") + +# Create the dbt resource +dbt_resource = DbtCliResource(project_dir=my_dbt_project.project_dir) + + +# Define assets from dbt models +@dbt_assets(manifest=my_dbt_project.manifest_path) +def my_dbt_assets(context: dg.AssetExecutionContext, dbt: DbtCliResource): + yield from dbt.cli(["build"], context=context).stream() + + +# Define definitions +defs = dg.Definitions(assets=[my_dbt_assets], resources={"dbt": dbt_resource}) +``` + +## Manifest Management + +The dbt manifest can be generated at runtime (development) or build time (production). + +### Runtime Compilation (Development) + +`DbtProject.prepare_if_dev()` automatically compiles the manifest during local development: + +```python nocheck +my_dbt_project = DbtProject(project_dir="path/to/dbt_project") +my_dbt_project.prepare_if_dev() # Runs dbt deps + dbt parse if needed +``` + +### Build-Time Compilation (Production) + +For production deployments, precompile the manifest in CI/CD to avoid recompilation overhead. The +manifest at `DbtProject.manifest_path` (typically `target/manifest.json`) should be included in your +deployed package. + +## Selecting Models + +Use dbt selection syntax to filter which models are included: + +```python nocheck +@dbt_assets( + manifest=my_dbt_project.manifest_path, + select="tag:daily", + exclude="tag:deprecated", +) +def my_dbt_assets(context, dbt: DbtCliResource): + yield from dbt.cli(["build"], context=context).stream() +``` + +## Translation and Customization + +### Using DagsterDbtTranslator + +Create a custom translator by subclassing `DagsterDbtTranslator` and overriding `get_asset_spec()`: + +```python nocheck +from collections.abc import Mapping +from typing import Any, Optional + +import dagster as dg +from dagster_dbt import DagsterDbtTranslator, DbtProject, dbt_assets + + +class CustomDbtTranslator(DagsterDbtTranslator): + def get_asset_spec( + self, + manifest: Mapping[str, Any], + unique_id: str, + project: Optional[DbtProject], + ) -> dg.AssetSpec: + base_spec = super().get_asset_spec(manifest, unique_id, project) + dbt_props = self.get_resource_props(manifest, unique_id) + + return base_spec.merge_attributes( + metadata={"model_name": dbt_props["name"]}, group_name="analytics" + ) + + +@dbt_assets( + manifest=my_dbt_project.manifest_path, + dagster_dbt_translator=CustomDbtTranslator(), +) +def my_dbt_assets(context, dbt: DbtCliResource): + yield from dbt.cli(["build"], context=context).stream() +``` + +The `get_asset_spec()` method is called once per dbt node and should return a complete `AssetSpec` +describing that node. + +### dbt Meta Config + +Define custom metadata in your dbt project files that can be consumed by your custom +`get_asset_spec()` implementation: + +```yaml +models: + - name: customers + meta: + custom_field: custom_value +``` + +Access this metadata via the manifest in your translator's `get_asset_spec()` method. + +## Incremental Models and Partitioning + +To partition incremental dbt models: + +1. Define a `PartitionsDefinition` +2. Pass it to the `@dbt_assets` decorator +3. Access `context.partition_time_window` in the asset function +4. Pass partition-specific vars to the dbt CLI + +```python nocheck +import json + +import dagster as dg +from dagster_dbt import dbt_assets + +daily_partitions = dg.DailyPartitionsDefinition(start_date="2023-01-01") + + +@dbt_assets(manifest=my_dbt_project.manifest_path, partitions_def=daily_partitions) +def my_dbt_assets(context: dg.AssetExecutionContext, dbt: DbtCliResource): + time_window = context.partition_time_window + + dbt_vars = { + "min_date": time_window.start.strftime("%Y-%m-%d"), + "max_date": time_window.end.strftime("%Y-%m-%d"), + } + + yield from dbt.cli( + ["build", "--vars", json.dumps(dbt_vars)], context=context + ).stream() +``` + +Reference these vars in your dbt SQL: + +```sql +{{ config(materialized='incremental', unique_key='order_date') }} + +select * from {{ ref('my_model') }} + +{% if is_incremental() %} +where order_date >= '{{ var('min_date') }}' and order_date <= '{{ var('max_date') }}' +{% endif %} +``` + +For multiple partitions definitions, create separate `@dbt_assets` definitions and use +`select`/`exclude` to filter models for each. + +## Metadata + +Enable automatic metadata fetching by chaining methods on the event iterator: + +```python nocheck +@dbt_assets(manifest=my_dbt_project.manifest_path) +def my_dbt_assets(context, dbt: DbtCliResource): + yield from ( + dbt.cli(["build"], context=context) + .stream() + .fetch_row_counts() + .fetch_column_metadata() + ) +``` + +- `fetch_row_counts()`: Fetch row counts for tables +- `fetch_column_metadata()`: Fetch column schema and lineage (extracted via sqlglot parsing) + +Both metadata types are fetched in parallel during dbt execution. + +## Asset Checks + +See [Asset Checks](asset-checks.md) for details on how dbt tests are loaded as Dagster asset checks. + +## Dependencies + +See [Dependencies](dependencies.md) for details on how Dagster parses dbt project dependencies and +patterns for defining additional dependencies. + +### Referencing dbt Models in Other Assets + +Use `get_asset_key_for_model()` to get asset keys for dbt models: + +```python nocheck +import dagster as dg +from dagster_dbt import get_asset_key_for_model + + +@dg.asset(deps=[get_asset_key_for_model([my_dbt_assets], "customers")]) +def export_customers(): + # Use the customers model + pass +``` + +## Scheduling + +### dbt-Only Jobs + +Use `build_schedule_from_dbt_selection()` for jobs that only materialize dbt assets: + +```python nocheck +from dagster_dbt import build_schedule_from_dbt_selection + +daily_dbt_schedule = build_schedule_from_dbt_selection( + [my_dbt_assets], + job_name="daily_dbt_models", + cron_schedule="0 0 * * *", + dbt_select="tag:daily", +) +``` + +### Mixed Jobs (dbt + Non-dbt Assets) + +Use `build_dbt_asset_selection()` combined with `AssetSelection` for jobs with dbt and non-dbt +assets: + +```python nocheck +import dagster as dg +from dagster_dbt import build_dbt_asset_selection + +# Select dbt models and all downstream assets +dbt_models = build_dbt_asset_selection([my_dbt_assets], dbt_select="tag:daily") +all_assets = dbt_models.downstream(include_self=True) + +daily_job = dg.define_asset_job(name="daily_pipeline", selection=all_assets) +daily_schedule = dg.ScheduleDefinition(job=daily_job, cron_schedule="0 0 * * *") +``` + +### Declarative Automation + +Configure AutomationConditions via custom `get_asset_spec()` implementation in your translator: + +```python nocheck +class AutomatedDbtTranslator(DagsterDbtTranslator): + def get_asset_spec(self, manifest, unique_id, project) -> dg.AssetSpec: + base_spec = super().get_asset_spec(manifest, unique_id, project) + return base_spec.replace_attributes( + automation_condition=dg.AutomationCondition.eager() + ) +``` + +## Runtime Configuration + +Use Dagster's config system to provide runtime parameters: + +```python nocheck +import dagster as dg +from dagster_dbt import dbt_assets + + +class DbtConfig(dg.Config): + full_refresh: bool = False + + +@dbt_assets(manifest=my_dbt_project.manifest_path) +def my_dbt_assets(context, dbt: DbtCliResource, config: DbtConfig): + args = ["build"] + if config.full_refresh: + args.append("--full-refresh") + + yield from dbt.cli(args, context=context).stream() +``` + +## dbt Cloud + +For dbt Cloud projects, see [dbt Cloud Integration](dbt-cloud.md). diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-dbt/scaffolding.md b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/scaffolding.md new file mode 100644 index 0000000..e31f20e --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-dbt/scaffolding.md @@ -0,0 +1,76 @@ +--- +title: "dbt: Scaffolding" +triggers: + - "scaffolding a new dbt component in a Dagster project" +--- + +# Scaffolding a dbt Component + +This guide covers the end-to-end process of creating a new `DbtProjectComponent` in a Dagster project. +For configuration and customization of an existing component, see +[Component-Based Integration](component-based-integration.md). + +## Prerequisites + +Ensure `dagster-dbt` is installed in the project: + +```bash +uv add dagster-dbt +``` + +## Scaffold the Component + +### Colocated dbt Project + +If the dbt project lives inside the Dagster repository: + +```bash +dg scaffold defs dagster_dbt.DbtProjectComponent --project-path +``` + +### Remote Git Repository + +If the dbt project lives in a separate git repository: + +```bash +dg scaffold defs dagster_dbt.DbtProjectComponent --git-url --project-path +``` + +## Install the Adapter Library (Required) + +dbt requires a database-specific adapter library to compile the project manifest. **Without it, +`dbt parse` will fail and no assets will be loaded.** + +1. Check the `profiles.yml` file in the dbt project for the adapter type (look for the `type:` field + under the target configuration). +2. If the adapter type is not clear from `profiles.yml`, ask the user which database they are + targeting. +3. Install the adapter: + +```bash +uv add dbt- +``` + +Common adapters: + +| Adapter | Package | +| ---------- | --------------- | +| DuckDB | `dbt-duckdb` | +| Snowflake | `dbt-snowflake` | +| BigQuery | `dbt-bigquery` | +| PostgreSQL | `dbt-postgres` | +| Redshift | `dbt-redshift` | + +## Verify + +Always run `dg list defs` to confirm the manifest compiled and assets are visible: + +```bash +dg list defs +``` + +If this returns no definitions, check that: + +- The adapter library is installed +- The dbt project path is correct +- `profiles.yml` is properly configured diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-deltalake/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-deltalake/INDEX.md new file mode 100644 index 0000000..0f40529 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-deltalake/INDEX.md @@ -0,0 +1,11 @@ +--- +title: dagster-deltalake +triggers: + - "lakehouse storage with Delta Lake" +--- + +# dagster-deltalake + +> **Community-supported integration.** + +Docs: https://docs.dagster.io/integrations/libraries/deltalake diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-docker/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-docker/INDEX.md new file mode 100644 index 0000000..c9c29a4 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-docker/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-docker +triggers: + - "containerized execution with Docker" +--- + +# dagster-docker + +Docs: https://docs.dagster.io/integrations/libraries/docker diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-duckdb/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-duckdb/INDEX.md new file mode 100644 index 0000000..f997759 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-duckdb/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-duckdb +triggers: + - "in-process analytical queries with DuckDB" +--- + +# dagster-duckdb + +Docs: https://docs.dagster.io/integrations/libraries/duckdb diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-fivetran/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-fivetran/INDEX.md new file mode 100644 index 0000000..b4d9f23 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-fivetran/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-fivetran +triggers: + - "managed extract-load connectors with Fivetran" +--- + +# dagster-fivetran + +Docs: https://docs.dagster.io/integrations/libraries/fivetran diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-gcp/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-gcp/INDEX.md new file mode 100644 index 0000000..b724fe3 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-gcp/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-gcp +triggers: + - "Google Cloud Platform (BigQuery, GCS) from Dagster" +--- + +# dagster-gcp + +Docs: https://docs.dagster.io/integrations/libraries/gcp diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-github/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-github/INDEX.md new file mode 100644 index 0000000..0ca7f56 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-github/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-github +triggers: + - "GitHub repository event handling from Dagster" +--- + +# dagster-github + +Docs: https://docs.dagster.io/integrations/libraries/github diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-great-expectations/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-great-expectations/INDEX.md new file mode 100644 index 0000000..fe0ad3b --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-great-expectations/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-great-expectations +triggers: + - "data validation and testing with Great Expectations" +--- + +# dagster-great-expectations + +Docs: https://docs.dagster.io/integrations/libraries/great-expectations diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-hightouch/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-hightouch/INDEX.md new file mode 100644 index 0000000..29fa9c0 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-hightouch/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-hightouch +triggers: + - "reverse ETL and data activation with Hightouch" +--- + +# dagster-hightouch + +Docs: https://docs.dagster.io/integrations/libraries/hightouch diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-iceberg/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-iceberg/INDEX.md new file mode 100644 index 0000000..a124ed6 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-iceberg/INDEX.md @@ -0,0 +1,11 @@ +--- +title: dagster-iceberg +triggers: + - "Apache Iceberg table format management" +--- + +# dagster-iceberg + +> **Community-supported integration.** + +Docs: https://docs.dagster.io/integrations/libraries/iceberg diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-jupyter/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-jupyter/INDEX.md new file mode 100644 index 0000000..7203c4a --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-jupyter/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-jupyter +triggers: + - "notebook-based assets with Jupyter" +--- + +# dagster-jupyter + +Docs: https://docs.dagster.io/integrations/libraries/jupyter diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-k8s/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-k8s/INDEX.md new file mode 100644 index 0000000..99f9187 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-k8s/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-k8s +triggers: + - "Kubernetes container orchestration and execution" +--- + +# dagster-k8s + +Docs: https://docs.dagster.io/integrations/libraries/k8s diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-looker/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-looker/INDEX.md new file mode 100644 index 0000000..1b7a0bf --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-looker/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-looker +triggers: + - "Looker BI dashboard assets" +--- + +# dagster-looker + +Docs: https://docs.dagster.io/integrations/libraries/looker diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-mlflow/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-mlflow/INDEX.md new file mode 100644 index 0000000..77c66f8 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-mlflow/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-mlflow +triggers: + - "ML experiment tracking and model management with MLflow" +--- + +# dagster-mlflow + +Docs: https://docs.dagster.io/integrations/libraries/mlflow diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-msteams/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-msteams/INDEX.md new file mode 100644 index 0000000..2c6e045 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-msteams/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-msteams +triggers: + - "Microsoft Teams notifications and alerts from Dagster" +--- + +# dagster-msteams + +Docs: https://docs.dagster.io/integrations/libraries/msteams diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-mysql/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-mysql/INDEX.md new file mode 100644 index 0000000..0fff826 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-mysql/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-mysql +triggers: + - "MySQL as a Dagster storage backend" +--- + +# dagster-mysql + +Docs: https://docs.dagster.io/integrations/libraries/mysql diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-omni/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-omni/INDEX.md new file mode 100644 index 0000000..6422653 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-omni/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-omni +triggers: + - "analytics and BI with Omni" +--- + +# dagster-omni + +Docs: https://docs.dagster.io/integrations/libraries/omni diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-openai/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-openai/INDEX.md new file mode 100644 index 0000000..a0e43e9 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-openai/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-openai +triggers: + - "LLM-powered assets with OpenAI" +--- + +# dagster-openai + +Docs: https://docs.dagster.io/integrations/libraries/openai diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-pagerduty/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-pagerduty/INDEX.md new file mode 100644 index 0000000..16b64bb --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-pagerduty/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-pagerduty +triggers: + - "incident management alerts with PagerDuty" +--- + +# dagster-pagerduty + +Docs: https://docs.dagster.io/integrations/libraries/pagerduty diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-pandas/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-pandas/INDEX.md new file mode 100644 index 0000000..3cd8b7f --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-pandas/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-pandas +triggers: + - "Pandas DataFrame type checking and validation" +--- + +# dagster-pandas + +Docs: https://docs.dagster.io/integrations/libraries/pandas diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-pandera/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-pandera/INDEX.md new file mode 100644 index 0000000..85211f9 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-pandera/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-pandera +triggers: + - "DataFrame schema validation with Pandera" +--- + +# dagster-pandera + +Docs: https://docs.dagster.io/integrations/libraries/pandera diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-papertrail/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-papertrail/INDEX.md new file mode 100644 index 0000000..159eb64 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-papertrail/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-papertrail +triggers: + - "log management with Papertrail" +--- + +# dagster-papertrail + +Docs: https://docs.dagster.io/integrations/libraries/papertrail diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-polars/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-polars/INDEX.md new file mode 100644 index 0000000..2c5e8f6 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-polars/INDEX.md @@ -0,0 +1,11 @@ +--- +title: dagster-polars +triggers: + - "fast DataFrame processing with Polars" +--- + +# dagster-polars + +> **Community-supported integration.** + +Docs: https://docs.dagster.io/integrations/libraries/polars diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-postgres/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-postgres/INDEX.md new file mode 100644 index 0000000..b25113f --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-postgres/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-postgres +triggers: + - "PostgreSQL as a Dagster storage backend" +--- + +# dagster-postgres + +Docs: https://docs.dagster.io/integrations/libraries/postgres diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-powerbi/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-powerbi/INDEX.md new file mode 100644 index 0000000..334f033 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-powerbi/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-powerbi +triggers: + - "Power BI dashboard assets" +--- + +# dagster-powerbi + +Docs: https://docs.dagster.io/integrations/libraries/powerbi diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-prometheus/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-prometheus/INDEX.md new file mode 100644 index 0000000..4103712 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-prometheus/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-prometheus +triggers: + - "metrics collection with Prometheus" +--- + +# dagster-prometheus + +Docs: https://docs.dagster.io/integrations/libraries/prometheus diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-pyspark/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-pyspark/INDEX.md new file mode 100644 index 0000000..5dc226b --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-pyspark/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-pyspark +triggers: + - "distributed data processing with PySpark" +--- + +# dagster-pyspark + +Docs: https://docs.dagster.io/integrations/libraries/pyspark diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-sigma/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-sigma/INDEX.md new file mode 100644 index 0000000..d3b4381 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-sigma/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-sigma +triggers: + - "BI and analytics assets with Sigma" +--- + +# dagster-sigma + +Docs: https://docs.dagster.io/integrations/libraries/sigma diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-slack/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-slack/INDEX.md new file mode 100644 index 0000000..8dde693 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-slack/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-slack +triggers: + - "Slack notifications or alerts from Dagster" +--- + +# dagster-slack + +Docs: https://docs.dagster.io/integrations/libraries/slack diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-sling/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-sling/INDEX.md new file mode 100644 index 0000000..9526b11 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-sling/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-sling +triggers: + - "EL data replication with Sling" +--- + +# dagster-sling + +Docs: https://docs.dagster.io/integrations/libraries/sling diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-snowflake/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-snowflake/INDEX.md new file mode 100644 index 0000000..8a9277f --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-snowflake/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-snowflake +triggers: + - "interacting with Snowflake from Dagster" +--- + +# dagster-snowflake + +Docs: https://docs.dagster.io/integrations/libraries/snowflake diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-spark/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-spark/INDEX.md new file mode 100644 index 0000000..812b4b7 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-spark/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-spark +triggers: + - "distributed data processing with Apache Spark" +--- + +# dagster-spark + +Docs: https://docs.dagster.io/integrations/libraries/spark diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-ssh/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-ssh/INDEX.md new file mode 100644 index 0000000..1c7b1b1 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-ssh/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-ssh +triggers: + - "remote command execution via SSH" +--- + +# dagster-ssh + +Docs: https://docs.dagster.io/integrations/libraries/ssh diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-tableau/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-tableau/INDEX.md new file mode 100644 index 0000000..51efe82 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-tableau/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-tableau +triggers: + - "Tableau BI dashboard assets" +--- + +# dagster-tableau + +Docs: https://docs.dagster.io/integrations/libraries/tableau diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-twilio/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-twilio/INDEX.md new file mode 100644 index 0000000..c4a0aa2 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-twilio/INDEX.md @@ -0,0 +1,9 @@ +--- +title: dagster-twilio +triggers: + - "SMS and communication with Twilio" +--- + +# dagster-twilio + +Docs: https://docs.dagster.io/integrations/libraries/twilio diff --git a/.agents/skills/dagster-expert/references/integrations/dagster-wandb/INDEX.md b/.agents/skills/dagster-expert/references/integrations/dagster-wandb/INDEX.md new file mode 100644 index 0000000..99afbf9 --- /dev/null +++ b/.agents/skills/dagster-expert/references/integrations/dagster-wandb/INDEX.md @@ -0,0 +1,11 @@ +--- +title: dagster-wandb +triggers: + - "ML experiment tracking with Weights & Biases" +--- + +# dagster-wandb + +> **Community-supported integration.** + +Docs: https://docs.dagster.io/integrations/libraries/wandb diff --git a/.agents/skills/dagster-init/SKILL.md b/.agents/skills/dagster-init/SKILL.md new file mode 100644 index 0000000..a41dc42 --- /dev/null +++ b/.agents/skills/dagster-init/SKILL.md @@ -0,0 +1,172 @@ +--- +name: dagster-init +description: Initialize a dagster project using the create-dagster cli. Create a dagster project, uv virtual environment, and everything needed for a user to run dg dev or dg check defs successfully. (project) +license: MIT +--- + +# Dagster Project Initialization + +## Overview + +This skill automates the creation of a new Dagster project using the `create-dagster` CLI tool with uv as the package manager. It creates a clean, Components-compatible project structure ready for local development. + +## What This Skill Does + +When invoked, this skill will: + +1. ✅ Create a new Dagster project using `create-dagster@latest` +2. ✅ Set up a uv virtual environment with all dependencies +3. ✅ Initialize project structure with Components architecture +4. ✅ Ensure the project is ready to run `dg dev` or `dg check defs` +5. ✅ Provide clear next steps for development + +## Prerequisites + +Before running this skill, ensure: +- `uv` is installed (check with `uv --version`) +- You have a project name in mind (or will use the default) +- You're in the directory where you want to create the project + +## Skill Workflow + +### Step 1: Validate Environment + +Check that uv is available: +```bash +uv --version +``` + +If uv is not installed, provide installation instructions: +```bash +curl -LsSf https://astral.sh/uv/install.sh | sh +``` + +### Step 2: Get Project Name + +Ask the user for a project name, or use a sensible default like `my-dagster-project`. Validate that: +- The name starts with a letter +- Contains only alphanumeric characters, hyphens, or underscores +- The directory doesn't already exist (or ask to overwrite) + +### Step 3: Create Project with create-dagster + +Use `uvx` to run the latest create-dagster CLI. The CLI requires interactive confirmation, so we pass "y" automatically using `printf`: + +```bash +printf "y\n" | uvx create-dagster@latest project +``` + +**Important:** The `printf "y\n"` automatically answers "yes" to the interactive prompt that asks for confirmation to proceed with project creation. + +This will: +- Scaffold a new Dagster project with Components structure +- Create `pyproject.toml` with project metadata +- Set up package structure with `definitions.py` +- Create `definitions/defs/` directory for components + +### Step 4: Install Dependencies + +Navigate into the project directory and run uv sync: + +```bash +cd +uv sync +``` + +This creates the virtual environment and installs all dependencies specified in `pyproject.toml`. + +### Step 5: Verify Installation + +Check that the project is properly set up by running: + +```bash +uv run dg check defs +``` + +This validates that: +- All dependencies are installed correctly +- The Dagster definitions are loadable +- The project structure is correct + +### Step 6: Display Success Message + +Provide the user with a clear summary and next steps: + +``` +✅ Successfully created Dagster project: + +📁 Project structure: + • pyproject.toml - Project configuration + • /definitions.py - Main definitions module + • /definitions/defs/ - Components directory + +🚀 Next steps: + 1. cd + 2. uv run dg dev # Start local development server + 3. Open http://localhost:3000 to view Dagster UI + +💡 Additional commands: + • uv run dg check defs # Validate definitions + • uv run pytest # Run tests (if configured) + • uv add # Add new dependencies +``` + +## Error Handling + +Handle common issues gracefully: + +1. **uv not installed**: Provide installation instructions +2. **Directory already exists**: Ask user to choose different name or overwrite +3. **create-dagster fails**: Show error details and suggest troubleshooting + - Note: The CLI requires interactive confirmation - we automatically pass "y" via `printf "y\n"` to avoid hanging +4. **Dependency installation fails**: Check network, suggest clearing cache +5. **dg check defs fails**: Show validation errors and help debug + +## Alternative: Using the Python Script + +You can also invoke the provided Python script directly: + +```bash +python .agents/skills/dagster-init/scripts/create-dagster.py +``` + +This provides an interactive workflow with the same functionality. The script automatically handles the interactive prompt by passing "y" to stdin, so it won't hang waiting for user input. + +## Project Structure + +After successful creation, the project will have: + +``` +/ +├── pyproject.toml # Project metadata and dependencies +├── / +│ ├── __init__.py +│ ├── definitions.py # Main Dagster definitions +│ └── definitions/ +│ └── defs/ # Components directory +│ ├── __init__.py +│ └── ... # Your components go here +├── _tests/ # Test directory +├── .venv/ # uv virtual environment +└── uv.lock # Locked dependencies +``` + +## Tips for Success + +- Use descriptive project names that reflect the purpose +- Run `dg check defs` regularly during development to catch issues early +- Keep dependencies minimal initially, add as needed +- Follow the Components pattern for scalable project organization +- Use `uv add` to add new dependencies (it updates pyproject.toml automatically) + +## Related Skills + +- **dg-plus-init**: For setting up Dagster+ Cloud deployments +- Use after creating a project with this skill to deploy to the cloud + +## Resources + +- [Dagster Documentation](https://docs.dagster.io/) +- [Components Guide](https://docs.dagster.io/guides/build/projects/moving-to-components) +- [uv Documentation](https://docs.astral.sh/uv/) +- [create-dagster CLI](https://github.com/dagster-io/dagster/tree/master/python_modules/dagster/dagster/_cli/create_dagster) diff --git a/.agents/skills/dagster-init/scripts/create-dagster.py b/.agents/skills/dagster-init/scripts/create-dagster.py new file mode 100755 index 0000000..d974eb8 --- /dev/null +++ b/.agents/skills/dagster-init/scripts/create-dagster.py @@ -0,0 +1,293 @@ +#!/usr/bin/env python3 +""" +Dagster Project Initialization Script + +Creates a new Dagster project using create-dagster CLI with uv package manager. +Sets up a complete development environment ready for `dg dev` or `dg check defs`. + +Usage: + python create-dagster.py [project-name] +""" + +import os +import sys +import subprocess +import shutil +import re +import argparse + + +def check_uv_installed(): + """Check if uv is installed and available.""" + try: + result = subprocess.run(["uv", "--version"], capture_output=True, text=True) + if result.returncode == 0: + print(f" uv is installed: {result.stdout.strip()}") + return True + return False + except FileNotFoundError: + return False + + +def install_uv_instructions(): + """Provide instructions for installing uv.""" + print("\nL uv is not installed.") + print("\n=� To install uv, run:") + print(" curl -LsSf https://astral.sh/uv/install.sh | sh") + print("\nOr visit: https://docs.astral.sh/uv/") + return False + + +def validate_project_name(name): + """Validate that the project name follows Python package naming conventions.""" + if not name: + return False + + # Must start with letter, contain only alphanumeric, hyphens, underscores + if not re.match(r"^[a-zA-Z][a-zA-Z0-9_-]*$", name): + return False + + return True + + +def get_project_name(provided_name=None): + """Get and validate project name from user or argument.""" + if provided_name: + if validate_project_name(provided_name): + return provided_name + else: + print(f"� Invalid project name: {provided_name}") + print( + "Project name must start with a letter and contain only letters, numbers, hyphens, and underscores." + ) + + # Interactive prompt + while True: + name = input( + "\nEnter project name (or press Enter for 'my-dagster-project'): " + ).strip() + + if not name: + return "my-dagster-project" + + if validate_project_name(name): + return name + else: + print("� Invalid project name.") + print( + "Project name must start with a letter and contain only letters, numbers, hyphens, and underscores." + ) + + +def check_directory_exists(project_name): + """Check if directory exists and handle accordingly.""" + if not os.path.exists(project_name): + return True + + print(f"\n� Directory '{project_name}' already exists.") + while True: + choice = input( + "Choose an option:\n 1. Use a different name\n 2. Remove and recreate\n 3. Cancel\nEnter choice (1-3): " + ).strip() + + if choice == "1": + return False # Will prompt for new name + elif choice == "2": + try: + shutil.rmtree(project_name) + print(f"=� Removed existing directory: {project_name}") + return True + except Exception as e: + print(f"L Failed to remove directory: {e}") + return False + elif choice == "3": + print("L Operation cancelled") + sys.exit(0) + else: + print("� Invalid choice. Please enter 1, 2, or 3.") + + +def create_project_with_uvx(project_name): + """Create Dagster project using uvx and create-dagster CLI.""" + print(f"\n=� Creating Dagster project: {project_name}") + print("=' Using create-dagster@latest via uvx...") + + try: + # Pass "y\n" to stdin to automatically answer the interactive prompt + result = subprocess.run( + ["uvx", "create-dagster@latest", "project", project_name], + capture_output=True, + text=True, + input="y\n", # Automatically answer "yes" to the interactive prompt + ) + + if result.returncode != 0: + print("L Failed to create project") + print(f"Error output: {result.stderr}") + return False + + # Show any output from create-dagster + if result.stdout: + print(result.stdout) + + print(f" Project scaffolded successfully: {project_name}/") + return True + + except FileNotFoundError: + print("L uvx command not found. Please ensure uv is installed correctly.") + return False + except Exception as e: + print(f"L Error creating project: {e}") + return False + + +def install_dependencies(project_name): + """Install project dependencies using uv sync.""" + print("\n=� Installing dependencies with uv sync...") + + project_dir = os.path.abspath(project_name) + + try: + result = subprocess.run( + ["uv", "sync"], cwd=project_dir, capture_output=True, text=True + ) + + if result.returncode == 0: + print(" Dependencies installed successfully") + + # Show the virtual environment location + venv_path = os.path.join(project_dir, ".venv") + if os.path.exists(venv_path): + print(f"=� Virtual environment created at: {venv_path}") + + return True + else: + print("� uv sync had issues:") + print(result.stderr) + return False + + except Exception as e: + print(f"L Error installing dependencies: {e}") + return False + + +def verify_installation(project_name): + """Verify the project is correctly set up by running dg check defs.""" + print("\n==> Verifying project setup...") + + project_dir = os.path.abspath(project_name) + + try: + result = subprocess.run( + ["uv", "run", "dg", "check", "defs"], + cwd=project_dir, + capture_output=True, + text=True, + timeout=30, + ) + + if result.returncode == 0: + print(" Project verification successful!") + print(" All definitions loaded correctly.") + return True + else: + print("� Project verification had warnings:") + if result.stdout: + print(result.stdout) + if result.stderr: + print(result.stderr) + return True # Still consider it success, might just be warnings + + except subprocess.TimeoutExpired: + print("� Verification timed out, but project is likely OK") + return True + except Exception as e: + print(f"� Could not verify installation: {e}") + print(" You can manually verify with: uv run dg check defs") + return True + + +def display_success_message(project_name): + """Display success message and next steps.""" + project_dir = os.path.abspath(project_name) + + print("\n" + "=" * 60) + print(f"<� Successfully created Dagster project: {project_name}") + print("=" * 60) + + print("\n==> Project structure:") + print(" - pyproject.toml - Project configuration") + print(f" - {project_name}/definitions.py - Main definitions module") + print(f" - {project_name}/definitions/defs/ - Components directory") + print(" - .venv/ - Virtual environment") + + print("\n==> Next steps:") + print(f" 1. cd {project_name}") + print(" 2. uv run dg dev") + print(" 3. Open http://localhost:3000 in your browser") + + print("\n==> Useful commands:") + print(" - uv run dg check defs - Validate your definitions") + print(" - uv run pytest - Run tests") + print(" - uv add - Add new dependencies") + print(" - uv run python - Run Python in the venv") + + print(f"\n=� Project location: {project_dir}") + print() + + +def main(): + """Main function for Dagster project initialization.""" + parser = argparse.ArgumentParser(description="Create a new Dagster project with uv") + parser.add_argument( + "project_name", nargs="?", help="Name of the Dagster project to create" + ) + + args = parser.parse_args() + + print("=� Dagster Project Initialization") + print("=" * 60) + + # Step 1: Check uv installation + if not check_uv_installed(): + install_uv_instructions() + sys.exit(1) + + # Step 2: Get and validate project name + project_name = None + while not project_name: + candidate_name = get_project_name(args.project_name) + if check_directory_exists(candidate_name): + project_name = candidate_name + else: + args.project_name = None # Clear arg so we prompt again + + # Step 3: Create project with create-dagster + if not create_project_with_uvx(project_name): + print("\nL Project creation failed") + sys.exit(1) + + # Step 4: Install dependencies + if not install_dependencies(project_name): + print("\n� Dependencies installation had issues, but project was created") + print(" You can try running 'uv sync' manually in the project directory") + + # Step 5: Verify installation + verify_installation(project_name) + + # Step 6: Display success message + display_success_message(project_name) + + +if __name__ == "__main__": + try: + main() + except KeyboardInterrupt: + print("\n\nL Operation cancelled by user") + sys.exit(0) + except Exception as e: + print(f"\nL Unexpected error: {e}") + import traceback + + traceback.print_exc() + sys.exit(1) diff --git a/.agents/skills/dbt-development/dbt_examples.md b/.agents/skills/dbt-development/dbt_examples.md new file mode 100644 index 0000000..152e3f8 --- /dev/null +++ b/.agents/skills/dbt-development/dbt_examples.md @@ -0,0 +1,919 @@ +# dbt Code Examples & Patterns + +This document provides comprehensive code examples for common dbt patterns and use cases. + +## Table of Contents +1. [Staging Models](#staging-models) +2. [Intermediate Models](#intermediate-models) +3. [Mart Models](#mart-models) +4. [Incremental Models](#incremental-models) +5. [Snapshots](#snapshots) +6. [Macros](#macros) +7. [Tests](#tests) +8. [Advanced Patterns](#advanced-patterns) + +--- + +## Staging Models + +### Basic Staging Model with Type Casting + +```sql +-- models/staging/salesforce/stg_salesforce__accounts.sql +with source as ( + select * from {{ source('salesforce', 'accounts') }} +), + +renamed as ( + select + -- Primary Key + id as account_id, + + -- Foreign Keys + owner_id, + parent_account_id, + + -- Timestamps - standardize to UTC with consistent naming + cast(created_date as timestamp) as account_created_at, + cast(last_modified_date as timestamp) as account_updated_at, + + -- Strings - clean and standardize + lower(trim(name)) as account_name, + lower(trim(type)) as account_type, + lower(trim(industry)) as industry, + + -- Numerics - proper casting + cast(annual_revenue as decimal(15,2)) as annual_revenue, + cast(number_of_employees as integer) as employee_count, + + -- Booleans - standardize + case + when is_deleted = 'true' then true + when is_deleted = 'false' then false + else null + end as is_deleted, + + -- JSON flattening (if needed) + json_extract_scalar(custom_fields, '$.region') as region, + json_extract_scalar(custom_fields, '$.segment') as segment + + from source + where not coalesce(is_deleted, false) -- Remove deleted records +) + +select * from renamed +``` + +### Staging with Union (Multiple Sources) + +```sql +-- models/staging/ecommerce/stg_ecommerce__orders.sql +with shopify_orders as ( + select + id as order_id, + 'shopify' as source_system, + customer_id, + created_at as order_created_at, + total_price as order_amount, + status as order_status + from {{ source('shopify', 'orders') }} +), + +amazon_orders as ( + select + order_id, + 'amazon' as source_system, + customer_id, + order_date as order_created_at, + total as order_amount, + order_status + from {{ source('amazon', 'orders') }} +), + +combined as ( + select * from shopify_orders + union all + select * from amazon_orders +) + +select * from combined +``` + +### Deduplication in Staging + +```sql +-- models/staging/mixpanel/stg_mixpanel__events.sql +with source as ( + select * from {{ source('mixpanel', 'events') }} +), + +deduplicated as ( + select + event_id, + user_id, + event_name, + event_timestamp, + event_properties, + row_number() over ( + partition by event_id + order by _loaded_at desc + ) as row_num + from source +), + +final as ( + select + event_id, + user_id, + event_name, + event_timestamp, + event_properties + from deduplicated + where row_num = 1 +) + +select * from final +``` + +--- + +## Intermediate Models + +### Grain Change with Aggregation + +```sql +-- models/intermediate/product/int_order_items_daily.sql +with order_items as ( + select * from {{ ref('stg_shopify__order_items') }} +), + +daily_aggregates as ( + select + date(order_created_at) as order_date, + product_id, + product_category, + + -- Aggregations + count(distinct order_id) as order_count, + count(*) as item_count, + sum(quantity) as total_quantity, + sum(item_revenue) as total_revenue, + + -- Statistical measures + avg(item_revenue) as avg_item_revenue, + min(item_revenue) as min_item_revenue, + max(item_revenue) as max_item_revenue, + stddev(item_revenue) as stddev_item_revenue + + from order_items + group by 1, 2, 3 +) + +select * from daily_aggregates +``` + +### Complex Business Logic + +```sql +-- models/intermediate/finance/int_customer_segments.sql +with customers as ( + select * from {{ ref('stg_customers') }} +), + +orders as ( + select * from {{ ref('fct_orders') }} +), + +customer_metrics as ( + select + customers.customer_id, + customers.customer_created_at, + customers.customer_email, + + -- Order metrics + count(orders.order_id) as lifetime_orders, + sum(orders.order_amount) as lifetime_revenue, + avg(orders.order_amount) as avg_order_value, + max(orders.order_created_at) as last_order_date, + min(orders.order_created_at) as first_order_date, + + -- Recency calculation + datediff('day', max(orders.order_created_at), current_date()) as days_since_last_order + + from customers + left join orders on customers.customer_id = orders.customer_id + group by 1, 2, 3 +), + +segmented as ( + select + *, + + -- RFM Segmentation + case + when lifetime_orders >= 10 and lifetime_revenue >= 1000 then 'VIP' + when lifetime_orders >= 5 and lifetime_revenue >= 500 then 'High Value' + when lifetime_orders >= 2 and lifetime_revenue >= 100 then 'Regular' + when lifetime_orders = 1 then 'One-Time' + else 'Prospect' + end as customer_segment, + + -- Churn risk + case + when days_since_last_order > 180 then 'High Risk' + when days_since_last_order > 90 then 'Medium Risk' + when days_since_last_order > 30 then 'Low Risk' + else 'Active' + end as churn_risk_category + + from customer_metrics +) + +select * from segmented +``` + +### Fan-out then Fan-in Pattern + +```sql +-- models/intermediate/finance/int_order_items_with_costs.sql +with order_items as ( + select * from {{ ref('stg_order_items') }} +), + +products as ( + select * from {{ ref('stg_products') }} +), + +-- Fan out to item level, join product costs +items_with_costs as ( + select + order_items.order_id, + order_items.item_id, + order_items.product_id, + order_items.quantity, + order_items.item_price, + + -- Cost information from products + products.unit_cost, + + -- Calculated fields + order_items.quantity * order_items.item_price as item_revenue, + order_items.quantity * products.unit_cost as item_cost, + (order_items.quantity * order_items.item_price) - + (order_items.quantity * products.unit_cost) as item_profit + + from order_items + left join products on order_items.product_id = products.product_id +) + +select * from items_with_costs +``` + +--- + +## Mart Models + +### Dimension Table (SCD Type 2) + +```sql +-- models/marts/core/dim_customers.sql +{{ config( + materialized='table', + tags=['core', 'daily'] +) }} + +with customers as ( + select * from {{ ref('stg_customers') }} +), + +customer_segments as ( + select * from {{ ref('int_customer_segments') }} +), + +final as ( + select + -- Surrogate key + {{ dbt_utils.generate_surrogate_key(['customers.customer_id']) }} as customer_key, + + -- Natural key + customers.customer_id, + + -- Attributes + customers.customer_name, + customers.customer_email, + customers.customer_created_at, + + -- Enriched attributes from intermediate model + customer_segments.customer_segment, + customer_segments.lifetime_orders, + customer_segments.lifetime_revenue, + customer_segments.avg_order_value, + customer_segments.churn_risk_category, + + -- SCD Type 2 fields + customers.customer_created_at as valid_from, + null as valid_to, + true as is_current + + from customers + left join customer_segments + on customers.customer_id = customer_segments.customer_id +) + +select * from final +``` + +### Fact Table with Multiple Grain Levels + +```sql +-- models/marts/core/fct_order_line_items.sql +{{ config( + materialized='table', + tags=['core', 'daily'] +) }} + +with order_items as ( + select * from {{ ref('int_order_items_with_costs') }} +), + +orders as ( + select * from {{ ref('stg_orders') }} +), + +customers as ( + select * from {{ ref('dim_customers') }} +), + +products as ( + select * from {{ ref('dim_products') }} +), + +final as ( + select + -- Grain: One row per order line item + + -- Fact table key + {{ dbt_utils.generate_surrogate_key([ + 'order_items.order_id', + 'order_items.item_id' + ]) }} as order_line_item_key, + + -- Foreign keys to dimensions + customers.customer_key, + products.product_key, + + -- Degenerate dimensions + order_items.order_id, + order_items.item_id, + orders.order_status, + + -- Time dimensions + date(orders.order_created_at) as order_date, + orders.order_created_at as order_timestamp, + + -- Measures + order_items.quantity, + order_items.item_price, + order_items.item_revenue, + order_items.item_cost, + order_items.item_profit, + + -- Derived measures + order_items.item_revenue / nullif(order_items.item_cost, 0) as profit_margin + + from order_items + inner join orders on order_items.order_id = orders.order_id + left join customers on orders.customer_id = customers.customer_id + left join products on order_items.product_id = products.product_id +) + +select * from final +``` + +--- + +## Incremental Models + +### Basic Incremental Pattern + +```sql +-- models/marts/core/fct_events.sql +{{ config( + materialized='incremental', + unique_key='event_id', + on_schema_change='append_new_columns' +) }} + +with events as ( + select * from {{ ref('stg_mixpanel__events') }} + + {% if is_incremental() %} + -- Only process new events since the last run + where event_timestamp > (select max(event_timestamp) from {{ this }}) + {% endif %} +), + +enriched as ( + select + event_id, + user_id, + event_name, + event_timestamp, + event_properties, + + -- Add processing metadata + current_timestamp() as dbt_loaded_at + + from events +) + +select * from enriched +``` + +### Incremental with Delete/Update Pattern + +```sql +-- models/marts/core/fct_orders_incremental.sql +{{ config( + materialized='incremental', + unique_key='order_id', + on_schema_change='fail' +) }} + +with new_and_updated_orders as ( + select * from {{ ref('stg_orders') }} + + {% if is_incremental() %} + -- Get orders created or updated since last run + where order_updated_at > (select max(order_updated_at) from {{ this }}) + {% endif %} +), + +final as ( + select + order_id, + customer_id, + order_created_at, + order_updated_at, + order_status, + order_amount, + + current_timestamp() as dbt_updated_at + + from new_and_updated_orders +) + +select * from final +``` + +### Incremental with Late-Arriving Facts + +```sql +-- models/marts/core/fct_transactions_incremental.sql +{{ config( + materialized='incremental', + unique_key='transaction_id', + on_schema_change='append_new_columns' +) }} + +with source_transactions as ( + select * from {{ ref('stg_transactions') }} + + {% if is_incremental() %} + -- Look back 7 days to catch late-arriving data + where transaction_date >= ( + select dateadd('day', -7, max(transaction_date)) + from {{ this }} + ) + {% endif %} +), + +deduplicated as ( + select + *, + row_number() over ( + partition by transaction_id + order by _loaded_at desc + ) as rn + from source_transactions +), + +final as ( + select + transaction_id, + customer_id, + transaction_date, + transaction_amount, + transaction_type + from deduplicated + where rn = 1 +) + +select * from final +``` + +--- + +## Snapshots + +### Basic Snapshot Configuration + +```sql +-- snapshots/customers_snapshot.sql +{% snapshot customers_snapshot %} + +{{ + config( + target_schema='snapshots', + unique_key='customer_id', + strategy='timestamp', + updated_at='updated_at', + invalidate_hard_deletes=True + ) +}} + +select + customer_id, + customer_name, + customer_email, + customer_status, + customer_segment, + updated_at +from {{ source('crm', 'customers') }} + +{% endsnapshot %} +``` + +### Check Strategy Snapshot + +```sql +-- snapshots/products_snapshot.sql +{% snapshot products_snapshot %} + +{{ + config( + target_schema='snapshots', + unique_key='product_id', + strategy='check', + check_cols=['price', 'category', 'is_active'] + ) +}} + +select + product_id, + product_name, + price, + category, + is_active, + updated_at +from {{ source('ecommerce', 'products') }} + +{% endsnapshot %} +``` + +--- + +## Macros + +### Reusable Field Cleaning Macro + +```sql +-- macros/clean_string_field.sql +{% macro clean_string_field(column_name) %} + lower(trim(regexp_replace({{ column_name }}, '[^a-zA-Z0-9\\s]', ''))) +{% endmacro %} +``` + +Usage: +```sql +select + {{ clean_string_field('customer_name') }} as customer_name_clean +from {{ ref('stg_customers') }} +``` + +### Date Spine Macro + +```sql +-- macros/create_date_spine.sql +{% macro create_date_spine(start_date, end_date, datepart='day') %} + +with date_spine as ( + {{ dbt_utils.date_spine( + datepart=datepart, + start_date="to_date('" ~ start_date ~ "', 'YYYY-MM-DD')", + end_date="to_date('" ~ end_date ~ "', 'YYYY-MM-DD')" + ) }} +) + +select + date_{{ datepart }} as date_value, + extract(year from date_{{ datepart }}) as year, + extract(month from date_{{ datepart }}) as month, + extract(day from date_{{ datepart }}) as day, + extract(dayofweek from date_{{ datepart }}) as day_of_week +from date_spine + +{% endmacro %} +``` + +### Grant Permissions Macro + +```sql +-- macros/grant_select.sql +{% macro grant_select_on_schemas(schemas, role) %} + {% for schema in schemas %} + grant usage on schema {{ schema }} to role {{ role }}; + grant select on all tables in schema {{ schema }} to role {{ role }}; + grant select on all views in schema {{ schema }} to role {{ role }}; + {% endfor %} +{% endmacro %} +``` + +--- + +## Tests + +### Custom Generic Test + +```sql +-- tests/generic/test_valid_email.sql +{% test valid_email(model, column_name) %} + +select * +from {{ model }} +where {{ column_name }} is not null + and not regexp_like( + {{ column_name }}, + '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}$' + ) + +{% endtest %} +``` + +### Singular Test + +```sql +-- tests/assert_positive_order_amounts.sql +-- This test will fail if any orders have negative amounts + +select + order_id, + order_amount +from {{ ref('fct_orders') }} +where order_amount < 0 +``` + +### Relationship Test with Custom Message + +```yaml +# models/marts/core/_core__models.yml +version: 2 + +models: + - name: fct_orders + tests: + - dbt_utils.expression_is_true: + expression: "order_amount >= 0" + config: + severity: error + error_if: ">0" + warn_if: ">10" +``` + +--- + +## Advanced Patterns + +### Dynamic Pivot with Jinja + +```sql +-- models/intermediate/product/int_product_metrics_pivoted.sql +{%- set metrics = ['revenue', 'units_sold', 'profit'] -%} +{%- set time_periods = ['7d', '30d', '90d'] -%} + +with daily_metrics as ( + select * from {{ ref('int_product_metrics_daily') }} +), + +pivoted as ( + select + product_id, + product_name, + + {% for period in time_periods %} + {% for metric in metrics %} + sum( + case + when datediff('day', metric_date, current_date()) <= {{ period[:-1] }} + then {{ metric }} + else 0 + end + ) as {{ metric }}_{{ period }} + {%- if not loop.last or not loop.parent.last %},{% endif %} + {% endfor %} + {% endfor %} + + from daily_metrics + group by 1, 2 +) + +select * from pivoted +``` + +### Slowly Changing Dimension Type 2 + +```sql +-- models/marts/core/dim_products_scd2.sql +{{ config( + materialized='incremental', + unique_key='product_scd_key' +) }} + +with source_data as ( + select + product_id, + product_name, + category, + price, + is_active, + updated_at + from {{ ref('stg_products') }} +), + +{% if is_incremental() %} + +-- Find changed records +changes as ( + select + source_data.*, + existing.product_scd_key, + existing.valid_from, + existing.valid_to, + existing.is_current + from source_data + inner join {{ this }} as existing + on source_data.product_id = existing.product_id + and existing.is_current = true + where ( + source_data.product_name != existing.product_name + or source_data.category != existing.category + or source_data.price != existing.price + or source_data.is_active != existing.is_active + ) +), + +-- Close out old records +updated_records as ( + select + product_scd_key, + product_id, + product_name, + category, + price, + is_active, + valid_from, + current_timestamp() as valid_to, + false as is_current + from changes +), + +-- Create new records +new_records as ( + select + {{ dbt_utils.generate_surrogate_key(['product_id', 'updated_at']) }} as product_scd_key, + product_id, + product_name, + category, + price, + is_active, + updated_at as valid_from, + null as valid_to, + true as is_current + from changes +), + +{% else %} + +-- Initial load +new_records as ( + select + {{ dbt_utils.generate_surrogate_key(['product_id', 'updated_at']) }} as product_scd_key, + product_id, + product_name, + category, + price, + is_active, + updated_at as valid_from, + null as valid_to, + true as is_current + from source_data +), + +updated_records as ( + select * from new_records limit 0 +), + +{% endif %} + +final as ( + select * from new_records + union all + select * from updated_records +) + +select * from final +``` + +--- + +## Project Configuration Examples + +### dbt_project.yml Best Practices + +```yaml +name: 'analytics' +version: '1.0.0' +config-version: 2 + +profile: 'analytics' + +model-paths: ["models"] +analysis-paths: ["analyses"] +test-paths: ["tests"] +seed-paths: ["seeds"] +macro-paths: ["macros"] +snapshot-paths: ["snapshots"] + +target-path: "target" +clean-targets: + - "target" + - "dbt_packages" + +models: + analytics: + # Staging layer - views by default + staging: + +materialized: view + +schema: staging + +tags: + - "staging" + + # Source-specific configs + salesforce: + +tags: + - "staging" + - "salesforce" + + stripe: + +tags: + - "staging" + - "stripe" + + # Intermediate layer + intermediate: + +materialized: view + +schema: intermediate + +tags: + - "intermediate" + + # Marts layer - tables by default + marts: + +materialized: table + +tags: + - "marts" + + core: + +schema: core + +tags: + - "marts" + - "core" + - "daily" + + finance: + +schema: finance + +tags: + - "marts" + - "finance" + - "daily" + + marketing: + +schema: marketing + +tags: + - "marts" + - "marketing" + - "hourly" + +seeds: + analytics: + +schema: seeds + +tags: + - "seeds" + +snapshots: + analytics: + +target_schema: snapshots + +tags: + - "snapshots" +``` + +This comprehensive reference should serve as your go-to resource for dbt development patterns! \ No newline at end of file diff --git a/.agents/skills/dbt-development/dbt_skill.md b/.agents/skills/dbt-development/dbt_skill.md new file mode 100644 index 0000000..3c6ebda --- /dev/null +++ b/.agents/skills/dbt-development/dbt_skill.md @@ -0,0 +1,490 @@ +--- +name: dbt-development +description: Expert guidance for developing dbt projects including project structure, model patterns, testing, and best practices. Use when working with dbt models, building data transformations, or setting up dbt projects. +--- + +# dbt Development Expert + +## Quick Reference + +### When to Use This Skill +- Building or refactoring dbt models +- Setting up dbt project structure +- Writing SQL transformations in dbt +- Implementing tests and documentation +- Optimizing dbt performance +- Following dbt best practices + +## Core Principles + +### 1. Project Structure +Organize dbt projects in three primary layers moving from source-conformed to business-conformed: + +``` +models/ +├── staging/ # Source-conformed, atomic building blocks +│ ├── source_a/ # One subdirectory per source system +│ └── source_b/ +├── intermediate/ # Purpose-built transformations +│ ├── finance/ # Business domain groupings +│ └── marketing/ +└── marts/ # Business-conformed, end-user ready + ├── core/ + ├── finance/ + └── marketing/ +``` + +### 2. Model Naming Conventions + +**Staging Models:** +- Format: `stg_[source]__[entity]s.sql` +- Example: `stg_stripe__payments.sql` +- Use double underscore between source and entity +- Always plural (unless truly singular) + +**Intermediate Models:** +- Format: `int_[entity]s_[verb]s.sql` +- Example: `int_payments_pivoted_to_orders.sql` + +**Mart Models:** +- Format: `fct_[entity]s.sql` or `dim_[entity]s.sql` +- Example: `fct_orders.sql`, `dim_customers.sql` + +## Staging Layer Best Practices + +### Standard Staging Pattern +Every staging model follows this exact pattern: + +```sql +-- models/staging/stripe/stg_stripe__payments.sql + +with source as ( + select * from {{ source('stripe', 'payments') }} +), + +renamed as ( + select + -- IDs + id as payment_id, + customer_id, + order_id, + + -- Timestamps (convert to UTC, standardize naming) + created_at as payment_created_at, + updated_at as payment_updated_at, + + -- Strings (clean, standardize) + lower(trim(payment_method)) as payment_method, + lower(trim(status)) as payment_status, + + -- Numerics (recast to correct types) + cast(amount as decimal(10,2)) as payment_amount, + cast(amount / 100.0 as decimal(10,2)) as payment_amount_usd, + + -- Booleans + case + when is_successful = 'true' then true + when is_successful = 'false' then false + else null + end as is_successful + + from source +) + +select * from renamed +``` + +### Staging Layer Guidelines +- **One source, one staging model** - 1:1 mapping +- **Rename fields** to fit project conventions (e.g., `_at` for timestamps) +- **Recast datatypes** (timestamps to UTC, prices to decimals) +- **Light cleansing** only (remove unwanted characters, NULL for empty strings) +- **No joins or aggregations** - preserve source grain +- **Materialize as views** (default) unless handling large volumes +- **Always test primary keys** for uniqueness and not null + +### Incremental Staging (Large Tables) + +```sql +-- models/staging/shopify/stg_shopify__orders.sql +{{ config( + materialized='incremental', + unique_key='order_id' +) }} + +with source as ( + select * from {{ source('shopify', 'orders') }} + + {% if is_incremental() %} + where created_at > (select max(order_created_at) from {{ this }}) + {% endif %} +), + +renamed as ( + select + id as order_id, + customer_id, + created_at as order_created_at, + status as order_status, + total_amount + from source +) + +select * from renamed +``` + +## Intermediate Layer Best Practices + +### Purpose +- Join staging models together +- Apply business logic +- Create reusable building blocks +- Reduce complexity in mart models + +### Example Intermediate Model + +```sql +-- models/intermediate/finance/int_payments_pivoted_to_orders.sql + +{%- set payment_methods = ['credit_card', 'bank_transfer', 'paypal'] -%} + +with payments as ( + select * from {{ ref('stg_stripe__payments') }} +), + +pivoted as ( + select + order_id, + {% for method in payment_methods %} + sum(case when payment_method = '{{ method }}' then payment_amount else 0 end) + as {{ method }}_amount + {%- if not loop.last %},{% endif %} + {% endfor %} + from payments + where payment_status = 'success' + group by 1 +) + +select * from pivoted +``` + +### Intermediate Guidelines +- **Subdirectories by business domain** (finance, marketing, etc.) +- **Single purpose per model** - do one thing well +- **Multiple inputs, single output** - join/combine here +- **Materialize as views** in dev, consider tables in prod +- **Use custom schemas** to separate from production + +## Mart Layer Best Practices + +### Purpose +- Analytics-ready tables for BI tools +- Business-friendly naming +- Denormalized for query performance +- Well-documented and tested + +### Example Fact Table + +```sql +-- models/marts/core/fct_orders.sql +{{ config( + materialized='table', + tags=['daily'] +) }} + +with orders as ( + select * from {{ ref('stg_jaffle_shop__orders') }} +), + +customers as ( + select * from {{ ref('stg_jaffle_shop__customers') }} +), + +payments as ( + select * from {{ ref('int_payments_pivoted_to_orders') }} +), + +final as ( + select + -- Primary Key + orders.order_id, + + -- Foreign Keys + orders.customer_id, + + -- Dimensions + customers.customer_name, + customers.customer_email, + orders.order_status, + orders.order_created_at, + + -- Metrics + payments.credit_card_amount, + payments.bank_transfer_amount, + payments.paypal_amount, + coalesce( + payments.credit_card_amount, 0 + ) + coalesce( + payments.bank_transfer_amount, 0 + ) + coalesce( + payments.paypal_amount, 0 + ) as total_amount + + from orders + left join customers on orders.customer_id = customers.customer_id + left join payments on orders.order_id = payments.order_id +) + +select * from final +``` + +### Mart Guidelines +- **Organize by business domain** +- **Materialize as tables** for performance +- **Include comprehensive documentation** +- **Test all metrics and joins** +- **Use prefixes** (fct_, dim_) for clarity + +## Testing Best Practices + +### Standard Tests in schema.yml + +```yaml +# models/staging/stripe/_stg_stripe__models.yml +version: 2 + +models: + - name: stg_stripe__payments + description: "Cleaned and standardized payment data from Stripe" + columns: + - name: payment_id + description: "Primary key for payments" + tests: + - unique + - not_null + + - name: payment_amount + description: "Payment amount in cents" + tests: + - not_null + + - name: payment_status + description: "Payment status (success, failed, pending)" + tests: + - accepted_values: + values: ['success', 'failed', 'pending'] +``` + +### Custom Generic Tests + +```sql +-- tests/generic/test_positive_values.sql +{% test positive_values(model, column_name) %} + +select * +from {{ model }} +where {{ column_name }} < 0 + +{% endtest %} +``` + +Usage: +```yaml +columns: + - name: payment_amount + tests: + - positive_values +``` + +### Testing Checklist +- [ ] Every model has a primary key tested for unique + not_null +- [ ] Accepted values for categorical fields +- [ ] Relationships between foreign keys +- [ ] Custom business logic tests +- [ ] Row count checks for incremental models + +## Jinja & Macros + +### DRY Principle with Macros + +```sql +-- macros/cents_to_dollars.sql +{% macro cents_to_dollars(column_name, decimal_places=2) %} + round({{ column_name }} / 100.0, {{ decimal_places }}) +{% endmacro %} +``` + +Usage: +```sql +select + payment_id, + {{ cents_to_dollars('amount_cents') }} as amount_dollars +from {{ ref('stg_payments') }} +``` + +### Common Jinja Patterns + +**Looping for DRY code:** +```sql +{%- set payment_methods = ['credit', 'debit', 'paypal'] -%} + +select + order_id, + {% for method in payment_methods %} + sum(case when payment_method = '{{ method }}' then amount else 0 end) as {{ method }}_amount + {%- if not loop.last %},{% endif %} + {% endfor %} +from payments +group by 1 +``` + +## Performance Optimization + +### 1. Materialization Strategy +- **Views**: Staging models, low-cost queries +- **Tables**: Mart models, frequently queried +- **Incremental**: Large fact tables, append-only data +- **Ephemeral**: Intermediate steps, no materialization needed + +### 2. Incremental Models Pattern + +```sql +{{ config( + materialized='incremental', + unique_key='order_id', + on_schema_change='append_new_columns' +) }} + +select + order_id, + order_date, + total_amount +from {{ source('raw', 'orders') }} + +{% if is_incremental() %} + where order_date > (select max(order_date) from {{ this }}) +{% endif %} +``` + +### 3. Execution Optimization +- Use `dbt run --select +model_name` for focused builds +- Use `dbt build` for production (runs + tests in DAG order) +- Use `state:modified` for CI/CD efficiency +- Apply tags for selective runs: `dbt run --select tag:daily` + +## Documentation Best Practices + +```yaml +# models/marts/core/_core__models.yml +version: 2 + +models: + - name: fct_orders + description: | + Order fact table containing one row per order with associated + customer information and payment totals. + + This model is refreshed daily at 6 AM UTC. + + columns: + - name: order_id + description: "Unique identifier for each order" + tests: + - unique + - not_null + + - name: total_amount + description: | + Total order amount in USD, calculated as the sum of all + successful payment methods (credit card + bank transfer + paypal). +``` + +## Git Workflow + +### Branch Strategy +```bash +# Create feature branch +git checkout -b feature/new-customer-metrics + +# Make changes +git add models/marts/core/fct_customers.sql +git commit -m "Add customer lifetime value metric to fct_customers" + +# Push and create PR +git push origin feature/new-customer-metrics +``` + +### Development Flow +1. **Development**: Use `dev` target, work in feature branch +2. **Testing**: Run `dbt build --select state:modified+` +3. **Code Review**: Submit PR with clear description +4. **Deployment**: Merge to main, runs against `prod` target + +## Common Patterns & Anti-Patterns + +### ✅ DO +- Use `ref()` for model dependencies +- Use `source()` for raw tables +- Keep staging models 1:1 with sources +- Test every primary key +- Document business logic +- Use consistent naming +- Materialize staging as views +- Group models by business domain + +### ❌ DON'T +- Reference raw tables in marts (use staging) +- Join or aggregate in staging +- Hardcode table names (use ref/source) +- Leave models without tests +- Use `SELECT *` in final models +- Skip documentation +- Over-optimize prematurely +- Nest CTEs more than 3 levels deep + +## Quick Command Reference + +```bash +# Development +dbt run --select model_name # Run single model +dbt run --select +model_name # Run model + upstream +dbt run --select model_name+ # Run model + downstream +dbt run --select tag:daily # Run tagged models + +# Testing +dbt test --select model_name # Test single model +dbt build --select model_name # Run + test single model + +# Documentation +dbt docs generate # Generate docs +dbt docs serve # Serve docs locally + +# State-based selection (CI/CD) +dbt run --select state:modified+ # Modified models + downstream +dbt test --select state:modified # Test only modified models +``` + +## References + +For deeper dives into specific topics: +- [Official dbt Best Practices](https://docs.getdbt.com/best-practices) +- [How We Structure Projects](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview) +- [dbt Style Guide](https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md) +- [dbt Utils Package](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) + +--- + +## Validation Checklist + +Before completing any dbt work, verify: + +- [ ] Models follow naming conventions (stg_, int_, fct_, dim_) +- [ ] Models are in correct layer/directory +- [ ] Primary keys tested (unique + not_null) +- [ ] All models use ref() or source() +- [ ] Business logic is documented in schema.yml +- [ ] Materialization strategy is appropriate +- [ ] No hardcoded values or table names +- [ ] CTEs are well-named and logical +- [ ] Model compiles: `dbt compile --select model_name` +- [ ] Tests pass: `dbt test --select model_name` \ No newline at end of file diff --git a/.agents/skills/design-philosophy/SKILL.md b/.agents/skills/design-philosophy/SKILL.md new file mode 100644 index 0000000..795bd88 --- /dev/null +++ b/.agents/skills/design-philosophy/SKILL.md @@ -0,0 +1,177 @@ +--- +name: design-philosophy +description: Data-first visualization design combining Tufte principles with Jobs/Ive simplicity for React + Nivo dashboards. +version: 1.1.0 +tags: [design, tufte, data-visualization, philosophy, nivo, react, typography] +--- + +# Design Philosophy: Economic Data Visualization + +This skill consolidates the prior `design-philosphy`, `nivo-charts`, and `tufte-visualization` skills into a single, canonical guide. + +## Use When + +- Creating or revising charts, dashboards, or data-dense UI +- Choosing chart types, encodings, or annotations +- Assessing visual integrity, readability, or hierarchy + +## Core Principle + +**Show the data.** Every visual choice must serve comprehension. Remove elements that do not improve clarity. + +## Tufte Principles (Applied) + +### 1. Data-Ink Ratio +- Remove ornamental elements (heavy grids, gradients, 3D effects). +- Favor thin strokes, direct labels, and whitespace. + +```typescript +// ✅ Clean Nivo Line + +``` + +### 2. Graphical Integrity +- Bar charts start at zero. +- Use linear scales unless data is explicitly log-scaled. +- Avoid perspective, drop-shadows, or faux depth. + +```typescript +const integrityConfig = { + yScale: { type: 'linear', min: 0, max: 'auto' }, +}; +``` + +### 3. Chartjunk Elimination +Remove: +- Background textures +- Excessive ticks +- Redundant legends +- Non-data icons + +### 4. Data Density +Prefer small multiples over cramming unrelated series. + +```typescript +function SmallMultiples({ data, renderChart }: { + data: Record; + renderChart: (data: T[], key: string) => React.ReactNode; +}) { + return ( +
+ {Object.entries(data).map(([key, values]) => ( +
+
{key}
+
{renderChart(values, key)}
+
+ ))} +
+ ); +} +``` + +### 5. Sparklines for Quick Context +Use sparklines to show trend and direction inline. + +```typescript +function Sparkline({ data, color = '#0f766e' }: { data: number[]; color?: string }) { + const series = [{ id: 'spark', data: data.map((y, x) => ({ x, y })) }]; + return ( + + ); +} +``` + +## Jobs/Ive Simplicity (Applied) + +- **Depth over decoration**: hierarchy through spacing, not ornament. +- **Single visual story**: every view should answer one question clearly. +- **Tactile clarity**: typography and rhythm make data feel deliberate. + +## Typography and Hierarchy + +- Use a strong humanist sans + mono pairing (e.g., `"IBM Plex Sans"` and `"IBM Plex Mono"`). +- Use tabular numerals for metrics. + +```css +:root { + --font-sans: "IBM Plex Sans", "Source Sans 3", system-ui, sans-serif; + --font-mono: "IBM Plex Mono", ui-monospace, monospace; +} +``` + +## Canonical Nivo Theme + +```typescript +import { Theme } from '@nivo/core'; + +export const dataFirstTheme: Theme = { + background: 'transparent', + text: { + fontSize: 11, + fontFamily: 'var(--font-sans)', + fill: '#334155', + }, + axis: { + domain: { line: { stroke: '#94a3b8', strokeWidth: 1 } }, + ticks: { + line: { stroke: 'transparent', strokeWidth: 0 }, + text: { fontSize: 10, fill: '#64748b' }, + }, + legend: { + text: { fontSize: 11, fill: '#334155', fontWeight: 500 }, + }, + }, + grid: { + line: { stroke: '#e2e8f0', strokeWidth: 1 }, + }, + tooltip: { + container: { + background: '#ffffff', + fontSize: 11, + borderRadius: 4, + boxShadow: '0 2px 8px rgba(0,0,0,0.12)', + padding: '8px 10px', + }, + }, +}; +``` + +## Chart Selection Guide + +- Time series trend: line chart +- Comparisons across categories: bar chart +- Part-to-whole with few categories: stacked bar (avoid pie) +- Distributions: histogram or box plot + +## Do / Avoid + +- Do: direct labels, minimal palettes, consistent scales +- Avoid: heavy borders, rainbow palettes, 3D, non-zero baselines for bars + +## Related Skills + +- `/tailwind-css-data-viz` +- `/react-component-architecture` +- `/typescript-financial-data-modeling` diff --git a/.agents/skills/design-philosphy/SKILL.md b/.agents/skills/design-philosphy/SKILL.md new file mode 100644 index 0000000..82cc32b --- /dev/null +++ b/.agents/skills/design-philosphy/SKILL.md @@ -0,0 +1,11 @@ +--- +name: design-philosphy +description: Deprecated alias for design-philosophy. Use that skill instead. +version: 1.0.0 +--- + +# Deprecated + +This skill has been consolidated into `design-philosophy`. + +Use `/design-philosophy` for the canonical guidance. diff --git a/.agents/skills/dignified-python/SKILL.md b/.agents/skills/dignified-python/SKILL.md new file mode 100644 index 0000000..14b45c8 --- /dev/null +++ b/.agents/skills/dignified-python/SKILL.md @@ -0,0 +1,170 @@ +--- +name: dignified-python +description: + Production Python coding standards with automatic version detection (3.10-3.13). Use when writing, + reviewing, or refactoring Python to ensure adherence to modern type syntax, LBYL exception + handling, pathlib operations, ABC-based interfaces, and production-tested patterns. Not + Dagster-specific - applies to any Python project. +references: + - core-standards + - cli-patterns + - versions/python-3.10 + - versions/python-3.11 + - versions/python-3.12 + - versions/python-3.13 + - advanced/api-design + - advanced/exception-handling + - advanced/interfaces + - advanced/typing-advanced +--- + +# Dignified Python Coding Standards Skill + +Production-quality Python coding standards for writing clean, maintainable, modern Python code +(versions 3.10-3.13). + +## When to Use This Skill + +Auto-invoke when users ask about: + +- "make this pythonic" / "is this good python" +- "type hints" / "type annotations" / "typing" +- "LBYL vs EAFP" / "exception handling" +- "pathlib vs os.path" / "path operations" +- "CLI patterns" / "click usage" +- "code review" / "improve this code" +- Any Python code quality or standards question + +**Note**: This skill is **general Python standards**, not Dagster-specific. Use +`/dagster-expert` for Dagster concepts/CLI and `/dagster-development` for repo-specific patterns. + +## When to Use This Skill vs. Others + +| User Need | Use This Skill | Alternative Skill | +| ---------------------------- | --------------------------- | ------------------------- | +| "make this pythonic" | ✅ Yes - Python standards | | +| "is this good python" | ✅ Yes - code quality | | +| "type hints" | ✅ Yes - typing guidance | | +| "LBYL vs EAFP" | ✅ Yes - exception patterns | | +| "pathlib vs os.path" | ✅ Yes - path handling | | +| "best practices for dagster" | ❌ No | `/dagster-expert` | +| "implement X pipeline" | ❌ No | `/dg` for implementation | +| "which integration to use" | ❌ No | `/dagster-expert` | +| "CLI argument parsing" | ✅ Yes - CLI patterns | | + +## Core Knowledge (ALWAYS Loaded) + +@dignified-python-core.md + +## Version Detection + +**Identify the project's minimum Python version** by checking (in order): + +1. `pyproject.toml` - Look for `requires-python` field (e.g., `requires-python = ">=3.12"`) +2. `setup.py` or `setup.cfg` - Look for `python_requires` +3. `.python-version` file - Contains version like `3.12` or `3.12.0` +4. Default to Python 3.12 if no version specifier found + +**Once identified, load the appropriate version-specific file:** + +- Python 3.10: Load `versions/python-3.10.md` +- Python 3.11: Load `versions/python-3.11.md` +- Python 3.12: Load `versions/python-3.12.md` +- Python 3.13: Load `versions/python-3.13.md` + +## Conditional Loading (Load Based on Task Patterns) + +Core files above cover 80%+ of Python code patterns. Only load these additional files when you +detect specific patterns: + +Pattern detection examples: + +- If task mentions "click" or "CLI" -> Load `cli-patterns.md` +- If task mentions "subprocess" -> Load subprocess patterns from core-standards + +## Reference Documentation Structure + +The skill root contains detailed guidance organized by topic: + +### Core References + +- **`core-standards.md`** - Essential standards (always loaded) +- **`cli-patterns.md`** - Command-line interface patterns (click, argparse) + +### Version-Specific References (`versions/`) + +- **`python-3.10.md`** - Features available in Python 3.10+ +- **`python-3.11.md`** - Features available in Python 3.11+ +- **`python-3.12.md`** - Features available in Python 3.12+ +- **`python-3.13.md`** - Features available in Python 3.13+ + +### Advanced Topics (`references/advanced/`) + +- **`exception-handling.md`** - LBYL patterns, error boundaries +- **`interfaces.md`** - ABC and Protocol patterns +- **`typing-advanced.md`** - Advanced typing patterns +- **`api-design.md`** - API design principles + +## When to Read Each Reference Document + +### `references/advanced/exception-handling.md` + +**Read when**: + +- Writing try/except blocks +- Wrapping third-party APIs that may raise +- Seeing or writing `from e` or `from None` +- Unsure if LBYL alternative exists + +### `references/advanced/interfaces.md` + +**Read when**: + +- Creating ABC or Protocol classes +- Writing @abstractmethod decorators +- Designing gateway layer interfaces +- Choosing between ABC and Protocol + +### `references/advanced/typing-advanced.md` + +**Read when**: + +- Using typing.cast() +- Creating Literal type aliases +- Narrowing types in conditional blocks + +### `references/module-design.md` + +**Read when**: + +- Creating new Python modules +- Adding module-level code (beyond simple constants) +- Using @cache decorator at module level +- Seeing Path() or computation at module level +- Considering inline imports + +### `references/api-design.md` + +**Read when**: + +- Adding default parameter values to functions +- Defining functions with 5 or more parameters +- Using ThreadPoolExecutor.submit() +- Reviewing function signatures + +### `references/checklists.md` + +**Read when**: + +- Final review before committing Python code +- Unsure if you've followed all rules +- Need a quick lookup of requirements + +## How to Use This Skill + +1. **Core knowledge** is loaded automatically (LBYL, pathlib, basic imports, anti-patterns) +2. **Version detection** happens once - identify the minimum Python version and load the appropriate + version file +3. **Reference documents** are loaded on-demand based on the triggers above +4. **Additional patterns** may require extra loading (CLI patterns, subprocess) +5. **Each file is self-contained** with complete guidance for its domain diff --git a/.agents/skills/dignified-python/cli-patterns.md b/.agents/skills/dignified-python/cli-patterns.md new file mode 100644 index 0000000..3ca9826 --- /dev/null +++ b/.agents/skills/dignified-python/cli-patterns.md @@ -0,0 +1,156 @@ +--- +--- + +# CLI Patterns - Click Best Practices + +## Core Rules + +1. **Use `click.echo()` for output, NEVER `print()`** +2. **Exit with `raise SystemExit(1)` for CLI errors** +3. **Error boundaries at command level** +4. **Use `err=True` for error output** +5. **Flush stderr before `click.confirm()`** (prevents buffering hangs) + +## Basic Click Patterns + +```python +import click +from pathlib import Path + +# ✅ CORRECT: Use click.echo for output +@click.command() +@click.argument("name") +def greet(name: str) -> None: + """Greet the user.""" + click.echo(f"Hello, {name}!") + +# ❌ WRONG: Using print() +@click.command() +def greet(name: str) -> None: + print(f"Hello, {name}!") # NEVER use print in CLI +``` + +## Error Handling in CLI + +```python +# ✅ CORRECT: CLI command error boundary +@click.command("create") +@click.argument("name") +def create(name: str) -> None: + """Create a resource.""" + try: + create_resource(name) + except subprocess.CalledProcessError as e: + click.echo(f"Error: Command failed: {e.stderr}", err=True) + raise SystemExit(1) + except ValueError as e: + click.echo(f"Error: {e}", err=True) + raise SystemExit(1) +``` + +## Output Patterns + +```python +# Regular output to stdout +click.echo("Processing complete") + +# Error output to stderr +click.echo("Error: Operation failed", err=True) + +# Colored output +click.echo(click.style("Success!", fg="green")) +click.echo(click.style("Warning!", fg="yellow", bold=True)) + +# Progress indication +with click.progressbar(items) as bar: + for item in bar: + process(item) +``` + +## Command Structure + +```python +@click.group() +@click.pass_context +def cli(ctx: click.Context) -> None: + """Main CLI entry point.""" + ctx.ensure_object(dict) + ctx.obj["config"] = load_config() + +@cli.command() +@click.option("--dry-run", is_flag=True, help="Perform dry run") +@click.argument("path", type=click.Path(exists=True)) +@click.pass_obj +def sync(obj: dict, path: str, dry_run: bool) -> None: + """Sync the repository.""" + config = obj["config"] + + if dry_run: + click.echo("DRY RUN: Would sync...") + else: + perform_sync(Path(path), config) + click.echo("✓ Sync complete") +``` + +## User Interaction + +```python +import sys + +# ✅ CORRECT: Flush stderr before confirmation prompts +# This prevents buffering hangs when mixing stderr output with stdin prompts +click.echo("Warning: This operation is destructive!", err=True) +sys.stderr.flush() # Flush before prompting +if click.confirm("Are you sure?"): + perform_dangerous_operation() + +# ❌ WRONG: click.confirm() after stderr output without flush +# This can hang because stderr isn't flushed before the prompt +click.echo("Warning: This operation is destructive!", err=True) +if click.confirm("Are you sure?"): # BAD: potential buffering hang + perform_dangerous_operation() + +# User input +name = click.prompt("Enter your name", default="User") + +# Password input +password = click.prompt("Password", hide_input=True) + +# Choice selection +choice = click.prompt( + "Select option", + type=click.Choice(["option1", "option2"]), + default="option1" +) +``` + +## Path Handling + +```python +@click.command() +@click.argument( + "input_file", + type=click.Path(exists=True, file_okay=True, dir_okay=False) +) +@click.argument( + "output_dir", + type=click.Path(exists=False, file_okay=False, dir_okay=True) +) +def process(input_file: str, output_dir: str) -> None: + """Process input file to output directory.""" + input_path = Path(input_file) + output_path = Path(output_dir) + + if not output_path.exists(): + output_path.mkdir(parents=True) + + click.echo(f"Processing {input_path} → {output_path}") +``` + +## Key Takeaways + +1. **Always click.echo()**: Never use print() in CLI code +2. **Error to stderr**: Use `err=True` for error messages +3. **Exit cleanly**: Use `raise SystemExit(1)` for errors +4. **User-friendly**: Provide clear messages and confirmations +5. **Type paths**: Use `click.Path()` for path arguments diff --git a/.agents/skills/dignified-python/dignified-python-core.md b/.agents/skills/dignified-python/dignified-python-core.md new file mode 100644 index 0000000..57122df --- /dev/null +++ b/.agents/skills/dignified-python/dignified-python-core.md @@ -0,0 +1,345 @@ +--- +--- + +# Dignified Python - Core Standards + +This document contains the core Python coding standards that apply to 80%+ of Python code. These +principles are loaded with every skill invocation. + +For conditional loading of specialized patterns: + +- CLI development -> Load `cli-patterns.md` +- Subprocess operations -> Load `subprocess.md` + +For detailed reference material, see the "When to Read Each Reference" section in SKILL.md. + +--- + +## The Cornerstone: LBYL Over EAFP + +**Look Before You Leap: Check conditions proactively, NEVER use exceptions for control flow.** + +This is the single most important rule in dignified Python. Every pattern below flows from this +principle. + +```python +# CORRECT: Check first +if key in mapping: + value = mapping[key] + process(value) + +# WRONG: Exception as control flow +try: + value = mapping[key] + process(value) +except KeyError: + pass +``` + +--- + +## Exception Handling Basics + +### Core Principle + +**ALWAYS use LBYL, NEVER EAFP for control flow** + +LBYL means checking conditions before acting. EAFP (Easier to Ask for Forgiveness than Permission) +means trying operations and catching exceptions. In dignified Python, we strongly prefer LBYL. + +### Dictionary Access Patterns + +```python +# CORRECT: Membership testing +if key in mapping: + value = mapping[key] + process(value) +else: + handle_missing() + +# ALSO CORRECT: .get() with default +value = mapping.get(key, default_value) +process(value) + +# CORRECT: Check before nested access +if "config" in data and "timeout" in data["config"]: + timeout = data["config"]["timeout"] + +# WRONG: KeyError as control flow +try: + value = mapping[key] +except KeyError: + handle_missing() +``` + +### When Exceptions ARE Acceptable + +Exceptions are ONLY acceptable at: + +1. **Error boundaries** (CLI/API level) +2. **Third-party API compatibility** (when no alternative exists) +3. **Adding context before re-raising** + +**Default: Let exceptions bubble up** + +For detailed exception handling patterns including B904 chaining, third-party API examples, and +anti-patterns, see `references/exception-handling.md`. + +--- + +## Path Operations + +### The Golden Rule + +**ALWAYS check `.exists()` BEFORE `.resolve()` or `.is_relative_to()`** + +### Why This Matters + +- `.resolve()` raises `OSError` for non-existent paths +- `.is_relative_to()` raises `ValueError` for invalid comparisons +- Checking `.exists()` first avoids exceptions entirely (LBYL!) + +### Correct Patterns + +```python +from pathlib import Path + +# CORRECT: Check exists first +for wt_path in worktree_paths: + if wt_path.exists(): + wt_path_resolved = wt_path.resolve() + if current_dir.is_relative_to(wt_path_resolved): + current_worktree = wt_path_resolved + break + +# WRONG: Using exceptions for path validation +try: + wt_path_resolved = wt_path.resolve() + if current_dir.is_relative_to(wt_path_resolved): + current_worktree = wt_path_resolved +except (OSError, ValueError): + continue +``` + +### Pathlib Best Practices + +**Always Use Pathlib (Never os.path)** + +```python +# CORRECT: Use pathlib.Path +from pathlib import Path + +config_file = Path.home() / ".config" / "app.yml" +if config_file.exists(): + content = config_file.read_text(encoding="utf-8") + +# WRONG: Use os.path +import os.path +config_file = os.path.join(os.path.expanduser("~"), ".config", "app.yml") +``` + +**Always Specify Encoding** + +```python +# CORRECT: Always specify encoding +content = path.read_text(encoding="utf-8") +path.write_text(data, encoding="utf-8") + +# WRONG: Default encoding +content = path.read_text() # Platform-dependent! +``` + +--- + +## Import Organization + +### Core Rules + +1. **Default: ALWAYS place imports at module level** +2. **Use absolute imports only** (no relative imports) +3. **Inline imports only for specific exceptions** (circular deps, TYPE_CHECKING, conditional + features) + +```python +# CORRECT: Module-level imports +import json +import click +from pathlib import Path +from erk.config import load_config + +def my_function() -> None: + data = json.loads(content) + +# CORRECT: Absolute import +from erk.config import load_config + +# WRONG: Relative import +from .config import load_config + +# WRONG: Inline imports without justification +def my_function() -> None: + import json # NEVER do this +``` + +For detailed inline import patterns and when they're legitimate, see `references/module-design.md`. + +--- + +## Performance Guidelines + +### Properties Must Be O(1) + +```python +# WRONG: Property doing I/O +@property +def size(self) -> int: + return self._fetch_from_db() + +# CORRECT: Explicit method name +def fetch_size_from_db(self) -> int: + return self._fetch_from_db() + +# CORRECT: O(1) property +@property +def size(self) -> int: + return self._cached_size +``` + +### Magic Methods Must Be O(1) + +```python +# WRONG: __len__ doing iteration +def __len__(self) -> int: + return sum(1 for _ in self._items) + +# CORRECT: O(1) __len__ +def __len__(self) -> int: + return self._count +``` + +--- + +## Anti-Patterns + +### No Backwards Compatibility Preservation (Default) + +```python +# WRONG: Keeping old API unnecessarily +def process_data(data: dict, legacy_format: bool = False) -> Result: + if legacy_format: + return legacy_process(data) + return new_process(data) + +# CORRECT: Break and migrate immediately +def process_data(data: dict) -> Result: + return new_process(data) +``` + +### No Re-Exports: One Canonical Import Path + +**Core Principle:** Every symbol has exactly one import path. Never re-export. + +```python +# WRONG: __all__ exports create duplicate import paths +# myapp/__init__.py +from myapp.core import Process +__all__ = ["Process"] + +# CORRECT: Empty __init__.py, import from canonical location +# from myapp.core import Process +``` + +**When re-exports ARE required** (plugin entry points): Use explicit `import X as X` syntax: + +```python +# CORRECT: Explicit re-export syntax for required entry points +from myapp.core.feature import my_function as my_function +``` + +### Declare Variables Close to Use + +```python +# WRONG: Variable declared far from use +def process_data(ctx, items): + result_path = compute_result_path(ctx) # Declared here... + # 20+ lines of other logic... + save_to_path(transformed, result_path) # ...used here + +# CORRECT: Inline at use site +def process_data(ctx, items): + validate_items(items) + transformed = transform_items(items) + save_to_path(transformed, compute_result_path(ctx)) +``` + +### Don't Destructure Objects Into Single-Use Locals + +```python +# WRONG: Unnecessary field extraction +result = fetch_user(user_id) +name = result.name # only used once below +email = result.email # only used once below +send_notification(name, email, role) + +# CORRECT: Access fields directly +user = fetch_user(user_id) +send_notification(user.name, user.email, user.role) +``` + +### Indentation Depth Limit + +**Maximum indentation: 4 levels** + +```python +# WRONG: Too deeply nested (5 levels) +def process_items(items): + for item in items: + if item.valid: + for child in item.children: + if child.enabled: + for grandchild in child.descendants: + pass # 5 levels deep! + +# CORRECT: Extract helper functions +def process_items(items): + for item in items: + if item.valid: + process_children(item.children) + +def process_children(children): + for child in children: + if child.enabled: + process_descendants(child.descendants) +``` + +--- + +## Backwards Compatibility Philosophy + +**Default stance: NO backwards compatibility preservation** + +Only preserve backwards compatibility when: + +- Code is clearly part of public API +- User explicitly requests it +- Migration cost is prohibitively high (rare) + +Benefits: + +- Cleaner, maintainable codebase +- Faster iteration +- No legacy code accumulation +- Simpler mental models + +--- + +## See Also + +For detailed guidance on specialized topics: + +- **Exception chaining (B904)**: `references/exception-handling.md` +- **ABC vs Protocol**: `references/interfaces.md` +- **typing.cast() assertions**: `references/typing-advanced.md` +- **Import-time side effects, @cache**: `references/module-design.md` +- **Default parameters, keyword-only args**: `references/api-design.md` +- **All decision checklists**: `references/checklists.md` diff --git a/.agents/skills/dignified-python/references/advanced/api-design.md b/.agents/skills/dignified-python/references/advanced/api-design.md new file mode 100644 index 0000000..5eb713a --- /dev/null +++ b/.agents/skills/dignified-python/references/advanced/api-design.md @@ -0,0 +1,230 @@ +--- +description: + Default parameter dangers, keyword-only arguments, ThreadPoolExecutor patterns, speculative test + infrastructure. +--- + +# API Design Reference + +**Read when**: Adding default parameters, functions with 5+ params, using ThreadPoolExecutor + +--- + +## Default Parameter Values Are Dangerous + +> **Scope:** This rule applies to **function definitions** (`def foo(bar: bool = False)`), NOT to +> **function calls** where you pass an argument named `default` (e.g., +> `click.confirm(default=True)`). Passing `default=True` to a function that accepts a `default` +> parameter is perfectly valid—you're not creating a default parameter value, you're explicitly +> providing a value. + +**Avoid default parameter values unless absolutely necessary.** They are a significant source of +bugs. + +**Why defaults are dangerous:** + +1. **Silent incorrect behavior** - Callers forget to pass a parameter and get unexpected results +2. **Hidden coupling** - The default encodes an assumption that may not hold for all callers +3. **Audit difficulty** - Hard to verify all call sites are using the right value +4. **Refactoring hazard** - Adding a new parameter with a default doesn't trigger errors at existing + call sites + +```python +# DANGEROUS: Default that might be wrong for some callers +def process_file(path: Path, encoding: str = "utf-8") -> str: + return path.read_text(encoding=encoding) + +# Caller forgets encoding, silently gets wrong behavior for legacy file +content = process_file(legacy_latin1_file) # Bug: should be encoding="latin-1" + +# SAFER: Require explicit choice +def process_file(path: Path, encoding: str) -> str: + return path.read_text(encoding=encoding) + +# Caller must think about encoding +content = process_file(legacy_latin1_file, encoding="latin-1") +``` + +**When you discover a default is never overridden, eliminate it:** + +```python +# If every call site uses the default... +activate_worktree(ctx, repo, path, script, "up", preserve_relative_path=True) # Always True +activate_worktree(ctx, repo, path, script, "down", preserve_relative_path=True) # Always True + +# CORRECT: Remove the parameter entirely +def activate_worktree(ctx, repo, path, script, command_name) -> None: + # Always preserve relative path - it's just the behavior + ... +``` + +**Acceptable uses of defaults:** + +1. **Truly optional behavior** - Where the default is correct for 95%+ of callers +2. **Backwards compatibility** - When adding a parameter to existing API (temporary) +3. **Test helper functions** - Functions in `tests/test_utils/` that exist to reduce test + boilerplate are explicitly exempt. These helpers often wrap complex constructors (like + `format_plan_header_body`) with sensible defaults, and having many default parameters is their + intended purpose—not a code smell + +**When reviewing code with defaults, ask:** + +- Do all call sites actually want this default? +- Would a caller forgetting this parameter cause a bug? +- Is there a safer design that makes the choice explicit? + +--- + +## Keyword-Only Arguments for Complex Functions + +**Functions with 5 or more parameters MUST use keyword-only arguments.** + +Use the `*` separator after the first positional parameter to enforce keyword-only at the language +level. This improves call-site readability by forcing explicit parameter names. + +```python +# CORRECT: Keyword-only after first param +def fetch_data( + url, + *, + timeout: float, + retries: int, + headers: dict[str, str], + auth_token: str, +) -> Response: + ... + +# Call site is self-documenting +response = fetch_data( + api_url, + timeout=30.0, + retries=3, + headers={"Accept": "application/json"}, + auth_token=token, +) + +# WRONG: All positional parameters +def fetch_data( + url, + timeout: float, + retries: int, + headers: dict[str, str], + auth_token: str, +) -> Response: + ... + +# Call site is unreadable - what do these values mean? +response = fetch_data(api_url, 30.0, 3, {"Accept": "application/json"}, token) +``` + +**Exceptions:** + +1. **`self`** - Always positional (Python requirement) +2. **`ctx` / context objects** - Can remain positional as the first parameter (convention) +3. **ABC/Protocol methods** - Exempt to avoid forcing all implementations to change signatures +4. **Click callbacks** - Click injects parameters; follow Click conventions + +```python +# CORRECT: ctx stays positional, rest are keyword-only +def create_worktree( + ctx: ErkContext, + *, + branch_name: str, + base_branch: str, + path: Path, + checkout: bool, +) -> WorktreeInfo: + ... +``` + +--- + +## ThreadPoolExecutor.submit() Pattern + +`ThreadPoolExecutor.submit()` passes arguments positionally to the callable. For functions with +keyword-only parameters, wrap the call in a lambda: + +```python +# WRONG: submit() passes args positionally - fails with keyword-only functions +future = executor.submit(fetch_data, url, timeout, retries, headers, token) + +# CORRECT: Lambda enables keyword arguments +future = executor.submit( + lambda: fetch_data( + url, + timeout=timeout, + retries=retries, + headers=headers, + auth_token=token, + ) +) +``` + +--- + +## Speculative Test Infrastructure + +**Don't add parameters to fakes "just in case" they might be useful for testing.** + +Fakes should mirror production interfaces. Adding test-only configuration knobs that never get used +creates dead code and false complexity. + +```python +# WRONG: Test-only parameter that's never used in production +class FakeGitHub: + def __init__( + self, + prs: dict[str, PullRequestInfo] | None = None, + rate_limited: bool = False, # "Might test this later" + ) -> None: + self._rate_limited = rate_limited # Never set to True anywhere + +# CORRECT: Only add infrastructure when you need it +class FakeGitHub: + def __init__( + self, + prs: dict[str, PullRequestInfo] | None = None, + ) -> None: + ... +``` + +**The test for this:** If grep shows a parameter is only ever passed in test files, and those tests +are testing hypothetical scenarios rather than actual production behavior, delete both the parameter +and the tests. + +--- + +## Speculative Tests + +```python +# FORBIDDEN: Tests for future features +# def test_feature_we_might_add(): +# pass + +# CORRECT: TDD for current implementation +def test_feature_being_built_now(): + result = new_feature() + assert result == expected +``` + +--- + +## Decision Checklist + +Before adding a default parameter value: + +- [ ] Do 95%+ of callers actually want this default? +- [ ] Would forgetting to pass this parameter cause a subtle bug? +- [ ] Is there a safer design that makes the choice explicit? +- [ ] If the default is never overridden anywhere, should this parameter exist at all? + +**Default: Require explicit values; eliminate unused defaults** + +Before adding a function with 5+ parameters: + +- [ ] Have I added `*` after the first (or ctx) parameter? +- [ ] Is only `self`/`ctx` positional? +- [ ] Is this an ABC/Protocol method? (exempt from rule) +- [ ] If using ThreadPoolExecutor.submit(), am I using a lambda wrapper? + +**Default: All parameters after the first should be keyword-only** diff --git a/.agents/skills/dignified-python/references/advanced/exception-handling.md b/.agents/skills/dignified-python/references/advanced/exception-handling.md new file mode 100644 index 0000000..614378b --- /dev/null +++ b/.agents/skills/dignified-python/references/advanced/exception-handling.md @@ -0,0 +1,185 @@ +--- +description: + Detailed exception handling patterns including B904 chaining, third-party API compatibility, and + anti-patterns. +--- + +# Exception Handling Reference + +**Read when**: Writing try/except blocks, wrapping third-party APIs, seeing `from e` or `from None` + +--- + +## When Exceptions ARE Acceptable + +Exceptions are ONLY acceptable at: + +1. **Error boundaries** (CLI/API level) +2. **Third-party API compatibility** (when no alternative exists) +3. **Adding context before re-raising** + +### 1. Error Boundaries + +```python +# ACCEPTABLE: CLI command error boundary +@click.command("create") +@click.pass_obj +def create(ctx: ErkContext, name: str) -> None: + """Create a worktree.""" + try: + create_worktree(ctx, name) + except subprocess.CalledProcessError as e: + click.echo(f"Error: Git command failed: {e.stderr}", err=True) + raise SystemExit(1) +``` + +### 2. Third-Party API Compatibility + +```python +# ACCEPTABLE: Third-party API forces exception handling +def _get_bigquery_sample(sql_client, table_name): + """ + BigQuery's TABLESAMPLE doesn't work on views. + There's no reliable way to determine a priori whether + a table supports TABLESAMPLE. + """ + try: + return sql_client.run_query(f"SELECT * FROM {table_name} TABLESAMPLE...") + except Exception: + return sql_client.run_query(f"SELECT * FROM {table_name} ORDER BY RAND()...") +``` + +> **The test for "no alternative exists"**: Can you validate or check the condition BEFORE calling +> the API? If yes (even using a different function/method), use LBYL. The exception only applies +> when the API provides NO way to determine success a priori—you literally must attempt the +> operation to know if it will work. + +### What Does NOT Qualify as Third-Party API Compatibility + +Standard library functions with known LBYL alternatives do NOT qualify: + +```python +# WRONG: int() has LBYL alternative (str.isdigit) +try: + port = int(user_input) +except ValueError: + port = 80 + +# CORRECT: Check before calling +if user_input.lstrip('-+').isdigit(): + port = int(user_input) +else: + port = 80 + +# WRONG: datetime.fromisoformat() can be validated first +try: + dt = datetime.fromisoformat(timestamp_str) +except ValueError: + dt = None + +# CORRECT: Validate format before parsing +def _is_iso_format(s: str) -> bool: + return len(s) >= 10 and s[4] == "-" and s[7] == "-" + +if _is_iso_format(timestamp_str): + dt = datetime.fromisoformat(timestamp_str) +else: + dt = None +``` + +### 3. Adding Context Before Re-raising + +```python +# ACCEPTABLE: Adding context before re-raising +try: + process_file(config_file) +except yaml.YAMLError as e: + raise ValueError(f"Failed to parse config file {config_file}: {e}") from e +``` + +--- + +## Exception Chaining (B904 Lint Compliance) + +**Ruff rule B904** requires explicit exception chaining when raising inside an `except` block. This +prevents losing the original traceback. + +```python +# CORRECT: Chain to preserve context +try: + parse_config(path) +except ValueError as e: + click.echo(json.dumps({"success": False, "error": str(e)})) + raise SystemExit(1) from e # Preserves traceback + +# CORRECT: Explicitly break chain when intentional +try: + fetch_from_cache(key) +except KeyError: + # Original exception is not relevant to caller + raise ValueError(f"Unknown key: {key}") from None + +# WRONG: Missing exception chain (B904 violation) +try: + parse_config(path) +except ValueError: + raise SystemExit(1) # Lint error: missing 'from e' or 'from None' + +# CORRECT: CLI error boundary with JSON output +try: + result = some_operation() +except RuntimeError as e: + click.echo(json.dumps({"success": False, "error": str(e)})) + raise SystemExit(0) from None # Exception is in JSON, traceback irrelevant to CLI user +``` + +**When to use each:** + +- `from e` - Preserve original exception for debugging +- `from None` - Intentionally suppress original (e.g., transforming exception type, CLI JSON output) + +--- + +## Exception Anti-Patterns + +**Never swallow exceptions silently** + +Even at error boundaries, you must at least log/warn so issues can be diagnosed: + +```python +# WRONG: Silent exception swallowing +try: + risky_operation() +except: + pass + +# WRONG: Silent swallowing even at error boundary +try: + optional_feature() +except Exception: + pass # Silent - impossible to diagnose issues + +# CORRECT: Let exceptions bubble up (default) +risky_operation() + +# CORRECT: At error boundaries, log the exception +try: + optional_feature() +except Exception as e: + logging.warning("Optional feature failed: %s", e) # Diagnosable +``` + +**Never use silent fallback behavior** + +```python +# WRONG: Silent fallback masks failure +def process_text(text: str) -> dict: + try: + return llm_client.process(text) + except Exception: + return regex_parse_fallback(text) + +# CORRECT: Let error bubble to boundary +def process_text(text: str) -> dict: + return llm_client.process(text) +``` diff --git a/.agents/skills/dignified-python/references/advanced/interfaces.md b/.agents/skills/dignified-python/references/advanced/interfaces.md new file mode 100644 index 0000000..04b02b4 --- /dev/null +++ b/.agents/skills/dignified-python/references/advanced/interfaces.md @@ -0,0 +1,183 @@ +--- +description: ABC vs Protocol decision guide, dependency injection patterns, and complete DI examples. +--- + +# Interface Design Reference + +**Read when**: Creating ABC/Protocol classes, writing @abstractmethod, designing gateway interfaces + +--- + +## ABC vs Protocol: Choosing the Right Interface + +**ABCs (nominal typing)** and **Protocols (structural typing)** serve different purposes. Choose +based on ownership and coupling needs. + +| Use Case | Recommended | Why | +| ----------------------------------------- | ----------- | ---------------------------------------------------- | +| Internal interfaces you control | ABC | Explicit enforcement, runtime validation, code reuse | +| Third-party library boundaries | Protocol | No inheritance required, loose coupling | +| Plugin systems with isinstance checks | ABC | Reliable runtime type validation | +| Minimal interface contracts (1-2 methods) | Protocol | Less boilerplate, focused contracts | + +**Default for erk internal code: ABC. Default for external library facades: Protocol.** + +--- + +## ABC Interface Pattern + +```python +# CORRECT: Use ABC for interfaces +from abc import ABC, abstractmethod + +class Repository(ABC): + @abstractmethod + def save(self, entity: Entity) -> None: + """Save entity to storage.""" + ... + + @abstractmethod + def load(self, id: str) -> Entity: + """Load entity by ID.""" + ... + +class PostgresRepository(Repository): + def save(self, entity: Entity) -> None: + # Implementation + pass + + def load(self, id: str) -> Entity: + # Implementation + pass +``` + +--- + +## Benefits of ABC (Internal Interfaces) + +1. **Explicit inheritance** - Clear class hierarchy, explicit opt-in +2. **Runtime validation** - Errors at instantiation if abstract methods missing +3. **Code reuse** - Can include concrete methods and shared logic +4. **Reliable isinstance()** - Full signature checking at runtime + +--- + +## Benefits of Protocol (External Boundaries) + +1. **No inheritance required** - Works with code you don't control +2. **Loose coupling** - Implementations don't know about the protocol +3. **Minimal contracts** - Define only the methods you need +4. **Duck typing** - Aligns with Python's philosophy + +--- + +## Complete DI Example + +```python +from abc import ABC, abstractmethod +from dataclasses import dataclass + +# Define the interface +class DataStore(ABC): + @abstractmethod + def get(self, key: str) -> str | None: + """Retrieve value by key.""" + ... + + @abstractmethod + def set(self, key: str, value: str) -> None: + """Store value with key.""" + ... + +# Real implementation +class RedisStore(DataStore): + def get(self, key: str) -> str | None: + return self.client.get(key) + + def set(self, key: str, value: str) -> None: + self.client.set(key, value) + +# Fake for testing +class FakeStore(DataStore): + def __init__(self) -> None: + self._data: dict[str, str] = {} + + def get(self, key: str) -> str | None: + if key not in self._data: + return None + return self._data[key] + + def set(self, key: str, value: str) -> None: + self._data[key] = value + +# Business logic accepts interface +@dataclass +class Service: + store: DataStore # Depends on abstraction + + def process(self, item: str) -> None: + cached = self.store.get(item) + if cached is None: + result = expensive_computation(item) + self.store.set(item, result) + else: + result = cached + use_result(result) +``` + +--- + +## When to Use Protocol + +**Protocols excel at defining interfaces for code you don't control:** + +```python +# CORRECT: Protocol for third-party library facade +from typing import Protocol + +class HttpClient(Protocol): + """Interface for HTTP operations - decouples from requests/httpx/aiohttp.""" + def get(self, url: str) -> Response: ... + def post(self, url: str, data: dict) -> Response: ... + +# Any HTTP library that has these methods works - no inheritance needed +def fetch_data(client: HttpClient, endpoint: str) -> dict: + response = client.get(endpoint) + return response.json() +``` + +**Protocols are also appropriate for minimal, focused interfaces:** + +```python +# CORRECT: Protocol for structural typing with minimal interface +from typing import Protocol + +class Closeable(Protocol): + def close(self) -> None: ... + +def cleanup_resources(resources: list[Closeable]) -> None: + for r in resources: + r.close() +``` + +--- + +## Protocol Limitations + +1. **No runtime validation** - `@runtime_checkable` only checks method existence, not signatures +2. **No code reuse** - Protocols shouldn't have method implementations +3. **Weaker isinstance() checks** - ABCs provide more reliable runtime type checking + +--- + +## Decision Checklist + +Before defining an interface (ABC or Protocol): + +- [ ] Do I own all implementations? -> Prefer ABC +- [ ] Am I wrapping a third-party library? -> Prefer Protocol +- [ ] Do I need runtime isinstance() validation? -> Use ABC +- [ ] Is this a minimal interface (1-2 methods)? -> Protocol may be simpler +- [ ] Do I need shared method implementations? -> Use ABC + +**Default for erk internal code: ABC. Default for external library facades: Protocol.** diff --git a/.agents/skills/dignified-python/references/advanced/typing-advanced.md b/.agents/skills/dignified-python/references/advanced/typing-advanced.md new file mode 100644 index 0000000..2827c52 --- /dev/null +++ b/.agents/skills/dignified-python/references/advanced/typing-advanced.md @@ -0,0 +1,158 @@ +--- +description: Advanced typing patterns including cast() with assertions, Literal types for programmatic strings. +--- + +# Advanced Typing Reference + +**Read when**: Using typing.cast(), creating Literal type aliases, narrowing types + +--- + +## Using `typing.cast()` + +### Core Rule + +**ALWAYS verify `cast()` with a runtime assertion, unless there's a documented reason not to.** + +`typing.cast()` is a compile-time only construct—it tells the type checker to trust you but performs +no runtime verification. If your assumption is wrong, you'll get silent misbehavior instead of a +clear error. + +### Required Pattern + +```python +from collections.abc import MutableMapping +from typing import Any, cast + +# CORRECT: Runtime assertion before cast +assert isinstance(doc, MutableMapping), f"Expected MutableMapping, got {type(doc)}" +cast(dict[str, Any], doc)["key"] = value + +# CORRECT: Alternative with hasattr for duck typing +assert hasattr(obj, '__setitem__'), f"Expected subscriptable, got {type(obj)}" +cast(dict[str, Any], obj)["key"] = value +``` + +### Anti-Pattern + +```python +# WRONG: Cast without runtime verification +cast(dict[str, Any], doc)["key"] = value # If doc isn't a dict-like, silent failure +``` + +### When to Skip Runtime Verification + +**Default: Always add the assertion when cost is trivial (O(1) checks like `in`, `isinstance`).** + +Skip the assertion only in these narrow cases: + +1. **Immediately after a type guard**: The check was just performed and would be redundant + + ```python + if isinstance(value, str): + # No assertion needed - we just checked + result = cast(str, value).upper() + ``` + +2. **Performance-critical hot path**: Add a comment explaining the measured overhead + ```python + # Skip assertion: called 10M times/sec, isinstance adds 15% overhead + # Type invariant maintained by _validate_input() at entry point + cast(int, cached_value) + ``` + +**What is NOT a valid reason to skip:** + +- "Click validates the choice set" - Add assertion anyway; cost is trivial +- "The library guarantees the type" - Add assertion anyway; defense in depth +- "It's obvious from context" - Add assertion anyway; future readers benefit + +### Why This Matters + +- **Silent bugs are worse than loud bugs**: An assertion failure gives you a stack trace and clear + error message +- **Documentation**: The assertion documents your assumption for future readers +- **Defense in depth**: Third-party libraries can change behavior between versions + +--- + +## Programmatically Significant Strings + +**Use `Literal` types for strings that have programmatic meaning.** + +When strings represent a fixed set of valid values (error codes, status values, command types), +model them in the type system using `Literal`. + +### Why This Matters + +1. **Type safety** - Typos caught at type-check time, not runtime +2. **IDE support** - Autocomplete shows valid options +3. **Documentation** - Valid values are explicit in the code +4. **Refactoring** - Rename operations work correctly + +### Naming Convention + +**Use kebab-case for all internal Literal string values:** + +```python +# CORRECT: kebab-case for internal values +IssueCode = Literal["orphan-state", "orphan-dir", "missing-branch"] +ErrorType = Literal["not-found", "invalid-format", "timeout-exceeded"] +``` + +**Exception: When modeling external systems, match the external API's convention:** + +```python +# CORRECT: Match GitHub API's UPPER_CASE +PRState = Literal["OPEN", "MERGED", "CLOSED"] + +# CORRECT: Match GitHub Actions API's lowercase +WorkflowStatus = Literal["completed", "in_progress", "queued"] +``` + +The rule is: kebab-case by default, external convention when modeling external APIs. + +### Pattern + +```python +from dataclasses import dataclass +from typing import Literal + +# CORRECT: Define a type alias for the valid values +IssueCode = Literal["orphan-state", "orphan-dir", "missing-branch"] + +@dataclass(frozen=True) +class Issue: + code: IssueCode + message: str + +def check_state() -> list[Issue]: + issues: list[Issue] = [] + if problem_detected: + issues.append(Issue(code="orphan-state", message="description")) # Type-checked! + return issues + +# WRONG: Bare strings without type constraint +def check_state() -> list[tuple[str, str]]: + issues: list[tuple[str, str]] = [] + issues.append(("orphen-state", "desc")) # Typo goes unnoticed! + return issues +``` + +### When to Use Literal + +- Error/issue codes +- Status values (pending, complete, failed) +- Command types or action names +- Configuration keys with fixed valid values +- Any string that is compared programmatically + +### Decision Checklist + +Before using a bare `str` type, ask: + +- Is this string compared with `==` or `in` anywhere? +- Is there a fixed set of valid values? +- Would a typo in this string cause a bug? + +If any answer is "yes", use `Literal` instead. diff --git a/.agents/skills/dignified-python/references/checklists.md b/.agents/skills/dignified-python/references/checklists.md new file mode 100644 index 0000000..7c272f6 --- /dev/null +++ b/.agents/skills/dignified-python/references/checklists.md @@ -0,0 +1,134 @@ +--- +description: All decision checklists consolidated for final review before committing Python changes. +--- + +# Decision Checklists Reference + +**Read when**: Final review before committing Python code, need quick lookup of requirements + +--- + +## Before Writing `try/except` + +- [ ] Is this at an error boundary? (CLI/API level) +- [ ] Can I check the condition proactively? (LBYL) +- [ ] Am I adding meaningful context, or just hiding? +- [ ] Is third-party API forcing me to use exceptions? (No LBYL check exists—not even format + validation) +- [ ] Have I encapsulated the violation? +- [ ] Am I catching specific exceptions, not broad? +- [ ] If catching at error boundary, am I logging/warning? (Never silently swallow) + +**Default: Let exceptions bubble up** + +--- + +## Before Path Operations + +- [ ] Did I check `.exists()` before `.resolve()`? +- [ ] Did I check `.exists()` before `.is_relative_to()`? +- [ ] Am I using `pathlib.Path`, not `os.path`? +- [ ] Did I specify `encoding="utf-8"`? + +--- + +## Before Using `typing.cast()` + +- [ ] Have I added a runtime assertion to verify the cast? +- [ ] Is the assertion cost trivial (O(1))? If yes, always add it. +- [ ] If skipping, is it because I just performed an isinstance check (redundant)? +- [ ] If skipping for performance, have I documented the measured overhead? + +**Default: Always add runtime assertion before cast when cost is trivial** + +--- + +## Before Defining an Interface (ABC or Protocol) + +- [ ] Do I own all implementations? -> Prefer ABC +- [ ] Am I wrapping a third-party library? -> Prefer Protocol +- [ ] Do I need runtime isinstance() validation? -> Use ABC +- [ ] Is this a minimal interface (1-2 methods)? -> Protocol may be simpler +- [ ] Do I need shared method implementations? -> Use ABC + +**Default for erk internal code: ABC. Default for external library facades: Protocol.** + +--- + +## Before Preserving Backwards Compatibility + +- [ ] Did the user explicitly request it? +- [ ] Is this a public API with external consumers? +- [ ] Have I documented why it's needed? +- [ ] Is migration cost prohibitively high? + +**Default: Break the API and migrate callsites immediately** + +--- + +## Before Inline Imports + +- [ ] Is this to break a circular dependency? +- [ ] Is this for TYPE_CHECKING? +- [ ] Is this for conditional features? +- [ ] If for startup time: Have I MEASURED the import cost? +- [ ] If for startup time: Is the cost significant (>100ms)? +- [ ] If for startup time: Have I documented the measured cost in a comment? +- [ ] Have I documented why the inline import is needed? + +**Default: Module-level imports** + +--- + +## Before Importing/Re-Exporting Symbols + +- [ ] Is there already a canonical location for this symbol? +- [ ] Am I creating a second import path for the same symbol? +- [ ] If this is a shim module, am I importing only what's needed for this module's purpose? +- [ ] Have I avoided `__all__` exports? + +**Default: Import from canonical location, never re-export** + +--- + +## Before Declaring a Local Variable + +- [ ] Is this variable used more than once? +- [ ] Is this variable used close to where it's declared? +- [ ] Would inlining the computation hurt readability? +- [ ] Am I extracting object fields into locals that are only used once? + +**Default: Inline single-use computations at the call site; access object attributes directly** + +--- + +## Before Adding a Default Parameter Value + +- [ ] Do 95%+ of callers actually want this default? +- [ ] Would forgetting to pass this parameter cause a subtle bug? +- [ ] Is there a safer design that makes the choice explicit? +- [ ] If the default is never overridden anywhere, should this parameter exist at all? + +**Default: Require explicit values; eliminate unused defaults** + +--- + +## Before Adding a Function with 5+ Parameters + +- [ ] Have I added `*` after the first (or ctx) parameter? +- [ ] Is only `self`/`ctx` positional? +- [ ] Is this an ABC/Protocol method? (exempt from rule) +- [ ] If using ThreadPoolExecutor.submit(), am I using a lambda wrapper? + +**Default: All parameters after the first should be keyword-only** + +--- + +## Before Writing Module-Level Code + +- [ ] Does this involve any computation (even `Path()` construction)? +- [ ] Does this involve I/O (file, network, environment)? +- [ ] Could this fail or raise exceptions? +- [ ] Would tests need to mock this value? + +If any answer is "yes", wrap in a `@cache`-decorated function instead. diff --git a/.agents/skills/dignified-python/references/module-design.md b/.agents/skills/dignified-python/references/module-design.md new file mode 100644 index 0000000..32b94f6 --- /dev/null +++ b/.agents/skills/dignified-python/references/module-design.md @@ -0,0 +1,214 @@ +--- +description: Import-time side effects, @cache for deferred computation, module-level code patterns. +--- + +# Module Design Reference + +**Read when**: Creating new modules, adding module-level code, using @cache decorator + +--- + +## Import-Time Side Effects + +### Core Rule + +**Avoid computation and side effects at import time. Defer to function calls.** + +Module-level code runs when the module is imported. Side effects at import time cause: + +1. **Slower startup** - Every import triggers computation +2. **Test brittleness** - Hard to mock/control behavior +3. **Circular import issues** - Dependencies evaluated too early +4. **Unpredictable order** - Import order affects behavior + +--- + +## Common Anti-Patterns + +```python +# WRONG: Path computed at import time +SESSION_ID_FILE = Path(".erk/scratch/current-session-id") + +def get_session_id() -> str | None: + if SESSION_ID_FILE.exists(): + return SESSION_ID_FILE.read_text(encoding="utf-8") + return None + +# WRONG: Config loaded at import time +CONFIG = load_config() # I/O at import! + +# WRONG: Connection established at import time +DB_CLIENT = DatabaseClient(os.environ["DB_URL"]) # Side effect at import! +``` + +--- + +## Correct Patterns + +**Use `@cache` for deferred computation:** + +```python +from functools import cache + +# CORRECT: Defer computation until first call +@cache +def _session_id_file_path() -> Path: + """Return path to session ID file (cached after first call).""" + return Path(".erk/scratch/current-session-id") + +def get_session_id() -> str | None: + session_file = _session_id_file_path() + if session_file.exists(): + return session_file.read_text(encoding="utf-8") + return None +``` + +**Use functions for resources:** + +```python +# CORRECT: Defer resource creation to function call +@cache +def get_config() -> Config: + """Load config on first call, cache result.""" + return load_config() + +@cache +def get_db_client() -> DatabaseClient: + """Create database client on first call.""" + return DatabaseClient(os.environ["DB_URL"]) +``` + +--- + +## When Module-Level Constants ARE Acceptable + +Simple, static values that don't involve computation or I/O: + +```python +# ACCEPTABLE: Static constants +DEFAULT_TIMEOUT = 30 +MAX_RETRIES = 3 +SUPPORTED_FORMATS = frozenset({"json", "yaml", "toml"}) +``` + +--- + +## Inline Import Patterns + +### Core Rules + +1. **Default: ALWAYS place imports at module level** +2. **Use absolute imports only** (no relative imports) +3. **Inline imports only for specific exceptions** (see below) + +### Legitimate Inline Import Patterns + +#### 1. Circular Import Prevention + +```python +# commands/sync.py +def register_commands(cli_group): + """Register commands with CLI group (avoids circular import).""" + from myapp.cli import sync_command # Breaks circular dependency + cli_group.add_command(sync_command) +``` + +**When to use:** + +- CLI command registration +- Plugin systems with bidirectional dependencies +- Lazy loading to break import cycles + +#### 2. Conditional Feature Imports + +```python +def process_data(data: dict, dry_run: bool = False) -> None: + if dry_run: + # Inline import: Only needed for dry-run mode + from myapp.dry_run import NoopProcessor + processor = NoopProcessor() + else: + processor = RealProcessor() + processor.execute(data) +``` + +**When to use:** + +- Debug/verbose mode utilities +- Dry-run mode wrappers +- Optional feature modules +- Platform-specific implementations + +#### 3. TYPE_CHECKING Imports + +```python +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from myapp.models import User # Only for type hints + +def process_user(user: "User") -> None: + ... +``` + +**When to use:** + +- Avoiding circular dependencies in type hints +- Forward declarations + +#### 4. Startup Time Optimization (Rare) + +Some packages have genuinely heavy import costs (pyspark, jupyter ecosystem, large ML frameworks). +Deferring these imports can improve CLI startup time. + +**However, apply "innocent until proven guilty":** + +- Default to module-level imports +- Only defer imports when you have MEASURED evidence of startup impact +- Document the measured cost in a comment + +```python +# ACCEPTABLE: Measured heavy import (adds 800ms to startup) +def run_spark_job(config: SparkConfig) -> None: + from pyspark.sql import SparkSession # Heavy: 800ms import time + session = SparkSession.builder.getOrCreate() + ... + +# WRONG: Speculative deferral without measurement +def check_staleness(project_dir: Path) -> None: + # Inline imports to avoid import-time side effects <- WRONG: no evidence + from myapp.staleness import get_version + ... +``` + +**When NOT to defer:** + +- Standard library modules +- Lightweight internal modules +- Modules you haven't measured +- "Just in case" optimization + +--- + +## Decision Checklist + +Before writing module-level code: + +- [ ] Does this involve any computation (even `Path()` construction)? +- [ ] Does this involve I/O (file, network, environment)? +- [ ] Could this fail or raise exceptions? +- [ ] Would tests need to mock this value? + +If any answer is "yes", wrap in a `@cache`-decorated function instead. + +Before inline imports: + +- [ ] Is this to break a circular dependency? +- [ ] Is this for TYPE_CHECKING? +- [ ] Is this for conditional features? +- [ ] If for startup time: Have I MEASURED the import cost? +- [ ] If for startup time: Is the cost significant (>100ms)? +- [ ] If for startup time: Have I documented the measured cost in a comment? +- [ ] Have I documented why the inline import is needed? + +**Default: Module-level imports** diff --git a/.agents/skills/dignified-python/subprocess.md b/.agents/skills/dignified-python/subprocess.md new file mode 100644 index 0000000..9ff2639 --- /dev/null +++ b/.agents/skills/dignified-python/subprocess.md @@ -0,0 +1,99 @@ +--- +--- + +# Subprocess Handling - Safe Execution + +## Core Rule + +**ALWAYS use `check=True` with `subprocess.run()`** + +## Basic Subprocess Pattern + +```python +import subprocess +from pathlib import Path + +# ✅ CORRECT: check=True to raise on error +result = subprocess.run( + ["git", "status"], + check=True, + capture_output=True, + text=True +) +print(result.stdout) + +# ❌ WRONG: No check - silently ignores errors +result = subprocess.run(["git", "status"]) +``` + +## Complete Subprocess Example + +```python +def run_git_command(args: list[str], cwd: Path | None = None) -> str: + """Run a git command and return output.""" + try: + result = subprocess.run( + ["git"] + args, + check=True, # Raise on non-zero exit + capture_output=True, # Capture stdout/stderr + text=True, # Return strings, not bytes + cwd=cwd # Working directory + ) + return result.stdout.strip() + except subprocess.CalledProcessError as e: + # Error boundary - add context + raise RuntimeError(f"Git command failed: {e.stderr}") from e +``` + +## Error Handling + +```python +try: + result = subprocess.run( + ["make", "test"], + check=True, + capture_output=True, + text=True + ) +except subprocess.CalledProcessError as e: + # Access error details + print(f"Command: {e.cmd}") + print(f"Exit code: {e.returncode}") + print(f"Stdout: {e.stdout}") + print(f"Stderr: {e.stderr}") + raise +``` + +## Common Patterns + +```python +# Silent execution (no output) +subprocess.run(["git", "fetch"], check=True, capture_output=True) + +# Stream output in real-time +process = subprocess.Popen( + ["pytest", "-v"], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True +) +for line in process.stdout: + print(line, end="") +process.wait() +if process.returncode != 0: + raise subprocess.CalledProcessError(process.returncode, process.args) + +# With timeout +try: + subprocess.run(["long-command"], check=True, timeout=30) +except subprocess.TimeoutExpired: + print("Command timed out") +``` + +## Key Takeaways + +1. **Always check=True**: Ensure errors are not silently ignored +2. **Capture output**: Use `capture_output=True` for stdout/stderr +3. **Text mode**: Use `text=True` for string output +4. **Error context**: Wrap in try/except at boundaries +5. **Timeout safety**: Set timeout for long-running commands diff --git a/.agents/skills/dignified-python/versions/python-3.10.md b/.agents/skills/dignified-python/versions/python-3.10.md new file mode 100644 index 0000000..5e9cb5c --- /dev/null +++ b/.agents/skills/dignified-python/versions/python-3.10.md @@ -0,0 +1,520 @@ +--- +--- + +# Type Annotations - Python 3.10 + +This document provides complete, canonical type annotation guidance for Python 3.10. This is the +baseline for modern Python type syntax. + +## Overview + +Python 3.10 introduced major improvements to type annotation syntax through PEP 604 (union types via +`|`) and PEP 585 (generic types in standard collections). These features eliminated the need for +most `typing` module imports and made type annotations more concise and readable. + +**What's new in 3.10:** + +- Union types with `|` operator (PEP 604) +- Built-in generic types: `list[T]`, `dict[K, V]`, etc. (PEP 585) +- No more need for `List`, `Dict`, `Union`, `Optional` from typing + +**What you need from typing module:** + +- `TypeVar` for generic functions/classes +- `Protocol` for structural typing (rare - prefer ABC) +- `TYPE_CHECKING` for conditional imports +- `Any` (use sparingly) + +## Complete Type Annotation Syntax for Python 3.10 + +### Basic Collection Types + +✅ **PREFERRED** - Use built-in generic types: + +```python +names: list[str] = [] +mapping: dict[str, int] = {} +unique_ids: set[str] = set() +coordinates: tuple[int, int] = (0, 0) +``` + +❌ **WRONG** - Don't use typing module equivalents: + +```python +from typing import List, Dict, Set, Tuple # Don't do this +names: List[str] = [] +mapping: Dict[str, int] = {} +``` + +**Why**: Built-in types are more concise, don't require imports, and are the modern Python standard. + +### Union Types + +✅ **PREFERRED** - Use `|` operator: + +```python +def process(value: str | int) -> str: + return str(value) + +def find_config(name: str) -> dict[str, str] | dict[str, int]: + ... + +# Multiple unions +def parse(input: str | int | float) -> str: + return str(input) +``` + +❌ **WRONG** - Don't use `typing.Union`: + +```python +from typing import Union +def process(value: Union[str, int]) -> str: # Don't do this + ... +``` + +### Optional Types + +✅ **PREFERRED** - Use `X | None`: + +```python +def find_user(id: str) -> User | None: + """Returns user or None if not found.""" + if id in users: + return users[id] + return None + +def get_config(key: str) -> str | None: + return config.get(key) +``` + +❌ **WRONG** - Don't use `typing.Optional`: + +```python +from typing import Optional +def find_user(id: str) -> Optional[User]: # Don't do this + ... +``` + +### Generic Functions with TypeVar + +✅ **PREFERRED** - Use TypeVar for generic functions: + +```python +from typing import TypeVar + +T = TypeVar("T") + +def first(items: list[T]) -> T | None: + """Return first item or None if empty.""" + if not items: + return None + return items[0] + +def identity(value: T) -> T: + """Return the value unchanged.""" + return value +``` + +**Note**: This is the standard way in Python 3.10. Python 3.12 introduces better syntax (PEP 695). + +### Generic Classes + +✅ **PREFERRED** - Use Generic with TypeVar: + +```python +from typing import Generic, TypeVar + +T = TypeVar("T") + +class Stack(Generic[T]): + """A generic stack data structure.""" + + def __init__(self) -> None: + self._items: list[T] = [] + + def push(self, item: T) -> None: + self._items.append(item) + + def pop(self) -> T | None: + if not self._items: + return None + return self._items.pop() + +# Usage +int_stack = Stack[int]() +int_stack.push(42) +``` + +**Note**: Python 3.12 introduces cleaner syntax for this pattern. + +### Constrained and Bounded TypeVars + +✅ **Use TypeVar constraints when needed**: + +```python +from typing import TypeVar + +# Constrained to specific types +Numeric = TypeVar("Numeric", int, float) + +def add(a: Numeric, b: Numeric) -> Numeric: + return a + b + +# Bounded to base class +T = TypeVar("T", bound=BaseClass) + +def process(obj: T) -> T: + return obj +``` + +### Callable Types + +✅ **PREFERRED** - Use `collections.abc.Callable`: + +```python +from collections.abc import Callable + +# Function that takes int, returns str +processor: Callable[[int], str] = str + +# Function with no args, returns None +callback: Callable[[], None] = lambda: None + +# Function with multiple args +validator: Callable[[str, int], bool] = lambda s, i: len(s) > i +``` + +### Type Aliases + +✅ **Use simple assignment for type aliases**: + +```python +# Simple alias +UserId = str +Config = dict[str, str | int | bool] + +# Complex nested type +JsonValue = dict[str, "JsonValue"] | list["JsonValue"] | str | int | float | bool | None + +def load_config() -> Config: + return {"host": "localhost", "port": 8080} +``` + +**Note**: Python 3.12 introduces `type` statement for better alias support. + +### when from **future** import annotations is Needed + +Use `from __future__ import annotations` when you encounter: + +**Forward references** (class referencing itself): + +```python +from __future__ import annotations + +class Node: + def __init__(self, value: int, parent: Node | None = None): + self.value = value + self.parent = parent +``` + +**Circular type imports**: + +```python +# a.py +from __future__ import annotations +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from b import B + +class A: + def method(self) -> B: + ... +``` + +**Complex recursive types**: + +```python +from __future__ import annotations + +JsonValue = dict[str, JsonValue] | list[JsonValue] | str | int | float | bool | None +``` + +### Interfaces: ABC vs Protocol + +✅ **PREFERRED** - Use ABC for interfaces: + +```python +from abc import ABC, abstractmethod + +class Repository(ABC): + @abstractmethod + def get(self, id: str) -> User | None: + """Get user by ID.""" + + @abstractmethod + def save(self, user: User) -> None: + """Save user.""" +``` + +🟡 **VALID** - Use Protocol only for structural typing: + +```python +from typing import Protocol + +class Drawable(Protocol): + def draw(self) -> None: ... + +# Any object with draw() method matches +def render(obj: Drawable) -> None: + obj.draw() +``` + +**Dignified Python prefers ABC** because it makes inheritance and intent explicit. + +## Complete Examples + +### Repository Pattern + +```python +from abc import ABC, abstractmethod + +class Repository(ABC): + """Abstract base class for data repositories.""" + + @abstractmethod + def get(self, id: str) -> dict[str, str] | None: + """Get entity by ID.""" + + @abstractmethod + def save(self, entity: dict[str, str]) -> None: + """Save entity.""" + + @abstractmethod + def delete(self, id: str) -> bool: + """Delete entity, return success.""" + +class UserRepository(Repository): + def __init__(self) -> None: + self._users: dict[str, dict[str, str]] = {} + + def get(self, id: str) -> dict[str, str] | None: + return self._users.get(id) + + def save(self, entity: dict[str, str]) -> None: + if "id" not in entity: + raise ValueError("Entity must have id") + self._users[entity["id"]] = entity + + def delete(self, id: str) -> bool: + if id in self._users: + del self._users[id] + return True + return False +``` + +### Generic Data Structures + +```python +from typing import Generic, TypeVar + +T = TypeVar("T") + +class Node(Generic[T]): + """A node in a tree structure.""" + + def __init__(self, value: T, children: list[Node[T]] | None = None) -> None: + self.value = value + self.children = children or [] + + def add_child(self, child: Node[T]) -> None: + self.children.append(child) + + def find(self, predicate: Callable[[T], bool]) -> Node[T] | None: + """Find first node matching predicate.""" + if predicate(self.value): + return self + for child in self.children: + result = child.find(predicate) + if result: + return result + return None + +# Usage +from collections.abc import Callable + +root = Node[int](1) +root.add_child(Node[int](2)) +root.add_child(Node[int](3)) +``` + +### Configuration Management + +```python +from dataclasses import dataclass + +@dataclass(frozen=True) +class DatabaseConfig: + host: str + port: int + username: str + password: str | None = None + ssl_enabled: bool = False + +@dataclass(frozen=True) +class AppConfig: + app_name: str + debug_mode: bool + database: DatabaseConfig + feature_flags: dict[str, bool] + +def load_config(path: str) -> AppConfig: + """Load application configuration from file.""" + import json + from pathlib import Path + + config_path = Path(path) + if not config_path.exists(): + raise FileNotFoundError(f"Config not found: {path}") + + data: dict[str, str | int | bool | dict[str, str | int | bool]] = json.loads( + config_path.read_text(encoding="utf-8") + ) + + # Parse and validate... + return AppConfig(...) +``` + +### API Client with Error Handling + +```python +from collections.abc import Callable +from typing import TypeVar + +T = TypeVar("T") + +class ApiResponse(Generic[T]): + """Container for API response with data or error.""" + + def __init__(self, data: T | None = None, error: str | None = None) -> None: + self.data = data + self.error = error + + def is_success(self) -> bool: + return self.error is None + + def map(self, func: Callable[[T], U]) -> ApiResponse[U]: + """Transform successful response data.""" + if self.is_success() and self.data is not None: + return ApiResponse(data=func(self.data)) + return ApiResponse(error=self.error) + +U = TypeVar("U") + +def fetch_user(id: str) -> ApiResponse[dict[str, str]]: + """Fetch user from API.""" + # Implementation... + return ApiResponse(data={"id": id, "name": "Alice"}) +``` + +## Type Checking Rules + +### What to Type + +✅ **MUST type**: + +- All public function parameters (except `self`, `cls`) +- All public function return values +- All class attributes (public and private) +- Module-level constants + +🟡 **SHOULD type**: + +- Internal function signatures +- Complex local variables + +🟢 **MAY skip**: + +- Simple local variables where type is obvious (`count = 0`) +- Lambda parameters in short inline lambdas +- Loop variables in short comprehensions + +### Running Type Checker + +```bash +uv run ty check +``` + +All code should pass type checking without errors. + +### Type Checking Configuration + +Configure ty in `pyproject.toml`: + +```toml +[tool.ty.environment] +python-version = "3.10" +``` + +## Common Patterns + +### Checking for None + +✅ **CORRECT** - Check before use: + +```python +def process_user(user: User | None) -> str: + if user is None: + return "No user" + return user.name +``` + +### Dict.get() with Type Safety + +✅ **CORRECT** - Handle None case: + +```python +def get_port(config: dict[str, int]) -> int: + port = config.get("port") + if port is None: + return 8080 + return port +``` + +### List Operations + +✅ **CORRECT** - Check before accessing: + +```python +def first_or_default(items: list[str], default: str) -> str: + if not items: + return default + return items[0] +``` + +## Migration from Python 3.9 + +If upgrading from Python 3.9, apply these changes: + +1. **Replace typing module types**: + - `List[X]` → `list[X]` + - `Dict[K, V]` → `dict[K, V]` + - `Set[X]` → `set[X]` + - `Tuple[X, Y]` → `tuple[X, Y]` + - `Union[X, Y]` → `X | Y` + - `Optional[X]` → `X | None` + +2. **Add future annotations if needed**: + - Add `from __future__ import annotations` for forward references + - Add for circular imports with `TYPE_CHECKING` + +3. **Remove unnecessary imports**: + - Remove `from typing import List, Dict, Optional, Union` + - Keep only `TypeVar`, `Generic`, `Protocol`, `TYPE_CHECKING`, `Any` + +## References + +- [PEP 604: Union Types](https://peps.python.org/pep-0604/) +- [PEP 585: Type Hinting Generics In Standard Collections](https://peps.python.org/pep-0585/) +- [PEP 563: Postponed Evaluation of Annotations](https://peps.python.org/pep-0563/) +- [Python 3.10 What's New - Type Hints](https://docs.python.org/3.10/whatsnew/3.10.html) diff --git a/.agents/skills/dignified-python/versions/python-3.11.md b/.agents/skills/dignified-python/versions/python-3.11.md new file mode 100644 index 0000000..8d87ce9 --- /dev/null +++ b/.agents/skills/dignified-python/versions/python-3.11.md @@ -0,0 +1,538 @@ +--- +--- + +# Type Annotations - Python 3.11 + +This document provides complete, canonical type annotation guidance for Python 3.11. + +## Overview + +Python 3.11 builds on 3.10's type syntax with the addition of the `Self` type (PEP 673), making +method chaining and builder patterns significantly cleaner. All modern syntax from 3.10 continues to +work. + +**What's new in 3.11:** + +- `Self` type for self-returning methods (PEP 673) +- Variadic generics with TypeVarTuple (PEP 646) +- Significantly improved error messages + +**Available from 3.10:** + +- Built-in generic types: `list[T]`, `dict[K, V]`, etc. (PEP 585) +- Union types with `|` operator (PEP 604) +- Optional with `X | None` + +**What you need from typing module:** + +- `Self` for self-returning methods (NEW) +- `TypeVar` for generic functions/classes +- `Generic` for generic classes +- `Protocol` for structural typing (rare - prefer ABC) +- `TYPE_CHECKING` for conditional imports +- `Any` (use sparingly) + +## Complete Type Annotation Syntax for Python 3.11 + +### Basic Collection Types + +✅ **PREFERRED** - Use built-in generic types: + +```python +names: list[str] = [] +mapping: dict[str, int] = {} +unique_ids: set[str] = set() +coordinates: tuple[int, int] = (0, 0) +``` + +❌ **WRONG** - Don't use typing module equivalents: + +```python +from typing import List, Dict, Set, Tuple # Don't do this +names: List[str] = [] +``` + +### Union Types + +✅ **PREFERRED** - Use `|` operator: + +```python +def process(value: str | int) -> str: + return str(value) + +def find_config(name: str) -> dict[str, str] | dict[str, int]: + ... + +# Multiple unions +def parse(input: str | int | float) -> str: + return str(input) +``` + +❌ **WRONG** - Don't use `typing.Union`: + +```python +from typing import Union +def process(value: Union[str, int]) -> str: # Don't do this + ... +``` + +### Optional Types + +✅ **PREFERRED** - Use `X | None`: + +```python +def find_user(id: str) -> User | None: + """Returns user or None if not found.""" + if id in users: + return users[id] + return None +``` + +❌ **WRONG** - Don't use `typing.Optional`: + +```python +from typing import Optional +def find_user(id: str) -> Optional[User]: # Don't do this + ... +``` + +### Self Type for Self-Returning Methods (NEW in 3.11) + +✅ **PREFERRED** - Use Self for methods that return the instance: + +```python +from typing import Self + +class Builder: + def set_name(self, name: str) -> Self: + self.name = name + return self + + def set_value(self, value: int) -> Self: + self.value = value + return self + +# Usage with type safety +builder = Builder().set_name("app").set_value(42) +``` + +❌ **WRONG** - Don't use bound TypeVar anymore: + +```python +from typing import TypeVar + +T = TypeVar("T", bound="Builder") + +class Builder: + def set_name(self: T, name: str) -> T: # Don't do this + ... +``` + +**When to use Self:** + +- Methods that return `self` +- Builder pattern methods +- Fluent interfaces with method chaining +- Factory classmethods + +**Self in classmethod:** + +```python +from typing import Self + +class Config: + def __init__(self, data: dict[str, str]) -> None: + self.data = data + + @classmethod + def from_file(cls, path: str) -> Self: + """Load config from file.""" + import json + with open(path, encoding="utf-8") as f: + data = json.load(f) + return cls(data) +``` + +### Generic Functions with TypeVar + +✅ **PREFERRED** - Use TypeVar for generic functions: + +```python +from typing import TypeVar + +T = TypeVar("T") + +def first(items: list[T]) -> T | None: + """Return first item or None if empty.""" + if not items: + return None + return items[0] + +def identity(value: T) -> T: + return value +``` + +**Note**: Python 3.12 introduces better syntax (PEP 695) for this pattern. + +### Generic Classes + +✅ **PREFERRED** - Use Generic with TypeVar: + +```python +from typing import Generic, TypeVar + +T = TypeVar("T") + +class Stack(Generic[T]): + """A generic stack data structure.""" + + def __init__(self) -> None: + self._items: list[T] = [] + + def push(self, item: T) -> Self: # Can combine with Self! + self._items.append(item) + return self + + def pop(self) -> T | None: + if not self._items: + return None + return self._items.pop() + +# Usage +int_stack = Stack[int]() +int_stack.push(42).push(43) # Method chaining works! +``` + +**Note**: Python 3.12 introduces cleaner syntax for generic classes. + +### Constrained and Bounded TypeVars + +✅ **Use TypeVar constraints when needed**: + +```python +from typing import TypeVar + +# Constrained to specific types +Numeric = TypeVar("Numeric", int, float) + +def add(a: Numeric, b: Numeric) -> Numeric: + return a + b + +# Bounded to base class +T = TypeVar("T", bound=BaseClass) + +def process(obj: T) -> T: + return obj +``` + +### Callable Types + +✅ **PREFERRED** - Use `collections.abc.Callable`: + +```python +from collections.abc import Callable + +# Function that takes int, returns str +processor: Callable[[int], str] = str + +# Function with no args, returns None +callback: Callable[[], None] = lambda: None + +# Function with multiple args +validator: Callable[[str, int], bool] = lambda s, i: len(s) > i +``` + +### Type Aliases + +✅ **Use simple assignment for type aliases**: + +```python +# Simple alias +UserId = str +Config = dict[str, str | int | bool] + +# Complex nested type +JsonValue = dict[str, "JsonValue"] | list["JsonValue"] | str | int | float | bool | None + +def load_config() -> Config: + return {"host": "localhost", "port": 8080} +``` + +**Note**: Python 3.12 introduces `type` statement for better alias support. + +### When from **future** import annotations is Needed + +Use `from __future__ import annotations` when you encounter: + +**Forward references** (class referencing itself): + +```python +from __future__ import annotations + +class Node: + def __init__(self, value: int, parent: Node | None = None): + self.value = value + self.parent = parent +``` + +**Circular type imports**: + +```python +# a.py +from __future__ import annotations +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from b import B + +class A: + def method(self) -> B: + ... +``` + +**Complex recursive types**: + +```python +from __future__ import annotations + +JsonValue = dict[str, JsonValue] | list[JsonValue] | str | int | float | bool | None +``` + +### Interfaces: ABC vs Protocol + +✅ **PREFERRED** - Use ABC for interfaces: + +```python +from abc import ABC, abstractmethod + +class Repository(ABC): + @abstractmethod + def get(self, id: str) -> User | None: + """Get user by ID.""" + + @abstractmethod + def save(self, user: User) -> None: + """Save user.""" +``` + +🟡 **VALID** - Use Protocol only for structural typing: + +```python +from typing import Protocol + +class Drawable(Protocol): + def draw(self) -> None: ... + +# Any object with draw() method matches +def render(obj: Drawable) -> None: + obj.draw() +``` + +**Dignified Python prefers ABC** because it makes inheritance and intent explicit. + +## Complete Examples + +### Builder Pattern with Self + +```python +from typing import Self + +class QueryBuilder: + """SQL query builder with fluent interface.""" + + def __init__(self) -> None: + self._select: list[str] = ["*"] + self._from: str | None = None + self._where: list[str] = [] + self._limit: int | None = None + + def select(self, *columns: str) -> Self: + """Specify columns to select.""" + self._select = list(columns) + return self + + def from_table(self, table: str) -> Self: + """Specify table to query.""" + self._from = table + return self + + def where(self, condition: str) -> Self: + """Add WHERE condition.""" + self._where.append(condition) + return self + + def limit(self, n: int) -> Self: + """Set LIMIT.""" + self._limit = n + return self + + def build(self) -> str: + """Build final SQL query.""" + if not self._from: + raise ValueError("FROM table not specified") + + parts = [f"SELECT {', '.join(self._select)}"] + parts.append(f"FROM {self._from}") + + if self._where: + parts.append(f"WHERE {' AND '.join(self._where)}") + + if self._limit: + parts.append(f"LIMIT {self._limit}") + + return " ".join(parts) + +# Usage with type-safe method chaining +query = ( + QueryBuilder() + .select("id", "name", "email") + .from_table("users") + .where("active = true") + .where("age > 18") + .limit(10) + .build() +) +``` + +### Factory Methods with Self + +```python +from typing import Self +from pathlib import Path +import json + +class Config: + """Application configuration with multiple factory methods.""" + + def __init__(self, data: dict[str, str | int]) -> None: + self.data = data + + @classmethod + def from_json(cls, path: Path) -> Self: + """Load configuration from JSON file.""" + if not path.exists(): + raise FileNotFoundError(f"Config not found: {path}") + + with path.open(encoding="utf-8") as f: + data = json.load(f) + return cls(data) + + @classmethod + def from_env(cls) -> Self: + """Load configuration from environment variables.""" + import os + data = { + k.lower(): v + for k, v in os.environ.items() + if k.startswith("APP_") + } + return cls(data) + + @classmethod + def default(cls) -> Self: + """Create default configuration.""" + return cls({"host": "localhost", "port": 8080}) + + def with_override(self, key: str, value: str | int) -> Self: + """Return new config with overridden value.""" + new_data = self.data.copy() + new_data[key] = value + return type(self)(new_data) + +# All factory methods return correct type +config = Config.from_json(Path("config.json")) +dev_config = config.with_override("debug", True) +``` + +## Type Checking Rules + +### What to Type + +✅ **MUST type**: + +- All public function parameters (except `self`, `cls`) +- All public function return values +- All class attributes (public and private) +- Module-level constants + +🟡 **SHOULD type**: + +- Internal function signatures +- Complex local variables + +🟢 **MAY skip**: + +- Simple local variables where type is obvious (`count = 0`) +- Lambda parameters in short inline lambdas +- Loop variables in short comprehensions + +### Running Type Checker + +```bash +uv run ty check +``` + +All code should pass type checking without errors. + +### Type Checking Configuration + +Configure ty in `pyproject.toml`: + +```toml +[tool.ty.environment] +python-version = "3.11" +``` + +## Common Patterns + +### Checking for None + +✅ **CORRECT** - Check before use: + +```python +def process_user(user: User | None) -> str: + if user is None: + return "No user" + return user.name +``` + +### Dict.get() with Type Safety + +✅ **CORRECT** - Handle None case: + +```python +def get_port(config: dict[str, int]) -> int: + port = config.get("port") + if port is None: + return 8080 + return port +``` + +### List Operations + +✅ **CORRECT** - Check before accessing: + +```python +def first_or_default(items: list[str], default: str) -> str: + if not items: + return default + return items[0] +``` + +## Migration from Python 3.10 + +If upgrading from Python 3.10: + +1. **Replace bound TypeVar with Self** for self-returning methods: + - Old: `T = TypeVar("T", bound="ClassName")` + - New: `from typing import Self` and use `-> Self` + +2. **Enjoy improved error messages** (no code changes needed) + +3. **All existing 3.10 syntax continues to work** + +## References + +- [PEP 673: Self Type](https://peps.python.org/pep-0673/) +- [PEP 646: Variadic Generics](https://peps.python.org/pep-0646/) +- [Python 3.11 What's New](https://docs.python.org/3.11/whatsnew/3.11.html) diff --git a/.agents/skills/dignified-python/versions/python-3.12.md b/.agents/skills/dignified-python/versions/python-3.12.md new file mode 100644 index 0000000..c7940d5 --- /dev/null +++ b/.agents/skills/dignified-python/versions/python-3.12.md @@ -0,0 +1,664 @@ +--- +--- + +# Type Annotations - Python 3.12 + +This document provides complete, canonical type annotation guidance for Python 3.12. + +## Overview + +Python 3.12 introduces PEP 695, a major syntactic improvement for generic types. The new type +parameter syntax makes generic functions and classes significantly more readable. All syntax from +3.10 and 3.11 continues to work. + +**What's new in 3.12:** + +- PEP 695 type parameter syntax: `def func[T](x: T) -> T` +- `type` statement for better type aliases +- Cleaner generic class syntax + +**Available from 3.11:** + +- `Self` type for self-returning methods + +**Available from 3.10:** + +- Built-in generic types: `list[T]`, `dict[K, V]`, etc. +- Union types with `|` operator +- Optional with `X | None` + +**What you need from typing module:** + +- `Self` for self-returning methods +- `TypeVar` only for constrained/bounded generics +- `Protocol` for structural typing (rare - prefer ABC) +- `TYPE_CHECKING` for conditional imports +- `Any` (use sparingly) + +## Complete Type Annotation Syntax for Python 3.12 + +### Basic Collection Types + +✅ **PREFERRED** - Use built-in generic types: + +```python +names: list[str] = [] +mapping: dict[str, int] = {} +unique_ids: set[str] = set() +coordinates: tuple[int, int] = (0, 0) +``` + +❌ **WRONG** - Don't use typing module equivalents: + +```python +from typing import List, Dict, Set, Tuple # Don't do this +names: List[str] = [] +``` + +### Union Types + +✅ **PREFERRED** - Use `|` operator: + +```python +def process(value: str | int) -> str: + return str(value) + +def find_config(name: str) -> dict[str, str] | dict[str, int]: + ... + +# Multiple unions +def parse(input: str | int | float) -> str: + return str(input) +``` + +❌ **WRONG** - Don't use `typing.Union`: + +```python +from typing import Union +def process(value: Union[str, int]) -> str: # Don't do this + ... +``` + +### Optional Types + +✅ **PREFERRED** - Use `X | None`: + +```python +def find_user(id: str) -> User | None: + """Returns user or None if not found.""" + if id in users: + return users[id] + return None +``` + +❌ **WRONG** - Don't use `typing.Optional`: + +```python +from typing import Optional +def find_user(id: str) -> Optional[User]: # Don't do this + ... +``` + +### Self Type for Self-Returning Methods + +✅ **PREFERRED** - Use Self for methods that return the instance: + +```python +from typing import Self + +class Builder: + def set_name(self, name: str) -> Self: + self.name = name + return self + + def set_value(self, value: int) -> Self: + self.value = value + return self +``` + +### Generic Functions with PEP 695 (NEW in 3.12) + +✅ **PREFERRED** - Use PEP 695 type parameter syntax: + +```python +def first[T](items: list[T]) -> T | None: + """Return first item or None if empty.""" + if not items: + return None + return items[0] + +def identity[T](value: T) -> T: + """Return value unchanged.""" + return value + +# Multiple type parameters +def zip_dicts[K, V](keys: list[K], values: list[V]) -> dict[K, V]: + """Create dict from separate key and value lists.""" + return dict(zip(keys, values)) +``` + +🟡 **VALID** - TypeVar still works: + +```python +from typing import TypeVar + +T = TypeVar("T") + +def first(items: list[T]) -> T | None: + if not items: + return None + return items[0] +``` + +**Note**: Prefer PEP 695 syntax for simple generics. TypeVar is still needed for constraints/bounds. + +### Generic Classes with PEP 695 (NEW in 3.12) + +✅ **PREFERRED** - Use PEP 695 class syntax: + +```python +class Stack[T]: + """A generic stack data structure.""" + + def __init__(self) -> None: + self._items: list[T] = [] + + def push(self, item: T) -> Self: + self._items.append(item) + return self + + def pop(self) -> T | None: + if not self._items: + return None + return self._items.pop() + +# Usage +int_stack = Stack[int]() +int_stack.push(42).push(43) +``` + +🟡 **VALID** - Generic with TypeVar still works: + +```python +from typing import Generic, TypeVar + +T = TypeVar("T") + +class Stack(Generic[T]): + def __init__(self) -> None: + self._items: list[T] = [] + # ... rest of implementation +``` + +**Note**: PEP 695 is cleaner - no imports needed, type parameter scope is local to class. + +### Type Parameter Bounds + +✅ **Use bounds with PEP 695**: + +```python +class Comparable: + def compare(self, other: object) -> int: + ... + +def max_value[T: Comparable](items: list[T]) -> T: + """Get maximum value from comparable items.""" + return max(items, key=lambda x: x) +``` + +### Constrained TypeVars (Still Use TypeVar) + +✅ **Use TypeVar for specific type constraints**: + +```python +from typing import TypeVar + +# Constrained to specific types - must use TypeVar +Numeric = TypeVar("Numeric", int, float) + +def add(a: Numeric, b: Numeric) -> Numeric: + return a + b +``` + +❌ **WRONG** - PEP 695 doesn't support constraints: + +```python +# This doesn't constrain to int|float +def add[Numeric](a: Numeric, b: Numeric) -> Numeric: + return a + b +``` + +### Type Aliases with type Statement (NEW in 3.12) + +✅ **PREFERRED** - Use `type` statement: + +```python +# Simple alias +type UserId = str +type Config = dict[str, str | int | bool] + +# Generic type alias +type Result[T] = tuple[T, str | None] + +def process(value: str) -> Result[int]: + try: + return (int(value), None) + except ValueError as e: + return (0, str(e)) +``` + +🟡 **VALID** - Simple assignment still works: + +```python +UserId = str # Still valid +Config = dict[str, str | int | bool] # Still valid +``` + +**Note**: `type` statement is more explicit and works better with generics. + +### Callable Types + +✅ **PREFERRED** - Use `collections.abc.Callable`: + +```python +from collections.abc import Callable + +# Function that takes int, returns str +processor: Callable[[int], str] = str + +# Function with no args, returns None +callback: Callable[[], None] = lambda: None + +# Function with multiple args +validator: Callable[[str, int], bool] = lambda s, i: len(s) > i +``` + +### When from **future** import annotations is Needed + +Use `from __future__ import annotations` when you encounter: + +**Forward references** (class referencing itself): + +```python +from __future__ import annotations + +class Node: + def __init__(self, value: int, parent: Node | None = None): + self.value = value + self.parent = parent +``` + +**Circular type imports**: + +```python +# a.py +from __future__ import annotations +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from b import B + +class A: + def method(self) -> B: + ... +``` + +**Complex recursive types**: + +```python +from __future__ import annotations + +type JsonValue = dict[str, JsonValue] | list[JsonValue] | str | int | float | bool | None +``` + +### Interfaces: ABC vs Protocol + +✅ **PREFERRED** - Use ABC for interfaces: + +```python +from abc import ABC, abstractmethod + +class Repository(ABC): + @abstractmethod + def get(self, id: str) -> User | None: + """Get user by ID.""" + + @abstractmethod + def save(self, user: User) -> None: + """Save user.""" +``` + +🟡 **VALID** - Use Protocol only for structural typing: + +```python +from typing import Protocol + +class Drawable(Protocol): + def draw(self) -> None: ... + +def render(obj: Drawable) -> None: + obj.draw() +``` + +**Dignified Python prefers ABC** because it makes inheritance and intent explicit. + +## Complete Examples + +### Generic Stack with PEP 695 + +```python +from typing import Self + +class Stack[T]: + """Type-safe stack with PEP 695 syntax.""" + + def __init__(self) -> None: + self._items: list[T] = [] + + def push(self, item: T) -> Self: + """Push item and return self for chaining.""" + self._items.append(item) + return self + + def pop(self) -> T | None: + """Pop item or return None if empty.""" + if not self._items: + return None + return self._items.pop() + + def peek(self) -> T | None: + """Peek at top item without removing.""" + if not self._items: + return None + return self._items[-1] + + def is_empty(self) -> bool: + """Check if stack is empty.""" + return len(self._items) == 0 + +# Usage +numbers = Stack[int]() +numbers.push(1).push(2).push(3) +top = numbers.pop() # Type checker knows this is int | None +``` + +### Generic Repository with PEP 695 + +```python +from abc import ABC, abstractmethod +from typing import Self + +class Repository[T]: + """Abstract repository with generic type parameter.""" + + @abstractmethod + def get(self, id: str) -> T | None: + """Get entity by ID.""" + + @abstractmethod + def save(self, entity: T) -> Self: + """Save entity, return self for chaining.""" + + @abstractmethod + def delete(self, id: str) -> bool: + """Delete entity, return success.""" + + def get_or_fail(self, id: str) -> T: + """Get entity or raise error.""" + entity = self.get(id) + if entity is None: + raise ValueError(f"Entity not found: {id}") + return entity + +class InMemoryRepository[T](Repository[T]): + """In-memory repository implementation.""" + + def __init__(self) -> None: + self._storage: dict[str, T] = {} + + def get(self, id: str) -> T | None: + return self._storage.get(id) + + def save(self, entity: T) -> Self: + # Assume entity has 'id' attribute + entity_id = str(getattr(entity, "id", id(entity))) + self._storage[entity_id] = entity + return self + + def delete(self, id: str) -> bool: + if id in self._storage: + del self._storage[id] + return True + return False + +# Usage +from dataclasses import dataclass + +@dataclass +class User: + id: str + name: str + +repo = InMemoryRepository[User]() +repo.save(User("1", "Alice")).save(User("2", "Bob")) +user = repo.get("1") # Type: User | None +``` + +### Type Aliases with type Statement + +```python +# Simple aliases +type UserId = str +type ErrorMessage = str + +# Complex nested types +type JsonValue = dict[str, JsonValue] | list[JsonValue] | str | int | float | bool | None + +# Generic type aliases +type Result[T] = tuple[T, ErrorMessage | None] +type AsyncResult[T] = tuple[T | None, ErrorMessage | None] + +def parse_int(value: str) -> Result[int]: + """Parse string to int, return result with optional error.""" + try: + return (int(value), None) + except ValueError as e: + return (0, str(e)) + +def fetch_user(id: UserId) -> AsyncResult[dict[str, str]]: + """Fetch user data asynchronously.""" + # Implementation... + return ({"id": id, "name": "Alice"}, None) +``` + +### Builder Pattern with Self and PEP 695 + +```python +from typing import Self + +class QueryBuilder[T]: + """Generic query builder with fluent interface.""" + + def __init__(self, result_type: type[T]) -> None: + self._result_type = result_type + self._filters: list[str] = [] + self._limit: int | None = None + + def filter(self, condition: str) -> Self: + """Add filter condition.""" + self._filters.append(condition) + return self + + def limit(self, n: int) -> Self: + """Set result limit.""" + self._limit = n + return self + + def build(self) -> str: + """Build query string.""" + query = " AND ".join(self._filters) + if self._limit: + query += f" LIMIT {self._limit}" + return query + +# Usage +@dataclass +class User: + name: str + age: int + +builder = QueryBuilder[User](User) +query = ( + builder + .filter("active = true") + .filter("age > 18") + .limit(10) + .build() +) +``` + +### Generic Function Utilities + +```python +def map_list[T, U](items: list[T], func: Callable[[T], U]) -> list[U]: + """Map function over list items.""" + from collections.abc import Callable + return [func(item) for item in items] + +def filter_list[T](items: list[T], predicate: Callable[[T], bool]) -> list[T]: + """Filter list by predicate.""" + from collections.abc import Callable + return [item for item in items if predicate(item)] + +def reduce_list[T, U]( + items: list[T], + func: Callable[[U, T], U], + initial: U, +) -> U: + """Reduce list to single value.""" + from collections.abc import Callable + result = initial + for item in items: + result = func(result, item) + return result + +# Usage +numbers = [1, 2, 3, 4, 5] +doubled = map_list(numbers, lambda x: x * 2) # list[int] +evens = filter_list(numbers, lambda x: x % 2 == 0) # list[int] +sum_val = reduce_list(numbers, lambda acc, x: acc + x, 0) # int +``` + +## Type Checking Rules + +### What to Type + +✅ **MUST type**: + +- All public function parameters (except `self`, `cls`) +- All public function return values +- All class attributes (public and private) +- Module-level constants + +🟡 **SHOULD type**: + +- Internal function signatures +- Complex local variables + +🟢 **MAY skip**: + +- Simple local variables where type is obvious (`count = 0`) +- Lambda parameters in short inline lambdas +- Loop variables in short comprehensions + +### Running Type Checker + +```bash +uv run ty check +``` + +All code should pass type checking without errors. + +### Type Checking Configuration + +Configure ty in `pyproject.toml`: + +```toml +[tool.ty.environment] +python-version = "3.12" +``` + +## Common Patterns + +### Checking for None + +✅ **CORRECT** - Check before use: + +```python +def process_user(user: User | None) -> str: + if user is None: + return "No user" + return user.name +``` + +### Dict.get() with Type Safety + +✅ **CORRECT** - Handle None case: + +```python +def get_port(config: dict[str, int]) -> int: + port = config.get("port") + if port is None: + return 8080 + return port +``` + +### List Operations + +✅ **CORRECT** - Check before accessing: + +```python +def first_or_default[T](items: list[T], default: T) -> T: + if not items: + return default + return items[0] +``` + +## When to Use PEP 695 vs TypeVar + +**Use PEP 695 for**: + +- Simple generic functions (no constraints/bounds) +- Simple generic classes +- Most common generic use cases +- New code + +**Still use TypeVar for**: + +- Constrained type variables: `TypeVar("T", str, bytes)` +- Bound type variables with complex bounds +- Covariant/contravariant type variables +- Reusing same TypeVar across multiple functions + +## Migration from Python 3.11 + +If upgrading from Python 3.11: + +1. **Consider migrating to PEP 695 syntax**: + - `TypeVar` + `def func(x: T) -> T` → `def func[T](x: T) -> T` + - `Generic[T]` + `class C(Generic[T])` → `class C[T]` + +2. **Consider using `type` statement for aliases**: + - `Config = dict[str, str]` → `type Config = dict[str, str]` + +3. **Keep TypeVar for constraints**: + - `TypeVar` with constraints still needed + +4. **All existing 3.11 syntax continues to work**: + - `Self` type still preferred + - Union with `|` still preferred + +## References + +- [PEP 695: Type Parameter Syntax](https://peps.python.org/pep-0695/) +- [Python 3.12 What's New - Type Hints](https://docs.python.org/3.12/whatsnew/3.12.html) diff --git a/.agents/skills/dignified-python/versions/python-3.13.md b/.agents/skills/dignified-python/versions/python-3.13.md new file mode 100644 index 0000000..42d35a5 --- /dev/null +++ b/.agents/skills/dignified-python/versions/python-3.13.md @@ -0,0 +1,657 @@ +--- +--- + +# Type Annotations - Python 3.13 + +This document provides complete, canonical type annotation guidance for Python 3.13. Python 3.13 +implements PEP 649 (Deferred Evaluation of Annotations), fundamentally changing how annotations are +evaluated. + +## Overview + +**The key change: forward references and circular imports work naturally without +`from __future__ import annotations`.** + +All type features from previous versions (3.10-3.12) continue to work. + +**What's new in 3.13:** + +- PEP 649 deferred annotation evaluation +- Forward references work naturally (no quotes, no `from __future__`) +- Circular imports no longer cause annotation errors +- **DO NOT use `from __future__ import annotations`** + +**Available from 3.12:** + +- PEP 695 type parameter syntax: `def func[T](x: T) -> T` +- `type` statement for better type aliases + +**Available from 3.11:** + +- `Self` type for self-returning methods + +## Universal Philosophy + +**Code Clarity:** + +- Types serve as inline documentation +- Make function contracts explicit +- Reduce cognitive load when reading code +- Help understand data flow without tracing through implementation + +**IDE Support:** + +- Enable autocomplete and intelligent suggestions +- Catch typos and attribute errors before runtime +- Support refactoring tools (rename, move, extract) +- Provide jump-to-definition for typed objects + +**Bug Prevention:** + +- Catch type mismatches during static analysis +- Prevent None-related errors with explicit optional types +- Document expected input/output without running code +- Enable early detection of API contract violations + +## Consistency Rules + +**All public APIs:** + +- 🔴 MUST: Type all function parameters (except `self` and `cls`) +- 🔴 MUST: Type all function return values +- 🔴 MUST: Type all class attributes +- 🟡 SHOULD: Type module-level constants + +**Internal code:** + +- 🟡 SHOULD: Type function signatures where helpful for clarity +- 🟢 MAY: Type complex local variables where type isn't obvious +- 🟢 MAY: Omit types for obvious cases (e.g., `count = 0`) + +## Basic Collection Types + +✅ **PREFERRED** - Use built-in generic types: + +```python +names: list[str] = [] +mapping: dict[str, int] = {} +unique_ids: set[str] = set() +coordinates: tuple[int, int] = (0, 0) +``` + +❌ **WRONG** - Don't use typing module equivalents: + +```python +from typing import List, Dict, Set, Tuple # Don't do this +names: List[str] = [] +``` + +**Why**: Built-in types are more concise, don't require imports, and are the modern Python standard +(available since 3.10). + +## Union Types + +✅ **PREFERRED** - Use `|` operator: + +```python +def process(value: str | int) -> str: + return str(value) + +def find_config(name: str) -> dict[str, str] | dict[str, int]: + ... + +# Multiple unions +def parse(input: str | int | float) -> str: + return str(input) +``` + +❌ **WRONG** - Don't use `typing.Union`: + +```python +from typing import Union +def process(value: Union[str, int]) -> str: # Don't do this + ... +``` + +## Optional Types + +✅ **PREFERRED** - Use `X | None`: + +```python +def find_user(id: str) -> User | None: + """Returns user or None if not found.""" + if id in users: + return users[id] + return None +``` + +❌ **WRONG** - Don't use `typing.Optional`: + +```python +from typing import Optional +def find_user(id: str) -> Optional[User]: # Don't do this + ... +``` + +## Callable Types + +✅ **PREFERRED** - Use `collections.abc.Callable`: + +```python +from collections.abc import Callable + +# Function that takes int, returns str +processor: Callable[[int], str] = str + +# Function with no args, returns None +callback: Callable[[], None] = lambda: None + +# Function with multiple args +validator: Callable[[str, int], bool] = lambda s, i: len(s) > i +``` + +## Interfaces: ABC vs Protocol + +✅ **PREFERRED** - Use ABC for interfaces: + +```python +from abc import ABC, abstractmethod + +class Repository(ABC): + @abstractmethod + def get(self, id: str) -> User | None: + """Get user by ID.""" + + @abstractmethod + def save(self, user: User) -> None: + """Save user.""" +``` + +🟡 **VALID** - Use Protocol only for structural typing: + +```python +from typing import Protocol + +class Drawable(Protocol): + def draw(self) -> None: ... + +def render(obj: Drawable) -> None: + obj.draw() +``` + +**Dignified Python prefers ABC** because it makes inheritance and intent explicit. + +## Self Type for Self-Returning Methods (3.11+) + +✅ **PREFERRED** - Use Self for methods that return the instance: + +```python +from typing import Self + +class Builder: + def set_name(self, name: str) -> Self: + self.name = name + return self + + def set_value(self, value: int) -> Self: + self.value = value + return self +``` + +## Generic Functions with PEP 695 (3.12+) + +✅ **PREFERRED** - Use PEP 695 type parameter syntax: + +```python +def first[T](items: list[T]) -> T | None: + """Return first item or None if empty.""" + if not items: + return None + return items[0] + +def identity[T](value: T) -> T: + """Return value unchanged.""" + return value + +# Multiple type parameters +def zip_dicts[K, V](keys: list[K], values: list[V]) -> dict[K, V]: + """Create dict from separate key and value lists.""" + return dict(zip(keys, values)) +``` + +🟡 **VALID** - TypeVar still works: + +```python +from typing import TypeVar + +T = TypeVar("T") + +def first(items: list[T]) -> T | None: + if not items: + return None + return items[0] +``` + +**Note**: Prefer PEP 695 syntax for simple generics. TypeVar is still needed for constraints/bounds. + +## Generic Classes with PEP 695 (3.12+) + +✅ **PREFERRED** - Use PEP 695 class syntax: + +```python +class Stack[T]: + """A generic stack data structure.""" + + def __init__(self) -> None: + self._items: list[T] = [] + + def push(self, item: T) -> Self: + self._items.append(item) + return self + + def pop(self) -> T | None: + if not self._items: + return None + return self._items.pop() + +# Usage +int_stack = Stack[int]() +int_stack.push(42).push(43) +``` + +🟡 **VALID** - Generic with TypeVar still works: + +```python +from typing import Generic, TypeVar + +T = TypeVar("T") + +class Stack(Generic[T]): + def __init__(self) -> None: + self._items: list[T] = [] + # ... rest of implementation +``` + +**Note**: PEP 695 is cleaner - no imports needed, type parameter scope is local to class. + +## Type Parameter Bounds (3.12+) + +✅ **Use bounds with PEP 695**: + +```python +class Comparable: + def compare(self, other: object) -> int: + ... + +def max_value[T: Comparable](items: list[T]) -> T: + """Get maximum value from comparable items.""" + return max(items, key=lambda x: x) +``` + +## Constrained TypeVars (Still Use TypeVar) + +✅ **Use TypeVar for specific type constraints**: + +```python +from typing import TypeVar + +# Constrained to specific types - must use TypeVar +Numeric = TypeVar("Numeric", int, float) + +def add(a: Numeric, b: Numeric) -> Numeric: + return a + b +``` + +❌ **WRONG** - PEP 695 doesn't support constraints: + +```python +# This doesn't constrain to int|float +def add[Numeric](a: Numeric, b: Numeric) -> Numeric: + return a + b +``` + +## Type Aliases with type Statement (3.12+) + +✅ **PREFERRED** - Use `type` statement: + +```python +# Simple alias +type UserId = str +type Config = dict[str, str | int | bool] + +# Generic type alias +type Result[T] = tuple[T, str | None] + +def process(value: str) -> Result[int]: + try: + return (int(value), None) + except ValueError as e: + return (0, str(e)) +``` + +🟡 **VALID** - Simple assignment still works: + +```python +UserId = str # Still valid +Config = dict[str, str | int | bool] # Still valid +``` + +**Note**: `type` statement is more explicit and works better with generics. + +## Forward References and Circular Imports (NEW in 3.13) + +✅ **CORRECT** - Just works naturally with PEP 649: + +```python +# Forward reference - no quotes needed! +class Node: + def __init__(self, value: int, parent: Node | None = None): + self.value = value + self.parent = parent + +# Circular imports - just works! +# a.py +from b import B + +class A: + def method(self) -> B: + ... + +# b.py +from a import A + +class B: + def method(self) -> A: + ... + +# Recursive types - no future needed! +type JsonValue = dict[str, JsonValue] | list[JsonValue] | str | int | float | bool | None +``` + +❌ **WRONG** - Don't use `from __future__ import annotations`: + +```python +from __future__ import annotations # DON'T DO THIS in Python 3.13 + +class Node: + def __init__(self, value: int, parent: Node | None = None): + ... +``` + +**Why avoid `from __future__ import annotations` in 3.13:** + +- Unnecessary - PEP 649 provides better default behavior +- Can cause confusion +- Masks the native 3.13 deferred evaluation +- Prevents you from leveraging improvements + +## Complete Examples + +### Tree Structure with Natural Forward References + +```python +from typing import Self +from collections.abc import Callable + +class Node[T]: + """Tree node - forward reference works naturally in 3.13!""" + + def __init__( + self, + value: T, + parent: Node[T] | None = None, # Forward ref, no quotes! + children: list[Node[T]] | None = None, # Forward ref, no quotes! + ) -> None: + self.value = value + self.parent = parent + self.children = children or [] + + def add_child(self, child: Node[T]) -> Self: + """Add child and return self for chaining.""" + self.children.append(child) + child.parent = self + return self + + def find(self, predicate: Callable[[T], bool]) -> Node[T] | None: + """Find first node matching predicate.""" + if predicate(self.value): + return self + + for child in self.children: + result = child.find(predicate) + if result: + return result + + return None + +# Usage - all type-safe with no __future__ import! +root = Node[int](1) +root.add_child(Node[int](2)).add_child(Node[int](3)) +``` + +### Generic Repository with PEP 695 + +```python +from abc import ABC, abstractmethod +from typing import Self + +class Entity[T]: + """Base class for entities.""" + + def __init__(self, id: T) -> None: + self.id = id + +class Repository[T](ABC): + """Generic repository interface.""" + + @abstractmethod + def get(self, id: str) -> T | None: + """Get entity by ID.""" + + @abstractmethod + def save(self, entity: T) -> None: + """Save entity.""" + + @abstractmethod + def delete(self, id: str) -> bool: + """Delete entity, return True if deleted.""" + +class User(Entity[str]): + def __init__(self, id: str, name: str) -> None: + super().__init__(id) + self.name = name + +class UserRepository(Repository[User]): + def __init__(self) -> None: + self._users: dict[str, User] = {} + + def get(self, id: str) -> User | None: + if id not in self._users: + return None + return self._users[id] + + def save(self, entity: User) -> None: + self._users[entity.id] = entity + + def delete(self, id: str) -> bool: + if id not in self._users: + return False + del self._users[id] + return True +``` + +## General Best Practices + +**Prefer specificity:** + +```python +# ✅ GOOD - Specific +def get_config() -> dict[str, str | int]: + ... + +# ❌ WRONG - Too vague +def get_config() -> dict: + ... +``` + +**Use Union sparingly:** + +```python +# ✅ GOOD - Union only when necessary +def process(value: str | int) -> str: + ... + +# ❌ WRONG - Too permissive +def process(value: str | int | list | dict) -> str | None | list: + ... +``` + +**Be explicit with None:** + +```python +# ✅ GOOD - Explicit optional +def find_user(id: str) -> User | None: + ... + +# ❌ WRONG - Implicit None return +def find_user(id: str) -> User: + return None # Type checker error! +``` + +**Avoid Any when possible:** + +```python +# ✅ GOOD - Specific type +def serialize(obj: User | Config) -> str: + ... + +# ❌ WRONG - Defeats purpose of types +from typing import Any +def serialize(obj: Any) -> str: + ... +``` + +## When to Use Types + +**Always type:** + +- Public function signatures (parameters + return) +- Class attributes (including private ones) +- Function parameters that cross module boundaries +- Return values that aren't immediately obvious + +**Type when helpful:** + +- Complex local variables +- Closures and nested functions +- Lambda expressions used as callbacks + +**Can skip:** + +- Obvious cases: `count = 0`, `name = "example"` +- Trivial private helpers +- Test fixture setup code (if types add no clarity) + +## Type Checking with ty + +Dignified Python uses ty for static type checking: + +```bash +# Check all files +ty check + +# Check specific file +ty check src/mymodule.py + +# Check with specific Python version +ty check --python-version 3.13 +``` + +**Configuration** (in `pyproject.toml`): + +```toml +[tool.ty.environment] +python-version = "3.13" +``` + +## Anti-Patterns + +**❌ Don't ignore type errors with `# type: ignore`** + +```python +# ❌ WRONG - Hiding type error +result = unsafe_function() # type: ignore + +# ✅ CORRECT - Fix the type error +result: Expected = cast(Expected, unsafe_function()) +``` + +**❌ Don't use bare Exception in type hints** + +```python +# ❌ WRONG - No value from typing exception +def risky() -> str | Exception: + ... + +# ✅ CORRECT - Let exceptions bubble +def risky() -> str: + ... # Raises ValueError on error +``` + +**❌ Don't over-type simple cases** + +```python +# ❌ WRONG - Obvious from context +def add_numbers(a: int, b: int) -> int: + result: int = a + b # Unnecessary type annotation + return result + +# ✅ CORRECT - Type only signature +def add_numbers(a: int, b: int) -> int: + result = a + b # Type is obvious + return result +``` + +## Migration from 3.10/3.11 + +If migrating from Python 3.10/3.11: + +1. **Remove `from __future__ import annotations`** - No longer needed +2. **Consider upgrading to PEP 695 syntax** - Cleaner generics +3. **Use `type` statement for aliases** - More explicit than assignment +4. **Remove quoted forward references** - They work naturally now + +```python +# Python 3.10/3.11 +from __future__ import annotations +from typing import TypeVar, Generic + +T = TypeVar("T") + +class Node(Generic[T]): + def __init__(self, value: T, parent: "Node[T] | None" = None): + ... + +# Python 3.13 +from typing import Self + +class Node[T]: + def __init__(self, value: T, parent: Node[T] | None = None): + ... +``` + +## What typing imports are still needed? + +**Very rare:** + +- `TypeVar` - Only for constrained/bounded type variables +- `Any` - Use sparingly when type truly unknown +- `Protocol` - Structural typing (prefer ABC) +- `TYPE_CHECKING` - Conditional imports to avoid circular dependencies + +**Never needed:** + +- `List`, `Dict`, `Set`, `Tuple` - Use built-in types +- `Union` - Use `|` operator +- `Optional` - Use `X | None` +- `Generic` - Use PEP 695 class syntax diff --git a/.agents/skills/docker-env/SKILL.md b/.agents/skills/docker-env/SKILL.md new file mode 100644 index 0000000..a26c801 --- /dev/null +++ b/.agents/skills/docker-env/SKILL.md @@ -0,0 +1,45 @@ +--- +name: docker-env +description: Standardize Docker/Dagster environment debugging and recovery workflows. +version: 1.0.0 +tags: [docker, dagster, debugging] +--- + +# Docker Environment Skill + +Use this skill for container health checks, logs, and safe restarts. Prefer the least destructive command that answers the question. + +## Common Commands + +- **Status** + - `docker compose ps -a` + +- **Logs** + - `docker compose logs --tail=200` + - `docker compose logs -f ` (stream) + +- **Restart (service only)** + - `docker compose restart ` + +- **Rebuild (non-destructive)** + - `docker compose up --build -d` + +## Destructive Operations (ask before running) + +- **Cleanup** + - `docker system prune -f` + +- **Full reset** + - `docker compose down -v` + - `docker compose up --build -d` + +## Compose Files + +- Default: `docker-compose.yml` +- Local dev override: `docker-compose.local.yml` + - Use `docker compose -f docker-compose.yml -f docker-compose.local.yml ...` if needed + +## Guidance + +- Prefer targeted restarts before rebuilding or resetting volumes. +- Always capture logs before teardown for root-cause analysis. diff --git a/.agents/skills/git-pr-workflow/SKILL.md b/.agents/skills/git-pr-workflow/SKILL.md new file mode 100644 index 0000000..29900c8 --- /dev/null +++ b/.agents/skills/git-pr-workflow/SKILL.md @@ -0,0 +1,263 @@ +--- +name: git-pr-workflow +description: Automates the complete git and GitHub workflow: initializes a git repository, creates a private GitHub repository if needed, commits changes, creates a pull request, monitors GitHub Actions for completion, and merges the PR if all checks pass. +--- + +# Git PR Workflow Skill + +## Description +Automates the complete git and GitHub workflow: initializes a git repository, creates a private GitHub repository if needed, commits changes, creates a pull request, monitors GitHub Actions for completion, and merges the PR if all checks pass. + +## Usage +Invoke this skill when you need to: +- Initialize a new git repository and push to GitHub +- Create a pull request with automatic CI/CD verification +- Wait for GitHub Actions to complete before merging +- Automatically merge PRs that pass all checks + +## Instructions + +You are an expert at git and GitHub workflows. Your task is to automate the complete process of initializing a repository, creating a PR, and merging it after CI passes. + +### Step 0: Confirm GitHub CLI Auth + +Check that the GitHub CLI is authenticated: + +```bash +gh auth status +``` + +If not authenticated, stop and ask the user to run `gh auth login`. + +### Step 1: Initialize Git Repository (if needed) + +First, check if the current directory is already a git repository: + +```bash +git rev-parse --is-inside-work-tree 2>/dev/null +``` + +If not a git repository, initialize it: + +```bash +git init +``` + +### Step 2: Check for GitHub Remote + +Check if a GitHub remote already exists: + +```bash +git remote -v +``` + +If no remote exists, check if a GitHub repository exists for this project: + +```bash +gh repo view --json name,owner 2>&1 +``` + +### Step 3: Create GitHub Repository (if needed) + +If the repository doesn't exist on GitHub, create a new private repository: + +```bash +gh repo create $(basename "$PWD") --private --source=. --remote=origin --push +``` + +This command: +- Creates a private repository with the current directory name +- Sets the current directory as the source +- Adds the remote as 'origin' +- Pushes the initial commit + +If the repository already exists but no remote is configured: + +```bash +gh repo view --json nameWithOwner --jq .nameWithOwner | xargs -I {} git remote add origin "https://github.com/{}.git" +``` + +### Step 4: Ensure Main Branch Exists + +Make sure we have an initial commit on main: + +```bash +git add . +git commit -m "chore: initial commit" || echo "Already has commits" +git branch -M main +git push -u origin main +``` + +If the push fails due to non-fast-forward, stop and ask the user before proceeding. + +### Step 5: Determine Base Ref + +Before creating a feature branch, determine the appropriate base ref: + +```bash +git fetch origin main || true +BASE_REF="origin/main" +if ! git show-ref --verify --quiet refs/remotes/origin/main; then + BASE_REF="main" +fi +``` + +### Step 6: Create Feature Branch + +Create a new feature branch from the base ref: + +```bash +BRANCH_NAME="feature/automated-changes-$(date +%s)" +git checkout -b "$BRANCH_NAME" "$BASE_REF" +``` + +### Step 7: Apply Changes + +Commit any uncommitted changes on the feature branch: + +```bash +if [ -n "$(git status --porcelain)" ]; then + git add . + git commit -m "chore: automated changes" +else + echo "No changes to commit; stop and confirm next steps." + exit 1 +fi +``` + +### Step 8: Push Branch + +Push the feature branch to GitHub: + +```bash +git push -u origin "$BRANCH_NAME" +``` + +### Step 9: Create Pull Request + +Create a pull request using the gh CLI: + +```bash +PR_URL=$(gh pr create \ + --title "Automated changes" \ + --body "$(cat <<'EOF' +## Summary +Automated changes created via agent-assisted workflow + +## Changes +- Repository initialized and configured +- All current changes committed + +🤖 Generated with an agent-assisted workflow. +EOF +)" \ + --base main \ + --head "$BRANCH_NAME" \ + --json url \ + --jq .url) + +echo "PR created: $PR_URL" +``` + +### Step 10: Extract PR Number + +Extract the PR number from the URL: + +```bash +PR_NUMBER=$(echo "$PR_URL" | grep -oE '[0-9]+$') +echo "PR Number: $PR_NUMBER" +``` + +### Step 11: Monitor GitHub Actions + +Wait for GitHub Actions to complete. First, wait a few seconds for checks to start: + +```bash +echo "Waiting for checks to start..." +sleep 10 +``` + +Then monitor the checks status: + +```bash +MAX_WAIT=600 # 10 minutes +ELAPSED=0 +CHECK_INTERVAL=15 + +while [ $ELAPSED -lt $MAX_WAIT ]; do + echo "Checking PR status... (${ELAPSED}s elapsed)" + + # Check if all checks have completed + PENDING=$(gh pr checks "$PR_NUMBER" --json state --jq '[.[] | select(.state == "PENDING" or .state == "IN_PROGRESS")] | length') + + if [ "$PENDING" -eq 0 ]; then + echo "All checks completed!" + + # Check if any checks failed + FAILED=$(gh pr checks "$PR_NUMBER" --json conclusion --jq '[.[] | select(.conclusion == "FAILURE" or .conclusion == "CANCELLED")] | length') + + if [ "$FAILED" -gt 0 ]; then + echo "❌ Some checks failed:" + gh pr checks "$PR_NUMBER" --json name,conclusion --jq '.[] | select(.conclusion == "FAILURE" or .conclusion == "CANCELLED") | " - \(.name): \(.conclusion)"' + exit 1 + fi + + echo "✅ All checks passed!" + break + fi + + echo "Checks still running: $PENDING pending" + sleep $CHECK_INTERVAL + ELAPSED=$((ELAPSED + CHECK_INTERVAL)) +done + +if [ $ELAPSED -ge $MAX_WAIT ]; then + echo "⏱️ Timeout waiting for checks to complete" + exit 1 +fi +``` + +### Step 12: Merge Pull Request + +If all checks pass, merge the PR: + +```bash +gh pr merge "$PR_NUMBER" --auto --squash --delete-branch +echo "✅ PR merged successfully!" +``` + +### Step 13: Switch Back to Main + +Finally, switch back to the main branch and pull the merged changes: + +```bash +git checkout main +git pull origin main +echo "✅ Workflow complete! All changes merged to main." +``` + +## Error Handling + +Throughout the process: +- Check return codes after each command +- Provide clear error messages if something fails +- If GitHub Actions fail, display which checks failed and why +- If the merge fails, explain what went wrong + +## Success Criteria + +The skill succeeds when: +1. Git repository is initialized (or already exists) +2. GitHub repository exists and is accessible +3. Changes are committed to a feature branch +4. Pull request is created +5. All GitHub Actions checks pass +6. PR is successfully merged to main +7. Local main branch is updated + +## Notes + +- The repository created will be private by default +- The skill waits up to 10 minutes for GitHub Actions to complete +- If checks don't start within the timeout period, the skill will fail +- The feature branch is automatically deleted after merge diff --git a/.agents/skills/github-explorer/SKILL.md b/.agents/skills/github-explorer/SKILL.md new file mode 100644 index 0000000..cb401e5 --- /dev/null +++ b/.agents/skills/github-explorer/SKILL.md @@ -0,0 +1,620 @@ +--- +name: github-explorer +description: General-purpose GitHub exploration and analysis tool for searching issues, creating PRs, analyzing git history, and reviewing PR comments +license: MIT +--- + +# GitHub Explorer Skill + +## Description + +This skill provides interactive GitHub exploration and analysis capabilities using the GitHub CLI (`gh`) and git commands. It enables searching and filtering issues, creating pull requests with custom templates, analyzing file history with git blame, and reviewing PR comments and feedback. Use this skill when you need flexible, exploratory interactions with GitHub repositories. + +## Usage + +Invoke this skill when you need to: +- Search for issues with specific labels, assignees, or text queries +- List and filter issues across repositories +- Create pull requests with customized content and templates +- Analyze who authored specific lines of code and when +- Review pull request comments and identify actionable feedback +- Investigate code history and track changes to specific functions +- Parse and analyze PR review threads + +## Prerequisites + +This skill requires the following tools to be installed and configured: + +1. **GitHub CLI (`gh`)**: Version 2.0 or higher + - Check installation: `gh --version` + - Install: https://cli.github.com/ + +2. **Git**: Any recent version + - Check installation: `git --version` + +3. **jq**: JSON processor for parsing API responses + - Check installation: `jq --version` + - Install: https://jqlang.github.io/jq/download/ + +4. **GitHub Authentication**: Must be logged in to `gh` CLI + - Check status: `gh auth status` + - Login: `gh auth login` + +## Instructions + +### Feature 1: Search and List Issues + +Use this feature to find and explore GitHub issues across repositories. + +#### Step 1: List Issues in Current Repository + +```bash +# List all open issues in current repository +gh issue list --state open + +# List all issues (open and closed) +gh issue list --state all + +# Limit results +gh issue list --state open --limit 20 +``` + +#### Step 2: Filter Issues by Labels + +```bash +# List issues with a specific label +gh issue list --label "bug" --state open + +# List issues with multiple labels (AND logic) +gh issue list --label "bug,high-priority" --state open + +# Combine label filters with limits +gh issue list --label "enhancement" --state open --limit 10 +``` + +#### Step 3: Filter Issues by Assignee + +```bash +# List issues assigned to you +gh issue list --assignee @me --state open + +# List issues assigned to a specific user +gh issue list --assignee username --state open + +# Combine assignee and label filters +gh issue list --assignee @me --label "bug,high-priority" --state open +``` + +#### Step 4: Search Issues with Text Queries + +```bash +# Search issues in current repository +gh search issues "authentication error" + +# Search issues in a specific repository +gh search issues "docker configuration" --repo owner/repo + +# Search across an organization +gh search issues "api timeout" --owner orgname + +# Combine search with filters +gh search issues "memory leak" --state open --label "bug" --limit 10 +``` + +#### Step 5: View Issue Details + +```bash +# View a specific issue +gh issue view 123 + +# View issue with comments +gh issue view 123 --comments + +# Get issue JSON for parsing +gh issue view 123 --json title,body,state,labels,assignees,createdAt +``` + +#### Step 6: Parse and Format Issue Data + +```bash +# Get issues as JSON and parse with jq +gh issue list --json number,title,state,labels,assignees,createdAt --limit 10 | \ + jq -r '.[] | "\(.number): \(.title) [\(.state)]"' + +# Create a table of issues +gh issue list --json number,title,assignees --limit 20 | \ + jq -r '.[] | "\(.number)\t\(.title)\t\(.assignees[0].login // "unassigned")"' + +# Count issues by label +gh issue list --state all --json labels --jq '[.[].labels[].name] | group_by(.) | map({label: .[0], count: length}) | sort_by(.count) | reverse' +``` + +### Feature 2: Create Pull Requests + +Use this feature to create pull requests with custom content and templates. + +#### Step 1: Review Current Branch Changes + +Before creating a PR, review what changes will be included: + +```bash +# Show files changed compared to main +git diff main...HEAD --stat + +# Show detailed diff +git diff main...HEAD + +# Show commit messages that will be in the PR +git log main...HEAD --oneline + +# Show all commits with details +git log main...HEAD --pretty=fuller +``` + +#### Step 2: Create Basic Pull Request + +```bash +# Create PR with auto-filled title and body from commits +gh pr create --fill + +# Create PR with custom title and body +gh pr create --title "Fix authentication bug" --body "This PR fixes the token validation issue" + +# Create PR targeting a specific base branch +gh pr create --base develop --head feature-branch +``` + +#### Step 3: Create PR with Custom Template + +```bash +# Create PR with formatted body using heredoc +gh pr create --title "Add user profile feature" --body "$(cat <<'EOF' +## Summary +- Implemented user profile page +- Added avatar upload functionality +- Created profile editing form + +## Changes Made +- New UserProfile component +- Profile API endpoints +- Avatar storage in GCS + +## Related Issues +Fixes #42 +Relates to #38 + +## Testing +- [ ] Manual testing completed +- [ ] Unit tests added +- [ ] Integration tests passing + +🤖 Generated with an agent-assisted workflow. +EOF +)" +``` + +#### Step 4: Create Draft or WIP Pull Requests + +```bash +# Create draft PR (not ready for review) +gh pr create --draft --title "WIP: Refactor authentication" + +# Create normal PR that can be reviewed immediately +gh pr create --title "Fix login bug" --body "Ready for review" +``` + +#### Step 5: Link PR to Issues + +```bash +# Link PR to issue with "Fixes" keyword +gh pr create --title "Fix memory leak" --body "Fixes #123" + +# Link to multiple issues +gh pr create --title "Bug fixes" --body "$(cat <<'EOF' +## Summary +Multiple bug fixes + +## Issues +Fixes #123 +Fixes #124 +Relates to #125 +EOF +)" +``` + +#### Step 6: Advanced PR Creation + +```bash +# Get repository info for PR context +REPO_URL=$(gh repo view --json url --jq .url) +CURRENT_BRANCH=$(git branch --show-current) + +# Auto-generate PR body from commits +COMMITS=$(git log main...HEAD --pretty=format:"- %s") + +gh pr create --title "Feature: $(git log -1 --pretty=%s)" --body "$(cat < blame_output.csv +git blame -L 1,100 path/to/file.py --date=short | \ + awk '{print NR "," $2 "," $3 "," $1}' >> blame_output.csv +``` + +#### Step 6: Advanced History Analysis + +```bash +# Find when specific text was added +git log -S "critical_function" --source --all -p path/to/file.py + +# Show file changes over time +git log --stat path/to/file.py + +# Show who last modified each line (concise format) +git blame path/to/file.py | awk '{print $2, $3}' | sort | uniq -c | sort -nr + +# Compare current version with specific commit +COMMIT=$(git blame -L 50,50 path/to/file.py | awk '{print $1}') +git diff $COMMIT path/to/file.py +``` + +### Feature 4: Analyze PR Comments + +Use this feature to analyze pull request review comments and feedback. + +#### Step 1: View PR with Comments + +```bash +# View PR details with all comments +gh pr view 123 --comments + +# View specific PR sections +gh pr view 123 --json title,body,state,comments +``` + +#### Step 2: Check PR Review Status + +```bash +# View PR checks (CI/CD status) +gh pr checks 123 + +# View PR reviews +gh pr reviews 123 + +# Get review decision +gh pr view 123 --json reviewDecision +``` + +#### Step 3: Get Detailed Review Comments + +```bash +# Get all review comments as JSON +gh api repos/:owner/:repo/pulls/123/comments | jq '.' + +# Parse comments with jq +gh api repos/:owner/:repo/pulls/123/comments --jq '.[] | { + author: .user.login, + body: .body, + path: .path, + line: .line, + created: .created_at +}' + +# Filter comments by file +gh api repos/:owner/:repo/pulls/123/comments --jq '.[] | select(.path == "src/main.py") | {author: .user.login, line: .line, body: .body}' +``` + +#### Step 4: Analyze Conversation Threads + +```bash +# Get all review threads +gh api repos/:owner/:repo/pulls/123/reviews --jq '.[] | { + author: .user.login, + state: .state, + body: .body, + submitted: .submitted_at +}' + +# Find unresolved conversations +gh pr view 123 --json reviews --jq '.reviews[] | select(.state == "CHANGES_REQUESTED")' + +# Count reviews by state +gh pr view 123 --json reviews --jq '.reviews | group_by(.state) | map({state: .[0].state, count: length})' +``` + +#### Step 5: Track Comment Authors + +```bash +# Count comments by author +gh api repos/:owner/:repo/pulls/123/comments --jq 'group_by(.user.login) | map({author: .[0].user.login, count: length})' + +# List unique commenters +gh api repos/:owner/:repo/pulls/123/comments --jq '[.[].user.login] | unique' + +# Find most active reviewer +gh api repos/:owner/:repo/pulls/123/comments --jq 'group_by(.user.login) | map({author: .[0].user.login, count: length}) | sort_by(.count) | reverse | .[0]' +``` + +#### Step 6: Identify Actionable Feedback + +```bash +# Find all "CHANGES_REQUESTED" reviews +gh pr reviews 123 --json author,state,body | \ + jq '.[] | select(.state == "CHANGES_REQUESTED")' + +# Get list of files with comments +gh api repos/:owner/:repo/pulls/123/comments --jq '[.[].path] | unique' + +# Create summary of comments by file +gh api repos/:owner/:repo/pulls/123/comments --jq 'group_by(.path) | map({file: .[0].path, comment_count: length})' + +# View PR diff for context +gh pr diff 123 + +# View PR diff for specific file +gh pr diff 123 -- path/to/file.py +``` + +## Error Handling + +### Authentication Errors + +If you encounter authentication issues: + +```bash +# Check authentication status +gh auth status + +# Re-authenticate if needed +gh auth login + +# Test API access +gh api user + +# Verify you have access to the repository +gh repo view +``` + +### Rate Limiting + +GitHub API has rate limits. If you encounter rate limit errors: + +```bash +# Check current rate limit status +gh api rate_limit + +# View rate limit details +gh api rate_limit --jq '.resources.core | { + limit: .limit, + remaining: .remaining, + reset: .reset, + reset_time: (.reset | strftime("%Y-%m-%d %H:%M:%S")) +}' + +# Wait if rate limit is reached (reset time shown in output above) +``` + +Rate limits: +- Authenticated requests: 5,000 per hour +- Unauthenticated requests: 60 per hour +- GraphQL API: 5,000 points per hour + +### Repository Context Errors + +If commands fail because you're not in a git repository: + +```bash +# Check if in a git repository +if ! git rev-parse --is-inside-work-tree 2>/dev/null; then + echo "Error: Not in a git repository" + echo "Navigate to a git repository or initialize one with: git init" + exit 1 +fi + +# Get repository information +gh repo view --json nameWithOwner,url + +# Clone repository if needed +gh repo clone owner/repo +``` + +### Missing Issues or PRs + +If an issue or PR doesn't exist: + +```bash +# Check if issue exists before viewing +if gh issue view 123 2>/dev/null; then + gh issue view 123 +else + echo "Issue #123 not found" + echo "Available issues:" + gh issue list --limit 10 +fi + +# List available PRs +gh pr list --state all --limit 20 +``` + +### Permission Errors + +If you lack permissions for certain operations: + +1. **For PR creation**: Ensure you have write access to the repository +2. **For issue access**: Check repository visibility (private vs public) +3. **For API operations**: Verify your authentication scope + +```bash +# Check repository permissions +gh api repos/:owner/:repo | jq '.permissions' + +# Re-authenticate with additional scopes if needed +gh auth refresh -s repo,read:org +``` + +## Success Criteria + +This skill succeeds when: + +1. GitHub CLI is properly installed and authenticated (`gh auth status` shows active login) +2. Issue searches return relevant, filtered results matching the specified criteria +3. PR creation completes successfully with properly formatted title and body +4. Git blame analysis identifies correct authors and timestamps for code sections +5. PR comment analysis retrieves and parses all review threads and feedback +6. All bash commands execute without errors +7. JSON parsing with `jq` produces readable, formatted output +8. Rate limits are monitored and respected +9. Repository context is verified before operations that require it +10. Error messages are clear and provide actionable next steps + +## Notes + +### Best Practices + +1. **Always authenticate first**: Run `gh auth login` and `gh auth status` before using this skill +2. **Use JSON output for parsing**: Add `--json` flags to `gh` commands when you need structured data +3. **Parse with jq**: Use `jq` for filtering and formatting JSON API responses +4. **Verify repository context**: Check you're in the correct repository with `gh repo view` +5. **Monitor rate limits**: Check `gh api rate_limit` if you're making many requests +6. **Test commands incrementally**: Start with simple commands before building complex queries + +### GitHub API Rate Limits + +Be mindful of GitHub's API rate limits: +- **5,000 requests/hour** for authenticated users +- **60 requests/hour** for unauthenticated requests +- Use `gh api rate_limit` to check current usage +- The `gh` CLI automatically handles authentication + +### Complementary with git-pr-workflow Skill + +This skill complements the existing `git-pr-workflow` skill: + +- **Use github-explorer when**: You need to explore issues, analyze code history, review PR feedback, or manually craft custom PRs +- **Use git-pr-workflow when**: You want fully automated PR creation, CI monitoring, and auto-merge + +Example combined workflow: +1. Use `github-explorer` to find bugs: `gh issue list --label "bug"` +2. Fix the bugs in your code +3. Use `git-pr-workflow` to automatically create a PR, monitor CI, and merge + +### Working with Multiple Repositories + +To work across different repositories: + +```bash +# Specify repository explicitly +gh issue list --repo owner/repo +gh pr list --repo owner/other-repo + +# Clone and switch to different repo +gh repo clone owner/repo +cd repo + +# Search across organization +gh search issues "bug" --owner orgname +``` + +### Output Formatting Tips + +For better readability: +- Use `--json` with `jq` for structured data +- Pipe to `grep` for text filtering +- Use `column -t` for table formatting +- Export to CSV for spreadsheet analysis + +## Resources + +- **GitHub CLI Manual**: https://cli.github.com/manual/ +- **GitHub REST API Documentation**: https://docs.github.com/en/rest +- **GitHub CLI Authentication**: https://cli.github.com/manual/gh_auth_login +- **jq Tutorial**: https://jqlang.github.io/jq/tutorial/ +- **jq Manual**: https://jqlang.github.io/jq/manual/ +- **Git Blame Documentation**: https://git-scm.com/docs/git-blame +- **Git Log Documentation**: https://git-scm.com/docs/git-log +- **GitHub API Rate Limiting**: https://docs.github.com/en/rest/overview/rate-limits-for-the-rest-api diff --git a/.agents/skills/nivo-charts/SKILL.md b/.agents/skills/nivo-charts/SKILL.md new file mode 100644 index 0000000..f946f8f --- /dev/null +++ b/.agents/skills/nivo-charts/SKILL.md @@ -0,0 +1,11 @@ +--- +name: nivo-charts +description: Deprecated alias for design-philosophy. Use that skill instead. +version: 1.0.0 +--- + +# Deprecated + +This skill has been consolidated into `design-philosophy`. + +Use `/design-philosophy` for the canonical guidance. diff --git a/.agents/skills/react-component-architecture/SKILL.md b/.agents/skills/react-component-architecture/SKILL.md new file mode 100644 index 0000000..9747766 --- /dev/null +++ b/.agents/skills/react-component-architecture/SKILL.md @@ -0,0 +1,457 @@ +--- +name: React Data Dashboard Architecture +description: React component patterns, hooks, and state management for data visualization dashboards +version: 1.0.0 +tags: [react, typescript, architecture, hooks, tanstack-query, zustand] +--- + +# React Data Dashboard Architecture + +## Compound Component Pattern +```typescript +import React, { createContext, useContext, useState, useMemo, ReactNode } from 'react'; + +interface ChartContextValue { + data: DataPoint[]; + selectedPoint: DataPoint | null; + setSelectedPoint: (point: DataPoint | null) => void; +} + +const ChartContext = createContext(undefined); + +function useChartContext() { + const ctx = useContext(ChartContext); + if (!ctx) throw new Error('Must be used within Chart'); + return ctx; +} + +export function Chart({ data, children }: { data: DataPoint[]; children: ReactNode }) { + const [selectedPoint, setSelectedPoint] = useState(null); + const value = useMemo(() => ({ data, selectedPoint, setSelectedPoint }), [data, selectedPoint]); + return {children}; +} + +Chart.Line = function Line({ color = '#1e40af' }) { + const { data } = useChartContext(); + return ; +}; + +Chart.Tooltip = function Tooltip() { + const { selectedPoint } = useChartContext(); + if (!selectedPoint) return null; + return
{selectedPoint.value}
; +}; + +// Usage + + + + +``` + +## TanStack Query for Data Fetching +```typescript +import { useQuery } from '@tanstack/react-query'; + +export function useStockData(symbol: string) { + return useQuery({ + queryKey: ['stock', symbol], + queryFn: () => fetchStockData(symbol), + staleTime: 30 * 1000, + refetchInterval: 5000, + }); +} + +export function useHistoricalData(symbol: string, timeRange: TimeRange) { + return useQuery({ + queryKey: ['historical', symbol, timeRange], + queryFn: () => fetchHistoricalData(symbol, timeRange), + staleTime: 5 * 60 * 1000, + placeholderData: (prev) => prev, + }); +} +``` + +## Zustand for Dashboard State +```typescript +import { create } from 'zustand'; +import { persist } from 'zustand/middleware'; + +interface DashboardState { + selectedSymbols: string[]; + timeRange: TimeRange; + chartType: 'line' | 'candlestick' | 'area'; + setTimeRange: (range: TimeRange) => void; + toggleSymbol: (symbol: string) => void; +} + +export const useDashboardStore = create()( + persist( + (set) => ({ + selectedSymbols: ['AAPL'], + timeRange: '1M', + chartType: 'line', + setTimeRange: (range) => set({ timeRange: range }), + toggleSymbol: (symbol) => set((s) => ({ + selectedSymbols: s.selectedSymbols.includes(symbol) + ? s.selectedSymbols.filter(x => x !== symbol) + : [...s.selectedSymbols, symbol] + })), + }), + { name: 'dashboard-storage' } + ) +); +``` + +## useEffect Best Practices + +### Async Effects with Cleanup +Always handle component unmount for async operations to prevent state updates on unmounted components: + +```typescript +useEffect(() => { + let isMounted = true; + + async function fetchData() { + try { + setLoading(true); + const data = await api.getData(); + if (isMounted) { + setData(data); + } + } catch (err) { + if (isMounted) { + setError(err instanceof Error ? err.message : 'Unknown error'); + } + } finally { + if (isMounted) { + setLoading(false); + } + } + } + + fetchData(); + return () => { + isMounted = false; + }; +}, []); +``` + +### AbortController for Fetch Requests +Use AbortController for cancelable network requests: + +```typescript +useEffect(() => { + const controller = new AbortController(); + + async function fetchData() { + try { + const response = await fetch(url, { signal: controller.signal }); + const data = await response.json(); + setData(data); + } catch (err) { + if (err instanceof Error && err.name !== 'AbortError') { + setError(err.message); + } + } + } + + fetchData(); + return () => controller.abort(); +}, [url]); +``` + +### Ref-Based Cleanup for Long-Lived Components +Use refs for cleanup flags when multiple effects need access: + +```typescript +function useAsyncOperation() { + const isMountedRef = useRef(true); + + useEffect(() => { + isMountedRef.current = true; + return () => { + isMountedRef.current = false; + }; + }, []); + + const execute = useCallback(async () => { + const result = await api.call(); + if (isMountedRef.current) { + setResult(result); + } + }, []); + + return { execute }; +} +``` + +### Race Condition Prevention +Prevent stale closures when dependencies change rapidly: + +```typescript +useEffect(() => { + let isCurrentRequest = true; + + async function loadData() { + const data = await fetchData(id); + if (isCurrentRequest) { + setData(data); + } + } + + loadData(); + return () => { + isCurrentRequest = false; + }; +}, [id]); +``` + +### Context Value Memoization +Always memoize context values to prevent unnecessary re-renders: + +```typescript +function MyProvider({ children }: { children: ReactNode }) { + const [state, setState] = useState(initialState); + + const doSomething = useCallback(() => { + setState(prev => ({ ...prev, updated: true })); + }, []); + + const value = useMemo( + () => ({ state, doSomething }), + [state, doSomething] + ); + + return {children}; +} +``` + +### When NOT to Use useEffect +Prefer alternatives when possible: + +```typescript +// BAD: Deriving state in useEffect +useEffect(() => { + setFilteredItems(items.filter(i => i.active)); +}, [items]); + +// GOOD: Derive during render with useMemo +const filteredItems = useMemo(() => items.filter(i => i.active), [items]); + +// BAD: Resetting state on prop change +useEffect(() => { + setSelectedId(null); +}, [items]); + +// GOOD: Use key to reset component state + + +// BAD: Fetching in useEffect when React Query is available +useEffect(() => { + fetchData().then(setData); +}, []); + +// GOOD: Use TanStack Query +const { data } = useQuery({ queryKey: ['data'], queryFn: fetchData }); +``` + +### Synchronous Effects (No Cleanup Needed) +Simple DOM updates and localStorage writes don't need cleanup: + +```typescript +// localStorage sync - no cleanup needed +useEffect(() => { + localStorage.setItem('theme', theme); +}, [theme]); + +// DOM class updates - no cleanup needed +useEffect(() => { + document.documentElement.classList.toggle('dark', isDark); +}, [isDark]); + +// Scroll to element - no cleanup needed +useEffect(() => { + messagesEndRef.current?.scrollIntoView({ behavior: 'smooth' }); +}, [messages]); +``` + +## Custom Hooks + +### useChartDimensions +```typescript +export function useChartDimensions(margins = { top: 40, right: 30, bottom: 40, left: 75 }) { + const ref = useRef(null); + const [dimensions, setDimensions] = useState({ width: 0, height: 0, boundedWidth: 0, boundedHeight: 0 }); + + useLayoutEffect(() => { + if (!ref.current) return; + const observer = new ResizeObserver(([entry]) => { + const { width, height } = entry.contentRect; + setDimensions({ + width, height, + boundedWidth: Math.max(width - margins.left - margins.right, 0), + boundedHeight: Math.max(height - margins.top - margins.bottom, 0), + }); + }); + observer.observe(ref.current); + return () => observer.disconnect(); + }, [margins]); + + return [ref, dimensions] as const; +} +``` + +### useTimeRange +```typescript +export function useTimeRange(initial: TimeRange = '1M') { + const [range, setRange] = useState(initial); + + const { startDate, endDate, interval } = useMemo(() => { + const end = new Date(); + let start: Date, interval: string; + switch (range) { + case '1D': start = new Date(end.getTime() - 86400000); interval = 'minute'; break; + case '1W': start = new Date(end.getTime() - 7 * 86400000); interval = 'hour'; break; + case '1M': start = new Date(end.getFullYear(), end.getMonth() - 1, end.getDate()); interval = 'day'; break; + case '1Y': start = new Date(end.getFullYear() - 1, end.getMonth(), end.getDate()); interval = 'week'; break; + default: start = new Date(2000, 0, 1); interval = 'month'; + } + return { startDate: start, endDate: end, interval }; + }, [range]); + + return { range, setRange, startDate, endDate, interval }; +} +``` + +### useChartInteraction +```typescript +export function useChartInteraction() { + const [hoveredPoint, setHoveredPoint] = useState(null); + const [selectedPoints, setSelectedPoints] = useState([]); + const [brushRange, setBrushRange] = useState<[Date, Date] | null>(null); + + const handlers = useMemo(() => ({ + onHover: setHoveredPoint, + onSelect: (point: T) => setSelectedPoints(prev => + prev.includes(point) ? prev.filter(p => p !== point) : [...prev, point] + ), + onBrush: setBrushRange, + onClear: () => { setSelectedPoints([]); setBrushRange(null); }, + }), []); + + return { hoveredPoint, selectedPoints, brushRange, ...handlers }; +} +``` + +## Performance Patterns +```typescript +// Memoized chart component +const LineChart = React.memo(function LineChart({ data, config }: Props) { + const processedData = useMemo(() => transformData(data), [data]); + return ; +}); + +// Memoized event handlers +function InteractiveChart({ data, onPointSelect }: Props) { + const handleMouseMove = useCallback((e: MouseEvent) => { + const point = findNearestPoint(e, data); + if (point) onPointSelect(point); + }, [data, onPointSelect]); + + return ; +} +``` + +## Error Boundary +```typescript +class ChartErrorBoundary extends Component<{ children: ReactNode; fallback?: ReactNode }, { hasError: boolean }> { + state = { hasError: false }; + static getDerivedStateFromError() { return { hasError: true }; } + render() { + if (this.state.hasError) { + return this.props.fallback ||
Chart failed to load
; + } + return this.props.children; + } +} +``` + +## Suspense Pattern +```typescript +function Dashboard() { + return ( +
+ }> + + + }> + + +
+ ); +} +``` + +## Accessibility +```typescript +export function AccessibleChart({ data, title }: Props) { + const [focusedIndex, setFocusedIndex] = useState(-1); + + const handleKeyDown = useCallback((e: KeyboardEvent) => { + switch (e.key) { + case 'ArrowRight': setFocusedIndex(i => Math.min(i + 1, data.length - 1)); break; + case 'ArrowLeft': setFocusedIndex(i => Math.max(i - 1, 0)); break; + } + }, [data.length]); + + return ( +
+ {/* Chart */} +
+ {focusedIndex >= 0 && `${data[focusedIndex].label}: ${data[focusedIndex].value}`} +
+
+ ); +} +``` + +## Testing +```typescript +import { render, screen, fireEvent } from '@testing-library/react'; +import { renderHook, act } from '@testing-library/react'; + +describe('Chart', () => { + it('renders data points', () => { + render(); + expect(screen.getByRole('figure')).toBeInTheDocument(); + }); +}); + +describe('useTimeRange', () => { + it('updates range correctly', () => { + const { result } = renderHook(() => useTimeRange()); + act(() => result.current.setRange('1Y')); + expect(result.current.range).toBe('1Y'); + }); +}); +``` + +## File Structure +src/ +├── features/ +│ ├── dashboard/ +│ │ ├── components/ +│ │ ├── hooks/ +│ │ ├── store/ +│ │ └── types/ +│ └── charts/ +│ ├── components/ +│ ├── hooks/ +│ └── types/ +├── shared/ +│ ├── components/ui/ +│ ├── hooks/ +│ └── utils/ +└── lib/ +└── queryClient.ts \ No newline at end of file diff --git a/.agents/skills/start-issue/SKILL.md b/.agents/skills/start-issue/SKILL.md new file mode 100644 index 0000000..c3cc84a --- /dev/null +++ b/.agents/skills/start-issue/SKILL.md @@ -0,0 +1,42 @@ +--- +name: start-issue +description: Automate issue triage, planning, and branch creation for issue-driven development. +version: 1.0.0 +tags: [github, workflow, planning] +--- + +# Start Issue Skill + +Use this skill to go from GitHub issue → plan → branch in a consistent, repeatable way. + +## Workflow + +1. **Load issue context** + - Use `gh issue view --json title,body,labels,assignees,comments,updatedAt` to avoid project-card GraphQL issues. + - Summarize requirements, constraints, and acceptance criteria. + +2. **Clarify scope** + - If requirements are ambiguous, ask targeted questions before code changes. + +3. **Locate code** + - Use `rg --files` to identify modules. + - Use `rg -n ""` to locate patterns. + +4. **Plan** + - Draft a short, ordered plan (2-5 steps). + - Confirm with the user before executing if the plan changes scope. + +5. **Create branch** + - Use `git checkout -b feat/issue--` (or `fix/issue--` for bugfixes). + +6. **Post plan comment (optional)** + - If requested, post the plan with `gh issue comment --body "..."`. + +7. **Implement and validate** + - Make focused edits, run relevant tests/lints, and summarize changes. + +## Notes + +- Never commit directly to `main`. +- Prefer small, focused commits with clear intent. +- If the issue asks for a plan only, stop before code changes. diff --git a/.agents/skills/tailwind-css-data-viz/SKILL.md b/.agents/skills/tailwind-css-data-viz/SKILL.md new file mode 100644 index 0000000..455dc35 --- /dev/null +++ b/.agents/skills/tailwind-css-data-viz/SKILL.md @@ -0,0 +1,176 @@ +--- +name: Tailwind CSS Data Visualization Patterns +description: Tailwind CSS configurations and patterns for building data-dense financial dashboard UIs +version: 1.0.0 +tags: [tailwind, css, dashboard, data-visualization, dark-mode] +--- + +# Tailwind CSS Data Visualization Patterns + +## Configuration +```typescript +// tailwind.config.ts +import type { Config } from 'tailwindcss'; + +const config: Config = { + content: ['./src/**/*.{js,ts,jsx,tsx,mdx}'], + darkMode: 'class', + theme: { + extend: { + colors: { + gain: { 500: '#10b981', 600: '#059669', 700: '#047857' }, + loss: { 500: '#ef4444', 600: '#dc2626', 700: '#b91c1c' }, + dashboard: { + bg: '#f8fafc', 'bg-dark': '#0f172a', + card: '#ffffff', 'card-dark': '#1e293b', + border: '#e2e8f0', 'border-dark': '#334155', + }, + chart: { primary: '#1e40af', grid: '#f3f4f6', 'grid-dark': '#1e293b' }, + }, + fontSize: { + 'metric-xs': ['0.75rem', { lineHeight: '1rem' }], + 'metric-sm': ['0.875rem', { lineHeight: '1.25rem' }], + 'kpi': ['1.5rem', { lineHeight: '2rem', fontWeight: '600' }], + 'kpi-lg': ['2rem', { lineHeight: '2.5rem', fontWeight: '700' }], + }, + spacing: { card: '1rem', 'card-lg': '1.5rem', section: '2rem' }, + gridTemplateColumns: { + sidebar: 'minmax(200px, 260px) 1fr', + metrics: 'repeat(auto-fit, minmax(200px, 1fr))', + }, + animation: { + 'flash-gain': 'flashGain 600ms ease-out', + 'flash-loss': 'flashLoss 600ms ease-out', + }, + keyframes: { + flashGain: { '0%': { backgroundColor: 'rgba(16, 185, 129, 0.25)' }, '100%': { backgroundColor: 'transparent' } }, + flashLoss: { '0%': { backgroundColor: 'rgba(239, 68, 68, 0.25)' }, '100%': { backgroundColor: 'transparent' } }, + }, + boxShadow: { + card: '0 1px 3px 0 rgba(0, 0, 0, 0.08)', + 'card-hover': '0 4px 12px 0 rgba(0, 0, 0, 0.1)', + }, + }, + }, +}; +export default config; +``` + +## Dashboard Layout +```html +
+ +
+
+

Dashboard

+
+
+
+
+``` + +## KPI Card +```html +
+
+
+

+ Total Revenue +

+

+ $2,847,293 +

+
+
+ +
+
+
+ + +12.5% + + vs last period +
+
+``` + +## Chart Container +```html +
+
+

GDP Growth Rate

+

Quarterly, seasonally adjusted

+
+
+
+``` + +## Data Table +```html +
+ + + + + + + + + + + + + + + +
SymbolPriceChange
AAPL$178.23+2.34%
+
+``` + +## States: Loading, Error, Empty +```html + +
+
+
+
+ + +
+

Failed to load data

+ +
+ + +
+

No data available

+
+``` + +## Tabular Numbers +```html +$1,234,567.89 ++12.34% +-5.67% +``` + +## Dark Mode Toggle +```javascript +function toggleDarkMode() { + document.documentElement.classList.toggle('dark'); + localStorage.setItem('theme', document.documentElement.classList.contains('dark') ? 'dark' : 'light'); +} + +// Initialize +if (localStorage.theme === 'dark' || (!localStorage.theme && window.matchMedia('(prefers-color-scheme: dark)').matches)) { + document.documentElement.classList.add('dark'); +} +``` \ No newline at end of file diff --git a/.agents/skills/testing-strategies/SKILL.md b/.agents/skills/testing-strategies/SKILL.md new file mode 100644 index 0000000..5273346 --- /dev/null +++ b/.agents/skills/testing-strategies/SKILL.md @@ -0,0 +1,715 @@ +--- +name: testing-strategies +description: Comprehensive testing patterns and examples for Dagster, dbt, FastAPI, and React in the economic data project. +--- + +# Testing Strategies for Economic Data Project + +This skill provides comprehensive testing patterns and examples for all components of the economic data project: Dagster (orchestration), dbt (transformations), FastAPI (backend), and React (frontend). + +## Quick Reference + +| Component | Framework | Run Command | Location | +|-----------|-----------|-------------|----------| +| Dagster | pytest | `make test` or `cd macro_agents && uv run pytest tests/ -v` | `macro_agents/tests/` | +| dbt | dbt test | `dbt test` or `dbt test --select model_name` | `dbt_project/tests/` + schema.yml | +| Backend API | pytest | `cd api && PYTHONPATH=.. uv run pytest tests/ -v` | `api/tests/` | +| Frontend | vitest | `cd frontend && npm test -- --run` | `frontend/tests/` | + +--- + +## 1. Dagster Testing + +### Test Categories + +1. **Unit Tests** - Test individual resources and functions +2. **Integration Tests** - Test Dagster definitions load correctly +3. **Schedule/Sensor Tests** - Test automation configuration +4. **Data Validation Tests** - Test data schemas and integrity + +### Resource Unit Tests + +Test Dagster resources in isolation with mocked dependencies: + +```python +import pytest +from unittest.mock import Mock, patch +from macro_agents.defs.resources.motherduck import MotherDuckResource + +class TestMotherDuckResource: + def test_initialization_dev_environment(self): + resource = MotherDuckResource( + md_token="test_token", + environment="dev", + local_path="test.duckdb" + ) + assert resource.environment == "dev" + assert resource.db_connection == "test.duckdb" + + def test_initialization_prod_environment(self): + resource = MotherDuckResource( + md_token="test_token", + environment="prod", + md_database="test_db" + ) + assert resource.environment == "prod" + assert resource.db_connection == "md:?motherduck_token=test_token" +``` + +### Testing with Temporary Database + +Use `tempfile` for tests requiring actual database operations: + +```python +import tempfile +import os +import polars as pl + +def test_table_exists(self): + with tempfile.NamedTemporaryFile(suffix=".duckdb", delete=False) as tmp_file: + resource = MotherDuckResource( + md_token="test_token", + environment="dev", + local_path=tmp_file.name + ) + + assert not resource.table_exists("non_existent_table") + + test_df = pl.DataFrame({"id": [1, 2, 3]}) + resource.drop_create_duck_db_table("test_table", test_df) + assert resource.table_exists("test_table") + + os.unlink(tmp_file.name) +``` + +### Integration Tests - Definitions Load + +Verify all Dagster definitions load without errors: + +```python +from macro_agents.definitions import defs + +class TestDagsterDefinitions: + def test_definitions_load(self): + assert defs is not None + assert len(defs.assets) > 0 + assert len(defs.resources) > 0 + + def test_all_resources_are_configurable(self): + for resource_key, resource in defs.resources.items(): + assert hasattr(resource, "model_config") + assert hasattr(type(resource), "model_fields") +``` + +### Schedule and Sensor Tests + +Test schedule configuration and timing: + +```python +from macro_agents.defs.replication import weekly_replication_schedule + +class TestSchedules: + def test_weekly_replication_schedule_configuration(self): + assert weekly_replication_schedule.name == "weekly_replication_schedule" + assert weekly_replication_schedule.cron_schedule == "0 2 * * 0" + assert weekly_replication_schedule.execution_timezone == "America/New_York" + + def test_schedule_timing_consistency(self): + replication_hour = 2 # 2 AM EST + assert replication_hour <= 6, "Weekly replication should run early in the morning" +``` + +### Mocking External Services + +Use `patch` for external API calls and credentials: + +```python +from unittest.mock import Mock, patch + +def test_sling_resource_setup_with_credentials(self): + test_creds = { + "type": "service_account", + "project_id": "test-project", + "private_key": "-----BEGIN PRIVATE KEY-----\ntest\n-----END PRIVATE KEY-----\n", + } + creds_json = json.dumps(test_creds) + mock_context = Mock() + mock_context.log = Mock() + + with patch.dict(os.environ, {"SLING_GOOGLE_APPLICATION_CREDENTIALS": creds_json}): + with patch("macro_agents.defs.replication.sling.SlingConnectionResource") as mock_conn: + resource = SlingResourceWithCredentials() + resource.setup_for_execution(mock_context) + assert hasattr(resource, "_sling_resource") +``` + +### Skip Tests When Environment Not Available + +Use `pytest.mark.skipif` for optional integration tests: + +```python +@pytest.mark.skipif( + not os.getenv("MOTHERDUCK_TOKEN"), + reason="MOTHERDUCK_TOKEN not set" +) +def test_motherduck_connection_parameters(self): + token = os.getenv("MOTHERDUCK_TOKEN") + assert len(token) > 0 +``` + +--- + +## 2. dbt Testing + +### Test Types + +1. **Schema Tests** - Defined in `schema.yml` files (not_null, unique, accepted_values) +2. **Data Tests** - Custom SQL in `tests/` folder +3. **dbt Project Validation** - Python tests that run dbt commands + +### Schema Tests in schema.yml + +```yaml +version: 2 + +models: + - name: major_indicies_summary + description: "Performance analysis of major stock market indices" + columns: + - name: symbol + tests: + - not_null + - unique + - name: time_period + tests: + - not_null + - accepted_values: + values: ['12_weeks', '6_months', '1_year', '5_years'] + - name: total_return_pct + tests: + - not_null +``` + +### Custom Data Tests + +Create SQL files in `dbt_project/tests/` that return failing rows: + +```sql +-- tests/test_forward_returns_not_zero.sql +{{ config(severity='warn') }} + +{% set models_to_test = [ + 'currency_analysis_return', + 'major_indicies_analysis_return', +] %} + +{% for model in models_to_test %} + SELECT + symbol, + month_date, + pct_change_q1_forward, + 'zero_forward_return' as issue_type + FROM {{ ref(model) }} + WHERE ABS(pct_change_q1_forward) < 0.01 + AND pct_change_q1_forward IS NOT NULL + + {% if not loop.last %} + UNION ALL + {% endif %} +{% endfor %} +``` + +### Data Completeness Tests + +```sql +-- tests/test_weekly_data_completeness.sql +WITH weekly_counts AS ( + SELECT + DATE_TRUNC('week', date) AS week_start, + COUNT(*) AS record_count + FROM {{ ref('stg_us_sectors') }} + WHERE date >= CURRENT_DATE - INTERVAL '12 weeks' + GROUP BY DATE_TRUNC('week', date) +) +SELECT week_start +FROM weekly_counts +WHERE record_count < 10 +``` + +### Python Tests for dbt Validation + +Test dbt project from Python (runs in Dagster test suite): + +```python +import subprocess +from pathlib import Path + +class TestDbtProjectValidation: + @pytest.fixture + def dbt_project_dir(self): + return Path(__file__).parent.parent.parent / "dbt_project" + + def test_dbt_parse_succeeds(self, dbt_project_dir): + original_cwd = os.getcwd() + try: + os.chdir(dbt_project_dir) + result = subprocess.run( + ["dbt", "parse", "--target", "local"], + capture_output=True, + text=True, + timeout=60, + ) + if result.returncode != 0: + pytest.fail(f"dbt parse failed:\n{result.stderr}") + finally: + os.chdir(original_cwd) + + def test_dbt_list_models_succeeds(self, dbt_project_dir): + original_cwd = os.getcwd() + try: + os.chdir(dbt_project_dir) + result = subprocess.run( + ["dbt", "list", "--target", "local", "--resource-type", "model"], + capture_output=True, + text=True, + ) + models = [l.strip() for l in result.stdout.split("\n") if l.strip()] + assert len(models) > 0, "No models found" + finally: + os.chdir(original_cwd) +``` + +--- + +## 3. FastAPI Backend Testing + +### Test Setup (conftest.py) + +```python +import pytest +from fastapi.testclient import TestClient +from api.main import app + +@pytest.fixture +def client(): + return TestClient(app) + +@pytest.fixture +def temp_duckdb_file(): + with tempfile.NamedTemporaryFile(suffix=".duckdb", delete=False) as tmp_file: + yield tmp_file.name + if os.path.exists(tmp_file.name): + os.unlink(tmp_file.name) + +@pytest.fixture +def mock_motherduck_resource(temp_duckdb_file): + return MotherDuckResource( + md_token="test_token", + environment="dev", + local_path=temp_duckdb_file, + ) +``` + +### API Endpoint Tests + +```python +from fastapi.testclient import TestClient +from api.main import app + +client = TestClient(app) + +def test_get_market_summaries(): + response = client.get("/api/markets/summary") + assert response.status_code in [200, 500] # 500 if no DB + if response.status_code == 200: + data = response.json() + assert "results" in data + assert "count" in data + assert isinstance(data["results"], list) + +def test_get_market_summaries_with_category(): + response = client.get("/api/markets/summary?category=sector") + assert response.status_code in [200, 500] + +def test_get_market_summaries_invalid_limit(): + response = client.get("/api/markets/summary?limit=0") + assert response.status_code == 400 + + response = client.get("/api/markets/summary?limit=1001") + assert response.status_code == 400 +``` + +### Testing Validation Errors + +```python +def test_invalid_limit_returns_400(): + response = client.get("/api/markets/summary?limit=0") + assert response.status_code == 400 + assert "detail" in response.json() +``` + +### Service Layer Tests + +```python +def test_data_service_initialization(): + service = DataService() + assert service.motherduck is not None + +def test_get_market_summaries_empty(): + service = DataService() + results = service.get_market_summaries() + assert isinstance(results, list) +``` + +--- + +## 4. React Frontend Testing + +### Test Setup (tests/setup.ts) + +```typescript +import { expect, afterEach } from 'vitest'; +import { cleanup } from '@testing-library/react'; +import * as matchers from '@testing-library/jest-dom/matchers'; + +expect.extend(matchers); + +afterEach(() => { + cleanup(); +}); +``` + +### Component Tests with Providers + +Wrap components with required providers: + +```tsx +import { describe, it, expect, vi } from 'vitest'; +import { render, screen } from '@testing-library/react'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; +import { ThemeProvider } from '../theme/ThemeContext'; + +function renderWithProviders(component: React.ReactElement) { + const queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false }, + }, + }); + + return render( + + {component} + + ); +} +``` + +### Mocking API Hooks + +```tsx +import * as apiHooks from '../hooks/useApi'; + +vi.mock('../hooks/useApi'); + +describe('Dashboard', () => { + it('should render loading state', () => { + vi.mocked(apiHooks.useLatestEconomyState).mockReturnValue({ + data: undefined, + isLoading: true, + error: null, + } as ReturnType); + + renderWithProviders(); + expect(screen.getAllByText('Loading...').length).toBeGreaterThan(0); + }); + + it('should render dashboard with data', async () => { + const mockData = { + analysis_date: '2024-01-01', + model_name: 'test-model', + analysis_content: 'Test content', + }; + + vi.mocked(apiHooks.useLatestEconomyState).mockReturnValue({ + data: mockData, + isLoading: false, + error: null, + } as ReturnType); + + renderWithProviders(); + await waitFor(() => { + expect(screen.getByText('Latest Economy State')).toBeInTheDocument(); + }); + }); +}); +``` + +### Testing User Interactions + +```tsx +import { fireEvent } from '@testing-library/react'; + +it('should call onRangeChange when a button is clicked', () => { + const onRangeChange = vi.fn(); + render(); + + fireEvent.click(screen.getByText('2Y')); + expect(onRangeChange).toHaveBeenCalledWith('2Y'); +}); +``` + +### Testing with Theme Provider + +```tsx +import { ThemeProvider } from '../../theme/ThemeContext'; + +function renderWithTheme(component: React.ReactElement) { + return render({component}); +} + +describe('TimeSeriesChart', () => { + const mockData = [ + { date: '2024-01-01', value: 100 }, + { date: '2024-01-02', value: 105 }, + ]; + + it('should render chart with data', () => { + renderWithTheme(); + expect(screen.getByText('Test Chart')).toBeInTheDocument(); + }); + + it('should render chart without title', () => { + const { container } = renderWithTheme(); + expect(container.querySelector('h3')).not.toBeInTheDocument(); + }); +}); +``` + +### Testing Async Effects and Cleanup + +Test components with useEffect cleanup to ensure no memory leaks: + +```tsx +import { render, screen, waitFor, act } from '@testing-library/react'; + +describe('DataLoader (async useEffect)', () => { + beforeEach(() => { + vi.useFakeTimers(); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + it('should not update state after unmount', async () => { + const consoleSpy = vi.spyOn(console, 'error'); + + const mockFetch = vi.fn().mockImplementation( + () => new Promise(resolve => setTimeout(() => resolve({ data: 'test' }), 1000)) + ); + vi.mocked(api.fetchData).mockImplementation(mockFetch); + + const { unmount } = renderWithProviders(); + + // Unmount before the fetch resolves + unmount(); + + // Fast-forward past the timeout + await act(async () => { + vi.advanceTimersByTime(1500); + }); + + // Should not have any "Can't perform state update on unmounted component" warnings + expect(consoleSpy).not.toHaveBeenCalled(); + }); + + it('should cancel previous request when dependencies change', async () => { + const abortSpy = vi.fn(); + global.AbortController = vi.fn().mockImplementation(() => ({ + signal: {}, + abort: abortSpy, + })); + + const { rerender } = renderWithProviders(); + + // Trigger a re-render with new props before first fetch completes + rerender( + + + + ); + + // First request should have been aborted + expect(abortSpy).toHaveBeenCalled(); + }); + + it('should handle loading, success, and error states', async () => { + vi.mocked(api.fetchData).mockResolvedValueOnce({ items: [] }); + + renderWithProviders(); + + // Loading state + expect(screen.getByTestId('loading-spinner')).toBeInTheDocument(); + + await waitFor(() => { + // Success state + expect(screen.queryByTestId('loading-spinner')).not.toBeInTheDocument(); + expect(screen.getByText('No items found')).toBeInTheDocument(); + }); + }); +}); +``` + +### Testing Custom Hooks with Async Operations + +```tsx +import { renderHook, waitFor, act } from '@testing-library/react'; + +describe('useAsyncData hook', () => { + it('should return data after fetch completes', async () => { + vi.mocked(api.getData).mockResolvedValueOnce({ result: 'test' }); + + const { result } = renderHook(() => useAsyncData('test-id')); + + expect(result.current.isLoading).toBe(true); + + await waitFor(() => { + expect(result.current.isLoading).toBe(false); + expect(result.current.data).toEqual({ result: 'test' }); + }); + }); + + it('should handle errors gracefully', async () => { + vi.mocked(api.getData).mockRejectedValueOnce(new Error('Network error')); + + const { result } = renderHook(() => useAsyncData('test-id')); + + await waitFor(() => { + expect(result.current.isLoading).toBe(false); + expect(result.current.error).toBe('Network error'); + }); + }); + + it('should cleanup on unmount', async () => { + const mockAbort = vi.fn(); + global.AbortController = vi.fn().mockImplementation(() => ({ + signal: { aborted: false }, + abort: mockAbort, + })); + + const { unmount } = renderHook(() => useAsyncData('test-id')); + + unmount(); + + expect(mockAbort).toHaveBeenCalled(); + }); +}); +``` + +--- + +## 5. Running Tests + +### All Tests (via Makefile) + +```bash +make test +``` + +### Dagster Tests + +```bash +cd macro_agents +uv run pytest tests/ -v + +# Run specific test file +uv run pytest tests/test_resources.py -v + +# Run specific test class +uv run pytest tests/test_resources.py::TestMotherDuckResource -v + +# Run with coverage +uv run pytest tests/ --cov=macro_agents --cov-report=html +``` + +### dbt Tests + +```bash +cd dbt_project + +# Run all tests +dbt test + +# Run tests for specific model +dbt test --select major_indicies_summary + +# Run data tests only +dbt test --select test_type:data + +# Run schema tests only +dbt test --select test_type:schema +``` + +### API Tests + +```bash +cd api +PYTHONPATH=.. uv run pytest tests/ -v + +# Run specific test file +uv run pytest tests/test_markets.py -v +``` + +### Frontend Tests + +```bash +cd frontend + +# Run tests (watch mode) +npm test + +# Run tests once +npm test -- --run + +# Run with UI +npm run test:ui +``` + +--- + +## 6. Best Practices + +### General + +1. **Test one thing per test** - Each test should verify a single behavior +2. **Use descriptive test names** - Name should explain what is being tested +3. **Arrange-Act-Assert** - Structure tests clearly +4. **Clean up resources** - Use fixtures with cleanup for temp files/databases + +### Dagster + +1. **Mock external services** - Never call real APIs in unit tests +2. **Test resource initialization** - Verify both dev and prod configurations +3. **Test definitions load** - Catch import/config errors early +4. **Use `pytest.mark.skipif`** - Skip integration tests when credentials unavailable + +### dbt + +1. **Add schema tests for every model** - At minimum: not_null on key columns +2. **Use custom tests for business logic** - Data completeness, value ranges +3. **Set severity levels** - Use `severity='warn'` for non-blocking issues +4. **Test incrementally** - Run `dbt test --select model_name` during development + +### FastAPI + +1. **Test both success and error cases** - Include validation errors +2. **Use TestClient** - Simulates HTTP requests without running server +3. **Accept multiple status codes** - `[200, 500]` for endpoints that depend on DB state +4. **Test with fixtures** - Use `conftest.py` for shared test setup + +### React + +1. **Mock API hooks** - Don't make real API calls in tests +2. **Use providers** - Wrap with QueryClientProvider, ThemeProvider +3. **Test loading/error states** - Cover all UI states +4. **Prefer user-centric queries** - Use `getByText`, `getByRole` over `querySelector` diff --git a/.agents/skills/tufte-visualization/SKILL.md b/.agents/skills/tufte-visualization/SKILL.md new file mode 100644 index 0000000..c0692de --- /dev/null +++ b/.agents/skills/tufte-visualization/SKILL.md @@ -0,0 +1,11 @@ +--- +name: tufte-visualization-principles +description: Deprecated alias for design-philosophy. Use that skill instead. +version: 1.0.0 +--- + +# Deprecated + +This skill has been consolidated into `design-philosophy`. + +Use `/design-philosophy` for the canonical guidance. diff --git a/.agents/skills/typescript-financial-data-modeling/SKILL.md b/.agents/skills/typescript-financial-data-modeling/SKILL.md new file mode 100644 index 0000000..428dc8c --- /dev/null +++ b/.agents/skills/typescript-financial-data-modeling/SKILL.md @@ -0,0 +1,271 @@ +--- +name: TypeScript Financial Data Modeling +description: Type-safe data modeling patterns for financial and economic data visualization applications +version: 1.0.0 +tags: [typescript, financial-data, type-safety, zod, branded-types] +--- + +# TypeScript Financial Data Modeling + +## Branded Types for Domain Safety +```typescript +type Brand = T & { readonly __brand: B }; + +// Currency types +type USD = Brand; +type EUR = Brand; + +export function usd(value: number): USD { return value as USD; } +export function eur(value: number): EUR { return value as EUR; } + +// Type-safe operations +export function addUSD(a: USD, b: USD): USD { return (a + b) as USD; } +// addUSD(usd(100), eur(50)); // Compile error! + +// Percentage and Ratio +type Percentage = Brand; +type Ratio = Brand; +type BasisPoints = Brand; + +export function percentage(value: number): Percentage { + if (value < 0 || value > 100) throw new Error(`Invalid percentage: ${value}`); + return value as Percentage; +} + +export function percentageToRatio(pct: Percentage): Ratio { + return (pct / 100) as Ratio; +} +``` + +## Time Series Data Structures +```typescript +export interface TimeSeriesPoint { + readonly timestamp: Date; + readonly value: T; +} + +export interface TimeSeries { + readonly id: string; + readonly name: string; + readonly unit: string; + readonly frequency: 'daily' | 'weekly' | 'monthly' | 'quarterly' | 'yearly'; + readonly data: readonly TimeSeriesPoint[]; + readonly metadata?: { + readonly source: string; + readonly lastUpdated: Date; + readonly seasonallyAdjusted?: boolean; + }; +} + +export interface OHLCPoint { + readonly timestamp: Date; + readonly open: number; + readonly high: number; + readonly low: number; + readonly close: number; + readonly volume?: number; +} + +export interface StockSeries { + readonly symbol: string; + readonly exchange: string; + readonly currency: 'USD' | 'EUR' | 'GBP'; + readonly interval: '1min' | '5min' | 'daily' | 'weekly'; + readonly data: readonly OHLCPoint[]; +} +``` + +## Economic Indicators with Discriminated Unions +```typescript +interface BaseEconomicIndicator { + readonly id: string; + readonly country: string; + readonly releaseDate: Date; + readonly period: string; +} + +export interface GDPIndicator extends BaseEconomicIndicator { + readonly type: 'gdp'; + readonly value: number; + readonly growthRate: number; + readonly unit: 'billions_usd' | 'trillions_usd'; + readonly seasonallyAdjusted: boolean; +} + +export interface InflationIndicator extends BaseEconomicIndicator { + readonly type: 'inflation'; + readonly rate: number; + readonly monthOverMonth: number; + readonly yearOverYear: number; +} + +export interface UnemploymentIndicator extends BaseEconomicIndicator { + readonly type: 'unemployment'; + readonly rate: number; + readonly laborForceParticipation: number; +} + +export type EconomicIndicator = GDPIndicator | InflationIndicator | UnemploymentIndicator; + +// Type guards +export function isGDPIndicator(i: EconomicIndicator): i is GDPIndicator { + return i.type === 'gdp'; +} + +// Usage with narrowing +function formatIndicator(indicator: EconomicIndicator): string { + switch (indicator.type) { + case 'gdp': return `GDP: $${indicator.value}B (${indicator.growthRate}% growth)`; + case 'inflation': return `Inflation: ${indicator.yearOverYear}% YoY`; + case 'unemployment': return `Unemployment: ${indicator.rate}%`; + } +} +``` + +## Zod Runtime Validation +```typescript +import { z } from 'zod'; + +export const OHLCSchema = z.object({ + timestamp: z.coerce.date(), + open: z.number().positive(), + high: z.number().positive(), + low: z.number().positive(), + close: z.number().positive(), + volume: z.number().int().nonnegative().optional(), +}).refine( + (data) => data.high >= data.low && data.high >= data.open && data.low <= data.close, + { message: 'Invalid OHLC: high >= all, low <= all' } +); + +export const TimeSeriesSchema = z.object({ + id: z.string().min(1), + name: z.string().min(1), + frequency: z.enum(['daily', 'weekly', 'monthly', 'quarterly', 'yearly']), + data: z.array(z.object({ + timestamp: z.coerce.date(), + value: z.number(), + })), +}); + +// API response transformation +export const AlphaVantageQuoteSchema = z.object({ + 'Global Quote': z.object({ + '01. symbol': z.string(), + '05. price': z.string().transform(Number), + '09. change': z.string().transform(Number), + '10. change percent': z.string().transform(s => parseFloat(s.replace('%', ''))), + }), +}).transform((data) => ({ + symbol: data['Global Quote']['01. symbol'], + price: data['Global Quote']['05. price'], + change: data['Global Quote']['09. change'], + changePercent: data['Global Quote']['10. change percent'], +})); +``` + +## Nivo Integration Types +```typescript +import type { Serie, Datum } from '@nivo/line'; + +export interface FinancialDatum extends Datum { + x: Date | string | number; + y: number | null; + metadata?: { volume?: number; change?: number }; +} + +export interface FinancialSerie extends Omit { + id: string; + data: readonly FinancialDatum[]; + color?: string; +} + +export function toNivoSeries(timeSeries: TimeSeries[]): FinancialSerie[] { + return timeSeries.map((series) => ({ + id: series.id, + data: series.data.map((point) => ({ x: point.timestamp, y: point.value })), + })); +} +``` + +## Data Transformation Utilities +```typescript +export function calculateMovingAverage( + data: readonly TimeSeriesPoint[], + windowSize: number +): readonly TimeSeriesPoint[] { + return data.map((point, index) => { + const start = Math.max(0, index - windowSize + 1); + const window = data.slice(start, index + 1); + const avg = window.reduce((sum, p) => sum + p.value, 0) / window.length; + return { timestamp: point.timestamp, value: avg }; + }); +} + +export interface ChangeResult { + readonly absolute: number; + readonly percentage: number; + readonly direction: 'up' | 'down' | 'unchanged'; +} + +export function calculateChange(current: number, previous: number): ChangeResult { + const absolute = current - previous; + const percentage = previous !== 0 ? (absolute / previous) * 100 : 0; + return { + absolute, + percentage, + direction: absolute > 0 ? 'up' : absolute < 0 ? 'down' : 'unchanged', + }; +} +``` + +## API Response Types +```typescript +export type ApiResponse = + | { status: 'success'; data: T; timestamp: Date } + | { status: 'error'; error: { code: string; message: string } } + | { status: 'loading' }; + +export interface PaginatedResponse { + data: readonly T[]; + pagination: { + page: number; + pageSize: number; + totalItems: number; + hasNext: boolean; + }; +} +``` + +## Testing Utilities +```typescript +type DataFactory = (overrides?: Partial) => T; + +export const createMockOHLC: DataFactory = (overrides = {}) => ({ + timestamp: new Date(), + open: 100, + high: 105, + low: 98, + close: 102, + volume: 1000000, + ...overrides, +}); + +export function generatePriceData(startPrice: number, days: number, volatility = 0.02): OHLCPoint[] { + let price = startPrice; + return Array.from({ length: days }, (_, i) => { + const change = (Math.random() - 0.5) * volatility * price; + const open = price; + const close = price + change; + const result = { + timestamp: new Date(2024, 0, i + 1), + open, close, + high: Math.max(open, close) * 1.01, + low: Math.min(open, close) * 0.99, + volume: Math.floor(Math.random() * 10000000), + }; + price = close; + return result; + }); +} +``` \ No newline at end of file diff --git a/dbt_project/dbt_project.yml b/dbt_project/dbt_project.yml index bf8bca4..ba3f286 100644 --- a/dbt_project/dbt_project.yml +++ b/dbt_project/dbt_project.yml @@ -17,8 +17,8 @@ clean-targets: - "dbt_packages" # Model configurations -# Schema values map to BigQuery datasets. generate_schema_name macro must -# return the custom schema as-is (see macros/generate_schema_name.sql). +# Schema values map to BigQuery datasets. generate_schema_name appends +# environment suffixes for dev and staging targets while leaving prod canonical. models: dbt_project: +materialized: table diff --git a/dbt_project/models/markets/schema.yml b/dbt_project/models/markets/schema.yml index dbb4b24..c854a90 100644 --- a/dbt_project/models/markets/schema.yml +++ b/dbt_project/models/markets/schema.yml @@ -436,19 +436,10 @@ models: description: > Average 1-year standard deviation of daily dollar price changes. Higher = more price dispersion. For a normalized volatility ratio, - see annualized_volatility_ratio. + see the cross-asset annualized_volatility_ratio metric on asset_daily_returns. type: simple agg: average expr: std_diff_1yr - - name: annualized_volatility_ratio - label: Annualized Volatility Ratio - description: > - Average ratio of 1-year price StdDev to close price — a dimensionless - proxy for annualized volatility (analogous to daily vol * sqrt(252) / price). - Higher = more volatile relative to price level. - type: simple - agg: average - expr: std_diff_1yr / nullif(current_price, 0) - name: unique_symbols label: Distinct Symbols description: Count of distinct ticker symbols in the selection. diff --git a/dbt_project/models/semantic_layer/asset_daily_returns.sql b/dbt_project/models/semantic_layer/asset_daily_returns.sql new file mode 100644 index 0000000..9d7b8f1 --- /dev/null +++ b/dbt_project/models/semantic_layer/asset_daily_returns.sql @@ -0,0 +1,107 @@ +WITH stocks AS ( + SELECT + CONCAT('stock:', exchange, ':', symbol) AS asset_key, + 'stock' AS asset_class, + symbol AS asset_id, + symbol AS asset_name, + symbol, + symbol AS stock_symbol, + CAST(NULL AS STRING) AS sector_etf_symbol, + CAST(NULL AS STRING) AS commodity_name, + CAST(NULL AS STRING) AS commodity_unit, + exchange, + date AS trade_date, + current_price, + std_diff_1yr, + pct_change_1yr + FROM {{ ref('sp500_companies_analysis_return') }} +), + +sector_etfs AS ( + SELECT + CONCAT('sector_etf:', exchange, ':', symbol) AS asset_key, + 'sector_etf' AS asset_class, + symbol AS asset_id, + symbol AS asset_name, + symbol, + CAST(NULL AS STRING) AS stock_symbol, + symbol AS sector_etf_symbol, + CAST(NULL AS STRING) AS commodity_name, + CAST(NULL AS STRING) AS commodity_unit, + exchange, + date AS trade_date, + current_price, + std_diff_1yr, + pct_change_1yr + FROM {{ ref('us_sector_analysis_return') }} +), + +commodities AS ( + SELECT + CONCAT('commodity:', commodity_name, ':', commodity_unit) AS asset_key, + 'commodity' AS asset_class, + commodity_name AS asset_id, + commodity_name AS asset_name, + CAST(NULL AS STRING) AS symbol, + CAST(NULL AS STRING) AS stock_symbol, + CAST(NULL AS STRING) AS sector_etf_symbol, + commodity_name, + commodity_unit, + CAST(NULL AS STRING) AS exchange, + date AS trade_date, + current_price, + std_diff_1yr, + pct_change_1yr + FROM {{ ref('input_commodities_analysis_return') }} +) + +SELECT + asset_key, + asset_class, + asset_id, + asset_name, + symbol, + stock_symbol, + sector_etf_symbol, + commodity_name, + commodity_unit, + exchange, + trade_date, + current_price, + std_diff_1yr, + pct_change_1yr +FROM stocks +UNION ALL +SELECT + asset_key, + asset_class, + asset_id, + asset_name, + symbol, + stock_symbol, + sector_etf_symbol, + commodity_name, + commodity_unit, + exchange, + trade_date, + current_price, + std_diff_1yr, + pct_change_1yr +FROM sector_etfs +UNION ALL +SELECT + asset_key, + asset_class, + asset_id, + asset_name, + symbol, + stock_symbol, + sector_etf_symbol, + commodity_name, + commodity_unit, + exchange, + trade_date, + current_price, + std_diff_1yr, + pct_change_1yr +FROM commodities diff --git a/dbt_project/models/semantic_layer/schema.yml b/dbt_project/models/semantic_layer/schema.yml index e33bb6b..b43bfb9 100644 --- a/dbt_project/models/semantic_layer/schema.yml +++ b/dbt_project/models/semantic_layer/schema.yml @@ -12,3 +12,146 @@ models: tests: - not_null - unique + + - name: asset_daily_returns + description: > + Normalized daily return fact table across stocks, sector ETFs, and input + commodities. Grain: one row per asset per trading day. + tests: + - unique_combination: + arguments: + combination_of_columns: ['asset_key', 'trade_date'] + columns: + - name: asset_key + description: Stable cross-asset identifier prefixed by asset class. + entity: + name: asset + type: primary + dimension: + type: categorical + tests: + - not_null + - name: asset_class + description: Asset class for the normalized instrument. + dimension: + type: categorical + tests: + - not_null + - accepted_values: + arguments: + values: ['stock', 'sector_etf', 'commodity'] + - name: asset_id + description: Source identifier for the asset within its asset class. + dimension: + type: categorical + tests: + - not_null + - name: asset_name + description: Display name for the asset. + dimension: + type: categorical + tests: + - not_null + - name: symbol + description: Ticker symbol when the asset is exchange-traded. + dimension: + type: categorical + - name: stock_symbol + description: Stock ticker symbol for equity rows. + dimension: + type: categorical + - name: sector_etf_symbol + description: Sector ETF ticker symbol for sector ETF rows. + dimension: + type: categorical + - name: commodity_name + description: Commodity name for commodity rows. + dimension: + type: categorical + - name: commodity_unit + description: Unit of measurement for commodity rows. + dimension: + type: categorical + - name: exchange + description: Exchange where the asset trades, when applicable. + dimension: + type: categorical + - name: trade_date + description: Trading date. + granularity: day + dimension: + type: time + tests: + - not_null + - name: current_price + description: Current close or spot price. + - name: std_diff_1yr + description: Standard deviation of daily price differences over the past 1 year. + - name: pct_change_1yr + description: Percentage change from 1 year ago to current price. + +semantic_models: + - name: asset_daily_returns + description: > + Normalized daily return fact table across stocks, sector ETFs, and input + commodities. Grain: one row per asset per trading day. + model: ref('asset_daily_returns') + defaults: + agg_time_dimension: trade_date + entities: + - name: asset + type: primary + expr: asset_key + dimensions: + - name: asset_class + type: categorical + - name: asset_id + type: categorical + - name: asset_name + type: categorical + - name: symbol + type: categorical + - name: stock_symbol + type: categorical + - name: sector_etf_symbol + type: categorical + - name: commodity_name + type: categorical + - name: commodity_unit + type: categorical + - name: exchange + type: categorical + - name: trade_date + type: time + type_params: + time_granularity: day + measures: + - name: asset_volatility_proxy + description: Average 1-year standard deviation of daily price changes across normalized assets. + agg: average + expr: std_diff_1yr + - name: annualized_volatility_ratio_measure + description: > + Average annualized ratio of daily price-difference volatility to + current price across normalized stocks, sector ETFs, and commodities. + agg: average + expr: std_diff_1yr * sqrt(252) / nullif(current_price, 0) + +metrics: + - name: asset_volatility_proxy + label: Asset Price Volatility (1-Year StdDev) + description: Average 1-year standard deviation of daily price changes across normalized assets. + type: simple + type_params: + measure: + name: asset_volatility_proxy + - name: annualized_volatility_ratio + label: Annualized Volatility Ratio + description: > + Average annualized ratio of daily price-difference volatility to current + price across normalized stocks, sector ETFs, and commodities. Higher = + more volatile relative to price level. + type: simple + type_params: + measure: + name: annualized_volatility_ratio_measure diff --git a/dbt_project/profiles.yml b/dbt_project/profiles.yml index 6a947a9..a34fce9 100644 --- a/dbt_project/profiles.yml +++ b/dbt_project/profiles.yml @@ -7,9 +7,9 @@ econ_database: # deployed environments should use an attached or federated service account. method: oauth project: "{{ env_var('BIGQUERY_PROJECT', env_var('BIGQUERY_PROJECT_ID', 'econ-data-project-478800')) }}" - # Models land in *_dev datasets: economics_staging_dev, economics_marts_dev, etc. - # The generate_schema_name macro appends the _dev suffix automatically. - dataset: economics_staging_dev + # Base dataset for unconfigured resources. generate_schema_name appends + # _dev for dev targets, so this resolves to economics_staging_dev. + dataset: economics_staging threads: 4 location: "{{ env_var('BIGQUERY_LOCATION', 'US') }}" timeout_seconds: 300 @@ -20,8 +20,9 @@ econ_database: # deployed environments should use an attached or federated service account. method: oauth project: "{{ env_var('BIGQUERY_PROJECT', env_var('BIGQUERY_PROJECT_ID', 'econ-data-project-478800')) }}" - # Models land in *_staging datasets for pre-prod validation. - dataset: economics_staging_staging + # Base dataset for unconfigured resources. generate_schema_name appends + # _staging for staging targets, so this resolves to economics_staging_staging. + dataset: economics_staging threads: 6 location: "{{ env_var('BIGQUERY_LOCATION', 'US') }}" timeout_seconds: 300