Skip to content

Commit 5dbbca6

Browse files
C00ldudeNoonanC00ldudeNoonan
authored
Merge pull request #27 from C00ldudeNoonan/deployment-fixes
update deployment logic
2 parents 9911773 + a2c644d commit 5dbbca6

4 files changed

Lines changed: 257 additions & 18 deletions

File tree

‎.github/workflows/deploy.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ env:
1515
ENABLE_FAST_DEPLOYS: 'true'
1616
PYTHON_VERSION: '3.10'
1717
DAGSTER_CLOUD_FILE: 'dagster_cloud.yaml'
18+
MOTHERDUCK_PROD_SCHEMA: ${{ secrets.MOTHERDUCK_PROD_SCHEMA }}
1819

1920
jobs:
2021
dagster_cloud_default_deploy:

‎Readme.md‎

Lines changed: 248 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,256 @@
1-
# Economic Data Application
1+
# Economic Data Project
22

3+
An end-to-end data application that ingests, transforms, and analyzes economic and financial market data using modern open-source tools. The application combines traditional data engineering workflows with AI-powered analysis agents to provide insights into economic cycles, market trends, and asset allocation strategies.
34

4-
## Introduction
5+
## Tools and Technologies
56

6-
This repository serves as a public reference on how to build a end to end data application using open-source tools. It covers the entire workflow from data collection and processing to visualization and deployment. This project will use python and the following open source tools/libraries:
7-
* duckdb
8-
* [duckdb Docs](https://duckdb.org/docs)
9-
* evidence
10-
* [evidence Docs](https://evidence.dev/docs)
11-
* polars
12-
* [polars Docs](https://docs.pola.rs/api/python/stable/reference/index.html)
7+
### Core Frameworks
8+
- **Dagster**: Orchestration framework for data pipelines, asset management, schedules, and sensors
9+
- [Dagster Documentation](https://docs.dagster.io)
10+
- **dbt**: SQL-based transformation framework for data modeling and analytics
11+
- [dbt Documentation](https://docs.getdbt.com)
12+
- **DSPy**: Framework for building and optimizing AI agents with LLMs
13+
- [DSPy Documentation](https://dspy-docs.vercel.app)
14+
- **DuckDB/MotherDuck**: Embedded analytical database with cloud sync capabilities
15+
- [DuckDB Documentation](https://duckdb.org/docs)
16+
- [MotherDuck Documentation](https://motherduck.com/docs)
17+
18+
### Supporting Technologies
19+
- **Polars**: High-performance dataframe library
20+
- [Polars Documentation](https://docs.pola.rs)
21+
- **Sling**: Data replication tool for syncing between databases
22+
- [Sling Documentation](https://docs.slingdata.io)
23+
- **BigQuery**: Cloud data warehouse for replication target
24+
- [BigQuery Documentation](https://cloud.google.com/bigquery/docs)
25+
- **Python**: Primary programming language (3.10-3.13)
1326

1427
## Data Sources
1528

16-
All data for this project is from publicly available sources. The data will be stored in a duckdb database embedded within the Evidence Directory.
29+
All data is sourced from publicly available APIs:
30+
31+
### Economic Data
32+
- **Federal Reserve Economic Data (FRED)**: Comprehensive economic indicators including GDP, inflation, employment, housing, trade, and financial conditions
33+
- [FRED API Documentation](https://fred.stlouisfed.org/docs/api/fred/)
34+
- **Bureau of Labor Statistics (BLS)**: Employment and labor market data
35+
- [BLS API Documentation](https://www.bls.gov/developers/api_signature.htm)
36+
- **Census Bureau**: Population and demographic data
37+
- [Census Bureau API Documentation](https://www.census.gov/data/developers/data-sets.html)
38+
39+
### Market Data
40+
- **Market Stack API**: Stock market data for major indices, sectors, global markets, currencies, fixed income, and commodities
41+
- [Market Stack API Documentation](https://marketstack.com/documentation)
42+
- **Treasury Yields**: U.S. Treasury bond yield curve data
43+
- **Realtor.com**: Housing market inventory and pricing data
44+
- [Realtor.com Research Data](https://www.realtor.com/research/data/)
45+
46+
## Project Structure
47+
48+
```
49+
economic-data-project/
50+
├── macro_agents/ # Main Dagster project
51+
│ ├── src/macro_agents/
52+
│ │ ├── definitions.py # Central Dagster definitions
53+
│ │ └── defs/
54+
│ │ ├── ingestion/ # Data ingestion assets
55+
│ │ │ ├── fred.py # FRED economic data
56+
│ │ │ ├── bls.py # Bureau of Labor Statistics
57+
│ │ │ ├── market_stack.py # Market data API
58+
│ │ │ └── treasury_yields.py
59+
│ │ ├── transformation/ # Data transformation
60+
│ │ │ ├── dbt.py # dbt integration
61+
│ │ │ └── financial_condition_index.py
62+
│ │ ├── agents/ # AI analysis agents (DSPy)
63+
│ │ │ ├── analysis_agent.py
64+
│ │ │ ├── economic_cycle_analyzer.py
65+
│ │ │ ├── asset_allocation_analyzer.py
66+
│ │ │ └── backtesting.py
67+
│ │ ├── resources/ # Dagster resources
68+
│ │ │ ├── motherduck.py # DuckDB/MotherDuck connection
69+
│ │ │ ├── fred.py # FRED API resource
70+
│ │ │ └── market_stack.py # Market Stack API resource
71+
│ │ ├── replication/ # Data replication
72+
│ │ │ └── sling.py # Sling replication to BigQuery
73+
│ │ └── schedules.py # Dagster schedules, sensors, jobs
74+
│ └── tests/ # Test suite
75+
├── dbt_project/ # dbt transformation project
76+
│ ├── dbt_project.yml # dbt configuration
77+
│ ├── profiles.yml # Connection profiles
78+
│ └── models/
79+
│ ├── staging/ # Staging layer models
80+
│ ├── government/ # Government data models
81+
│ ├── markets/ # Market data models
82+
│ ├── commodities/ # Commodity data models
83+
│ └── analysis/ # Analysis layer models
84+
├── dagster_cloud.yaml # Dagster Cloud deployment config
85+
└── makefile # Build and automation commands
86+
```
87+
88+
## Data Flow
89+
90+
### 1. Ingestion Layer (Dagster Assets)
91+
Raw data assets pull from external APIs and store in DuckDB/MotherDuck:
92+
- **FRED Data**: Partitioned by 70+ economic series codes, scheduled weekly
93+
- **Market Data**: Partitioned by ticker and month for indices, sectors, commodities, currencies
94+
- **Treasury Yields**: Daily yield curve data
95+
- **Housing Data**: Inventory and pricing data from BLS and Realtor.com
96+
97+
### 2. Transformation Layer (dbt Models)
98+
SQL-based transformations organized in layers:
99+
- **Staging**: Standardizes and cleans raw data (`stg_*` models)
100+
- **Government**: Aggregates economic indicators (`fred_*`, `housing_*` models)
101+
- **Markets**: Analyzes market returns and summaries (`*_summary`, `*_analysis_return` models)
102+
- **Commodities**: Commodity-specific analysis
103+
- **Analysis**: Combines economic and market data for advanced analytics (`base_historical_analysis`, `leading_econ_return_indicator`)
104+
105+
### 3. AI Analysis Layer (DSPy Agents)
106+
AI-powered analysis agents that operate on transformed data:
107+
- **Economic Cycle Analysis**: Identifies economic phases (expansion, peak, contraction, trough)
108+
- **Asset Allocation**: Generates portfolio recommendations based on economic conditions
109+
- **Backtesting**: Tests investment strategies against historical data
110+
- **Model Evaluation**: Continuous improvement of AI models using DSPy metrics
111+
112+
### 4. Replication Layer (Sling)
113+
Replicates transformed data from MotherDuck to BigQuery for downstream consumption.
114+
115+
## Environment Variables
116+
117+
Create a `.env` file in the `macro_agents` directory with the following variables:
118+
119+
### Required
120+
- `MODEL_NAME`: OpenAI model to use (e.g., `gpt-4-turbo-preview`, `gpt-3.5-turbo`)
121+
- `OPENAI_API_KEY`: OpenAI API authentication key
122+
- `FRED_API_KEY`: Federal Reserve Economic Data API key
123+
- `MARKETSTACK_API_KEY`: Market Stack API key
124+
- `MOTHERDUCK_TOKEN`: MotherDuck authentication token (for cloud sync)
125+
126+
### Optional (Development)
127+
- `ENVIRONMENT`: Environment setting (`dev` or `prod`, defaults to `dev`)
128+
- `DBT_TARGET`: dbt target environment (`local`, `dev`, or `prod`, defaults to `local`)
129+
- `DBT_PROJECT_DIR`: Path to dbt project directory (auto-detected if not set)
130+
131+
### Optional (Production/Replication)
132+
- `MOTHERDUCK_DATABASE`: MotherDuck database name
133+
- `MOTHERDUCK_PROD_SCHEMA`: MotherDuck production schema
134+
- `BIGQUERY_PROJECT_ID`: Google Cloud project ID for BigQuery
135+
- `BIGQUERY_LOCATION`: BigQuery dataset location
136+
- `BIGQUERY_DATASET`: BigQuery dataset name
137+
- `GOOGLE_APPLICATION_CREDENTIALS`: Path to Google Cloud service account credentials JSON file
138+
- `CENSUS_API_KEY`: Census Bureau API key (if using Census data)
139+
140+
## Quick Start
141+
142+
### Prerequisites
143+
- Python 3.10-3.13
144+
- uv (recommended) or pip for package management
145+
- DuckDB and MotherDuck account (for cloud sync)
146+
- API keys for data sources
147+
- OpenAI API key for AI agents
148+
149+
### Installation
150+
151+
1. **Clone the repository**
152+
```bash
153+
git clone <repository-url>
154+
cd economic-data-project
155+
```
156+
157+
2. **Install dependencies**
158+
```bash
159+
cd macro_agents
160+
uv sync # or pip install -e .[dev]
161+
```
162+
163+
3. **Install dbt packages**
164+
```bash
165+
cd ../dbt_project
166+
dbt deps
167+
```
168+
169+
4. **Set up environment variables**
170+
Create a `.env` file in the `macro_agents` directory with required variables (see Environment Variables section above).
171+
172+
5. **Validate setup**
173+
```bash
174+
# Test Dagster definitions
175+
cd macro_agents
176+
dg check defs
177+
178+
# Test dbt models
179+
cd ../dbt_project
180+
dbt compile
181+
dbt parse
182+
```
183+
184+
### Running Locally
185+
186+
**Start Dagster UI:**
187+
```bash
188+
cd macro_agents
189+
dagster dev
190+
```
191+
Navigate to `http://localhost:3000` to view and materialize assets.
192+
193+
**Run dbt models manually:**
194+
```bash
195+
cd dbt_project
196+
dbt run # Run all models
197+
dbt run --select staging.* # Run specific layer
198+
```
199+
200+
**Run tests:**
201+
```bash
202+
cd macro_agents
203+
pytest tests/ -v
204+
# Or use the makefile
205+
make test
206+
```
207+
208+
## Deployment
209+
210+
The project is configured for deployment on Dagster Cloud using the `dagster_cloud.yaml` configuration file. The deployment builds from the `macro_agents` directory and uses `macro_agents.definitions` as the entry point.
211+
212+
## Development
213+
214+
### Common Commands
215+
216+
```bash
217+
# Run tests
218+
make test
219+
220+
# Lint Python code
221+
make ruff
222+
223+
# Lint SQL code
224+
make lint
225+
226+
# Fix SQL linting issues
227+
make fix
228+
229+
# Run pre-PR checks (linting, type checking, tests, security scans)
230+
make pre-pr
231+
```
232+
233+
### First Run Workflow
234+
235+
1. **Materialize ingestion assets** - Start with FRED data or Market Stack data via Dagster UI
236+
2. **Run dbt transformations** - Transform raw data through staging → marts → analysis layers (automated via eager assets)
237+
3. **Run analysis agents** - Execute DSPy agents on transformed data via Dagster UI
238+
4. **View results** - Check DuckDB/MotherDuck for analysis outputs
239+
240+
## Automation
241+
242+
- **Ingestion Assets**: Scheduled weekly on Mondays at midnight (FRED data)
243+
- **dbt Models**: Eager automation (run automatically when upstream data changes)
244+
- **Analysis Agents**: On-demand or scheduled via Dagster jobs
245+
- **Replication**: Monthly partitioned replication to BigQuery via Sling
246+
247+
## Testing
17248

18-
* Census Bureau
19-
* [Census Bureau API](https://www.census.gov/content/dam/Census/data/developers/api-user-guide/api-guide.pdf)
20-
* St. Louis Federal Reserve
21-
* [FRED API](https://fred.stlouisfed.org/docs/api/fred/)
22-
* Realtor.com
23-
* [Realtor.com API](https://www.realtor.com/research/data/)
249+
Test suite located in `macro_agents/tests/`:
250+
- Unit tests for analysis agents
251+
- Integration tests for end-to-end workflows
252+
- Tests for Dagster asset descriptions
253+
- Tests for dbt model descriptions
254+
- Resource and schedule tests
24255

256+
Run tests using `make test` or `pytest tests/ -v` from the `macro_agents` directory.

‎dagster_cloud.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,6 @@ locations:
44
module_name: macro_agents.definitions
55
working_directory: ./macro_agents
66
build:
7-
directory: ./macro_agents
7+
directory: ./macro_agents
8+
env_vars:
9+
GIT_PYTHON_REFRESH: quiet

‎macro_agents/src/macro_agents/defs/transformation/dbt.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,12 @@
11
from pathlib import Path
2+
import os
3+
4+
# Set GIT_PYTHON_REFRESH to quiet to suppress git executable errors in environments where git is not available
5+
# This is needed because dagster-dbt imports GitPython, which requires git to be in PATH
6+
os.environ.setdefault("GIT_PYTHON_REFRESH", "quiet")
27

38
from dagster_dbt import DbtProject, dbt_assets, DbtCliResource, DagsterDbtTranslator
49
import dagster as dg
5-
import os
610
from typing import Any, Optional
711
from collections.abc import Mapping
812

0 commit comments

Comments
 (0)