-
Notifications
You must be signed in to change notification settings - Fork 8
Expand file tree
/
Copy path.env.dagster.example
More file actions
112 lines (96 loc) · 4.51 KB
/
Copy path.env.dagster.example
File metadata and controls
112 lines (96 loc) · 4.51 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
# Dagster OSS Environment Configuration
# Copy this file to .env and fill in your actual values
# NEVER commit .env to git!
# ============================================================================
# PostgreSQL - Dagster Storage Backend
# ============================================================================
# This database stores Dagster's run history, event logs, and schedule state
# It is separate from your data warehouse (BigQuery)
DAGSTER_PG_PASSWORD=change-me-to-secure-password
# ============================================================================
# Dagster UI Configuration
# ============================================================================
DAGSTER_UI_PORT=3000
# ============================================================================
# Dagster Log Pruning + PostgreSQL Backups (Optional) — see
# docker/DAGSTER_RETENTION_AND_BACKUP.md
# ============================================================================
# Log pruning deletes ONLY plain log-message rows (event_logs with
# dagster_event_type IS NULL). Materialization/observation/asset-check state is
# NEVER deleted; schedule/sensor tick retention is handled in dagster.yaml.
DAGSTER_LOG_RETENTION_DAYS=30
DAGSTER_LOG_CLEANUP_BATCH_SIZE=5000
DAGSTER_EVENT_CLEANUP_INTERVAL_SECONDS=3600
# Automated PostgreSQL backups (dagster_db_backup service).
DAGSTER_BACKUP_INTERVAL_SECONDS=86400
DAGSTER_BACKUP_RETENTION_DAYS=14
# Optional app-level backup encryption (disk-at-rest encryption applies regardless).
DAGSTER_BACKUP_GPG_PASSPHRASE=
# ============================================================================
# Google Cloud Platform — Primary Data Warehouse (BigQuery)
# ============================================================================
# GCS Bucket for DSPy models and FOMC data
GCS_BUCKET_NAME=your-gcs-bucket-name
# BigQuery - Analytics Warehouse
BIGQUERY_PROJECT=your-gcp-project-id
BIGQUERY_DATASET=economics_raw
BIGQUERY_LOCATION=US
# Note: GCP service account JSON should be placed at ./gcp-service-account.json
# ============================================================================
# Data API Keys
# ============================================================================
# FRED (Federal Reserve Economic Data)
# Get key from: https://fred.stlouisfed.org/docs/api/api_key.html
FRED_API_KEY=your-fred-api-key
# MarketStack (Market Data)
# Get key from: https://marketstack.com/
MARKETSTACK_API_KEY=your-marketstack-api-key
# ============================================================================
# LLM API Keys
# ============================================================================
# OpenAI (GPT-4, GPT-3.5)
# Get key from: https://platform.openai.com/api-keys
OPENAI_API_KEY=your-openai-api-key
# Anthropic (Claude)
# Get key from: https://console.anthropic.com/
ANTHROPIC_API_KEY=your-anthropic-api-key
# Google Gemini
# Get key from: https://aistudio.google.com/app/apikey
GOOGLE_GENERATIVE_AI_API_KEY=your-google-api-key
# ============================================================================
# Social Media & Other APIs
# ============================================================================
# Reddit
# Get credentials from: https://www.reddit.com/prefs/apps
REDDIT_CLIENT_ID=your-reddit-client-id
REDDIT_CLIENT_SECRET=your-reddit-client-secret
# GitHub
# Get token from: https://github.com/settings/tokens
# Needs: repo scope for creating issues
GITHUB_TOKEN=your-github-token
# ============================================================================
# Development vs Production
# ============================================================================
# Uncomment for local development
# DAGSTER_HOME=/tmp/dagster_home
# ENV=dev
# ============================================================================
# Local Docker Development
# ============================================================================
# To run Dagster locally via Docker Compose (works on Windows/Mac/Linux):
#
# make run-dagster-local # Start all services
# make dagster-local-logs # Tail logs
# make dagster-local-restart # Restart user code after code changes
# make dagster-local-stop # Stop everything
# make dagster-local-rebuild # Rebuild after dependency changes
#
# This uses docker-compose.local.yml overlay which:
# - Uses DefaultRunLauncher (no Docker socket needed)
# - Mounts source code + dbt project for live changes
# - Runs with lower memory limits for laptops
# - Disables DB cleanup (not needed locally)
#
# Override environment for local dev:
# ENVIRONMENT=dev
# DBT_TARGET=dev