Skip to content

Commit 9939de2

Browse files
C00ldudeNoonanC00ldudeNoonan
committed
mulit tenant project
1 parent 527f8a9 commit 9939de2

57 files changed

Lines changed: 2634 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.buildkite/dagster-buildkite/dagster_buildkite/steps/packages.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -506,6 +506,12 @@ def _example_packages_with_custom_config(ctx: BuildkiteContext) -> list[PackageS
506506
AvailablePythonVersion.V3_14, # duckdb
507507
],
508508
),
509+
PackageSpec(
510+
oss_path("examples/project_multi_tenant"),
511+
unsupported_python_versions=[
512+
AvailablePythonVersion.V3_10, # requires-python >= 3.11
513+
],
514+
),
509515
PackageSpec(
510516
oss_path("examples/with_great_expectations"),
511517
unsupported_python_versions=[

‎examples/project_multi_tenant/.env.example‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
LLM_BACKEND=embedded
2+
OLLAMA_BASE_URL=http://localhost:11434
3+
OLLAMA_TIMEOUT_SECONDS=300
4+
OLLAMA_LOCK_TIMEOUT_SECONDS=900
5+
OLLAMA_LOCK_PATH=data/.ollama_generate.lock
6+
HARBOR_OUTFITTERS_MODEL=qwen2.5:0.5b
7+
SUMMIT_FINANCIAL_MODEL=qwen2.5:1.5b
8+
BEACON_HQ_MODEL=qwen2.5:0.5b

‎examples/project_multi_tenant/.gitignore‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
__pycache__/
2+
.pytest_cache/
3+
.ruff_cache/
4+
.venv/
5+
.dagster_home/
6+
.code_locations/
7+
.env
8+
*.pyc
9+
data/
10+
*.egg-info/
11+
tmp*/

‎examples/project_multi_tenant/README.md‎

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
# Dagster Multi-Tenant Example
2+
3+
This example project is a runnable Dagster demo that shows how to:
4+
5+
- split one project into multiple Dagster code locations
6+
- keep tenant pipelines isolated while coordinating them from Dagster
7+
- treat LLM work as data engineering and context engineering, not just prompt calls
8+
- run different model/runtime combinations per code location
9+
- run end to end with a bundled local LLM backend and optionally swap to Ollama
10+
11+
## Getting Started
12+
13+
Bootstrap your own Dagster project with this example:
14+
15+
```bash
16+
dagster project from-example --name my-dagster-project --example project_multi_tenant
17+
```
18+
19+
To work with the copy that lives in this repository directly:
20+
21+
```bash
22+
cd dagster/examples/project_multi_tenant
23+
```
24+
25+
Then continue with the local setup below.
26+
27+
The example uses three business-facing code locations:
28+
29+
- `harbor_outfitters`: retail catalog enrichment
30+
- `summit_financial`: transaction risk scoring
31+
- `beacon_hq`: executive reporting built from upstream business outputs
32+
33+
## Architecture
34+
35+
```mermaid
36+
flowchart LR
37+
subgraph Dagster["Dagster Workspace"]
38+
H["harbor_outfitters\nbronze -> silver -> gold"]
39+
S["summit_financial\nbronze -> silver -> gold"]
40+
B["beacon_hq\nreporting + executive summary"]
41+
end
42+
43+
subgraph Shared["Shared Runtime"]
44+
O["Embedded local backend\nor optional Ollama"]
45+
D["DuckDB\nper-code-location databases"]
46+
R["shared/resources.py\nshared/io_managers.py"]
47+
end
48+
49+
H -->|sales_summary| B
50+
S -->|account_summary| B
51+
52+
H --> R
53+
S --> R
54+
B --> R
55+
56+
R --> O
57+
R --> D
58+
```
59+
60+
## What The Demo Shows
61+
62+
- `harbor_outfitters` builds catalog context from product, brand, and taxonomy data before generating enriched product copy.
63+
- `summit_financial` builds investigation context from transactions, accounts, and rules before generating transaction rationales.
64+
- `beacon_hq` depends on upstream business outputs and turns them into a short executive briefing.
65+
66+
The local setup is intentionally opinionated:
67+
68+
- one root development environment
69+
- one bundled deterministic LLM backend that works out of the box
70+
- one optional Ollama backend if you want live model calls
71+
- optional copied Python environments per code location for runtime marker demos
72+
73+
## Repository Layout
74+
75+
```text
76+
harbor_outfitters/ Retail assets, jobs, and schedules
77+
summit_financial/ Financial assets, jobs, and schedules
78+
beacon_hq/ Reporting assets, jobs, and sensors
79+
shared/ Shared LLM resource, IO manager, and metadata helpers
80+
scripts/ Local setup scripts for models and per-location envs
81+
vendor/ Tiny runtime marker packages installed per code location
82+
tests/ End-to-end unit tests with fake LLM resources
83+
workspace.yaml Multi-code-location Dagster workspace
84+
dagster_cloud.yaml Dagster+ style code location config
85+
```
86+
87+
## Local Stack
88+
89+
The default backend is fully in-project:
90+
91+
- `LLM_BACKEND=embedded` uses deterministic local responses and does not require Docker or model downloads.
92+
93+
If you switch to Ollama, each code location uses its own default model:
94+
95+
- `harbor_outfitters`: `qwen2.5:0.5b`
96+
- `summit_financial`: `qwen2.5:1.5b`
97+
- `beacon_hq`: `qwen2.5:0.5b`
98+
99+
Business-facing environment variables:
100+
101+
- `LLM_BACKEND`
102+
- `HARBOR_OUTFITTERS_MODEL`
103+
- `SUMMIT_FINANCIAL_MODEL`
104+
- `BEACON_HQ_MODEL`
105+
106+
The project also includes optional runtime marker packages:
107+
108+
- `harbor_outfitters`: `catalog_coach_runtime==1.4.0`
109+
- `summit_financial`: `risk_reviewer_runtime==2.2.0`
110+
- `beacon_hq`: `briefing_writer_runtime==0.9.0`
111+
112+
Those markers are bundled under `vendor/` and can be installed into isolated code-location environments if you want to demonstrate per-location runtime differences.
113+
114+
Assets are persisted to per-code-location DuckDB databases under `data/`.
115+
116+
## Quickstart
117+
118+
Copy the local environment file:
119+
120+
```bash
121+
cp .env.example .env
122+
```
123+
124+
Create the root development environment and install the project with its vendor runtime
125+
marker packages:
126+
127+
```bash
128+
python -m venv .venv
129+
source .venv/bin/activate
130+
pip install -e ".[dev]"
131+
pip install -e vendor/catalog_coach_runtime \
132+
-e vendor/risk_reviewer_runtime \
133+
-e vendor/briefing_writer_runtime
134+
```
135+
136+
Load the environment and start Dagster:
137+
138+
```bash
139+
set -a
140+
source .env
141+
set +a
142+
export DAGSTER_HOME=$(pwd)/.dagster_home
143+
mkdir -p "$DAGSTER_HOME"
144+
dagster dev -w workspace.yaml
145+
```
146+
147+
Then open `http://127.0.0.1:3000`. The webserver and daemon (schedules, sensors,
148+
backfills) both start automatically.
149+
150+
> **Note:** `dagster dev` is the standard way to start Dagster locally. If you have
151+
> `dagster-dg-cli` installed (included in the `[dev]` extras), you can also use
152+
> `dg dev -w workspace.yaml`.
153+
154+
## Optional Ollama Backend
155+
156+
If you want live model calls instead of the bundled local backend:
157+
158+
```bash
159+
source .venv/bin/activate
160+
export LLM_BACKEND=ollama
161+
docker compose up -d ollama
162+
./scripts/pull_ollama_models.sh
163+
dagster dev -w workspace.yaml
164+
```
165+
166+
## Optional Isolated Code-Location Environments
167+
168+
If you want each code location to carry its own runtime marker package in a copied environment:
169+
170+
```bash
171+
source .venv/bin/activate
172+
./scripts/setup_code_location_envs.sh
173+
```
174+
175+
## Validation
176+
177+
Run the local checks with:
178+
179+
```bash
180+
source .venv/bin/activate
181+
set -a
182+
source .env
183+
set +a
184+
export DAGSTER_HOME=$(pwd)/.dagster_home
185+
mkdir -p "$DAGSTER_HOME"
186+
ruff check .
187+
pytest -q -o cache_dir=/tmp/pytest-cache
188+
dagster definitions validate -w workspace.yaml
189+
```
190+
191+
## Demo Flow
192+
193+
With `dagster dev` running, open the Dagster UI at `http://127.0.0.1:3000` and
194+
materialize assets through the UI, or launch the pipeline jobs in order from
195+
the command line using the GraphQL API or the Jobs page.
196+
197+
The recommended order is:
198+
199+
1. **Harbor Outfitters + Summit Financial** (can run in parallel):
200+
- `harbor_catalog_publish_job` materializes all Harbor assets (bronze through gold)
201+
- `summit_risk_scoring_job` materializes all Summit assets (bronze through gold)
202+
2. **Beacon HQ** (depends on upstream outputs from step 1):
203+
- `beacon_reporting_inputs_job` builds `consolidated_revenue`, `risk_overview`, and `briefing_highlights`
204+
- `beacon_executive_briefing_job` produces the `executive_summary` and `executive_llm_audit_log`
205+
206+
If you enable Ollama, avoid launching the LLM-heavy runs in parallel unless you increase machine capacity.
207+
208+
## Jobs And Automation
209+
210+
The repo includes pipeline-shaped jobs for each business unit:
211+
212+
- `harbor_catalog_publish_job`
213+
- `summit_risk_scoring_job`
214+
- `beacon_executive_briefing_job`
215+
216+
It also includes simple cross-location orchestration:
217+
218+
- `harbor_daily_refresh_schedule`
219+
- `summit_daily_refresh_schedule`
220+
- `beacon_after_upstream_success_sensor`
221+
222+
The key point is that lineage crosses code locations, but execution stays per code location. Beacon reacts to Harbor and Summit outputs instead of collapsing everything into one monolithic job.
223+
224+
## Notes
225+
226+
- Runtime DataFrame metadata includes `row_count` and `top_5_rows` on asset materializations.
227+
- Tests use fake LLM resources from `tests/fakes.py`; the default project runtime uses the bundled local backend in `shared/resources.py`.
228+
- Ollama remains available as an opt-in backend for a live local-model demo.

‎examples/project_multi_tenant/beacon_hq/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
from .definitions import defs
2+
3+
__all__ = ["defs"]

‎examples/project_multi_tenant/beacon_hq/assets/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""Tenant Gamma asset modules."""

‎examples/project_multi_tenant/beacon_hq/assets/reports.py‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
from __future__ import annotations
2+
3+
import pandas as pd
4+
import dagster as dg
5+
6+
from shared.metadata import add_dataframe_preview_metadata
7+
from shared.resources import SupportsGenerate
8+
9+
10+
@dg.asset(
11+
group_name="silver",
12+
deps=[dg.AssetKey(["harbor_outfitters", "sales_summary"])],
13+
)
14+
def consolidated_revenue(context) -> pd.DataFrame:
15+
return add_dataframe_preview_metadata(
16+
context,
17+
pd.DataFrame(
18+
[
19+
{"tenant": "harbor_outfitters", "metric": "total_revenue", "value": 308.43},
20+
{
21+
"tenant": "harbor_outfitters",
22+
"metric": "top_category_revenue",
23+
"value": 149.97,
24+
},
25+
]
26+
),
27+
)
28+
29+
30+
@dg.asset(
31+
group_name="silver",
32+
deps=[dg.AssetKey(["summit_financial", "account_summary"])],
33+
)
34+
def risk_overview(context) -> pd.DataFrame:
35+
return add_dataframe_preview_metadata(
36+
context,
37+
pd.DataFrame(
38+
[
39+
{"tenant": "summit_financial", "metric": "avg_score", "value": 53.33},
40+
{
41+
"tenant": "summit_financial",
42+
"metric": "high_risk_transactions",
43+
"value": 2.0,
44+
},
45+
]
46+
),
47+
)
48+
49+
50+
@dg.asset(group_name="gold")
51+
def briefing_highlights(
52+
context,
53+
consolidated_revenue: pd.DataFrame,
54+
risk_overview: pd.DataFrame,
55+
) -> pd.DataFrame:
56+
revenue_lines = "; ".join(
57+
f"{row.metric}={row.value:.2f}" for row in consolidated_revenue.itertuples(index=False)
58+
)
59+
risk_lines = "; ".join(
60+
f"{row.metric}={row.value:.2f}" for row in risk_overview.itertuples(index=False)
61+
)
62+
return add_dataframe_preview_metadata(
63+
context,
64+
pd.DataFrame(
65+
[
66+
{
67+
"section": "weekly_briefing",
68+
"highlight_text": (
69+
f"Retail metrics: {revenue_lines}. Risk metrics: {risk_lines}."
70+
),
71+
}
72+
]
73+
),
74+
)
75+
76+
77+
@dg.asset(group_name="gold")
78+
def executive_context_packet(
79+
context,
80+
briefing_highlights: pd.DataFrame,
81+
) -> pd.DataFrame:
82+
return add_dataframe_preview_metadata(
83+
context,
84+
pd.DataFrame(
85+
[
86+
{
87+
"section": "weekly_briefing",
88+
"context_packet": (
89+
"Write a concise executive summary using this prepared context. "
90+
+ str(briefing_highlights.iloc[0]["highlight_text"])
91+
),
92+
}
93+
]
94+
),
95+
)
96+
97+
98+
@dg.asset(group_name="gold")
99+
def executive_summary(
100+
context,
101+
executive_context_packet: pd.DataFrame,
102+
llm: dg.ResourceParam[SupportsGenerate],
103+
) -> pd.DataFrame:
104+
context.add_output_metadata(llm.runtime_metadata())
105+
context_packet = str(executive_context_packet.iloc[0]["context_packet"])
106+
response = llm.generate(context_packet)
107+
return add_dataframe_preview_metadata(context, pd.DataFrame([{"summary": response}]))
108+
109+
110+
@dg.asset(group_name="gold")
111+
def executive_llm_audit_log(
112+
context,
113+
executive_context_packet: pd.DataFrame,
114+
executive_summary: pd.DataFrame,
115+
) -> pd.DataFrame:
116+
return add_dataframe_preview_metadata(
117+
context,
118+
pd.DataFrame(
119+
[
120+
{
121+
"context_packet": str(executive_context_packet.iloc[0]["context_packet"]),
122+
"summary": str(executive_summary.iloc[0]["summary"]),
123+
}
124+
]
125+
),
126+
)

0 commit comments

Comments
 (0)