Skip to content

Commit 7d1c336

Browse files
RonaldHensbergenSemTiOnerenovate[bot]
authored
Fixes for test case T3 - part 1 (RonaldHensbergen#127)
* Fixing Dagster user-code being unhealthy * Test-case T1 succeeds! * Updating the test checklist * Updated test with correct Dockerfile * Linting issues (why a file I didn't change?) * Updating Dockerfile * Adding healthcheck to docker-daemon image * Added test for dagster daemon running when dagster is part of the profile * Get Postgres ready for other modules * init-db loads at compose up and databseas will survive restarts * Get branch up to date (RonaldHensbergen#126) * Fix/dagster round 2 (RonaldHensbergen#117) * Fixing Dagster user-code being unhealthy * Test-case T1 succeeds! * Updating the test checklist * Updated test with correct Dockerfile * Linting issues (why a file I didn't change?) * Fix/dagster round 2 (RonaldHensbergen#121) * Fixing Dagster user-code being unhealthy * Test-case T1 succeeds! * Updating the test checklist * Updated test with correct Dockerfile * Linting issues (why a file I didn't change?) * Updating Dockerfile * Adding healthcheck to docker-daemon image * Added test for dagster daemon running when dagster is part of the profile * Fix/dagster round 2 (RonaldHensbergen#117) (RonaldHensbergen#120) * Fixing Dagster user-code being unhealthy * Test-case T1 succeeds! * Updating the test checklist * Updated test with correct Dockerfile * Linting issues (why a file I didn't change?) * docs: add pre-commit setup for docs and python quality checks (RonaldHensbergen#118) Closes RonaldHensbergen#31 * Update all dependencies (RonaldHensbergen#123) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * docs: add PowerShell task runner parity for Makefile targets (RonaldHensbergen#124) Adds Makefile.ps1 as a Windows equivalent for install, validate, validate-profile, and package targets. lint and docker-build are intentionally not ported (see PowerShell script help text for why). Manually tested on Windows PowerShell: help, install, validate, validate-profile (with and without -P), unknown target handling, package (sdist + wheel build), and CRLF line-ending normalization via .gitattributes. Closes RonaldHensbergen#55 * Update Makefile to check all Dockerfiles (RonaldHensbergen#119) * Update Makefile to check all Dockerfiles * Update Dockerfile search pattern in CI workflow * Fix/dagster round 2 (RonaldHensbergen#121) (RonaldHensbergen#122) * Fixing Dagster user-code being unhealthy * Test-case T1 succeeds! * Updating the test checklist * Updated test with correct Dockerfile * Linting issues (why a file I didn't change?) * Updating Dockerfile * Adding healthcheck to docker-daemon image * Added test for dagster daemon running when dagster is part of the profile * Fix/dagster round 2 (RonaldHensbergen#117) (RonaldHensbergen#120) * Fixing Dagster user-code being unhealthy * Test-case T1 succeeds! * Updating the test checklist * Updated test with correct Dockerfile * Linting issues (why a file I didn't change?) * user-code image build check is good again --------- Co-authored-by: Dane Parin <emphyst80@gmail.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * linting issue * End of file line --------- Co-authored-by: Dane Parin <emphyst80@gmail.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
1 parent 05ec7f0 commit 7d1c336

7 files changed

Lines changed: 188 additions & 4 deletions

File tree

‎cli/renderer.py‎

Lines changed: 92 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,13 @@ def _render_services(
144144
service_copy["healthcheck"] = _substitute_values(hc_copy, context)
145145

146146
service_copy = _substitute_values(service_copy, context)
147-
service_copy = _rewrite_service_volumes(service_copy, module["id"])
147+
service_copy = _rewrite_service_volumes(
148+
service_copy,
149+
module,
150+
profile_dir=profile_dir,
151+
project_root=project_root,
152+
compose_dir=compose_dir,
153+
)
148154
service_copy = _rewrite_depends_on(service_copy, module)
149155
service_copy = _rewrite_build_context(
150156
service_copy,
@@ -272,7 +278,10 @@ def _resolve_expr(expr: str, context: dict[str, Any]) -> Any:
272278

273279
def _rewrite_service_volumes(
274280
service_def: dict[str, Any],
275-
module_id: str,
281+
module: dict[str, Any],
282+
profile_dir: Path | None,
283+
project_root: Path | None,
284+
compose_dir: Path,
276285
) -> dict[str, Any]:
277286
volumes = service_def.get("volumes")
278287
if not isinstance(volumes, list):
@@ -283,12 +292,92 @@ def _rewrite_service_volumes(
283292
if isinstance(item, str):
284293
parts = item.split(":", 1)
285294
if len(parts) == 2 and _is_named_volume(parts[0]):
286-
item = f"{module_id}-{parts[0]}:{parts[1]}"
295+
item = f"{module['id']}-{parts[0]}:{parts[1]}"
296+
elif len(parts) >= 2:
297+
source = parts[0]
298+
rewritten_source = _rewrite_local_path(
299+
source,
300+
module=module,
301+
profile_dir=profile_dir,
302+
project_root=project_root,
303+
compose_dir=compose_dir,
304+
)
305+
if rewritten_source != source:
306+
item = f"{rewritten_source}:{parts[1]}"
307+
elif isinstance(item, dict):
308+
item_copy = deepcopy(item)
309+
if item_copy.get("type") == "bind" and isinstance(item_copy.get("source"), str):
310+
item_copy["source"] = _rewrite_local_path(
311+
item_copy["source"],
312+
module=module,
313+
profile_dir=profile_dir,
314+
project_root=project_root,
315+
compose_dir=compose_dir,
316+
)
317+
item = item_copy
287318
rewritten.append(item)
288319

289320
return {**service_def, "volumes": rewritten}
290321

291322

323+
def _rewrite_local_path(
324+
path_value: str,
325+
module: dict[str, Any],
326+
profile_dir: Path | None,
327+
project_root: Path | None,
328+
compose_dir: Path,
329+
) -> str:
330+
if Path(path_value).is_absolute() or _looks_remote_context(path_value) or "${" in path_value:
331+
return path_value
332+
333+
candidates: list[Path] = []
334+
for base in _local_path_bases(module, profile_dir, project_root, compose_dir):
335+
candidate = (base / path_value).resolve()
336+
if candidate not in candidates:
337+
candidates.append(candidate)
338+
339+
if not candidates:
340+
return path_value
341+
342+
chosen = _choose_best_local_path_candidate(candidates)
343+
try:
344+
rel = Path(chosen).relative_to(compose_dir)
345+
except ValueError:
346+
rel = Path(os.path.relpath(chosen, compose_dir))
347+
return rel.as_posix()
348+
349+
350+
def _local_path_bases(
351+
module: dict[str, Any],
352+
profile_dir: Path | None,
353+
project_root: Path | None,
354+
compose_dir: Path,
355+
) -> list[Path]:
356+
bases: list[Path] = []
357+
358+
if profile_dir is not None:
359+
bases.append(profile_dir)
360+
361+
module_dir = _resolve_module_dir(module, profile_dir)
362+
if module_dir is not None:
363+
bases.append(module_dir)
364+
365+
bases.append(compose_dir)
366+
367+
if project_root is not None:
368+
bases.append((project_root / "build").resolve())
369+
370+
return bases
371+
372+
373+
def _choose_best_local_path_candidate(candidates: list[Path]) -> Path:
374+
for candidate in candidates:
375+
if candidate.exists():
376+
return candidate
377+
378+
return candidates[0]
379+
380+
292381
def _rewrite_depends_on(
293382
service_def: dict[str, Any],
294383
module: dict[str, Any],

‎images/dagster/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ RUN pip install --upgrade pip && \
1717

1818
ENV DAGSTER_HOME=/opt/dagster/dagster_home/
1919

20-
RUN mkdir -p $DAGSTER_HOME
20+
RUN apt-get update && apt-get install -y postgresql-client && rm -rf /var/lib/apt/lists/*
2121

2222
COPY dagster.yaml workspace.yaml $DAGSTER_HOME
2323

‎modules/warehouse/postgres/module.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,11 @@ spec:
6060
type: string
6161
default: 5Gi
6262

63+
initDbScript:
64+
type: string
65+
minLength: 1
66+
default: init-db.sql
67+
6368
healthcheck:
6469
type: object
6570
additionalProperties: false
@@ -94,6 +99,10 @@ spec:
9499
POSTGRES_USER: "${config.username}"
95100
POSTGRES_PASSWORD: "${config.passwordFrom}"
96101
volumes:
102+
- type: bind
103+
source: "${config.initDbScript}"
104+
target: /docker-entrypoint-initdb.d/init-db.sql
105+
read_only: true
97106
- postgres-data:/var/lib/postgresql/data
98107
healthcheck:
99108
conditionallyEnabledFrom: spec.config.healthcheck.enabled

‎profiles/local-dagster-postgres-superset-vault/init-db.sql‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
-- Initialize databases for the local-dagster-postgres-superset profile.
2+
-- Safe to run multiple times.
3+
4+
-- PostgreSQL does not support CREATE DATABASE IF NOT EXISTS, so use psql \gexec.
5+
SELECT 'CREATE DATABASE analytics'
6+
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'analytics')
7+
\gexec
8+
9+
SELECT 'CREATE DATABASE dagster'
10+
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'dagster')
11+
\gexec
12+
13+
SELECT 'CREATE DATABASE superset'
14+
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'superset')
15+
\gexec
16+
17+
-- Grant database-level privileges to the analytics user.
18+
GRANT ALL PRIVILEGES ON DATABASE analytics TO analytics;
19+
GRANT ALL PRIVILEGES ON DATABASE dagster TO analytics;
20+
GRANT ALL PRIVILEGES ON DATABASE superset TO analytics;
21+
22+
-- Per-database schema/default privileges.
23+
\connect analytics
24+
GRANT ALL ON SCHEMA public TO analytics;
25+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO analytics;
26+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO analytics;
27+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO analytics;
28+
29+
\connect dagster
30+
GRANT ALL ON SCHEMA public TO analytics;
31+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO analytics;
32+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO analytics;
33+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO analytics;
34+
35+
\connect superset
36+
GRANT ALL ON SCHEMA public TO analytics;
37+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO analytics;
38+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO analytics;
39+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO analytics;
40+
41+
-- Verify databases exist.
42+
\l

‎profiles/local-dagster-postgres-superset-vault/profile.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ spec:
3535
username: analytics
3636
passwordFrom: secrets.postgres_password
3737
port: 5432
38+
initDbScript: init-db.sql
3839
storage:
3940
enabled: true
4041
size: 5Gi

‎profiles/local-dagster-postgres-superset/init-db.sql‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
-- Initialize databases for the local-dagster-postgres-superset profile.
2+
-- Safe to run multiple times.
3+
4+
-- PostgreSQL does not support CREATE DATABASE IF NOT EXISTS, so use psql \gexec.
5+
SELECT 'CREATE DATABASE analytics'
6+
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'analytics')
7+
\gexec
8+
9+
SELECT 'CREATE DATABASE dagster'
10+
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'dagster')
11+
\gexec
12+
13+
SELECT 'CREATE DATABASE superset'
14+
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'superset')
15+
\gexec
16+
17+
-- Grant database-level privileges to the analytics user.
18+
GRANT ALL PRIVILEGES ON DATABASE analytics TO analytics;
19+
GRANT ALL PRIVILEGES ON DATABASE dagster TO analytics;
20+
GRANT ALL PRIVILEGES ON DATABASE superset TO analytics;
21+
22+
-- Per-database schema/default privileges.
23+
\connect analytics
24+
GRANT ALL ON SCHEMA public TO analytics;
25+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO analytics;
26+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO analytics;
27+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO analytics;
28+
29+
\connect dagster
30+
GRANT ALL ON SCHEMA public TO analytics;
31+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO analytics;
32+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO analytics;
33+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO analytics;
34+
35+
\connect superset
36+
GRANT ALL ON SCHEMA public TO analytics;
37+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO analytics;
38+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO analytics;
39+
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO analytics;
40+
41+
-- Verify databases exist.
42+
\l

‎profiles/local-dagster-postgres-superset/profile.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ spec:
2525
username: analytics
2626
passwordFrom: secrets.postgres_password
2727
port: 5432
28+
initDbScript: init-db.sql
2829
storage:
2930
enabled: true
3031
size: 5Gi

0 commit comments

Comments
 (0)