feat: freeze product_name instead of line_revenue (+ prod trigger & drop-script fixes) - #43
Merged
Merged
Conversation
Remove the synthetic line_revenue/unit_price_at_sale columns; gold total_value is now SUM(item_total) (the line value the source already freezes on the order). Re-point the silver insert-only-MERGE / streaming- table freeze at the mutable product_name, which the seed now changes by renaming 2 products per run (Product N -> Product N.k); unit_price stays a static attribute. The AI/BI "by product" chart consolidates by product_id labeled with each product's latest name (one line per physical product across renames), while frozen historical names remain in report.order_agg and the Product filter. Tests, schemas, SDP pipeline, and data-model/test-plan docs updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AI/BI wraps each dataset query as `WITH q AS (<query>) SELECT ... FROM q` and concatenates queryLines with no separator. The multi-line query both mashed tokens at line boundaries (total_orders+FROM) and produced a nested WITH inside the wrapper. Rewrite ds_orders as one line using a derived-table subquery (no leading WITH) for the latest-name lookup; verified the wrapped by-country aggregation parses and runs on the dev warehouse. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…on is sole trigger Remove the daily 0 0 5 CronSchedule from _build_job's prod branch so the batch job1 (job1_prod) no longer self-triggers. In prod, job1_prod_integration (0 0 6 Sao_Paulo) is the single scheduled entry point: it seeds, then triggers the batch job (RunJobTask) and the SDP pipeline (PipelineTask) in parallel. The SDP pipeline already had no schedule. Prod failure/duration alerting on job1 is retained. Documented in architecture.md Jobs DAG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Some SQL warehouse channels reject the DROP STREAMING TABLE grammar with PARSE_SYNTAX_ERROR (while accepting a plain DROP TABLE on the streaming table), which aborted `make drop` mid-loop and left the catalog half- dropped. Try the kind-specific statement first, then fall back to DROP TABLE. Keeps DROP MATERIALIZED VIEW for MVs (DROP TABLE type-mismatches there). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ta-model.md Add a narrative overview of the medallion flow (external_source -> bronze -> silver -> gold -> dashboard), explaining frozen item_total revenue and how a product rename keeps both old and new names. Document the field naming rules (entity_id suffix, entity-qualified names, item_*/total_* prefixes, no abbreviations, DateType dates, _sdp suffix) that were previously implicit. Point CLAUDE.md at both. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ever edit Add an explicit constraint mirroring the jobs.yml rule: the _deploy copy is gitignored and regenerated from orders_dashboard.lvdash.json on every deploy (DABs can't substitute bundle vars inside .lvdash.json), so edit the source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… history The filter was bound to the frozen per-row product_name, so selecting a renamed product's current name (e.g. "Product 11.19") matched only the rows physically stamped with it (today's post-rename orders) and dropped the pre-rename history (stamped "Product 11"). Bind the filter to the consolidated latest name (`product`) instead, matching the chart's color encoding: the dropdown lists each product once and selecting it shows full pre-/post-rename history. Old names stay in report.order_agg (frozen product_name) for audit. Verified on prod: filtering "Product 11.19" now returns 1820 rows spanning 2025-06-27..today. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add "git_commit": "${bundle.git.commit}" to _tags() so bundle deploy
substitutes the HEAD SHA into every job's tags. This makes a deployed
environment's exact commit identifiable from `databricks jobs list`
(alongside the existing git_branch tag), instead of diffing deployed
bundle source file-by-file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 task
andre-salvati
added a commit
that referenced
this pull request
Jul 16, 2026
…ixes (#48) ## What? Three related things, all in the docs/repo-hygiene layer — no runtime behavior changes. 1. **Star-history chart is generated locally now.** New `scripts/star_history.py` + `make star-history` render committed light/dark SVGs into `assets/`; the README's `<picture>` points at them. 2. **README refresh + a setup-breaking fix.** Resolved the two `#TODO`s (keyword-first H1; badges stay at top) and corrected the CI secret names in step 8. 3. **Docs reconciled with the code**, plus `scripts/` added to the ruff lint gate. 4. **Changelog conventions tightened**: entries are now written at merge time and capped at ~5 rendered lines; `#42`/`#43` headers standardized onto the PR-URL format. ## Why? The chart stopped rendering because `api.star-history.com` returns `503 All GitHub API tokens are rate-limited` — for *every* repo, reproduced against `facebook/react`. Nothing was wrong with our markup; the free tier had exhausted its GitHub token pool. Caching their SVG would have kept the dependency, so the chart is now built from the GitHub API directly. The most important fix is the smallest diff: **README step 8 told people to configure `DATABRICKS_PRINCIPAL_ID` / `DATABRICKS_SECRET`, but `onpush.yml` reads `DATABRICKS_CLIENT_ID` / `DATABRICKS_CLIENT_SECRET` / `TEMPLATE_ALERT_EMAILS`.** Anyone following the setup guide configured CI auth that silently failed. For a template whose pitch is "fork this and go," that's the worst place to be wrong. `specs/architecture.md` already had the correct names, so the docs also contradicted each other. The rest came out of a documentation audit: the specs claimed a `bundle validate` CI step that doesn't exist, an `--env local` value the parser rejects, omitted `--load-test` and `job1_sdp/` entirely, and over-generalized the retry rule. CLAUDE.md is loaded every session, and the `resources/jobs.yml` rule was stated in it four separate times. ## How? - **No new dependencies.** The SVG is hand-rolled; stargazer timestamps come from `gh api` (`Accept: application/vnd.github.star+json`). - **Two committed SVGs, not one**, to keep the existing `<picture>` light/dark switch. - **Chart form**: single-series line, so no legend (the title carries identity) and exactly one direct label (the current total) — a static README image has no hover layer to carry values. - **`known-first-party = ["_sdk_sql"]`** was needed once `scripts/` was linted: isort otherwise files that local sibling under the third-party `databricks` imports. - **Changelog rule** (`CLAUDE.md`): an entry is written *immediately before merge* rather than during development (scope grows; an early entry just gets rewritten), and is capped at **~5 rendered lines (~475 chars)** on top of the 3-sentence rule — "3 sentences" alone had drifted into 17-line run-ons. Applies to new entries only; the changelog stays append-only, so the other 47 entries are untouched. - Constraints and Keep It Simple in CLAUDE.md were left verbatim — they're the rules most often broken. Only genuine duplication was cut (~90 → 88 lines). ## Validation? - `make unit-test` — **16 passed**. - `pre-commit run --all-files` — both hooks pass; `ruff check src/ scripts/` clean; 23 files formatted. - **Lint gate proven, not assumed**: a deliberately-broken probe in `scripts/` initially reported "Passed" — pre-commit skips *untracked* files. Once git-added, the hook caught it and failed. The gate genuinely covers `scripts/`. - Both SVGs rendered in a real browser and inspected (no label collisions, clean ticks, correct in both themes); the committed files are byte-identical to what was reviewed. - Downsampling (`MAX_POINTS`) and `nice_ticks` edge cases exercised with 1,000 synthetic stars. - Merged `origin/main` (which brought in #47, merged mid-branch); the only conflict was both branches adding a changelog entry at the top — resolved keeping #47. - Every corrected doc claim re-checked against source — README secret names now diff-match `onpush.yml` exactly. **Verified on the pushed branch:** GitHub *does* rewrite relative paths inside `<source srcset>` — both sources resolve to `/andre-salvati/databricks-template/raw/<branch>/assets/…` and serve `200 image/svg+xml`. This had been an open risk (`<picture>` has no fallback if a matched source fails, so a bad path would break dark mode outright); it is now confirmed working in both themes, and no absolute-URL workaround is needed. ## Impact in prod - [x] No table schema/data change — no production impact. No table, schema, cluster key, or task logic touched. The only `src/` change is a corrected comment in `job1_sdp/pipeline.py` (it said the streaming silver freezes *price*; it freezes `product_name` since #43). Docs-only pushes are skipped by `paths-ignore`, but this PR also touches `scripts/`, `pyproject.toml`, and `src/`, so CI will run. **Chosen strategy:** leave as-is (no schema change). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-points the medallion "freeze at sale time" demonstration from a synthetic price column onto the product name, and cleans up a few prod/ops rough edges found along the way.
Data model — freeze
product_nameinstead ofline_revenueline_revenue/unit_price_at_salecolumns fromorder_enriched_schema. Goldtotal_valueis nowSUM(item_total)— the line value the source already freezes on the order.MERGE, SDP streaming table) now freezes the mutableproduct_nameonto each order line at processing time. A later rename never relabels already-booked orders; a materialized view would restate it, which is why silver stays a streaming table on the SDP path.product_nameinstead ofunit_price: it renames 2 products per incremental run (Product N→Product N.k, wherekis the cumulative rename count), day-anchored so reruns of a date are idempotent.unit_priceis now a static attribute (set once at initial load, never bumped).Dashboard
product_id, one line per physical product across renames); the frozen historical names remain inreport.order_aggfor audit.Prod / ops
job1's standalone prod schedule (the SDP pipeline already had none), sojob1_prod_integrationis the single prod trigger orchestrating seed → batch + SDP.sdk_drop_tables.pynow falls back toDROP TABLEwhen a warehouse's parser rejectsDROP STREAMING TABLE(kind-specific statement first, plainDROP TABLEas last resort).git_commit(${bundle.git.commit}) deploy tag on every job, so a deployed environment's exact commit is identifiable fromdatabricks jobs list(alongside the existinggit_branchtag) instead of diffing deployed bundle source file-by-file.Tests & docs
commonSchemas, the SDP pipeline/transforms, and the specs (data-model.md,test-plan.md,architecture.md) to match. All 16 unit tests pass.specs/CHANGELOG.mdentry added.🤖 Generated with Claude Code