Skip to content

Commit 449b486

Browse files
C00ldudeNoonanC00ldudeNoonan
authored
Fix dbt Fusion BigQuery build (#100)
* feat: prepare dbt platform fusion orchestration * chore: apply ruff formatting * fix: make dbt project build with fusion * docs: clarify codex PR review request
1 parent 0741989 commit 449b486

107 files changed

Lines changed: 1116 additions & 2081 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env.example‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -62,12 +62,21 @@ GEMINI_API_KEY=your_gemini_api_key_here
6262
# =============================================================================
6363
# GOOGLE CLOUD (for GCS storage, Google Docs, and BigQuery)
6464
# =============================================================================
65-
# Service account credentials (JSON file path or JSON string)
66-
# Used for: GCS storage, Google Docs API, and BigQuery
65+
# Use Application Default Credentials (ADC) for Google auth.
66+
# Local setup:
67+
# gcloud auth application-default login
68+
# Service account impersonation, if preferred:
69+
# gcloud auth application-default login --impersonate-service-account service-account@project.iam.gserviceaccount.com
70+
# On GCE/Cloud Run/GitHub WIF, ADC uses the attached or federated service account.
71+
#
72+
# GOOGLE_APPLICATION_CREDENTIALS is intentionally unset for normal ADC auth.
73+
# Set it only if you must use a local service-account keyfile fallback.
74+
# GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
75+
#
76+
# Used for: GCS storage, Google Docs API, and BigQuery.
6777
# Required scopes for Google Docs:
6878
# - https://www.googleapis.com/auth/documents
6979
# - https://www.googleapis.com/auth/drive
70-
GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
7180
GCS_BUCKET_NAME=your_gcs_bucket_name
7281

7382
# Google Docs Integration (optional)
@@ -89,9 +98,8 @@ GOOGLE_DRIVE_FOLDER_ID=your_google_drive_folder_id_here
8998
# Note: This can be the same as GOOGLE_DRIVE_FOLDER_ID if using one folder
9099
REALTOR_GDRIVE_FOLDER_ID=your_realtor_folder_id_here
91100

92-
# BigQuery credentials (path to service account JSON on local dev)
93-
# On GCE the attached VM service account is used automatically via ADC.
94-
# GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
101+
# BigQuery auth is through ADC. No service-account keyfile is required when
102+
# local ADC, attached service accounts, or workload identity federation are configured.
95103

96104
# =============================================================================
97105
# DBT

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@
6666
- Prefer small, focused commits with clear intent.
6767
- PRs should include a concise summary, linked issues (if applicable), and screenshots for UI changes.
6868
- Never merge PRs. Always leave merging to a human reviewer.
69-
- Add a PR comment tagging `@codex` to request review.
69+
- Add a PR comment tagging `@codex` to request review immediately after opening every PR. Use `gh pr comment <PR_NUMBER> --body "@codex please review this PR."` and mention the review request in the handoff.
7070

7171
## Data Request Scoping (Wizard Loop 1)
7272

‎dbt_project/macros/calculate_commodity_analysis_return.sql‎

Lines changed: 30 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -68,18 +68,18 @@ rolling_stats AS (
6868

6969
MAX(current_price) OVER (
7070
PARTITION BY commodity_name, commodity_unit
71-
ORDER BY date
72-
RANGE BETWEEN INTERVAL 365 DAY PRECEDING AND CURRENT ROW
71+
ORDER BY UNIX_DATE(date)
72+
RANGE BETWEEN 365 PRECEDING AND CURRENT ROW
7373
) AS high_1yr,
7474
MIN(current_price) OVER (
7575
PARTITION BY commodity_name, commodity_unit
76-
ORDER BY date
77-
RANGE BETWEEN INTERVAL 365 DAY PRECEDING AND CURRENT ROW
76+
ORDER BY UNIX_DATE(date)
77+
RANGE BETWEEN 365 PRECEDING AND CURRENT ROW
7878
) AS low_1yr,
7979
STDDEV(daily_diff) OVER (
8080
PARTITION BY commodity_name, commodity_unit
81-
ORDER BY date
82-
RANGE BETWEEN INTERVAL 365 DAY PRECEDING AND CURRENT ROW
81+
ORDER BY UNIX_DATE(date)
82+
RANGE BETWEEN 365 PRECEDING AND CURRENT ROW
8383
) AS std_diff_1yr,
8484
price_365d_ago AS price_start_1yr,
8585
CASE
@@ -90,18 +90,18 @@ rolling_stats AS (
9090

9191
MAX(current_price) OVER (
9292
PARTITION BY commodity_name, commodity_unit
93-
ORDER BY date
94-
RANGE BETWEEN INTERVAL 270 DAY PRECEDING AND CURRENT ROW
93+
ORDER BY UNIX_DATE(date)
94+
RANGE BETWEEN 270 PRECEDING AND CURRENT ROW
9595
) AS high_9mo,
9696
MIN(current_price) OVER (
9797
PARTITION BY commodity_name, commodity_unit
98-
ORDER BY date
99-
RANGE BETWEEN INTERVAL 270 DAY PRECEDING AND CURRENT ROW
98+
ORDER BY UNIX_DATE(date)
99+
RANGE BETWEEN 270 PRECEDING AND CURRENT ROW
100100
) AS low_9mo,
101101
STDDEV(daily_diff) OVER (
102102
PARTITION BY commodity_name, commodity_unit
103-
ORDER BY date
104-
RANGE BETWEEN INTERVAL 270 DAY PRECEDING AND CURRENT ROW
103+
ORDER BY UNIX_DATE(date)
104+
RANGE BETWEEN 270 PRECEDING AND CURRENT ROW
105105
) AS std_diff_9mo,
106106
price_270d_ago AS price_start_9mo,
107107
CASE
@@ -112,18 +112,18 @@ rolling_stats AS (
112112

113113
MAX(current_price) OVER (
114114
PARTITION BY commodity_name, commodity_unit
115-
ORDER BY date
116-
RANGE BETWEEN INTERVAL 180 DAY PRECEDING AND CURRENT ROW
115+
ORDER BY UNIX_DATE(date)
116+
RANGE BETWEEN 180 PRECEDING AND CURRENT ROW
117117
) AS high_6mo,
118118
MIN(current_price) OVER (
119119
PARTITION BY commodity_name, commodity_unit
120-
ORDER BY date
121-
RANGE BETWEEN INTERVAL 180 DAY PRECEDING AND CURRENT ROW
120+
ORDER BY UNIX_DATE(date)
121+
RANGE BETWEEN 180 PRECEDING AND CURRENT ROW
122122
) AS low_6mo,
123123
STDDEV(daily_diff) OVER (
124124
PARTITION BY commodity_name, commodity_unit
125-
ORDER BY date
126-
RANGE BETWEEN INTERVAL 180 DAY PRECEDING AND CURRENT ROW
125+
ORDER BY UNIX_DATE(date)
126+
RANGE BETWEEN 180 PRECEDING AND CURRENT ROW
127127
) AS std_diff_6mo,
128128
price_180d_ago AS price_start_6mo,
129129
CASE
@@ -134,18 +134,18 @@ rolling_stats AS (
134134

135135
MAX(current_price) OVER (
136136
PARTITION BY commodity_name, commodity_unit
137-
ORDER BY date
138-
RANGE BETWEEN INTERVAL 90 DAY PRECEDING AND CURRENT ROW
137+
ORDER BY UNIX_DATE(date)
138+
RANGE BETWEEN 90 PRECEDING AND CURRENT ROW
139139
) AS high_3mo,
140140
MIN(current_price) OVER (
141141
PARTITION BY commodity_name, commodity_unit
142-
ORDER BY date
143-
RANGE BETWEEN INTERVAL 90 DAY PRECEDING AND CURRENT ROW
142+
ORDER BY UNIX_DATE(date)
143+
RANGE BETWEEN 90 PRECEDING AND CURRENT ROW
144144
) AS low_3mo,
145145
STDDEV(daily_diff) OVER (
146146
PARTITION BY commodity_name, commodity_unit
147-
ORDER BY date
148-
RANGE BETWEEN INTERVAL 90 DAY PRECEDING AND CURRENT ROW
147+
ORDER BY UNIX_DATE(date)
148+
RANGE BETWEEN 90 PRECEDING AND CURRENT ROW
149149
) AS std_diff_3mo,
150150
price_90d_ago AS price_start_3mo,
151151
CASE
@@ -156,18 +156,18 @@ rolling_stats AS (
156156

157157
MAX(current_price) OVER (
158158
PARTITION BY commodity_name, commodity_unit
159-
ORDER BY date
160-
RANGE BETWEEN INTERVAL 30 DAY PRECEDING AND CURRENT ROW
159+
ORDER BY UNIX_DATE(date)
160+
RANGE BETWEEN 30 PRECEDING AND CURRENT ROW
161161
) AS high_1mo,
162162
MIN(current_price) OVER (
163163
PARTITION BY commodity_name, commodity_unit
164-
ORDER BY date
165-
RANGE BETWEEN INTERVAL 30 DAY PRECEDING AND CURRENT ROW
164+
ORDER BY UNIX_DATE(date)
165+
RANGE BETWEEN 30 PRECEDING AND CURRENT ROW
166166
) AS low_1mo,
167167
STDDEV(daily_diff) OVER (
168168
PARTITION BY commodity_name, commodity_unit
169-
ORDER BY date
170-
RANGE BETWEEN INTERVAL 30 DAY PRECEDING AND CURRENT ROW
169+
ORDER BY UNIX_DATE(date)
170+
RANGE BETWEEN 30 PRECEDING AND CURRENT ROW
171171
) AS std_diff_1mo,
172172
price_30d_ago AS price_start_1mo,
173173
CASE

‎dbt_project/macros/calculate_commodity_summary.sql‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,8 +87,13 @@ start_prices AS (
8787
FROM period_boundaries AS pb
8888
INNER JOIN filtered_data AS fd ON
8989
pb.commodity_name = fd.commodity_name
90+
AND pb.commodity_unit = fd.commodity_unit
9091
AND pb.time_period = fd.time_period
9192
AND pb.period_start_date = fd.trade_date
93+
QUALIFY ROW_NUMBER() OVER (
94+
PARTITION BY pb.commodity_name, pb.commodity_unit, pb.time_period
95+
ORDER BY fd.trade_date ASC, fd.price ASC
96+
) = 1
9297
),
9398

9499
end_prices AS (
@@ -100,8 +105,13 @@ end_prices AS (
100105
FROM period_boundaries AS pb
101106
INNER JOIN filtered_data AS fd ON
102107
pb.commodity_name = fd.commodity_name
108+
AND pb.commodity_unit = fd.commodity_unit
103109
AND pb.time_period = fd.time_period
104110
AND pb.period_end_date = fd.trade_date
111+
QUALIFY ROW_NUMBER() OVER (
112+
PARTITION BY pb.commodity_name, pb.commodity_unit, pb.time_period
113+
ORDER BY fd.trade_date DESC, fd.price DESC
114+
) = 1
105115
),
106116

107117
-- Main aggregation
@@ -140,10 +150,12 @@ combined_results AS (
140150
LEFT JOIN start_prices AS sp
141151
ON
142152
ar.commodity_name = sp.commodity_name
153+
AND ar.commodity_unit = sp.commodity_unit
143154
AND ar.time_period = sp.time_period
144155
LEFT JOIN end_prices AS ep
145156
ON
146157
ar.commodity_name = ep.commodity_name
158+
AND ar.commodity_unit = ep.commodity_unit
147159
AND ar.time_period = ep.time_period
148160
),
149161

‎dbt_project/macros/calculate_market_analysis_return.sql‎

Lines changed: 30 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -88,18 +88,18 @@ rolling_stats AS (
8888

8989
MAX(current_high) OVER (
9090
PARTITION BY symbol, exchange
91-
ORDER BY date
92-
RANGE BETWEEN INTERVAL 365 DAY PRECEDING AND CURRENT ROW
91+
ORDER BY UNIX_DATE(date)
92+
RANGE BETWEEN 365 PRECEDING AND CURRENT ROW
9393
) AS high_1yr,
9494
MIN(current_low) OVER (
9595
PARTITION BY symbol, exchange
96-
ORDER BY date
97-
RANGE BETWEEN INTERVAL 365 DAY PRECEDING AND CURRENT ROW
96+
ORDER BY UNIX_DATE(date)
97+
RANGE BETWEEN 365 PRECEDING AND CURRENT ROW
9898
) AS low_1yr,
9999
STDDEV(daily_diff) OVER (
100100
PARTITION BY symbol, exchange
101-
ORDER BY date
102-
RANGE BETWEEN INTERVAL 365 DAY PRECEDING AND CURRENT ROW
101+
ORDER BY UNIX_DATE(date)
102+
RANGE BETWEEN 365 PRECEDING AND CURRENT ROW
103103
) AS std_diff_1yr,
104104
price_365d_ago AS price_start_1yr,
105105
CASE
@@ -110,18 +110,18 @@ rolling_stats AS (
110110

111111
MAX(current_high) OVER (
112112
PARTITION BY symbol, exchange
113-
ORDER BY date
114-
RANGE BETWEEN INTERVAL 270 DAY PRECEDING AND CURRENT ROW
113+
ORDER BY UNIX_DATE(date)
114+
RANGE BETWEEN 270 PRECEDING AND CURRENT ROW
115115
) AS high_9mo,
116116
MIN(current_low) OVER (
117117
PARTITION BY symbol, exchange
118-
ORDER BY date
119-
RANGE BETWEEN INTERVAL 270 DAY PRECEDING AND CURRENT ROW
118+
ORDER BY UNIX_DATE(date)
119+
RANGE BETWEEN 270 PRECEDING AND CURRENT ROW
120120
) AS low_9mo,
121121
STDDEV(daily_diff) OVER (
122122
PARTITION BY symbol, exchange
123-
ORDER BY date
124-
RANGE BETWEEN INTERVAL 270 DAY PRECEDING AND CURRENT ROW
123+
ORDER BY UNIX_DATE(date)
124+
RANGE BETWEEN 270 PRECEDING AND CURRENT ROW
125125
) AS std_diff_9mo,
126126
price_270d_ago AS price_start_9mo,
127127
CASE
@@ -132,18 +132,18 @@ rolling_stats AS (
132132

133133
MAX(current_high) OVER (
134134
PARTITION BY symbol, exchange
135-
ORDER BY date
136-
RANGE BETWEEN INTERVAL 180 DAY PRECEDING AND CURRENT ROW
135+
ORDER BY UNIX_DATE(date)
136+
RANGE BETWEEN 180 PRECEDING AND CURRENT ROW
137137
) AS high_6mo,
138138
MIN(current_low) OVER (
139139
PARTITION BY symbol, exchange
140-
ORDER BY date
141-
RANGE BETWEEN INTERVAL 180 DAY PRECEDING AND CURRENT ROW
140+
ORDER BY UNIX_DATE(date)
141+
RANGE BETWEEN 180 PRECEDING AND CURRENT ROW
142142
) AS low_6mo,
143143
STDDEV(daily_diff) OVER (
144144
PARTITION BY symbol, exchange
145-
ORDER BY date
146-
RANGE BETWEEN INTERVAL 180 DAY PRECEDING AND CURRENT ROW
145+
ORDER BY UNIX_DATE(date)
146+
RANGE BETWEEN 180 PRECEDING AND CURRENT ROW
147147
) AS std_diff_6mo,
148148
price_180d_ago AS price_start_6mo,
149149
CASE
@@ -154,18 +154,18 @@ rolling_stats AS (
154154

155155
MAX(current_high) OVER (
156156
PARTITION BY symbol, exchange
157-
ORDER BY date
158-
RANGE BETWEEN INTERVAL 90 DAY PRECEDING AND CURRENT ROW
157+
ORDER BY UNIX_DATE(date)
158+
RANGE BETWEEN 90 PRECEDING AND CURRENT ROW
159159
) AS high_3mo,
160160
MIN(current_low) OVER (
161161
PARTITION BY symbol, exchange
162-
ORDER BY date
163-
RANGE BETWEEN INTERVAL 90 DAY PRECEDING AND CURRENT ROW
162+
ORDER BY UNIX_DATE(date)
163+
RANGE BETWEEN 90 PRECEDING AND CURRENT ROW
164164
) AS low_3mo,
165165
STDDEV(daily_diff) OVER (
166166
PARTITION BY symbol, exchange
167-
ORDER BY date
168-
RANGE BETWEEN INTERVAL 90 DAY PRECEDING AND CURRENT ROW
167+
ORDER BY UNIX_DATE(date)
168+
RANGE BETWEEN 90 PRECEDING AND CURRENT ROW
169169
) AS std_diff_3mo,
170170
price_90d_ago AS price_start_3mo,
171171
CASE
@@ -176,18 +176,18 @@ rolling_stats AS (
176176

177177
MAX(current_high) OVER (
178178
PARTITION BY symbol, exchange
179-
ORDER BY date
180-
RANGE BETWEEN INTERVAL 30 DAY PRECEDING AND CURRENT ROW
179+
ORDER BY UNIX_DATE(date)
180+
RANGE BETWEEN 30 PRECEDING AND CURRENT ROW
181181
) AS high_1mo,
182182
MIN(current_low) OVER (
183183
PARTITION BY symbol, exchange
184-
ORDER BY date
185-
RANGE BETWEEN INTERVAL 30 DAY PRECEDING AND CURRENT ROW
184+
ORDER BY UNIX_DATE(date)
185+
RANGE BETWEEN 30 PRECEDING AND CURRENT ROW
186186
) AS low_1mo,
187187
STDDEV(daily_diff) OVER (
188188
PARTITION BY symbol, exchange
189-
ORDER BY date
190-
RANGE BETWEEN INTERVAL 30 DAY PRECEDING AND CURRENT ROW
189+
ORDER BY UNIX_DATE(date)
190+
RANGE BETWEEN 30 PRECEDING AND CURRENT ROW
191191
) AS std_diff_1mo,
192192
price_30d_ago AS price_start_1mo,
193193
CASE

0 commit comments

Comments
 (0)