fix: replace DuckDB-style quoted INTERVAL literals with BigQuery syntax#142
Merged
Merged
Conversation
…ax (#139) BigQuery rejects `INTERVAL '3 years'` (quoted string, DuckDB/Postgres style) with a confusing downstream parse error reported at the next clause rather than at the INTERVAL itself. The fix is BigQuery's native unquoted form: `INTERVAL 3 YEAR`. Fixed 4 confirmed-broken jobs plus 3 more instances found by inspection: - signals/fear_greed_composite.py (6 occurrences, f-string lookback var) - signals/entropy_complexity.py - signals/absorption_ratio.py - signals/network_correlation.py - domains/sec/cik_history.py - analysis/economy_state/domain_data_fetchers.py (4 occurrences) - analysis/economy_state/data_access.py (3 occurrences) Verified by running each of the 4 job-level fixes directly against a live BigQuery target (dagster job execute) before and after: all four previously failed with "Syntax error: Unexpected keyword ORDER"; after the fix, none show that error. fear_greed_job and entropy_complexity_job now fail on an unrelated, separate bug (unqualified stg_* table references resolving to the wrong dataset — filed separately); absorption_ratio_job and network_correlation_job now fail only because this local dev environment's BigQuery dataset has never been backfilled with sp500_companies_prices_raw (data gap, not a code defect). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #139 — replaces DuckDB/Postgres-style quoted
INTERVAL '3 years'literals with BigQuery's native unquotedINTERVAL 3 YEARsyntax across all instances found in the codebase.Why the error was confusing
BigQuery's parser doesn't fail cleanly at the malformed
INTERVAL— it loses sync and reports the error at the next clause, almost alwaysORDER BY:That pointed everywhere except the actual bug, which is why 4+ jobs sat broken.
Files changed
signals/fear_greed_composite.py— 6 occurrences via an f-stringlookbackvariable, changed to an int (lookback_years = 3)signals/entropy_complexity.py,signals/absorption_ratio.py,signals/network_correlation.py— hardcoded literalsdomains/sec/cik_history.py—INTERVAL '90 days'→INTERVAL 90 DAYanalysis/economy_state/domain_data_fetchers.py,analysis/economy_state/data_access.py— 7 occurrences total, all driven by an existingintparameter (max_months,months_per_series,months_limit), confirmed by reading each function signature before editingVerification
Ran each of the 4 job-level fixes directly against a live BigQuery target (
dagster job execute) before and after the fix:fear_greed_job400 Syntax error: Unexpected keyword ORDERstg_major_indicesreference resolves to the wrong dataset; filed separately)entropy_complexity_jobstg_*dataset bugabsorption_ratio_jobsp500_companies_prices_raw(a data gap specific to this sandbox, not a code defect — that table exists in the prod dataset)network_correlation_jobNo remaining
INTERVAL '...'(quoted) instances anywhere inmacro_agents/src— confirmed by repo-wide grep.Not in scope here
Filed separately since they're distinct root causes surfaced only after this fix removed the syntax error masking them: unqualified
stg_*table references infear_greed_composite.py/entropy_complexity.pyresolving against the wrong BigQuery dataset.🤖 Generated with Claude Code