[TICKET-04] Migrate bteq/02_stg_txn_summary.bteq to Snowflake SQL/dbt

[hrabbani]

Summary

MBA-2205 / TICKET-04. Adds a Snowflake dbt model that replaces bteq/02_stg_txn_summary.bteq; the Teradata/SAS assets are left in place and untouched.

New: dbt/models/staging/stg_txn_summary.sql (+ sources, tests, log_etl_run macro, reconciliation analysis) and docs/modernization/TICKET-04_stg_txn_summary.md.

Mapping decisions (full table in the doc):

Teradata BTEQ	Snowflake / dbt
CREATE VOLATILE TABLE VT_RUN_PARAMS + ${LOOKBACK_MONTHS} shell substitution	run_params CTE off the lookback_months dbt var — dbt run --vars '{lookback_months: 24}' changes the window with no code edit
ADD_MONTHS(CURRENT_DATE, -n) / CURRENT_DATE - MAX(d) / NULLIFZERO(x) / CURRENT_TIMESTAMP(6)	DATEADD(month, -n, CURRENT_DATE) / DATEDIFF(day, MAX(d), PERIOD_END) / NULLIF(x, 0) / CURRENT_TIMESTAMP()
DECIMAL(18,2), DECIMAL(5,2)	NUMBER(18,2), NUMBER(5,2)
DROP + CREATE MULTISET TABLE ... WITH DATA PRIMARY INDEX (CUSTOMER_ID, ACCOUNT_ID)	materialized='table', cluster_by=['CUSTOMER_ID','ACCOUNT_ID']
COLLECT STATISTICS	dropped (Snowflake maintains statistics)
.IF ERRORCODE <> 0 THEN .EXIT	dbt run failure
.IF ACTIVITYCOUNT = 0 THEN .EXIT 99	singular test assert_stg_txn_summary_not_empty (post-hook also logs ZERO_ROWS)
INSERT INTO ETL_RUN_LOG ...	post_hook="{{ log_etl_run('02_stg_txn_summary') }}"

Three behavioural deltas, all required by the acceptance criteria:

• Grain. The BTEQ query left-joined the top-merchant-category subquery on ACCOUNT_ID and then grouped by top_cat.MERCHANT_CATEGORY, which can emit multiple rows per account. The model resolves one top category per account (ties broken alphabetically) before joining, so the grain is one row per ACCOUNT_ID — what the target DDL's primary index implies and what the uniqueness test asserts.
• PCT_OTHER_CHANNEL added. CHANNEL_CODE also carries ACH/WIRE, so the four legacy percentages did not sum to 100. With the extra bucket the mix totals 100 per account (asserted within 0.05 rounding tolerance).
• Revenue-flag aggregation added (in scope per the ticket, absent from the BTEQ): TXN_COUNT_REVENUE, AMT_TOTAL_REVENUE from TRANSACTION_TYPES.IS_REVENUE.

config/pipeline_config.cfg now derives DBT_VARS from the existing LOOKBACK_MONTHS; ddl/01_staging_tables.sql gets a pointer comment on the STG_TXN_SUMMARY block (Teradata DDL unchanged).

Assumptions about unmerged predecessors

• TICKET-01 (Snowflake DDL): source tables are assumed to land as CORE_BANKING.ACCOUNTS, TXN_PROCESSING.TRANSACTIONS, TXN_PROCESSING.TRANSACTION_TYPES in database SNOWFLAKE_DATABASE (default RETAIL_BANKING), with column names preserved from the Teradata DDL. Only the source YAMLs need editing if TICKET-01 lands different names. ETL_RUN_LOG is assumed to keep its (JOB_NAME, STEP_NAME, STATUS, ROW_COUNT, START_TS, END_TS) shape in schema var('etl_staging_schema').
• TICKET-02 (env/role/secret setup): no profile or credential material is committed here; the project expects a retail_banking_analytics dbt profile from TICKET-02.
• TICKET-03: the dbt project scaffolding (dbt_project.yml, source YAMLs, log_etl_run) is added here because TICKET-03 is not merged; if it lands the same files, keep one copy — the model itself is unaffected.

Validation

No warehouse available, so static only: dbt parse succeeds (1 model, 1 analysis, 9 tests, 3 sources), and sqlfluff parse with dialect = snowflake is clean for the model, all three singular tests and the analysis (dbt/.sqlfluff added so this is reproducible offline). analyses/recon_stg_txn_summary.sql compares row counts and per-column sums against the legacy output landed as LEGACY_STG_TXN_SUMMARY for the same window.

Devin-Org: engineering

Link to Devin session: https://app.devin.ai/sessions/125e1a3f81694bfda7efc5c5b59d2dce
Requested by: @hrabbani

---

Devin Review

Status	Commit
⚪ Not started	—

Run Devin Review

💡 Connect your GitHub account to enable automatic code reviews.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring