[TICKET-10] Migrate shell orchestration to Azure Synapse Pipelines (MBA-2211)

[hrabbani]

Summary

MBA-2211 / TICKET-10. Replaces the bash orchestration of the pipeline with Synapse pipeline definitions held as code under synapse/, plus a renderer/deployer and CI. The Teradata/SAS assets stay in place; orchestration/run_full_pipeline.sh and config/pipeline_config.cfg only gain a "superseded" header.

Control flow lands as five pipelines:

pl_retail_banking_analytics            # entry: runId, dryRun gate, failure notification
└─ (dryRun=false) pl_retail_banking_analytics_run
   ├─ StagingPhase   If skipStaging=false   -> pl_staging_snowflake   (nb_run_dbt x3, sequential)
   ├─ AnalyticsPhase If skipAnalytics=false -> pl_analytics_spark     (4 notebooks, sequential)
   └─ PostRunValidation                     -> pl_post_run_validation (counts -> Fail | run-log)

Mapping decisions worth calling out (full table in docs/modernization/orchestration_synapse_migration.md):

• Fail-fast — set -euo pipefail + exit ${RC} becomes dependsOn: [{Succeeded}] chains; every non-first activity in every scope has one, and the validator enforces that so the ordering can't silently regress.
• --dry-run — dryRun parameter gates the whole run: the true branch only writes the planned activity list to a variable, so nothing touches Snowflake.
• --skip-bteq / --skip-sas — skipStaging / skipAnalytics IfConditions; a skipped phase still counts as Succeeded for the next phase, matching the shell.
• Post-run validation — the inline BTEQ SELECT COUNT(*) heredoc becomes one Script activity producing BREACH_COUNT/DETAIL against per-environment minimums, then FailOnRowCountBreach (error code 99, the shell's zero-row exit code). ETL_RUN_LOG inserts become <staging_db>.OPS.PIPELINE_RUN_LOG.
• Notification — on failure the entry pipeline reads the webhook URL from Key Vault with the workspace MSI, POSTs the failure, then fails the run.
• Config/secrets — every pipeline_config.cfg value becomes a pipeline parameter supplied from synapse/config/{dev,uat,prod}.parameters.json (LOOKBACK_MONTHS, RISK_SCORE_THRESHOLD, LOG_LEVEL, DB_* -> Snowflake databases, RUN_DATE -> runDate/runId). SAS/edge-server paths are dropped. Only vault and secret names are in the repo; Snowflake auth is key-pair with the private key resolved from Key Vault by ls_snowflake.
• Env portability — artifacts are environment-neutral; scripts/deploy_synapse.py substitutes the Spark pool reference and the entry pipeline's default parameters per environment and deploys callee-before-caller, so DEV/UAT/PROD deploy from identical definitions with no manual edits.

Assumptions about unmerged predecessors

TICKET-01..09 are in flight, so this codes against the interfaces the ticket describes rather than merged artifacts: schemas <staging_db>.STAGING / <products_db>.DATA_PRODUCTS and the run-log table <staging_db>.OPS.PIPELINE_RUN_LOG; a dbt runner notebook nb_run_dbt taking dbt_select; and notebooks nb_customer_segments, nb_txn_analytics, nb_risk_scoring, nb_data_products. If a predecessor lands other names, only the referenceName values and EXPECTED_NOTEBOOKS in the validator change. Note also that bteq/run_bteq_pipeline.sh and sas/run_sas_pipeline.sh are referenced by the README but do not exist in the repo — their phase behaviour is reconstructed from run_full_pipeline.sh and the numbered BTEQ/SAS assets.

Validation

No Snowflake or Azure environment is available, so checks are static:

• python scripts/validate_synapse_artifacts.py — JSON parse, name/file agreement, resolvable pipeline/linked-service/notebook references, complete linked-service parameter sets, dependsOn present on every non-first activity, DEV/UAT/PROD parameter keys consistent and covering every required entry-pipeline parameter, no credential literals. Negative-tested by injecting a missing dependsOn, a bogus notebook, a dropped env parameter and a plaintext privateKey — all five were reported.
• python scripts/deploy_synapse.py --environment {dev,uat,prod} --dry-run — renders each environment and prints the az synapse commands in dependency order.
• bash -n on the two touched shell/config files.

CI (.github/workflows/synapse-ci.yml) runs the validation and the three dry-run renders on PRs, and deploys dev -> uat -> prod on main via Azure OIDC.

Devin-Org: engineering

Link to Devin session: https://app.devin.ai/sessions/9947f7c8c39540429878f2339628a5b7
Requested by: @hrabbani

---

Devin Review

Status	Commit
⚪ Not started	—

Run Devin Review

💡 Connect your GitHub account to enable automatic code reviews.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring