LDE: Deploy to AWS dev + ensure external access

[bjagg]

Part of the LDE rollout — see #906.

Deploy the lif_learner_data_export_api microservice to AWS dev and confirm it is reachable from the external web (the LDE service is intended to be open to the external internet, unlike the internal services behind it).

• Deploy via the GHA/AWS config from the config issue
• Verify external access end-to-end (GET /exports, health)
• Confirm it can reach its internal deps (Query Planner, MDR API, Translator) from the dev network

Related: #184 (update dev environment to support URI pathing) — likely a prerequisite/overlap for external access.

Depends on: GHA + AWS config issue. Blocks: deploy-to-demo.

[bjagg]

Deploy runbook — LDE to AWS dev (#998)

Prepped while #997 config is in review (PR #1001). #1001 was extended to include everything a deploy needs: the workflow, {dev,demo} params, taskdef-includes, the ECR repo (repositories.yml), the MDR server-side key mapping (lif-mdr-api-taskdef-includes.yml), and the LDE key group (setup-mdr-api-keys.sh). MDR's code already accepts the LDE key (mdr_auth/core.py, config.py, on main since #920/#958).

Prerequisites

• Issue #997: Add GHA + AWS deploy config for LDE microservice #1001 merged to main.
• aws sso login --profile lif (account 381492161417).

Deploy sequence (run sequentially — parallel aws-deploy.sh causes SSO conflicts)

1. Create the MDR↔LDE shared key in SSM
   • Preview: AWS_PROFILE=lif ./scripts/setup-mdr-api-keys.sh dev
   • Apply: AWS_PROFILE=lif ./scripts/setup-mdr-api-keys.sh dev --apply
   • Creates /dev/mdr-api/MdrAuthServiceApiKeyLearnerDataExport (MDR server) + /dev/learner-data-export-api/MdrApiKey (LDE client), same value.
2. Create the ECR repo (must precede the image push)
   • AWS_PROFILE=lif ./aws-deploy.sh -s dev --only-stack dev-lif-repositories
   • Creates lif/dev/lif_learner_data_export_api.
3. Build + push the image
   • Merging Issue #997: Add GHA + AWS deploy config for LDE microservice #1001 to main triggers the workflow (paths filter matches), or manual: gh workflow run lif_learner_data_export_api.yml --repo LIF-Initiative/lif-core.
   • Confirm :latest lands in ECR.
4. Redeploy MDR to read the new server-side key env (taskdef-includes change). Requires step 1 (the SSM param must exist or the task won't start).
   • AWS_PROFILE=lif ./aws-deploy.sh -s dev --only-stack dev-lif-mdr-api
5. Deploy the LDE service stack
   • AWS_PROFILE=lif ./aws-deploy.sh -s dev --only-stack dev-lif-learner-data-export-api
   • Creates the ECS service, ALB target group + listener rule (priority 3), and the Route53 record lde.dev.lif.unicon.net (UseLbForService=true).

Verify — "ensure external access"

• ECS service stable: running=1, /health checks green (aws ecs describe-services --cluster dev --services learner-data-export-api-FARGATE).
• External: curl https://lde.dev.lif.unicon.net/health → 200 {"status":"ok"} (public allowlist).
• CloudWatch (log group dev, stream prefix learner-data-export-api) clean at startup — no missing-SSM/config errors.
• End-to-end: curl "https://lde.dev.lif.unicon.net/exports?learnerId=…&dataModelName=…&dataModelVersion=…&dataModelContributorOrganization=…" (with the inbound key) → exercises LDE→MDR (authed) + LDE→QP + LDE→Translator.

Risks / notes

• LDE→QP and LDE→Translator are keyless — verified: the clients send no X-API-Key and neither query_planner_restapi nor translator_restapi adds AuthMiddleware (internal CloudMap-only). Confirm at deploy.
• MDR redeploy ordering: step 4 must follow step 1 (SSM param must exist first).
• TLS cert: assumes the dev ALB HTTPS listener cert covers lde.dev.lif.unicon.net (wildcard *.dev.lif.unicon.net). Verify; a non-wildcard cert would need a SAN/cert update.
• Inbound auth to LDE: externally reachable but guarded by the static service key for now (/health is public). Hardening past the static token = LDE: Expand auth beyond a static API token #1000.
• The changeme6 default in config.py is only a local fallback — confirm MDR and LDE both read the SSM-sourced key from step 1, not the default.

[bjagg]

Dev deploy executed ✅

Ran the runbook against AWS dev (acct 381492161417):

1. SSM key — created /dev/mdr-api/MdrAuthServiceApiKeyLearnerDataExport + /dev/learner-data-export-api/MdrApiKey (other service keys untouched).
2. ECR repo — dev-lif-repositories updated; lif/dev/lif_learner_data_export_api created.
3. Image — GHA build/push succeeded; :latest (…-66cd748) in ECR (ECS-update step no-op'd, as expected pre-service).
4. MDR redeploy — dev-lif-mdr-api rolled cleanly (1/1, rollout COMPLETED) so MDR accepts the LDE key.
5. LDE stack — deployed CREATE_COMPLETE; learner-data-export-api-FARGATE 1/1, rollout COMPLETED.

Verify — external access ✅

• curl https://lde.dev.lif.unicon.net/health → 200 {"status":"ok"}, valid TLS (wildcard *.dev.lif.unicon.net).
• CloudWatch (dev / learner-data-export-api) clean at startup — Uvicorn on :8013, no SSM/config errors.
• Deps healthy: mdr-api, query-planner-org1, translator-org1 all 1/1.

One fix needed mid-deploy

The params hardcoded ALB listener Priority 3, already held by advisor.dev → first attempt rolled back ("Priority '3' is currently in use"). Fixed to 6 (free on dev {2,3,4,5,8} and demo {1,2,4,5,8}) → PR #1009 (also fixes #999's demo deploy). The dev stack is already live at priority 6 (deployed from that branch); #1009 should merge before any LDE redeploy from main, else a redeploy would try to revert the rule to 3 and collide again.

Not yet done

• End-to-end /exports not exercised (needs an inbound LDE key + a real learner + a target data model w/ Org-LIF→target transformation). /health + deps confirm the plumbing; a real /exports call is the remaining functional check.
• Cosmetic: the stack also created learner-data-export-api.dev.lif.unicon.net (A record) with no matching listener rule, so it falls through to the ALB default (Unicon 404). Canonical host is lde.dev.lif.unicon.net.
• /available-data-formats still returns the hardcoded stub (Support learner data export in non-LIF formats #906 item).