Run first production pilot

[alexeygrigorev]

Run first production pilot

Status: pending; primarily blocked on HUMAN production access, DNS, and live-send coordination
Tags: P0, infra, email, ses, tracking, campaigns, ops, human
Depends on: #11, #12, #13
Blocks: —

Scope

• Run the first production Datamailer pilot after Add SES webhook processing and suppression #11 SES webhook/suppression handling, Add operator campaign/contact UI #12 operator campaign/contact UI, and Add production infrastructure and monitoring #13 production infrastructure/monitoring are available.
• [HUMAN] Verify AWS SES production access for the chosen account/region: sandbox exit, daily quota, max send rate, verified sender identity, configuration set, event publishing target, and any account-level suppression settings.
• [HUMAN] Configure and verify sending DNS for the selected production domain/subdomain: SES identity DKIM records, SPF/MAIL FROM record if used, DMARC alignment policy, and any provider-required verification records.
• [HUMAN] Configure and verify tracking/unsubscribe domains: public HTTPS routing to Datamailer, open pixel endpoint, click redirect endpoint, hosted unsubscribe endpoint, and any DNS/TLS records needed for those domains.
• Perform a staging smoke before any live production send: web health, migrations, queue enqueue/dequeue, Lambda/worker invocation, SES webhook path with non-production fixtures, alarms/dashboard visibility, and operator UI access.
• [HUMAN] Prepare the internal pilot recipient list and content using real operator workflow: correct client/audience, campaign body, unsubscribe link, tracking links, sender/from/reply-to, and suppression-aware recipient snapshot.
• Send an internal pilot to a small trusted list through production Datamailer/SES and verify delivery, rendering, links, unsubscribe, bounce/complaint handling where testable, event capture, and operator UI visibility.
• [HUMAN] Choose a limited real campaign segment, expected volume, send window, on-call owner, rollback owner, and current-provider comparison baseline before queuing the real pilot.
• Send a limited real campaign through Datamailer and compare delivery/stat behavior with the current provider for the same or comparable audience segment.
• Verify bounce, complaint, unsubscribe, delivery, open, and click evidence is captured in email_events, summarized on campaign/contact records, and visible in the operator UI without shell/database access.
• Define and enforce pause/rollback criteria for the pilot: elevated SES bounce/complaint/failure rate, queue/DLQ growth, stuck campaign state, worker/Lambda errors, web/API health issues, DB connection pressure, tracking/unsubscribe failures, or operator inability to audit the send.
• Collect pilot evidence: screenshots/links for operator UI views, SES/account metrics, CloudWatch alarms/dashboard status, sample email_events, campaign stats, current-provider comparison notes, incident/pause decisions, and final go/no-go recommendation.

Acceptance Criteria

• [HUMAN] AWS SES production access is confirmed in the intended account/region, including sandbox exit, sending quota/rate, verified identity, configuration set, and event publishing path.
• [HUMAN] Sending DNS is configured and verified for the selected domain/subdomain: DKIM passes, SPF/MAIL FROM alignment is correct if used, and DMARC behavior is acceptable for the pilot.
• [HUMAN] Tracking and unsubscribe domains resolve over HTTPS to Datamailer production, and test open/click/unsubscribe requests reach the expected public endpoints.
• Staging smoke passes before production sending: web health, migrations, DB connectivity, queue round trip, worker invocation, SES webhook fixture processing, alarms/dashboard visibility, and logs are verified.
• Internal pilot campaign is created through the operator workflow with a real client/audience, reviewed content, working tracking/unsubscribe links, and a small trusted recipient list.
• [HUMAN] Internal pilot is sent through production Datamailer/SES and at least one recipient confirms receipt, rendering, link behavior, unsubscribe path, and no obvious spam-folder/delivery regression.
• Internal pilot delivery/open/click/unsubscribe/bounce/complaint events that are exercised are captured as auditable email_events and update campaign recipient/contact/campaign summaries idempotently.
• Operator UI from Add operator campaign/contact UI #12 can inspect the internal pilot campaign list/detail, recipient filters, contact detail, suppression state, and event timeline without shell/database access.
• [HUMAN] Limited real campaign segment, expected recipient count, send window, on-call owner, rollback/pause owner, and current-provider comparison baseline are documented before queuing the real send.
• [HUMAN] Limited real campaign is sent through production Datamailer/SES only after pause criteria and monitoring responsibilities are active for the agreed on-call window.
• Real pilot campaign produces campaign recipient snapshots, sent/delivered/open/click/unsubscribe/bounce/complaint/failure summaries, and operator-visible audit trails consistent with the data model.
• Delivery and engagement stats are compared against the current provider or a comparable recent campaign, with differences documented and investigated when materially outside expectations.
• Bounce, complaint, and unsubscribe behavior is verified end to end: future marketing sends suppress affected contacts, event history is visible, and operators can explain the state from UI/audit data.
• Pause/rollback criteria are documented and were monitored during the on-call window; any triggered criterion pauses sending or routes a documented incident before further sends.
• Pilot evidence is attached in an issue comment: staging smoke results, SES/DNS verification notes, send window/on-call notes, campaign/operator UI screenshots or links, metric comparison, event samples, and final go/no-go recommendation.

Test Notes

• Unit/service tests: no new unit tests are expected unless this pilot uncovers a code defect. Any code defect found during pilot should be filed or fixed in a separate implementation issue with focused tests.
• API/view tests: use existing Add operator campaign/contact UI #12 operator UI coverage as the baseline. During pilot, manually verify authenticated campaign/contact/operator views against the production/staging pilot data and collect screenshots or links as evidence.
• LocalStack/mocked AWS tests: use existing Add SES webhook processing and suppression #11/Add production infrastructure and monitoring #13 mocked AWS and staging smoke coverage for non-live verification. Do not use real AWS sends in automated tests.
• Staging smoke: record exact environment, timestamp, health URL result, migration status, queue/worker result, SES webhook fixture result, alarm/dashboard visibility, and log location.
• Production smoke: [HUMAN] record SES identity status, DNS checks, send quota/rate, configuration set/event publishing status, tracking/unsubscribe domain checks, and first internal send result.
• Real-send verification: [HUMAN] record current-provider comparison baseline, Datamailer campaign stats, SES metrics, bounce/complaint/unsubscribe observations, suppression checks, and any pause/rollback decisions.
• UI/screenshot checks: capture operator campaign list/detail, recipient filters for at least sent/delivered/unsubscribed/bounced/complained where available, contact detail/event timeline for sampled recipients, and any monitoring dashboard evidence needed for the issue comment.

Blocked by: #11, #12, and #13 are closed, so there is no known code prerequisite. This issue remains blocked for execution until HUMAN owners confirm real AWS SES production access, DNS/domain control, production credentials, pilot recipient/content approval, and an on-call send window. Keep human on the issue until all live AWS/SES/DNS/send criteria are verified.