[RCA incident] Multiple Application Unreachable

[JamesGeorg]

RCA Discussion Thread
https://juspay.slack.com/archives/C04QY9BFM7Y/p1784268633489389?thread_ts=1784268601.064029&cid=C04QY9BFM7Y

Detection Method
Pingdom Alerts

Incident Triggered By
AWS outage

Executive Summary

AWS VPC origins started returning 504.
Affected domains

Service	Sandbox	Production
superposition.juspay.io	DOWN	DOWN
aarokya.juspay.io	UP	DOWN
airborne.juspay.in	DOWN	DOWN

Timelines - 16th July 2026

Timestamp	what happened
1:15	AWS VPC origin outage event start time
1:25	Pingdom alerts raised for aarokya
1:29	Pingdom alerts raised for superposition
1:40	Meet was started
1:46	Raise a support ticket with AWS support
1:54	Started a chime call with AWS support
??	Curls to envoy
2:19	Created NLB to proxy to ALB
2:27	Switch traffic to NLB for superposition. Pingdom up alert fired
2:43	Switch traffic to NLB for aarokya. Pingdom up alert fired

IMPACT
All applications were not reachable via cloudfront VPC origins.
Aarokya

1. Servers were unreachable
2. 1 user mandate execution failed

Superposition

1. All superposition client which rebooted during the window would fail. Actual numbers are unknown.
2. Releases were not possible in superposition
3. In memory caches in CAC clients protected all users using the CAC client

Airborne

1. Releases were not possible
2. Fresh users are stuck on older versions
3. Cloudfront caches protected most of airborne traffic.

There was 1 and half hour downtime across all applications

EVIDENCE

Superposition Cloudfront

Airborne Cloudfront (Low rates since it is highly protected by cache)

Superposition + Airborne ALB

Aarokya Cloudfront

Aarokya ALB

Q. Why did we take 80 mins to recover?
A. Because we were not sure of the diagnosis

Q. Why were we not sure of the diagnosis?
A. Because we did not follow a systematic triage. And we did not set up a standard monitoring dashboard

COEs