Skip to content

docs(blog): 1 Day, 2 Engineers, 3 Lines of Code — Shard Manager incident blog post#325

Draft
jakobht wants to merge 21 commits intomasterfrom
incident-blog
Draft

docs(blog): 1 Day, 2 Engineers, 3 Lines of Code — Shard Manager incident blog post#325
jakobht wants to merge 21 commits intomasterfrom
incident-blog

Conversation

@jakobht
Copy link
Copy Markdown
Member

@jakobht jakobht commented Mar 27, 2026

What changed?

Added a new blog post: "1 Day, 2 Engineers, 3 Lines of Code" — an incident writeup describing how the new Cadence Shard Manager found and mitigated a latent deadlock in the Matching service.

Includes interactive React components (architecture diagram, log histograms, log tables, timeline, time series charts) embedded in the MDX post.

Why?

Sharing the incident story publicly as a demonstration of the Shard Manager's automatic mitigation capabilities.

How did you verify it?

Previewed locally via Docusaurus dev server.

Potential risks

None — documentation only.

Related changes

None.

@gitar-bot
Copy link
Copy Markdown

gitar-bot bot commented Mar 27, 2026

Code Review ✅ Approved 2 resolved / 2 findings

Blog post documenting the Shard Manager incident with timeline, graphics, and post-mortem analysis. Timezone inconsistency between TimeSeriesChart and other components has been resolved.

✅ 2 resolved
Bug: Timezone inconsistency: TimeSeriesChart uses local time, others use UTC

📄 blog/2026-03-23-shard-manager-incident/TimeSeriesChart/index.tsx:14-19 📄 blog/2026-03-23-shard-manager-incident/LogHistogram/index.tsx:15-19 📄 blog/2026-03-23-shard-manager-incident/LogTimeline/index.tsx:9-14
TimeSeriesChart's formatTime uses getHours()/getMinutes() (local timezone), while LogHistogram and LogTimeline use getUTCHours()/getUTCMinutes() (UTC). Since this is an incident blog post where readers need to correlate timestamps across the heartbeat chart, CPU chart, log histogram, and timeline, the time axes will show different offsets depending on the reader's timezone. For example, a reader in UTC-8 would see the log histogram spike at 22:00 but the heartbeat drop at 14:00, making the narrative confusing.

Bug: Module-level mutable hostColors leaks across renders/SSR

📄 blog/2026-03-23-shard-manager-incident/LogTimeline/index.tsx:22-29
The hostColors object is declared at module scope and mutated inside hostColor(). In a Docusaurus SSR build (or with React strict mode / hot reloading), this state persists across renders and can produce inconsistent color assignments. Since there are only two hosts it's unlikely to cause visible breakage, but it's a subtle footgun.

Rules ❌ No requirements met

Repository Rules

PR Description Quality Standards: Replace the vague verification statement with concrete steps, e.g. 'Ran `npm run start` and verified the new blog post renders at `/blog/shard-manager-incident`.'
Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change:

Auto-apply Compact
gitar auto-apply:on         
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

jakobht added 20 commits March 27, 2026 10:34
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Signed-off-by: Jakob Haahr Taankvist <jht@uber.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant