Skip to content

README.md

Latest commit

 

History

History
149 lines (107 loc) · 5.1 KB

README.md

File metadata and controls

149 lines (107 loc) · 5.1 KB

External Storage Demo — NickelAndDime

NickelAndDime.com runs an employee spending platform. This workflow audits each employee's transaction history for recurring charges and large payments.

This repo is organized as a sequence of git branches that incrementally add Temporal features. This branch (no external storage) is the starting point: everything works for normal employees, but two AI-heavy employees' transaction lists exceed Temporal's 2MB payload limit and their audits fail.

Cast

20 employees in expenses.json.

id name role txns size
emp-007 Yuki Tanaka CTO 13,118 2.7 MB
emp-013 Luis Hernandez VP AI Research 13,068 2.7 MB
18 others ... ... 30-150 7-30 KB

Yuki and Luis both spend heavily on Anthropic / OpenAI API tokens — thousands of small charges. Their raw transaction arrays are >2MB JSON-encoded.

Workflow shape

ExpenseAuditWorkflow(employee_id)
  └─ fetch_transactions(employee_id)           # returns the whole array
  ├─ review_recurring_payments(all_txns)       # full list in — human review
  └─ review_large_payments(filtered_txns)      # workflow filters (>= $1000) first

Recurring + large run in parallel via asyncio.gather.

Retry policy: activities retry forever on a fixed 7-second interval. Workflows aren't designed to fail — they keep trying until the underlying problem is fixed. The fixed interval (vs. exponential backoff) keeps demo timing predictable. In the next branch, once external storage is wired up, the in-flight retries will succeed on their next attempt and the stuck workflows will unstick on their own.

Setup

# Terminal A — Temporal dev server
temporal server start-dev

# This directory
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python generate_expenses.py      # writes expenses.json (~8MB)

Run

# Terminal B
python worker.py

# Terminal C
python starter.py

Expected summary:

OK:     18/20
STUCK:   2/20  (still retrying — inspect in UI)

Stuck workflows (still retrying every 7s):
  emp-007  Yuki Tanaka          CTO
  emp-013  Luis Hernandez       VP AI Research

The two stuck workflows stay running; their fetch_transactions activity keeps retrying every 7s with a blob-size error. They will recover automatically once external storage is wired up in the next branch.

Temporal UI: http://localhost:8233/namespaces/default/workflows

Demo script

Setup (pre-demo)

  1. Terminal A: temporal server start-dev
  2. Terminal B: python worker.py
  3. Have the Temporal UI open at http://localhost:8233.

Act 1 — Launch a bunch of jobs

"NickelAndDime built a Temporal workflow that audits each employee's transaction history. Let's run it for all 20 employees tonight."

  1. Terminal C: python starter.py
  2. Point at the UI as workflows start to complete.
  3. Starter prints the summary after 20s: 18 OK, 2 STUCK.

"Most finished cleanly. Two are stuck — they're still retrying. Our CTO spends a lot on AI tokens, and it looks like the VP of AI Research does too. Workflows don't give up; they'll keep trying on a 7-second interval until we fix whatever's wrong."

Act 2 — Investigate in the UI

  1. In the UI, the two stuck runs are Running (not Failed). Click into audit-emp-007-....
  2. Look at the fetch_transactions activity. Each attempt completes locally but the completion is rejected, so Temporal schedules another attempt 7 seconds later. The attempt counter is climbing.
  3. Open the activity's pending failure. Call out the error code: TMPRL1103.

"Temporal has a 2MB default limit on payload size. The CTO's transaction list is about 2.7MB — too big to store in Event History. The workflow isn't failing, it's patiently retrying and waiting for us to fix it."

Act 3 — Find the fix

  1. Search TMPRL1103 in the Temporal docs.
  2. The docs describe the fix: wire up external storage via a payload codec. Large payloads get stored outside Temporal; a "claim check" goes into Event History instead.

"This is exactly what external storage is built for. In the next branch, we'll wire it up and watch these workflows unstick."

Exit

Leave the worker and dev server running; the next branch builds on top of this same environment.

What's intentionally missing on this branch

  • Payload codec with external storage (next branch).
  • Codec server for inspecting blob-backed payloads in the UI (branch after).
  • A $0 charge is seeded inside emp-007's transactions — that's a planted bug for the codec-server debugging arc. It causes no failure here.

Files

  • generate_expenses.py — seeded synthetic data generator. Reproducible.
  • expenses.json — generated; gitignored. Run the generator to produce it.
  • activities.py — three activities, dataclass return types.
  • workflows.py — ExpenseAuditWorkflow.
  • worker.py / starter.py — local dev runners.