Steadfast Support Ticket Triage Pipeline

Overview

You're building a support ticket triage pipeline for Steadfast, a B2B SaaS company. The pipeline processes incoming support tickets and must:

1. Classify each ticket into a category and priority level
2. Generate a short initial response to the customer

You're given a knowledge base of 300 historical tickets (CSV) and a 40-ticket labeled dev set for evaluation. A hidden test set (which you do not have access to) will be used in the final assessment.

Your job is to implement the pipeline described below, iterate on it using the dev set, and submit your best version.

Don't assume the knowledge base is clean. It is a raw export of historical tickets as real support agents filed them — not a curated training set, and nobody has audited it. Whatever you feed the model becomes the ground truth it reasons from, so anything wrong in there propagates into your predictions, confidently and invisibly. Look at the data properly before you build on it, and say in your write-up what you found and what you did about it.

---

The Pipeline

You must implement a pipeline with the following stages. Each stage has a clear input and output. You have freedom in how you implement each stage, but all stages must be present and functional.

┌─────────────┐    ┌──────────────┐    ┌──────────────────────┐
│  1. LOAD     │───▶│ 2. PREPROCESS │───▶│ 3. LLM CLASSIFICATION │
│  DATA        │    │              │    │                      │
└─────────────┘    └──────────────┘    └──────────┬───────────┘
                                                  │
                                                  ▼
┌─────────────┐    ┌──────────────┐    ┌──────────────────────┐
│  6. EVALUATE │◀───│ 5. HEURISTICS │◀───│ 4. VALIDATE          │
│              │    │  & POST-PROC │    │    OUTPUT             │
└──────┬──────┘    └──────────────┘    └──────────────────────┘
       │
       ▼
┌─────────────┐    ┌──────────────┐
│  7. ERROR    │───▶│ 8. ITERATE   │──── (loop back to any stage)
│  ANALYSIS    │    │              │
└─────────────┘    └──────────────┘

Stage 1 — Load Data

Read the knowledge base CSV and dev set JSON. Parse, inspect structure, understand what you're working with. Pay attention to the vocabulary and concepts that appear in the knowledge base — Steadfast has platform-specific features and terminology that will appear in incoming tickets.

Stage 2 — Preprocess

Clean and normalize the knowledge base. Prepare it for use as context in later stages. Document any decisions you make here.

Stage 3 — LLM Classification

Send each incoming ticket to an LLM with a structured prompt. The model should return JSON with category, priority, and a customer response. Use the knowledge base as context (RAG, in-context examples, embeddings — your choice). Design your system prompt carefully.

Important: The knowledge base contains Steadfast-specific features, internal terminology, and historical resolution context that the model won't know about otherwise. Your responses should be grounded in this context — referencing specific workarounds, configuration steps, known issues, or KB articles where relevant. Generic responses like "we're looking into it" that could apply to any product are significantly lower quality than responses that demonstrate knowledge of Steadfast's platform.

Stage 4 — Validate Output

Check every LLM response for: valid JSON structure, allowed enum values for category and priority, required fields present, response not empty. Handle failures gracefully — fall back to unknown or flag for human review. Track validation failure rates.

Stage 5 — Heuristics & Post-Processing

Add rule-based corrections on top of the LLM output. Examples: tickets mentioning specific keywords might need priority adjustments, certain patterns might reliably indicate a category, plan-based business rules might apply. These rules should be informed by patterns you observe in the data and in your error analysis.

Stage 6 — Evaluate

Run the full pipeline on the dev set. Compute:

• Category accuracy (exact match + partial credit for acceptable alternatives)
• Priority accuracy
• A response quality score (you define the metric — justify your choice)
• Per-category and per-priority breakdowns

Your response quality metric should capture whether the response is actually helpful to the customer — not just whether it's polite and the right length. Consider whether the response references the correct issue, provides actionable next steps, and avoids giving advice that's related but wrong (e.g., fixing the wrong endpoint, referencing the wrong integration provider).

Output a structured report.

Stage 7 — Error Analysis

Review incorrect predictions. Categorize them by root cause and document your findings.

Stage 8 — Iterate

Based on error analysis, improve any stage: prompt, preprocessing rules, heuristics, context selection, validation logic. Rerun eval. Repeat.

---

Output Format

The pipeline must produce a JSON result per ticket:

{
  "ticket_id": "EVAL-001",
  "category": "bug",
  "priority": "high",
  "response": "Hi — thanks for reaching out. We're looking into the dashboard issue...",
  "confidence": 0.85,
  "flags": []
}

Field	Required	Description
ticket_id	Yes	From the input ticket
category	Yes	One of: billing, bug, feature_request, account, integration, onboarding, security, performance
priority	Yes	One of: low, medium, high, critical
response	Yes	Initial customer-facing response
confidence	No	Model's confidence (0-1). Encouraged but optional.
flags	No	Array of strings for anything notable: ["ambiguous_category", "possible_duplicate", "escalate_to_human"]

---

Constraints

• Use any LLM provider (OpenAI, Anthropic, open-source — your choice)
• No fine-tuning. Prompt engineering and RAG only.
• Total latency per ticket should be under 30 seconds
• Pipeline must be re-runnable end-to-end from a single command

---

Rules

1. AI / coding tools are welcome for the code (Copilot, Cursor, Claude, ChatGPT, etc.) — we expect you to use them. How you use them, and where you apply your own judgment, is part of what we evaluate. The write-up (WRITEUP.md) is the one exception: write it entirely yourself, with no LLM/AI help — no AI generation, drafting, editing, rephrasing, or touch-ups. We want to read your reasoning in your own words, not a model's. A short, plainly-written write-up in your own voice is exactly what we're after.
2. Time budget: ~6 hours of focused work. Not timed, but scope accordingly. We value a well-reasoned 80% solution over a sloppy 95%.
3. Include a git log. Init a repo and commit as you go. We want to see how your work evolved.
4. Code must run. We will clone, install deps, set an API key env var, and run your pipeline. If it breaks, that's a signal.

---

Deliverables

1. The pipeline code — all 8 stages, modular, runnable

2. Evaluation output — your latest eval results as JSON

3. Write-up (max 2 pages), written entirely by you with no LLM/AI help, covering:

   • Data exploration: how you approached the knowledge base and eval set
   • Pipeline design decisions: why you chose your approach at each stage
   • Iteration log: what you tried, what worked, what didn't (with metrics)
   • Response quality metric: what you chose and why
   • What you'd do differently with more time

4. Your EDA notebook (e.g. eda.ipynb) — the exploratory analysis you did to understand the knowledge base and eval set before building the pipeline. We strongly suggest starting here: actually look at the data, profile it, and sanity-check it before computing anything — that's usually where the real insight lives. Include the notebook; it's one of the most useful artifacts for us to see how you reasoned about the problem. (Use AI freely for the notebook code — just remember WRITEUP.md is yours alone.)

---

Project Structure

/src
  pipeline.py          # main entry point
  preprocess.py        # stage 2
  agent.py             # stage 3 (LLM classification)
  validate.py          # stage 4
  postprocess.py       # stage 5
  evaluate.py          # stage 6
  analyze.py           # stage 7 (error analysis)
/data
  knowledge_base.csv   # 300 historical tickets
  eval_set.json        # 40-ticket labeled dev set
/output
  eval_results.json    # your latest eval run
  error_analysis.json  # your error analysis output
WRITEUP.md
README.md
requirements.txt
.env.example

You may restructure however you like, but all stages must be identifiable and the full pipeline must run from a single entry point.

---

Setup

git clone <this-repo>
cd steadfast-triage-assignment
pip install -r requirements.txt
cp .env.example .env
# Add your API key to .env

Run

# Run pipeline on eval set
python src/pipeline.py

# Run pipeline + evaluation + error analysis
python src/pipeline.py --eval

---

Self-check (optional)

You can preview how your pipeline scores on a practice held-out set — different tickets than your dev eval_set.json — using the provided CLI:

./run.sh validate                 # runs your pipeline on the held-out set
.\run.ps1 validate                # Windows (PowerShell)

It packages your repository, runs it in an isolated sandbox on our side, and returns your category accuracy, priority accuracy, and how many tickets your pipeline scored. You get 3 attempts.

This is a practice check to see whether your method generalizes — it is not your grade, and this practice set is not the set we grade on.

How we run your pipeline

The self-check — and our final evaluation — run your pipeline the same way: from your repo root, invoking your entry point on a fresh set of tickets. For that to work, src/pipeline.py must accept --input and --output:

python src/pipeline.py --input <tickets.json> --output <predictions.json>

• --input is a JSON array of unlabeled tickets, each with ticket_id, subject, body, customer_name, and plan (the same fields as your dev set, minus the labels). Classify every one.
• --output is where you write a JSON array of predictions — one per input ticket, in input order:

  [{"ticket_id": "...", "category": "...", "priority": "...", "response": "..."}]

• Read your knowledge base from your own repo (e.g. data/knowledge_base.csv). It's bundled with your submission; the held-out tickets arrive separately via --input, so don't assume they live under data/.
• LLM access is provided inside the sandbox: calls made with the anthropic or openai Python SDK the standard way (e.g. anthropic.Anthropic()) are routed to our model proxy — you don't need your own key for the self-check. The sandbox has no other network access.
• Declare extra dependencies in requirements.txt so we can install them before running.

---

Submission

When you're ready to submit your completed assignment, use the provided CLI tool to bundle and upload your repository.

Quick Submit

From the root of your assignment repository:

./run.sh publish

This will:

1. Bundle your git repository (respecting .gitignore)
2. Upload directly to our evaluation system
3. Ask you to confirm your email address
4. Print a confirmation message and instructions

Providing Your Email

The CLI needs your email address to notify you that the upload succeeded. It will:

1. First try to read your email from git config user.email
2. If it is not set, prompt you to enter it interactively
3. Alternatively, pass it directly: ./run.sh publish --email you@example.com

Confirmation

After the upload completes, you will be asked to reply to the email you received for this assignment with your name and the repository name. Please send that reply so we can match your submission to your application.

What Gets Uploaded

The CLI bundles:

• All git-tracked files in your repository
• The .git directory (we review your commit history)

Note: The archive respects .gitignore and excludes files like .env, node_modules, .venv, etc.

Troubleshooting

"Not inside a git repository"

• Make sure you've initialized git: git init && git add -A && git commit -m "initial"

"Repository must have at least one commit after initialization"

• The initial repository setup commit is not enough. Make at least one additional commit with your work before submitting.

Binary not found

• Run directly: bin/latent-cli-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m) publish

Upload fails

• Ensure you're connected to the internet

---

FAQ

Q: Can I use frameworks like LangChain, LlamaIndex, DSPy? A: Yes. But if your pipeline is opaque framework calls, we want evidence you understand what's happening at each stage.

Q: What model should I use? A: Your choice. Model selection is a decision we want to see you justify.

Q: How is the hidden test set scored? A: Same pipeline — category accuracy, priority accuracy, response quality (scored by an LLM judge with a rubric we share post-review). The test set has harder edge cases. We expect a 5-15% accuracy drop from dev to test.

Q: What if my accuracy is low? A: Raw accuracy is not the primary signal. Sound evaluation + clear error analysis + well-reasoned iteration beats high accuracy with no explanation.

Q: Do I need to implement all 8 stages? A: Yes. A stage can be minimal (e.g., "no heuristics applied — here's why") but it must be present. Skipping a stage entirely is a signal.

Q: How important is the knowledge base for classification? A: Very. Some tickets reference Steadfast-specific features and terminology that an LLM won't know about without KB context. Your pipeline should use the KB, not just classify from ticket text alone.

Q: How is response quality scored on the hidden test set? A: By an LLM judge evaluating relevance, tone, and actionability. Responses that reference specific Steadfast features, workarounds, or known issues from the KB score higher than generic empathetic responses. Getting the right advice for the wrong issue (e.g., fixing the wrong endpoint) is penalized.