Skip to content

feat: setup script, evaluation walkthrough, portals, and batch retry #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
RajjjAryan merged 1 commit into from
Apr 10, 2026
Merged

feat: setup script, evaluation walkthrough, portals, and batch retry #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,11 @@ AI-powered job search pipeline built on **GitHub Copilot CLI**. Evaluate offers,
### Setup

```bash
# 1. Clone and install
# Option A: One-liner setup
git clone <your-repo-url>
cd career-copilot && bash setup.sh

# Option B: Manual setup
git clone <your-repo-url>
cd career-copilot && npm install
npx playwright install chromium # Required for PDF generation
Expand Down
205 changes: 205 additions & 0 deletions docs/evaluation-walkthrough.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
# Evaluation Walkthrough

> A step-by-step example of how career-copilot evaluates a job offer, showing what each block produces and how the scoring works.

---

## Starting the Evaluation

Paste a job URL into Copilot CLI:

```
Evaluate this: https://boards.greenhouse.io/example/jobs/12345
```

career-copilot reads your `cv.md`, `config/profile.yml`, `modes/_profile.md`, and `modes/_shared.md`, then runs the full A-F evaluation pipeline.

---

## Block A — Role Summary

The AI extracts structured metadata from the JD:

```markdown
## A — Role Summary

| Field | Value |
|-------|-------|
| **Company** | Acme Corp |
| **Role** | Senior Backend Engineer |
| **Archetype** | AI Platform / LLMOps |
| **Domain** | Developer Tools |
| **Function** | Engineering |
| **Seniority** | Senior |
| **Remote** | Hybrid (NYC, 3 days/week) |
| **Location** | New York, USA |
| **Team size** | 8-person platform team |
| **URL** | https://boards.greenhouse.io/example/jobs/12345 |

**TL;DR**: Build and maintain the AI inference platform serving 10M+ daily requests.
The team owns model deployment, observability, and cost optimization.
```

The **archetype** classification determines which scoring weights and profile sections apply.

---

## Block B — CV Match (Score: 4.2/5)

A requirement-by-requirement comparison against your CV:

```markdown
## B — CV Match (Score: 4.2/5)

### Requirements Mapping

| # | JD Requirement | CV Evidence | Strength |
|---|---------------|-------------|----------|
| 1 | 5+ years backend engineering | 7 years across 3 companies | ✅ Strong |
| 2 | Kubernetes & container orchestration | Led K8s migration for 200-service platform | ✅ Strong |
| 3 | ML model serving (TensorFlow Serving, Triton) | Deployed PyTorch models via custom pipeline | ⚠️ Partial |
| 4 | Observability (Datadog, Prometheus) | Built Grafana dashboards, Prometheus alerting | ✅ Strong |
| 5 | Cost optimization at scale | No direct evidence | ❌ Gap |

### Gaps & Mitigation

| Gap | Severity | Mitigation Strategy |
|-----|----------|---------------------|
| ML serving frameworks (Triton) | Medium | Adjacent: custom serving pipeline + quick Triton ramp-up |
| Cloud cost optimization | Low | Portfolio project on spot instance optimization |
```

**How scoring works**: Each requirement is weighted by how prominently it appears in the JD. Strong matches contribute fully, partial matches at 50%, gaps at 0%. The weighted average gives the block score.

---

## Block C — Level & Strategy (Score: 4.0/5)

Compares your seniority to what the JD expects:

```markdown
## C — Level & Strategy (Score: 4.0/5)

**JD Level**: Senior (IC4-IC5 equivalent)
**Candidate Level**: Senior-to-Staff trajectory
**Assessment**: At-level to slightly over-leveled

### Positioning Strategy
- Lead with: Platform ownership and cross-team impact
- Key proof points: Led K8s migration (cv.md §3), mentored 4 engineers (cv.md §5)
- Risk: None — strong at-level match with growth headroom
```

---

## Block D — Comp & Market (Score: 3.8/5)

Market research via web search:

```markdown
## D — Comp & Market (Score: 3.8/5)

### Market Compensation
| Source | Range |
|--------|-------|
| levels.fyi | $180K-$240K (Senior, NYC) |
| Glassdoor | $170K-$220K |
| H1B data | $195K (median, similar roles) |

### JD Stated Comp
$175K-$210K base + equity (if mentioned)

### Assessment
Slightly below market median for NYC Senior Backend. Equity could compensate.
Negotiate toward $200K+ base if offer comes in.
```

---

## Block E — Red Flags & Cultural Signals (Score: 4.5/5)

Proactive detection of concerns:

```markdown
## E — Red Flags & Cultural Signals (Score: 4.5/5)

### 🟢 Positive Signals
- Clear team structure and reporting line mentioned
- Concrete technical challenges (not vague "fast-paced")
- Remote flexibility (hybrid, not forced 5-day)

### 🟡 Watch
- "Startup mentality" in a 500-person company — could mean under-resourced

### 🔴 Red Flags
- None detected
```

---

## Block F — Final Verdict

The weighted global score and recommendation:

```markdown
## F — Final Verdict

| Dimension | Score | Weight |
|-----------|-------|--------|
| CV Match | 4.2 | 30% |
| Level & Strategy | 4.0 | 20% |
| Comp & Market | 3.8 | 20% |
| Cultural Signals | 4.5 | 15% |
| North Star Alignment | 4.0 | 15% |

### Global Score: 4.1 / 5 — Grade: B+

**Recommendation**: Good match, worth applying. Strong technical alignment
with moderate comp upside. The platform ownership scope matches your
growth trajectory.

### Suggested Next Steps
1. Apply with tailored CV (run `generate PDF` mode)
2. Prepare STAR stories for system design and K8s migration
3. Research the team lead on LinkedIn for culture fit signals
```

---

## Score Interpretation

| Score Range | Grade | Meaning |
|------------|-------|---------|
| 4.5+ | A | Strong match — apply immediately |
| 4.0–4.4 | B | Good match — worth applying |
| 3.5–3.9 | C | Decent but not ideal — apply only with specific reason |
| Below 3.5 | D-F | Recommend against applying |

---

## Output Files

After evaluation, career-copilot creates:

| File | Location | Content |
|------|----------|---------|
| Evaluation report | `reports/001-acme-corp-2026-04-10.md` | Full A-F analysis |
| Tailored PDF | `output/cv-acme-corp-2026-04-10.pdf` | ATS-optimized resume |
| Tracker entry | `data/applications.md` | Pipeline row with score and status |

---

## Try It Yourself

```bash
# 1. Setup (if not done)
bash setup.sh

# 2. Open Copilot CLI in this directory

# 3. Paste any job URL
# "Evaluate this: https://boards.greenhouse.io/company/jobs/123456"

# 4. For batch evaluation, add URLs to batch/batch-input.tsv and run:
# "Process the batch"
```
36 changes: 27 additions & 9 deletions modes/batch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,13 @@ task(
- Wait for all to complete before dispatching next batch of 3
- Update `batch/batch-state.tsv` after each completion

### Step 4 — Collect Results
**Rate limiting:**
- Wait **2 seconds** between dispatching each batch of workers to avoid overwhelming job board APIs
- If a `web_fetch` or `browser_navigate` returns a 429 (rate limited), wait **10 seconds** then retry
- If fetching from the same domain (e.g., multiple Greenhouse URLs), space requests **3 seconds** apart
- Maximum **3 concurrent workers** — do not increase even if system resources allow it

### Step 4 — Collect Results & Retry

After each worker completes:

Expand All @@ -154,6 +160,14 @@ After each worker completes:
3. Update `batch/batch-state.tsv` with status, score, paths
4. Log any errors

**Retry logic for failed items:**
- After all items in the batch are processed, check `batch-state.tsv` for `failed` items
- Retry failed items up to **2 additional times** (3 total attempts)
- Increment the `retries` column in `batch-state.tsv` each attempt
- Wait **5 seconds** before each retry attempt
- If an item fails 3 times, mark it as `failed` permanently with the last error message
- Do NOT retry items that failed due to: login required, page not found (404), or job listing removed

### Step 5 — Merge Tracker

After all workers complete:
Expand All @@ -166,19 +180,23 @@ This consolidates `batch/tracker-additions/*.tsv` into `data/applications.md`.

---

## Error Handling
## Error Handling & Retry Policy

| Error | Recovery |
|-------|----------|
| URL inaccessible | Worker fails → conductor marks `failed`, continues |
| JD behind login | Use `browser_navigate` to render page, extract JD from snapshot. If login required → `failed` |
| Worker crashes | Mark `failed`, continue. Re-run batch to retry failed items |
| PDF generation fails | Report .md saved. PDF marked as ❌ in tracker |
| All workers fail | Check `npm run doctor` for setup issues |
| Error | Recovery | Retryable? |
|-------|----------|-----------|
| URL inaccessible / timeout | Wait 5s, retry up to 2 more times | ✅ Yes |
| Rate limited (429) | Wait 10s, retry | ✅ Yes |
| JD behind login | Use `browser_navigate` first. If login wall persists → `failed` | ❌ No |
| Page not found (404) | Mark `failed`, listing likely removed | ❌ No |
| Worker crashes | Mark `failed`, retry in next pass | ✅ Yes |
| PDF generation fails | Report .md saved. PDF marked as ❌ in tracker | ✅ Yes (PDF only) |
| All workers fail | Check `npm run doctor` for setup issues | — |

- **Individual failures do NOT block other items** — each URL is independent
- **Resumability**: Running batch again skips completed items, retries failed ones
- **Timeout**: If a worker exceeds 5 minutes, mark as `failed`
- **Max retries**: 3 total attempts per item (1 initial + 2 retries)
- **Backoff**: 5 second delay between retry attempts, 10 seconds after rate limit

---

Expand Down
112 changes: 112 additions & 0 deletions setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
#!/usr/bin/env bash
set -euo pipefail

# career-copilot setup script
# One command: curl -sL <repo-url>/setup.sh | bash
# Or locally: bash setup.sh

GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m'

pass() { echo -e " ${GREEN}✅ $1${NC}"; }
fail() { echo -e " ${RED}❌ $1${NC}"; exit 1; }
warn() { echo -e " ${YELLOW}⚠️ $1${NC}"; }
info() { echo -e " $1"; }

echo ""
echo "🚀 career-copilot setup"
echo "========================"
echo ""

# 1. Check Node.js
echo "1. Checking Node.js..."
if command -v node &>/dev/null; then
NODE_VERSION=$(node -v | sed 's/v//')
NODE_MAJOR=$(echo "$NODE_VERSION" | cut -d. -f1)
if [ "$NODE_MAJOR" -ge 18 ]; then
pass "Node.js v${NODE_VERSION}"
else
fail "Node.js >= 18 required (found v${NODE_VERSION}). Install from https://nodejs.org"
fi
else
fail "Node.js not found. Install from https://nodejs.org (v18+)"
fi

# 2. Install npm dependencies
echo ""
echo "2. Installing npm dependencies..."
if [ -f package.json ]; then
npm install --no-fund --no-audit 2>&1 | tail -1
pass "npm packages installed"
else
fail "package.json not found — run this from the career-copilot directory"
fi

# 3. Install Playwright Chromium
echo ""
echo "3. Installing Playwright Chromium (for PDF generation)..."
npx playwright install chromium 2>&1 | tail -3
pass "Playwright Chromium installed"

# 4. Check Go (optional — for TUI dashboard)
echo ""
echo "4. Checking Go (optional — for TUI dashboard)..."
if command -v go &>/dev/null; then
GO_VERSION=$(go version | awk '{print $3}' | sed 's/go//')
pass "Go ${GO_VERSION} — dashboard available"
info " Build with: cd dashboard && go build -o career-dashboard ."
else
warn "Go not installed — TUI dashboard won't build (everything else works fine)"
info " Install from https://go.dev/dl/ if you want the dashboard"
fi

# 5. Setup config files
echo ""
echo "5. Setting up configuration..."
if [ ! -f config/profile.yml ]; then
if [ -f config/profile.example.yml ]; then
cp config/profile.example.yml config/profile.yml
warn "Created config/profile.yml from template — edit with your details"
else
warn "No profile template found — create config/profile.yml manually"
fi
else
pass "config/profile.yml exists"
fi

if [ ! -f portals.yml ]; then
if [ -f templates/portals.example.yml ]; then
cp templates/portals.example.yml portals.yml
warn "Created portals.yml from template — customize for your job search"
else
warn "No portals template found"
fi
else
pass "portals.yml exists"
fi

if [ ! -f cv.md ]; then
warn "cv.md not found — create it with your CV in markdown format"
info " See examples/ for reference CV formats"
else
pass "cv.md exists"
fi

# 6. Run doctor
echo ""
echo "6. Running doctor check..."
echo ""
node doctor.mjs 2>&1 || true

echo ""
echo "========================"
echo -e "${GREEN}✅ Setup complete!${NC}"
echo ""
echo "Next steps:"
echo " 1. Edit config/profile.yml with your details"
echo " 2. Create cv.md with your CV in markdown"
echo " 3. Open GitHub Copilot CLI in this directory"
echo " 4. Paste a job URL to evaluate it"
echo ""
Loading
Loading