feat(claude_agent_sdk): add vulnerability detection agent cookbook#595
Conversation
06_The_vulnerability_detection_agent.ipynb runs threat-model -> find -> triage -> report on a self-contained 45-line canary C target using claude-agent-sdk: ClaudeSDKClient for the multi-turn bootstrap-then- interview threat model, stateless query() for find/triage/report, with Claude Code's built-in Read/Grep/Glob in place of a hand-rolled tool loop. Supporting changes: - vulnerability_detection_agent/canary/canary.c: three unlabeled memory-safety bugs (heap OOB, stack OOB, UAF) - registry.yaml: new entry under Claude Agent SDK + Cybersecurity; threat_intel_enrichment_agent also tagged Cybersecurity - authors.yaml: add eugeneyan-ant - pyproject.toml: +claude-agent-sdk so root 'uv sync' covers the series - .gitignore: generated THREAT_MODEL.md - .claude/commands/add-registry.md: add Cybersecurity to category list
Notebook ChangesThis PR modifies the following notebooks: 📓
|
There was a problem hiding this comment.
PR Review
Recommendation: REQUEST_CHANGES
Summary
This PR ports a well-structured vulnerability detection agent cookbook from the private repo, demonstrating a threat-model → find → triage → report pipeline using the Claude Agent SDK. The architecture and pedagogy are solid, but there are two blocking issues that need resolution before merge.
Actionable Feedback (7 items)
Blocking:
-
06_The_vulnerability_detection_agent.ipynb(kernel metadata) — Python version mismatch: notebook was executed on Python 3.13.12 butpyproject.tomlrestricts torequires-python = ">=3.11,<3.13". Re-run the notebook under Python 3.12, confirm all cells execute cleanly, and commit the regenerated output. If 3.13 is genuinely required, widen the constraint viauv(not a direct edit) and update Prerequisites. -
06_The_vulnerability_detection_agent.ipynb(setup cell,MODEL_NAME = "claude-opus-4-7") — Uses a different model than every other notebook in theclaude_agent_sdk/series (00_through05_all useclaude-opus-4-6) and differs from the CLAUDE.md canonical alias. Either align toclaude-opus-4-6, or open a separate PR that upgrades the entire series together and updates CLAUDE.md. Silent divergence within the same series is confusing.
Important (non-blocking but should be fixed):
-
06_The_vulnerability_detection_agent.ipynb(setup cell,collect()docstring) — Multi-paragraph docstring violates project style (one short line max). Trim to"""Drain a message stream, print tool calls, return final assistant text."""and move the "why factored out" prose to the preceding markdown cell. -
06_The_vulnerability_detection_agent.ipynb(report cell, code fence stripping) — Theif raw.startswith("```"):strip is fragile: it fails if the model adds prose after the closing fence, or if the JSON contains a literal```. At minimum add a comment noting the limitation; ideally use a regex or rely solely on theJSONDecodeErrorhandler. -
06_The_vulnerability_detection_agent.ipynb(find/triage cells) —collect()silently returns""if the model emits only tool calls with noTextBlock. Add a guard after each stage that feeds the next:if not findings_text.strip(): raise RuntimeError("find agent produced no text; check trace above"). -
06_The_vulnerability_detection_agent.ipynb(report cell,REPORT_SCHEMA) — The"file"field conflates path and line range (e.g.,"canary.c:8-9"), making programmatic extraction of line numbers require string parsing. Add a"line_range"field or rename to"location"with a comment explaining the format. - General:
assert TARGET_DIR.is_dir()— Per the pattern used in other Agent SDK notebooks (e.g.,03_), preferif not TARGET_DIR.is_dir(): raise RuntimeError(...)to avoid silent no-ops under-O.
Detailed Review
Code Quality
The three-stage pipeline (threat model, find, triage, then a separate report step) is architecturally clean and pedagogically sound. The decision to run triage as an independent pass that re-scores severity without inheriting the find agent's confidence scores is a real production pattern and is explained well in the markdown. The collect() helper is a sensible factoring that avoids repeating the isinstance ladder four times.
The prompt engineering is high quality: the quality-tier rubric in FIND_PROMPT (HIGH VALUE vs LOW VALUE bug classes) is the most important part of a production vuln-hunting prompt and the notebook calls this out explicitly. REPORT_SCHEMA with all fields in required and null for not-applicable is a solid pattern for structured LLM output.
The %%capture install cell, dotenv.load_dotenv(), and assert TARGET_DIR.is_dir() with explicit cwd message are all aligned with project conventions.
Security
disallowed_tools=["Bash"] is applied consistently across all agents, keeping the pipeline read-only and scoped to the vendored canary. The ENGAGEMENT_CONTEXT system prompt block documents authorization scope on every call — a good defensive pattern. The CVP link in Step 1's markdown is responsible and directly relevant. The .gitignore entry for THREAT_MODEL.md correctly prevents generated agentic artifacts from being committed.
Suggestions
jsonschemais mentioned in Step 6's prose ("in production you'd validate withjsonschema") but not installed in the%%capturecell; readers who want to follow through on the suggestion without leaving the notebook will hit anImportError.- The principle of least privilege demonstrated through scoped
allowed_toolsper agent (Read/Write/Edit→Read/Grep/Glob→Read/Grep→[]) is worth calling out explicitly in the Step 4 markdown or the Summary, since it's one of the strongest architectural points in the notebook.
Positive Notes
- The introduction leads with the pain point (false-positive fatigue, fuzzer harness cost) before naming any technology — the right structure for a problem-driven cookbook.
- The two-turn
ClaudeSDKClientthreat model (bootstrap → interview) vs. statelessquery()for the remaining stages cleanly illustrates when to use each SDK primitive, and the markdown explains the tradeoff. - The canary target is a perfect teaching artifact: 45 lines, three distinct bug classes, one-byte dispatch, no false complexity.
Add Cybersecurity to .github/registry_schema.json so the new vuln detection cookbook entry passes registry-check.
Notebook ChangesThis PR modifies the following notebooks: 📓
|
There was a problem hiding this comment.
PR Review
Recommendation: COMMENT
Summary
Ports a well-structured vulnerability-detection agent cookbook from the private repo. The multi-turn threat-model session + stateless find/triage/report pipeline is a clean, pedagogically sound demonstration of ClaudeSDKClient vs query() usage patterns.
Actionable Feedback (4 items)
-
06_The_vulnerability_detection_agent.ipynb(setup cell withassert TARGET_DIR.is_dir()) — Replaceassertwithraise RuntimeError. Python's-Oflag silently disablesassertstatements, so a CI runner with optimization enabled would skip the guard entirely. ARuntimeErrorwith explicit instructions is more robust:if not TARGET_DIR.is_dir(): raise RuntimeError( f"TARGET_DIR not found: {TARGET_DIR}\n" "Open this notebook from the claude_agent_sdk/ directory, " "or set TARGET_DIR to an absolute path." )
-
.gitignore— The entryclaude_agent_sdk/vulnerability_detection_agent/canary/THREAT_MODEL.mdis a sub-project artifact. Move it toclaude_agent_sdk/.gitignore(or a newvulnerability_detection_agent/canary/.gitignore) to avoid polluting the root gitignore with path-specific patterns. -
06_The_vulnerability_detection_agent.ipynb(Step 6 report cell,REPORT_SCHEMA) — The schema collapses path and line number into a singlefilefield (e.g.,"canary.c:8-9"). This will be hard for downstream consumers (SARIF, issue trackers) to parse reliably. Consider adding a separate"line"field, especially since the summary cell already recommends wiring the output into trackers. -
06_The_vulnerability_detection_agent.ipynb(collect()docstring) — The 7-line docstring restates what the adjacent markdown cells already explain. Per project style, a one-liner suffices:"""Drain an Agent SDK message stream, print tool calls, and return the final text."""
Detailed Review
Code Quality
The async patterns are correct throughout — async with ClaudeSDKClient for the multi-turn session and await collect(query(...)) for stateless one-shots. The collect() helper is a nice factoring that avoids repeating the isinstance ladder four times. The JSON fence-stripping in Step 6 (raw.split("\n", 1)[1].rsplit("```", 1)[0]) handles the common case; a .strip() call after the fence block would close the edge case where a leading newline remains, but this is minor for a demo notebook.
Security
The canary.c intentionally contains memory-safety bugs — this is clearly labeled in both the notebook prose and the file header. The disallowed_tools=["Bash"] constraint keeping agents read-only is a good safety default, and the explanation for why Bash was excluded (needs a sandboxed container) sets the right expectation for readers who want to extend the notebook.
The ENGAGEMENT_CONTEXT system prompt appended to every agent is excellent practice — it surfaces authorization scope as a first-class concern rather than a buried comment.
Suggestions
- The three learning-objective bullets in the introduction are slightly implementation-focused ("Run a bootstrap-then-interview session…") rather than outcome-focused ("Produce a structured threat model from source code alone…"). Not blocking, but outcome framing reads better for readers skimming to decide whether the notebook is relevant to them.
- A
## Setupmarkdown cell before the%%capture %pip installcell would separate dependency installation from the conceptual steps, consistent with the pattern used in the other notebooks in this series.
Positive Notes
load_dotenv()is used correctly; no hardcoded API keys anywhere.claude-opus-4-7is the current latest Opus model — correct choice.- The registry entry format,
Cybersecuritycategory addition to the schema enum, andeugeneyan-antauthor entry all follow established patterns exactly. canary.cis well-designed as a teaching artifact: three distinct bug classes, each reachable via a one-byte dispatch, in 45 lines — clean and self-contained.- The multi-turn
ClaudeSDKClientsession for the threat model versus single-shotquery()calls for stateless steps is a textbook demonstration of when to use each pattern, and the prose in Step 3 makes the distinction explicit.
- Rebase onto current main (resolves DIRTY merge state from PRs anthropics#573, anthropics#595) - Re-execute notebook end-to-end against live XPOZ MCP + Claude Sonnet 4.6 (804 posts sampled across Twitter/Reddit/Instagram, structured analysis committed as cell outputs per CONTRIBUTING.md guidance) - ruff check + ruff format clean (sorted imports, formatted code) - authors.yaml sorted alphabetically (validate_authors_sorted.py passes) - registry.yaml description: 'analyse' -> 'analyze' (US English)
- Rebase onto current main (resolves DIRTY merge state from PRs anthropics#573, anthropics#595) - Re-execute notebook end-to-end against live XPOZ MCP + Claude Sonnet 4.6 (804 posts sampled across Twitter/Reddit/Instagram, structured analysis committed as cell outputs per CONTRIBUTING.md guidance) - ruff check + ruff format clean (sorted imports, formatted code) - authors.yaml sorted alphabetically (validate_authors_sorted.py passes) - registry.yaml description: 'analyse' -> 'analyze' (US English)
Ports the vulnerability detection agent cookbook from the private repo (anthropics/claude-cookbooks-private#59, authored by @eugeneyan-ant) to the public repo.
What's new
claude_agent_sdk/06_The_vulnerability_detection_agent.ipynbwalks through threat-model → find → triage → report on a self-contained 45-line canary C target usingclaude-agent-sdk:ClaudeSDKClientfor the multi-turn bootstrap-then-interview threat modelquery()for find/triage/reportRenumbered from
04_(private) to06_(public) —04_and05_are already taken in public by the OpenAI migration and session browser cookbooks.Supporting changes
vulnerability_detection_agent/canary/canary.c: three unlabeled memory-safety bugs (heap OOB, stack OOB, UAF)registry.yaml: new entry under Claude Agent SDK + Cybersecurity;threat_intel_enrichment_agentalso tagged Cybersecurityauthors.yaml: addeugeneyan-antpyproject.toml: +claude-agent-sdkso rootuv synccovers the series.gitignore: generatedTHREAT_MODEL.md.claude/commands/add-registry.md: add Cybersecurity to category list