Phase 3: LLM Client + Templates + Generator

[sabyasm]

Goal

Generate structured eval prompts from scraped content via LLM. Core creative engine.

Files to Create

• src/generator/__init__.py
• src/generator/llm_client.py — Unified LLMClient. Supports openai/anthropic/mistral/custom. generate(prompt, system_prompt) -> str. Retry: 3 attempts w/ exponential backoff via tenacity. Logs raw responses.
• src/generator/prompt_builder.py — Load templates, fill generation_instruction with chunk + target metadata, call LLM, parse JSON response, attach UUID/version/template_id. Handle linked prompts (H4, C2, P1-P5, P7).
• src/generator/generation_plan.py — Given config targets (dimension_split, domain_split, difficulty 20/40/40), compute how many prompts per (template, domain, difficulty). Output generation manifest.
• src/generator/templates/ — 20 YAML files (see template table below)
• tests/test_generator.py — test template loading, generation_plan math sums to ~2910, LLM client with mock

Template Table

Templates	Dimension	Notes
H1-H5	hallucination	H4 = 3 linked variants (self_contradiction)
C1-C3	consistency	C2 = linked pairs (position_coherence)
P1-P5	robustness	5 phrasing variants per core intent (linked)
P6	robustness	instruction_following
P7a, P7b	robustness	zero/few-shot pairs (linked)
P8	robustness	ambiguity_handling
S1-S5	safety	should_refuse, over-refusal, toxicity, bias, tone

Each template: id, dimension, subdimension, description, system_prompt, user_message_template, generation_instruction (w/ adversarial strategies for hard), 2-3 negative_examples.

Key Decisions

• Linked prompts: single LLM call per group (cheaper, more coherent variants)
• Negative examples: LLM-assisted drafts, then hand-reviewed for quality
• Hard prompts: generation_instruction explicitly names adversarial strategies from Section 9.1, requires adversarial_strategy + expected_failure_modes in LLM JSON response
• Raw LLM output saved to data/generated/raw_llm_responses/ before parsing

Verification

python -c "from src.generator.prompt_builder import PromptBuilder; ..."  # generate 5 H1 prompts
pytest tests/test_generator.py -v

Blocked by

• Phase 1 (schema)
• Phase 2 (scraped content needed for end-to-end generation)