-
Notifications
You must be signed in to change notification settings - Fork 0
feat(enricher): enriched context, rewritten templates, custom prompts #64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 16 commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
c032f71
feat(enricher): add enriched context fields to FieldContext, ParamFie…
spencercjh 8c2a917
feat(enricher): populate enriched context (format, enum, constraints)…
spencercjh fe4d0ec
feat(enricher): populate enriched context (tags, format, enum, constr…
spencercjh c0daed4
feat(enricher): rewrite built-in prompts with type-specific system pr…
spencercjh 27f943d
feat(enricher): add custom prompt configuration and wire through enri…
spencercjh d26aa7d
chore: fix lint issues and add custom prompts integration test
spencercjh 9dfba58
docs: add P5 prompt optimization design and implementation plan
spencercjh 1f8db91
refactor(enricher): extract shared BuildConstraintsString and BuildEn…
spencercjh 288deda
fix(enricher): warn on invalid custom prompt type keys
spencercjh fdf9c39
feat(enricher): validate custom prompt templates at startup
spencercjh b25b564
refactor(cmd): deduplicate custom prompt mapping with shared helper
spencercjh cd16e54
fix(enricher): remove format from FieldType to avoid duplicate in prompt
spencercjh 3b12f39
fix(enricher): populate ExistingDescription for parameter field items
spencercjh d954451
docs: update implementation plan to reflect shared helper reuse and r…
spencercjh b5ed3e4
fix(enricher): merge custom prompts with built-in templates and rejec…
spencercjh 556716c
refactor(enricher): extract applyCustomPrompts to reduce Enrich cyclo…
spencercjh bdc1da2
fix(enricher): remove unsafe quoting of ExistingDescription in templa…
spencercjh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| # P5 Prompt Optimization Design | ||
|
|
||
| > **Status:** Implemented | ||
| > **Date:** 2026-03-31 | ||
| > **Parent Issue:** #40 (Phase 4 - LangchainGo Features) | ||
|
|
||
| ## Overview | ||
|
|
||
| Improve LLM enrichment output quality by enriching context passing, rewriting built-in prompt templates, and adding custom prompt file support. | ||
|
|
||
| ## Current State | ||
|
|
||
| | Feature | Status | Detail | | ||
| |-------------------|---------------|----------------------------------------------------------| | ||
| | Context passing | Minimal | Only name, type, required passed to templates | | ||
| | System prompts | Generic | All 4 types use "You are an API documentation expert" | | ||
| | Few-shot examples | None | Templates contain no input/output examples | | ||
| | Constraints/enums | Ignored | OpenAPI format, enum, min/max, pattern not passed to LLM | | ||
| | API tags | Ignored | Operation tags not included in context | | ||
| | Custom prompts | Not supported | No way to override built-in templates | | ||
|
|
||
| ## Design | ||
|
|
||
| ### Optimization 1: Enriched Context Passing | ||
|
|
||
| **Problem:** Templates receive only `Name`, `Type`, `Required` for fields/params. The OpenAPI spec contains much richer metadata that would help LLMs generate more precise descriptions. | ||
|
|
||
| **Solution:** Pass additional spec metadata to templates: | ||
|
|
||
| **FieldContext additions:** | ||
| - `Format` — e.g., `"email"`, `"date-time"`, `"uuid"` | ||
| - `Enum` — allowed values, e.g., `["active", "inactive"]` | ||
| - `Constraints` — human-readable string: `"min: 0, max: 100, pattern: ^[a-z]+$"` | ||
| - `ExistingDescription` — existing description from spec (useful in `--force` mode) | ||
|
|
||
| **ParamFieldContext additions:** Same as FieldContext. | ||
|
|
||
| **TemplateContext additions (API-specific):** | ||
| - `Tags` — operation tags from the spec | ||
| - `ExistingSummary` / `ExistingDescription` — existing partial documentation | ||
|
|
||
| **Example impact on Schema prompt:** | ||
|
|
||
| ``` | ||
| Before: | ||
| - email (string, required) | ||
|
|
||
| After: | ||
| - email (string, required, format: email, maxLength: 255) | ||
| - role (string, optional, enum: [admin, user, guest]) | ||
| ``` | ||
|
|
||
| ### Optimization 2: Improved Built-in Prompts | ||
|
|
||
| **Problem:** Generic system prompts produce generic descriptions. No examples, no quality guidelines, no output constraints. | ||
|
|
||
| **Solution:** Type-specific system prompts with: | ||
|
|
||
| 1. **Role definition** — Different expert roles per type (API writer, data modeler, parameter documenter) | ||
| 2. **Quality guidelines** — Specific rules per type (e.g., "Summary starts with a verb", "Avoid repeating field name") | ||
| 3. **Few-shot examples** — Input/output pairs showing expected quality | ||
| 4. **Explicit output format** — JSON schema with constraints | ||
|
|
||
| **API Template (before):** | ||
| ``` | ||
| System: You are an API documentation expert. Generate concise, clear descriptions. | ||
| Respond in {{.Language}} language. | ||
| Output format: JSON with "summary" and "description" fields. | ||
|
|
||
| User: API Endpoint: {{.Path}} | ||
| HTTP Method: {{.Method}} | ||
|
|
||
| Generate the summary (one line) and description (1-3 sentences) for this API. | ||
| ``` | ||
|
|
||
| **API Template (after):** | ||
| ``` | ||
| System: You are an expert OpenAPI documentation writer specializing in REST API descriptions. | ||
| Your task is to write clear, concise, and informative API summaries and descriptions. | ||
|
|
||
| Guidelines: | ||
| - Summary: A single line (max 80 chars) starting with a verb (e.g., "List", "Create", "Delete") | ||
| - Description: 1-3 sentences explaining what the endpoint does, when to use it, and notable behavior | ||
| - Be specific: mention resource names, ID formats, and key constraints | ||
| - Avoid generic phrases like "This API is used for..." | ||
|
|
||
| Respond in {{.Language}} language. | ||
| Output MUST be valid JSON: {"summary": "...", "description": "..."} | ||
|
|
||
| Example input: | ||
| POST /users | ||
| Example output: | ||
| {"summary": "Create a new user", "description": "Registers a new user account..."} | ||
|
|
||
| User: API Endpoint: {{.Method}} {{.Path}} | ||
| {{- if .Tags}} | ||
| Tags: {{join .Tags ", "}} | ||
| {{- end}} | ||
| ... | ||
| ``` | ||
|
|
||
| ### Optimization 3: Custom Prompt File Support | ||
|
|
||
| **Problem:** Users cannot customize prompts for their domain without modifying source code. | ||
|
|
||
| **Solution:** Add `customPrompts` section to `.spec-forge.yaml`: | ||
|
|
||
| ```yaml | ||
| enrich: | ||
| customPrompts: | ||
| api: | ||
| system: "You are a Chinese API documentation writer..." | ||
| user: "API: {{.Method}} {{.Path}}\n用中文描述这个接口。" | ||
| schema: | ||
| system: "You are a data model expert..." | ||
| ``` | ||
|
|
||
| **Implementation:** Config loads custom prompts via Viper, passes through `enricher.Config.CustomPrompts`, and applies via `TemplateManager.Set()`. | ||
|
|
||
| ## Architecture | ||
|
|
||
| ``` | ||
| ┌─────────────────────┐ | ||
| │ .spec-forge.yaml │ | ||
| │ customPrompts: │ | ||
| │ api/system/user │ | ||
| └──────────┬──────────┘ | ||
| │ | ||
| ▼ | ||
| ┌──────────────┐ ┌─────────────────────┐ ┌──────────────┐ | ||
| │ OpenAPI │───▶│ Collection Layer │───▶│ Template │ | ||
| │ Spec │ │ (enricher.go) │ │ Manager │ | ||
| │ │ │ - format │ │ │ | ||
| │ - format │ │ - enum │ │ Built-in │ | ||
| │ - enum │ │ - constraints │ │ + Custom │ | ||
| │ - min/max │ │ - tags │ │ overrides │ | ||
| │ - pattern │ │ - existing desc │ │ │ | ||
| │ - tags │ └─────────────────────┘ └──────┬───────┘ | ||
| └──────────────┘ │ | ||
| ▼ | ||
| ┌──────────────────┐ | ||
| │ LLM Provider │ | ||
| │ (OpenAI/etc.) │ | ||
| └──────────────────┘ | ||
| ``` | ||
|
|
||
| ## Key Decisions | ||
|
|
||
| 1. **Backward compatible** — Output format stays the same (`{"summary": "...", "description": "..."}` for API, `{"field": "desc"}` for schema/param). No response parsing changes. | ||
|
|
||
| 2. **Constraint helper reuse** — `enricher.go` reuses `processor.BuildConstraintsString` and `processor.BuildEnumStrings` to keep constraint/enum formatting logic centralized in the `processor` package, avoiding duplicated helpers. | ||
|
|
||
| 3. **ExistingDescription in templates** — Only visible in `--force` mode (fields with existing descriptions are skipped otherwise). When force is on, the LLM can improve or translate existing descriptions. | ||
|
|
||
| 4. **Template FuncMap** — Added `join` function (maps to `strings.Join`) for rendering enum/tag lists. Registered in `renderString` via `template.FuncMap`. | ||
|
|
||
| 5. **Config key mapping** — Custom prompt keys (`"api"`, `"schema"`, `"param"`, `"response"`) directly match `TemplateType` string constants. |
152 changes: 152 additions & 0 deletions
152
docs/plans/2026-03-31-p5-prompt-optimization-implementation.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,152 @@ | ||
| # P5 Prompt Optimization Implementation Plan | ||
|
|
||
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. | ||
|
|
||
| **Goal:** Improve LLM enrichment quality by enriching context passing, rewriting built-in prompts with few-shot examples, and adding custom prompt file support. | ||
|
|
||
| **Architecture:** Three-layer improvement: (1) pass more OpenAPI spec metadata (format, enum, constraints, tags) to templates, (2) rewrite templates with type-specific system prompts and examples, (3) allow users to override prompts via `.spec-forge.yaml`. Output format stays the same — no response parsing changes needed. | ||
|
|
||
| **Tech Stack:** Go 1.26, text/template, Viper config, kin-openapi | ||
|
|
||
| --- | ||
|
|
||
| ## File Structure | ||
|
|
||
| | File | Responsibility | | ||
| |----------------------------------------------|-------------------------------------------------------------| | ||
| | `internal/enricher/prompt/templates.go` | Context types, built-in templates, FuncMap, TemplateManager | | ||
| | `internal/enricher/prompt/templates_test.go` | Template rendering tests | | ||
| | `internal/enricher/processor/schema.go` | Schema field collection with enriched metadata | | ||
| | `internal/enricher/processor/processor.go` | FieldElement/ParamFieldItem types, conversion helpers | | ||
| | `internal/enricher/enricher.go` | Parameter collection, API context, custom prompt wiring | | ||
| | `internal/enricher/config.go` | CustomPrompts field on enricher Config | | ||
| | `internal/config/config.go` | CustomPrompts on EnrichConfig | | ||
| | `cmd/enrich.go` | Wire custom prompts from config to enricher | | ||
| | `cmd/generate.go` | Wire custom prompts in generate pipeline | | ||
| | `.spec-forge.example.yaml` | Document new config option | | ||
|
|
||
| --- | ||
|
|
||
| ### Task 1: Enrich FieldContext, ParamFieldContext, and TemplateContext types | ||
|
|
||
| **Files:** | ||
| - Modify: `internal/enricher/prompt/templates.go` | ||
| - Modify: `internal/enricher/prompt/templates_test.go` | ||
|
|
||
| - [x] Add enriched fields to `FieldContext` (Format, Enum, Constraints, ExistingDescription) | ||
| - [x] Add enriched fields to `ParamFieldContext` (Format, Enum, Constraints, ExistingDescription) | ||
| - [x] Add Tags, ExistingSummary, ExistingDescription to `TemplateContext` | ||
| - [x] Add `join` func to `renderString` FuncMap | ||
| - [x] Add tests for enriched field rendering | ||
| - [x] Commit: `feat(enricher): add enriched context fields to FieldContext, ParamFieldContext, and TemplateContext` | ||
|
|
||
| --- | ||
|
|
||
| ### Task 2: Populate enriched context in schema field collection | ||
|
|
||
| **Files:** | ||
| - Modify: `internal/enricher/processor/processor.go` | ||
| - Modify: `internal/enricher/processor/schema.go` | ||
| - Test: `internal/enricher/processor/schema_test.go` | ||
|
|
||
| - [x] Add enriched fields to `FieldElement` and `ParamFieldItem` | ||
| - [x] Update `convertFieldElements` and `convertParamFieldItems` to propagate enriched fields | ||
| - [x] Add `buildConstraintsString` and `buildEnumStrings` helpers to schema.go | ||
| - [x] Update `CollectSchemaFields` to populate Format, Enum, Constraints, ExistingDescription | ||
| - [x] Add `TestCollectSchemaFields_EnrichedContext` | ||
| - [x] Commit: `feat(enricher): populate enriched context (format, enum, constraints) in schema field collection` | ||
|
|
||
| --- | ||
|
|
||
| ### Task 3: Populate enriched context in parameter and API collection | ||
|
|
||
| **Files:** | ||
| - Modify: `internal/enricher/enricher.go` | ||
| - Modify: `internal/enricher/enricher_test.go` | ||
|
|
||
| - [x] Reuse shared `processor.BuildConstraintsString` and `processor.BuildEnumStrings` helpers for parameter constraint/enum extraction | ||
| - [x] Update `collectParameterGroups` to extract format, enum, constraints from param schemas | ||
| - [x] Update `collectElements` to pass Tags, ExistingSummary, ExistingDescription for API operations | ||
| - [x] Add `TestEnricher_CollectParameters_EnrichedContext` and `TestEnricher_CollectElements_APITags` | ||
| - [x] Commit: `feat(enricher): populate enriched context (tags, format, enum, constraints) in parameter and API collection` | ||
|
|
||
| --- | ||
|
|
||
| ### Task 4: Rewrite built-in prompt templates | ||
|
|
||
| **Files:** | ||
| - Modify: `internal/enricher/prompt/templates.go` | ||
| - Modify: `internal/enricher/prompt/templates_test.go` | ||
|
|
||
| - [x] Replace all 4 templates with type-specific system prompts, few-shot examples, quality guidelines | ||
| - [x] API template: verb-led summaries, specificity guidelines, tags/existing desc support | ||
| - [x] Schema template: constraint-aware descriptions, enum explanation guidance | ||
| - [x] Param template: location context, enum guidance, format hints | ||
| - [x] Response template: error cause guidance, success content hints | ||
| - [x] Add `TestNewTemplateManager_RendersAllTypesWithEnrichedContext` and `TestNewTemplateManager_APITemplateUsesTags` | ||
| - [x] Commit: `feat(enricher): rewrite built-in prompts with type-specific system prompts, few-shot examples, and enriched context` | ||
|
|
||
| --- | ||
|
|
||
| ### Task 5: Add custom prompt config | ||
|
|
||
| **Files:** | ||
| - Modify: `internal/config/config.go` | ||
| - Modify: `internal/enricher/config.go` | ||
| - Modify: `.spec-forge.example.yaml` | ||
|
|
||
| - [x] Add `CustomPrompts map[string]CustomPromptCfg` to `config.EnrichConfig` | ||
| - [x] Add `CustomPromptConfig` type and `CustomPrompts` field to `enricher.Config` | ||
| - [x] Update `.spec-forge.example.yaml` with commented customPrompts section | ||
|
|
||
| --- | ||
|
|
||
| ### Task 6: Wire custom prompts through enricher pipeline | ||
|
|
||
| **Files:** | ||
| - Modify: `internal/enricher/enricher.go` | ||
| - Modify: `cmd/enrich.go` | ||
| - Modify: `cmd/generate.go` | ||
|
|
||
| - [x] Apply custom prompts via `TemplateManager.Set()` in `Enrich()` method | ||
| - [x] Map custom prompts from config in `cmd/enrich.go` | ||
| - [x] Map custom prompts from config in `cmd/generate.go` | ||
| - [x] Commit: `feat(enricher): add custom prompt configuration and wire through enricher pipeline` | ||
|
|
||
| --- | ||
|
|
||
| ### Task 7: Integration test and verification | ||
|
|
||
| **Files:** | ||
| - Modify: `internal/enricher/enricher_test.go` | ||
|
|
||
| - [x] Add `TestEnricher_CustomPrompts` with `trackingMockProvider` | ||
| - [x] Fix lint issues (perfsprint, gocritic rangeValCopy) | ||
| - [x] `make fmt`, `make lint` (0 issues), `make test` (all pass) | ||
| - [x] Commit: `chore: fix lint issues and add custom prompts integration test` | ||
|
|
||
| --- | ||
|
|
||
| ## Verification | ||
|
|
||
| ```bash | ||
| # Build | ||
| go build -o ./build/spec-forge . | ||
|
|
||
| # Test with real LLM | ||
| LLM_API_KEY="your-key" ./build/spec-forge enrich \ | ||
| ./integration-tests/maven-springboot-openapi-demo/target/openapi.json \ | ||
| --provider custom --model deepseek-chat \ | ||
| --custom-base-url https://api.deepseek.com/v1 \ | ||
| --language zh -v | ||
|
|
||
| # Test custom prompts via config | ||
| # Add to .spec-forge.yaml: | ||
| # enrich: | ||
| # customPrompts: | ||
| # api: | ||
| # system: "You are a Chinese API writer..." | ||
| # user: "Endpoint: {{.Method}} {{.Path}}\nWrite summary+description in JSON." | ||
| ``` | ||
|
|
||
| Expected: Descriptions leverage format/enum/constraint context for more specific output. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.