Skip to content

Commit 9814f13

Browse files
bugerclaude
andauthored
fix: Prevent early returns with schema instructions (#292)
* fix: Prevent early returns when schema instructions are shown Fixes issue where AI returns empty results immediately instead of performing the requested analysis when schema is present in initial message. ## Problem PR #291 added schema instructions to initial message to prevent JSON validation loops. However, the example JSON (e.g., {"issues": []}) looked like a "completed" result, causing AI to return immediately without doing the work. ## Root Cause - Example showed empty arrays/default values - For code review schemas, {"issues": []} looks like "no issues found" - AI interpreted this as the expected immediate answer - Result: Empty results on iteration 1 with no analysis ## Solution 1. Changed example label to clarify it's a FORMAT placeholder: "Example:" → "Example format (populate with actual data...)" 2. Added explicit instruction to perform work first: "You should still perform the requested analysis/task thoroughly before providing the final JSON response." 3. Clarified schema purpose: "The schema defines the format for your answer, not a shortcut to return immediately." ## Impact - Preserves JSON validation loop fix from #291 - Prevents premature returns with empty results - AI now performs analysis before returning formatted JSON - No breaking changes, backward compatible 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove example JSON from schema instructions Following feedback, removed the example JSON entirely from schema instructions. The schema definition itself is sufficient to show the structure, and examples were causing confusion. ## Rationale The example JSON (e.g., {"issues": []}) was: - Looking like a "completed answer" - Adding unnecessary tokens to every message - Creating potential confusion between placeholder and expected result The JSON schema definition already clearly shows: - Field names and types - Required vs optional fields - Nested structure - Validation constraints This is sufficient for the AI to understand the format. ## Changes - Removed generateExampleFromSchema() call from generateSchemaInstructions() - Simplified instruction to: "First complete the requested analysis/task thoroughly, then provide your final answer in the JSON format above." - Cleaner, more direct, less ambiguous ## Benefits - Eliminates risk of examples looking like expected answers - Reduces token usage in initial messages - Simpler, clearer communication - Schema definition alone is sufficient 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove schema instructions from no-tool-call reminder Schema instructions should only appear once at the beginning of the conversation, not repeated in every "no tool call" reminder. ## Rationale The original PR #291 added schema instructions in TWO places: 1. Initial user message (line 1462) - Good, sets expectations upfront 2. "No tool call" reminder (line 2023) - Bad, unnecessary repetition This caused: - Redundant schema information in conversation - Extra tokens on every reminder - Potential confusion from repetition - Schema appearing in wrong context (when AI doesn't use tool) ## Solution Remove schema instructions from the "no tool call" reminder. The schema is already shown in the initial message, which is sufficient. **Before:** ```javascript if (options.schema) { reminderContent = `Please use tools...` + generateSchemaInstructions(options.schema, { debug: this.debug }); } ``` **After:** ```javascript const reminderContent = `Please use tools...`; // Schema was already shown in initial message ``` ## Flow Now 1. User asks question with schema requirement 2. Initial message shows schema instructions ONCE ✅ 3. AI works on the task 4. If AI doesn't use tool → standard reminder (no schema repetition) 5. When AI calls attempt_completion → validate against schema 6. If validation fails → correction prompt can remind about format ## Benefits - Cleaner conversation flow - Fewer tokens (schema not repeated in reminders) - Schema shown once at the right time (beginning) - Validation happens at the right time (when attempt_completion called) - Less noise in the conversation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: Remove internal analysis document This was just for internal analysis and shouldn't be in the PR --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 0afef33 commit 9814f13

2 files changed

Lines changed: 3 additions & 23 deletions

File tree

npm/src/agent/ProbeAgent.js

Lines changed: 2 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2009,21 +2009,8 @@ When troubleshooting:
20092009
// Add assistant response and ask for tool usage
20102010
currentMessages.push({ role: 'assistant', content: assistantResponseContent });
20112011

2012-
// Build appropriate reminder message based on whether schema is provided
2013-
let reminderContent;
2014-
if (options.schema) { // Apply for ANY schema, not just JSON schemas
2015-
// When schema is provided, use the same instructions as initial message
2016-
reminderContent = `Please use one of the available tools to help answer the question, or use attempt_completion if you have enough information to provide a final answer.
2017-
2018-
Remember: Use proper XML format with BOTH opening and closing tags:
2019-
2020-
<tool_name>
2021-
<parameter>value</parameter>
2022-
</tool_name>
2023-
` + generateSchemaInstructions(options.schema, { debug: this.debug }).trim();
2024-
} else {
2025-
// Standard reminder without schema
2026-
reminderContent = `Please use one of the available tools to help answer the question, or use attempt_completion if you have enough information to provide a final answer.
2012+
// Standard reminder - schema was already provided in initial message
2013+
const reminderContent = `Please use one of the available tools to help answer the question, or use attempt_completion if you have enough information to provide a final answer.
20272014
20282015
Remember: Use proper XML format with BOTH opening and closing tags:
20292016
@@ -2035,7 +2022,6 @@ Or for quick completion if your previous response was already correct and comple
20352022
<attempt_complete>
20362023
20372024
IMPORTANT: When using <attempt_complete>, this must be the ONLY content in your response. No additional text, explanations, or other content should be included. This tag signals to reuse your previous response as the final answer.`;
2038-
}
20392025

20402026
currentMessages.push({
20412027
role: 'user',

npm/src/agent/schemaUtils.js

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -63,20 +63,14 @@ export function generateSchemaInstructions(schema, options = {}) {
6363
try {
6464
const parsedSchema = typeof schema === 'string' ? JSON.parse(schema) : schema;
6565
instructions += `${JSON.stringify(parsedSchema, null, 2)}\n\n`;
66-
67-
// Generate example using helper function
68-
const exampleObj = generateExampleFromSchema(parsedSchema, { debug });
69-
if (exampleObj) {
70-
instructions += `Example:\n<attempt_completion>\n${JSON.stringify(exampleObj, null, 2)}\n</attempt_completion>\n\n`;
71-
}
7266
} catch (e) {
7367
if (debug) {
7468
console.error('[DEBUG] generateSchemaInstructions: Failed to parse schema:', e.message);
7569
}
7670
instructions += `${schema}\n\n`;
7771
}
7872

79-
instructions += 'Your response inside attempt_completion must be ONLY valid JSON - no plain text, no explanations, no markdown.';
73+
instructions += 'Your response inside attempt_completion must be ONLY valid JSON - no plain text, no explanations, no markdown.\n\nIMPORTANT: First complete the requested analysis/task thoroughly, then provide your final answer in the JSON format above.';
8074

8175
return instructions;
8276
}

0 commit comments

Comments
 (0)