JSON-PROMPTING-SUMMARY.md

JSON Prompting Migration Summary

What Changed

All three agent implementations have been migrated to use JSON prompting for structured reasoning and responses.

Key Improvements

1. Structured Reasoning

• Agents now explicitly document their thinking process
• Reasoning is separated from final output
• Transparent decision-making

2. Consistent Response Format

• All agents follow predictable schemas
• Easy to parse and validate
• Better for downstream processing

3. Enhanced Metadata

• Confidence scores
• Source tracking
• Completeness indicators

4. Better Formatting

• JSON responses are automatically formatted into readable text
• Structured sections (headline, key points, insights, conclusion)
• Consistent presentation across all agents

Files Modified

1. multi-agent.ts

Changes:

• Added Zod schemas for ResearcherResponseSchema and SummarizerResponseSchema
• Updated createAgent() to accept optional responseSchema parameter
• Implemented strict JSON schema enforcement via OpenAI's response_format
• Added formatSummaryFromJSON() helper function
• Enhanced logging to display reasoning sections
• Modified generateNews() to parse and format JSON responses

Schemas:

• Researcher: reasoning (query_analysis, search_strategy, tool_selection), action, metadata
• Summarizer: reasoning (information_assessment, key_themes, synthesis_approach), summary, metadata

2. cust-agent.ts

Changes:

• Added NewsAnalysisSchema with Zod
• Implemented conditional JSON mode (only after tool results)
• Added formatAnalysisFromJSON() helper function
• Modified callModel() to switch between tool mode and JSON mode
• Updated generateNews() to parse and format JSON responses

Schema:

• reasoning (query_understanding, information_gaps, search_plan)
• analysis (headline, summary, key_developments, context, outlook)
• metadata (sources_consulted, confidence_level)

3. simple-agent.js

Changes:

• Updated system message to include JSON schema structure
• Added formatReportFromJSON() helper function
• Modified generateNews() to parse and format JSON responses
• Uses prompt-based JSON enforcement (not strict schema)

Schema (prompt-based):

• reasoning (query_analysis, sources_evaluated, synthesis_approach)
• report (headline, summary, key_points, insights, conclusion)
• metadata (sources_count, confidence)

4. package.json

Added dependencies:

• zod: ^3.22.4
• zod-to-json-schema: ^3.22.4

5. New Files Created

• JSON-PROMPTING-GUIDE.md: Comprehensive guide to JSON prompting
• JSON-PROMPTING-SUMMARY.md: This summary document

How It Works

For TypeScript Agents (multi-agent.ts, cust-agent.ts)

1. Define Schema with Zod:

const MySchema = z.object({
  reasoning: z.object({...}),
  output: z.object({...}),
  metadata: z.object({...})
})

2. Enforce with OpenAI API:

model.bind({
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "agent_response",
      strict: true,
      schema: zodToJsonSchema(MySchema)
    }
  }
})

3. Parse and Format:

const jsonResponse = JSON.parse(response.content);
const formatted = formatFromJSON(jsonResponse.output);

For JavaScript Agent (simple-agent.js)

1. Include Schema in Prompt:

systemMessage: `You MUST respond in JSON format:
{
  "reasoning": {...},
  "report": {...},
  "metadata": {...}
}`

2. Parse Response:

const jsonResponse = JSON.parse(response.content);
const formatted = formatReportFromJSON(jsonResponse.report);

Example JSON Response

Researcher Agent

{
  "reasoning": {
    "query_analysis": "User wants current news about AI developments",
    "search_strategy": "Search for recent AI news from 2025",
    "tool_selection": "Using SerpAPI for comprehensive web results"
  },
  "action": {
    "type": "use_tool",
    "tool_name": "serpapi"
  },
  "metadata": {
    "confidence": 0.9,
    "sources_needed": true
  }
}

Summarizer Agent

{
  "reasoning": {
    "information_assessment": "Gathered 10+ sources about AI developments",
    "key_themes": ["LLM advances", "Regulation", "Industry adoption"],
    "synthesis_approach": "Organize by theme, highlight trends"
  },
  "summary": {
    "headline": "AI Industry Sees Major Advances in 2025",
    "overview": "The AI landscape has transformed significantly...",
    "key_points": [
      {
        "point": "GPT-5 Release",
        "details": "OpenAI launched GPT-5 with improved reasoning..."
      }
    ],
    "insights": "These developments indicate a maturation...",
    "conclusion": "The AI industry is entering a new phase..."
  },
  "metadata": {
    "completeness": 0.95,
    "topic_coverage": ["technology", "business", "regulation"]
  }
}

Benefits Realized

1. Transparency

• Can see exactly how agents are thinking
• Reasoning is logged and traceable
• Easier to debug issues

2. Consistency

• All responses follow same structure
• Predictable output format
• Easier to test and validate

3. Quality

• Forced step-by-step thinking improves accuracy
• Confidence scores help identify uncertain responses
• Metadata enables better decision-making

4. Maintainability

• Schemas document expected behavior
• Type safety with Zod/TypeScript
• Easy to extend with new fields

Testing

To test the JSON prompting:

1. Install dependencies:

npm install

2. Build TypeScript:

npm run build

3. Run an agent:

node dist/multi-agent.js
# or
node dist/cust-agent.js
# or
node simple-agent.js

4. Check console logs:

• Look for "🧠 Reasoning:" logs showing structured thinking
• Look for "📄 Formatted [summary/analysis/report] generated" confirmations
• Verify JSON structure in output

Next Steps

Recommended Enhancements

1. Add Validation:

const result = MySchema.safeParse(jsonResponse);
if (!result.success) {
  console.error('Validation failed:', result.error);
}

2. Track Metrics:

• Log confidence scores over time
• Track reasoning quality
• Monitor schema compliance

3. A/B Testing:

• Compare JSON vs non-JSON responses
• Measure quality improvements
• Optimize schemas based on results

4. Schema Evolution:

• Version schemas for compatibility
• Add new fields as needed
• Document schema changes

Troubleshooting

Issue: Agent not returning JSON

Solution: Check that:

• Model supports structured outputs (GPT-4, GPT-4o)
• Schema is valid (test with zodToJsonSchema())
• System prompt includes JSON requirement

Issue: JSON parsing fails

Solution:

• Add try-catch around JSON.parse()
• Log raw response for debugging
• Verify response_format is correctly set

Issue: Schema validation errors

Solution:

• Check Zod schema definitions
• Ensure all required fields are present
• Use .optional() for optional fields

Resources

• See JSON-PROMPTING-GUIDE.md for detailed documentation
• Check OpenAI docs for structured outputs
• Review Zod documentation for schema design

Questions?

For issues or questions about the JSON prompting implementation:

1. Review the guide: JSON-PROMPTING-GUIDE.md
2. Check console logs for reasoning output
3. Verify schema definitions in each file
4. Test with simple queries first