fix: Fix JSON parsing to handle single quotes and prevent incorrect extraction

[buger]

Summary

This PR fixes two related issues in JSON validation that were causing attempt_completion results to fail with "Unexpected token '''" errors:

1. Single-quote normalization: AI responses sometimes return JavaScript syntax with single quotes instead of valid JSON
2. Incorrect fragment extraction: Single-backtick pattern was extracting inline code from within JSON objects

Problem

When AI returned attempt_completion with results like:

{
  "text": "When `allowedTools` contains `['*']`, all tools are enabled",
  "intent": "comment_reply"
}

The system would incorrectly extract ['*'] (14 chars) from the inline code instead of preserving the full JSON object (1060+ chars), then fail validation with:

Unexpected token ''', "['*']" is not valid JSON

Solution

1. Added normalizeJsonQuotes() Function

Converts JavaScript array/object syntax to valid JSON by replacing single quotes with double quotes:

• ['*'] → ["*"]
• {'key': 'value'} → {"key": "value"}

2. Fixed Single-Backtick Pattern

Changed from /\([{[][\s\S]?[}]])`/to/^`([{[][\s\S]?[}]])`$/` with anchors to:

• ✅ Extract when entire input is `{"test": "value"}`
• ❌ Prevent extraction from {"text": "... `['*']` ..."}

Testing

New Test Suite

Added single-quote-json-bug.test.js with 17 comprehensive tests:

• Single-quote array/object normalization
• Code block extraction with normalization
• Real-world bug scenarios from error logs
• Edge cases with mixed quotes

Test Results

✅ New tests: 17/17 passed
✅ Existing schemaUtils tests: 96/96 passed
✅ Overall: 1114/1116 passed (2 unrelated failures)

Impact

• Fixes "Unexpected token '''" errors in attempt_completion validation
• Preserves full JSON objects instead of extracting embedded fragments
• Backward compatible - all existing tests pass

Files Changed

• npm/src/agent/schemaUtils.js: Added normalization and fixed extraction
• npm/tests/unit/single-quote-json-bug.test.js: Comprehensive test coverage

🤖 Generated with Claude Code

[probelabs]

🔍 Code Analysis Results

['*']

🐛 Debug Information

Provider: anthropic
Model: glm-4.6
API Key Source: ANTHROPIC_API_KEY
Processing Time: 152986ms
Timestamp: 2025-11-16T17:32:42.723Z
Prompt Length: 36944 characters
Response Length: 10 characters
JSON Parse Success: ❌

AI Prompt

[overview]
<instructions>
PR Title: fix: Fix JSON parsing to handle single quotes and prevent incorrect extraction

You are generating PR overview, to help owners of the repository to understand what this PR is above, and help reviewer to point to the right parts of the code. First you should provide detailed but concise description, mentioning all the changes.

## Files Changed Analysis
After you need to summarize insights from `<files_summary>`: changed files, additions/deletions, notable patterns.

Next ensure you cover all below:

## Architecture & Impact Assessment
  - What this PR accomplishes
  - Key technical changes introduced
  - Affected system components
  - Include one or more mermaid diagrams when useful to visualize component relationships or flow.
  
## Scope Discovery & Context Expansion
- From the `<files_summary>` and code diffs, infer the broader scope of impact across modules, services, and boundaries.
- If your environment supports code search/extract tools, use them to peek at immediately-related files (tests, configs, entrypoints) for better context. If tools are not available, infer and list what you would search next.

You may also be asked to assign labels to PR; if so use this:
- `tags.review-effort`: integer 1–5 estimating review effort (1=trivial, 5=very high).
- `tags.label`: one of [bug, chore, documentation, enhancement, feature]. Choose the best fit.

Important:
- Propose `tags.review-effort` and `tags.label` only for the initial PR open event.
- Do not change or re-suggest labels on PR update events; the repository applies labels only on `pr_opened`.

Be concise, specific, and actionable. Avoid praise or celebration.

</instructions>

<context>
<pull_request>
  <!-- Core pull request metadata including identification, branches, and change statistics -->
  <metadata>
    <number>296</number>
    <title>fix: Fix JSON parsing to handle single quotes and prevent incorrect extraction</title>
    <author>buger</author>
    <base_branch>main</base_branch>
    <target_branch>json-parse-single-quotes</target_branch>
    <total_additions>314</total_additions>
    <total_deletions>7</total_deletions>
    <files_changed_count>2</files_changed_count>
  </metadata>
  <raw_diff_header>
diff --git a/npm/src/agent/schemaUtils.js b/npm/src/agent/schemaUtils.js
  </raw_diff_header>
  <!-- Full pull request description provided by the author -->
  <description>
## Summary

This PR fixes two related issues in JSON validation that were causing `attempt_completion` results to fail with "Unexpected token '''" errors:

1. **Single-quote normalization**: AI responses sometimes return JavaScript syntax with single quotes instead of valid JSON
2. **Incorrect fragment extraction**: Single-backtick pattern was extracting inline code from within JSON objects

## Problem

When AI returned `attempt_completion` with results like:
```json
{
  "text": "When `allowedTools` contains `['*']`, all tools are enabled",
  "intent": "comment_reply"
}

The system would incorrectly extract ['*'] (14 chars) from the inline code instead of preserving the full JSON object (1060+ chars), then fail validation with:

Unexpected token ''', "['*']" is not valid JSON

Solution

1. Added normalizeJsonQuotes() Function

Converts JavaScript array/object syntax to valid JSON by replacing single quotes with double quotes:

• ['*'] → ["*"]
• {'key': 'value'} → {"key": "value"}

2. Fixed Single-Backtick Pattern

Changed from /\([{[][\s\S]?[}]])`/to/^`([{[][\s\S]?[}]])`$/` with anchors to:

• ✅ Extract when entire input is `{"test": "value"}`
• ❌ Prevent extraction from {"text": "... `['*']` ..."}

Testing

New Test Suite

Added single-quote-json-bug.test.js with 17 comprehensive tests:

• Single-quote array/object normalization
• Code block extraction with normalization
• Real-world bug scenarios from error logs
• Edge cases with mixed quotes

Test Results

✅ New tests: 17/17 passed
✅ Existing schemaUtils tests: 96/96 passed
✅ Overall: 1114/1116 passed (2 unrelated failures)

Impact

• Fixes "Unexpected token '''" errors in attempt_completion validation
• Preserves full JSON objects instead of extracting embedded fragments
• Backward compatible - all existing tests pass

Files Changed

• npm/src/agent/schemaUtils.js: Added normalization and fixed extraction
• npm/tests/unit/single-quote-json-bug.test.js: Comprehensive test coverage

🤖 Generated with Claude Code

<full_diff>
--- npm/src/agent/schemaUtils.js
@@ -165,6 +165,74 @@ export function decodeHtmlEntities(text) {
return decoded;
}

+/**

• • Normalize JavaScript syntax to valid JSON syntax
• • Converts single quotes to double quotes for strings in JSON-like structures
• • @param {string} str - String that might contain JavaScript array/object syntax
• • @returns {string} - String with single quotes normalized to double quotes
• */
  +function normalizeJsonQuotes(str) {
• if (!str || typeof str !== 'string') {
• return str;
• }
• // Quick check: if there are no single quotes, no need to normalize
• if (!str.includes("'")) {
• return str;
• }
• let result = '';
• let inDoubleQuote = false;
• let inSingleQuote = false;
• let escaped = false;
• for (let i = 0; i < str.length; i++) {
• const char = str[i];
• const prevChar = i > 0 ? str[i - 1] : '';
• // Handle escape sequences
• if (escaped) {

•  result += char;
  

•  escaped = false;
  

•  continue;
  

• }
• if (char === '\') {

•  escaped = true;
  

•  result += char;
  

•  continue;
  

• }
• // Track when we're inside double-quoted strings
• if (char === '"' && !inSingleQuote) {

•  inDoubleQuote = !inDoubleQuote;
  

•  result += char;
  

•  continue;
  

• }
• // Convert single quotes to double quotes (when not inside double quotes)
• if (char === "'" && !inDoubleQuote) {

•  // Check if this is a single quote inside a string value (like "It's")
  

•  // If we're already in a single-quoted string, toggle the state
  

•  if (inSingleQuote) {
  

•    // Closing single quote - convert to double quote
  

•    result += '"';
  

•    inSingleQuote = false;
  

•  } else {
  

•    // Opening single quote - convert to double quote
  

•    result += '"';
  

•    inSingleQuote = true;
  

•  }
  

•  continue;
  

• }
• result += char;
• }
• return result;
  +}

/**

• Clean AI response by extracting JSON content when response contains JSON
• Only processes responses that contain JSON structures { or [
  @@ -189,29 +257,36 @@ export function cleanSchemaResponse(response) {
  // Try with json language specifier
  const jsonBlockMatch = trimmed.match(/json\s*\n([\s\S]*?)\n/);
  if (jsonBlockMatch) {

• return jsonBlockMatch[1].trim();

• return normalizeJsonQuotes(jsonBlockMatch[1].trim());
  }

// Try any code block with JSON content
const anyBlockMatch = trimmed.match(/\s*\n([{\[][\s\S]*?[}\]])\s*/);
if (anyBlockMatch) {

• return anyBlockMatch[1].trim();

• return normalizeJsonQuotes(anyBlockMatch[1].trim());
  }

// Legacy patterns for more specific matching
const codeBlockPatterns = [
/json\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/,

• /\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/,
• /([{\[][\s\S]*?[}\]])/

• /\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/
  ];

for (const pattern of codeBlockPatterns) {
const match = trimmed.match(pattern);
if (match) {

•  return match[1].trim();
  

•  return normalizeJsonQuotes(match[1].trim());
  

  }
  }

• // Single backtick pattern - ONLY if the entire input is just the code block

• // This prevents extracting inline code from within JSON objects (e.g., ['*'] from markdown text)

• const singleBacktickPattern = /^([{\[][\s\S]*?[}\]])$/;

• const singleBacktickMatch = trimmed.match(singleBacktickPattern);

• if (singleBacktickMatch) {

• return normalizeJsonQuotes(singleBacktickMatch[1].trim());

• }

• // Look for code block start followed immediately by JSON
  const codeBlockStartPattern = /```(?:json)?\s*\n?\s*([{[])/;
  const codeBlockMatch = trimmed.match(codeBlockStartPattern);
  @@ -236,7 +311,7 @@ export function cleanSchemaResponse(response) {
  }

  if (bracketCount === 0) {

•  return trimmed.substring(startIndex, endIndex);
  

•  return normalizeJsonQuotes(trimmed.substring(startIndex, endIndex));
  

  }
  }

@@ -261,7 +336,8 @@ export function cleanSchemaResponse(response) {
const isJsonArray = firstChar === '[' && lastChar === ']';

if (isJsonObject || isJsonArray) {

• return cleaned;

• // Normalize JavaScript syntax (single quotes) to valid JSON syntax (double quotes)
• return normalizeJsonQuotes(cleaned);
  }

return response; // Return original if no extractable JSON found

--- npm/tests/unit/single-quote-json-bug.test.js
@@ -0,0 +1,231 @@
+/**

• • Test for single-quote JSON bug
• • Bug Description:
• • When AI responses contain JavaScript array syntax with single quotes like ['*']
• • instead of valid JSON with double quotes ["*"], the JSON parser fails with:
• • "Unexpected token ''', "['*']" is not valid JSON"
• • This test replicates the bug seen in the debug logs where:
• • 1. AI returns response with JavaScript array syntax: ['*', '!bash']
• • 2. cleanSchemaResponse extracts it as-is (no syntax normalization)
• • 3. JSON.parse fails because single quotes are invalid in JSON
• */

+import { describe, test, expect } from '@jest/globals';
+import { cleanSchemaResponse, validateJsonResponse } from '../../src/agent/schemaUtils.js';
+
+describe('Single-quote JSON Bug', () => {

• describe('JavaScript array syntax with single quotes', () => {
• test('should fail to parse JavaScript array syntax with single quotes', () => {

•  // This is what the AI returns - JavaScript syntax, not JSON
  

•  const invalidJson = "['*']";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toBeDefined();
  

•  // Should contain error about single quote
  

•  expect(result.error.toLowerCase()).toMatch(/unexpected token|unexpected character|'|quote/);
  

• });
• test('should fail to parse array with single-quoted strings', () => {

•  // More complex example from the bug report
  

•  const invalidJson = "['*', '!bash']";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toBeDefined();
  

•  expect(result.errorContext).toBeDefined();
  

•  expect(result.errorContext.position).toBe(1); // Points to the first single quote
  

• });
• test('should succeed with double-quoted JSON array syntax', () => {

•  // This is what SHOULD be returned - valid JSON
  

•  const validJson = '["*"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should succeed with double-quoted JSON array with exclusions', () => {

•  // Valid JSON version of the bug example
  

•  const validJson = '["*", "!bash"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash']);
  

• });
• });
• describe('cleanSchemaResponse normalizes quote syntax (FIX)', () => {
• test('should not extract from javascript code blocks (only json/generic blocks)', () => {

•  // AI response with JavaScript array syntax in ```javascript block
  

•  const input = "```javascript\n['*', '!bash']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // cleanSchemaResponse doesn't extract from ```javascript blocks, only ```json and ```
  

•  // So it returns the original input unchanged
  

•  expect(cleaned).toBe(input);
  

•  // The uncleaned version with code block markers is not valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should extract JavaScript array from json code block AND normalize quotes', () => {

•  // Even in a ```json block, the AI might use single quotes
  

•  const input = "```json\n['*']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // FIX: Extraction happens AND quote normalization (single -> double quotes)
  

•  expect(cleaned).toBe('["*"]');
  

•  // Now it's valid JSON!
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should not modify valid JSON arrays with double quotes', () => {

•  const input = '```json\n["*", "!bash"]\n```';
  

•  const cleaned = cleanSchemaResponse(input);
  

•  expect(cleaned).toBe('["*", "!bash"]');
  

•  // This is valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash']);
  

• });
• test('should normalize single quotes in complex arrays', () => {

•  const input = "```json\n['*', '!bash', '!docker']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // Should normalize all single quotes to double quotes
  

•  expect(cleaned).toBe('["*", "!bash", "!docker"]');
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash', '!docker']);
  

• });
• test('should normalize single quotes in objects', () => {

•  const input = "```json\n{'key': 'value', 'num': 42}\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // Should normalize object keys and string values
  

•  expect(cleaned).toBe('{"key": "value", "num": 42}');
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual({ key: 'value', num: 42 });
  

• });
• });
• describe('Real-world examples from bug report', () => {
• test('should replicate exact error from debug log line 9', () => {

•  // From log line 9: [DEBUG] JSON validation: Preview: ['*', '!bash']
  

•  const buggyResponse = "['*', '!bash']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  // From log line 10: Parse failed with error: Unexpected token ''', "['*', '!bash']" is not valid JSON
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toContain("Unexpected token");
  

• });
• test('should replicate exact error from debug log line 48', () => {

•  // From log line 48: [DEBUG] JSON validation: Preview: ['*']
  

•  const buggyResponse = "['*']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  // From log line 49: Parse failed with error: Unexpected token ''', "['*']" is not valid JSON
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toContain("Unexpected token");
  

• });
• test('should show error context pointing to single quote', () => {

•  const buggyResponse = "['*']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.errorContext).toBeDefined();
  

•  // Error should point to position 1 (the first single quote after '[')
  

•  expect(result.errorContext.position).toBe(1);
  

•  expect(result.errorContext.snippet).toContain("['*']");
  

• });
• });
• describe('Edge cases with mixed quotes', () => {
• test('should fail with mixed single and double quotes', () => {

•  const invalidJson = '["foo", \'bar\']';
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should fail with object using single quotes', () => {

•  const invalidJson = "{'key': 'value'}";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should succeed with properly escaped single quotes inside double quotes', () => {

•  const validJson = '["It\'s valid"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(["It's valid"]);
  

• });
• });
• describe('Attempt completion context', () => {
• test('should succeed when attempt_completion result contains JavaScript array syntax (FIX)', () => {

•  // This simulates the exact scenario from the bug report where
  

•  // attempt_completion's result field contains ['*'] instead of ["*"]
  

•  const attemptCompletionResult = "['*']";
  

•  // First, cleanSchemaResponse is called (now it DOES normalize quotes - FIX!)
  

•  const cleaned = cleanSchemaResponse(attemptCompletionResult);
  

•  expect(cleaned).toBe('["*"]'); // Single quotes normalized to double quotes
  

•  // Then validateJsonResponse is called without schema
  

•  const result = validateJsonResponse(cleaned);
  

•  // With the fix, this should now succeed!
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should handle multi-line response with single-quote arrays', () => {

•  // AI might return explanation followed by array
  

•  const response = `Looking at the _parseAllowedTools method, when allowedTools is undefined,
  

+the system returns ['*'] which enables all tools.`;
+

•  // cleanSchemaResponse won't extract this (text before JSON)
  

•  const cleaned = cleanSchemaResponse(response);
  

•  expect(cleaned).toBe(response);
  

•  // Validation will fail because it's not valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(false);
  

• });
• });
  +});
  </full_diff>

<files_summary>

npm/src/agent/schemaUtils.js
modified
83
7


npm/tests/unit/single-quote-json-bug.test.js
added
231
0

</files_summary>
</pull_request>

[external-label]

PR Title: fix: Fix JSON parsing to handle single quotes and prevent incorrect extraction

You are generating PR overview, to help owners of the repository to understand what this PR is above, and help reviewer to point to the right parts of the code. First you should provide detailed but concise description, mentioning all the changes.

Files Changed Analysis

After you need to summarize insights from <files_summary>: changed files, additions/deletions, notable patterns.

Next ensure you cover all below:

Architecture & Impact Assessment

• What this PR accomplishes
• Key technical changes introduced
• Affected system components
• Include one or more mermaid diagrams when useful to visualize component relationships or flow.

Scope Discovery & Context Expansion

• From the <files_summary> and code diffs, infer the broader scope of impact across modules, services, and boundaries.
• If your environment supports code search/extract tools, use them to peek at immediately-related files (tests, configs, entrypoints) for better context. If tools are not available, infer and list what you would search next.

You may also be asked to assign labels to PR; if so use this:

• tags.review-effort: integer 1–5 estimating review effort (1=trivial, 5=very high).
• tags.label: one of [bug, chore, documentation, enhancement, feature]. Choose the best fit.

Important:

• Propose tags.review-effort and tags.label only for the initial PR open event.
• Do not change or re-suggest labels on PR update events; the repository applies labels only on pr_opened.

Be concise, specific, and actionable. Avoid praise or celebration.

296 <title>fix: Fix JSON parsing to handle single quotes and prevent incorrect extraction</title> buger main json-parse-single-quotes 314 7 2 diff --git a/npm/src/agent/schemaUtils.js b/npm/src/agent/schemaUtils.js ## Summary

This PR fixes two related issues in JSON validation that were causing attempt_completion results to fail with "Unexpected token '''" errors:

1. Single-quote normalization: AI responses sometimes return JavaScript syntax with single quotes instead of valid JSON
2. Incorrect fragment extraction: Single-backtick pattern was extracting inline code from within JSON objects

Problem

When AI returned attempt_completion with results like:

{
  "text": "When `allowedTools` contains `['*']`, all tools are enabled",
  "intent": "comment_reply"
}

The system would incorrectly extract ['*'] (14 chars) from the inline code instead of preserving the full JSON object (1060+ chars), then fail validation with:

Unexpected token ''', "['*']" is not valid JSON

Solution

1. Added normalizeJsonQuotes() Function

Converts JavaScript array/object syntax to valid JSON by replacing single quotes with double quotes:

• ['*'] → ["*"]
• {'key': 'value'} → {"key": "value"}

2. Fixed Single-Backtick Pattern

Changed from /\([{[][\s\S]?[}]])`/to/^`([{[][\s\S]?[}]])`$/` with anchors to:

• ✅ Extract when entire input is `{"test": "value"}`
• ❌ Prevent extraction from {"text": "... `['*']` ..."}

Testing

New Test Suite

Added single-quote-json-bug.test.js with 17 comprehensive tests:

• Single-quote array/object normalization
• Code block extraction with normalization
• Real-world bug scenarios from error logs
• Edge cases with mixed quotes

Test Results

✅ New tests: 17/17 passed
✅ Existing schemaUtils tests: 96/96 passed
✅ Overall: 1114/1116 passed (2 unrelated failures)

Impact

• Fixes "Unexpected token '''" errors in attempt_completion validation
• Preserves full JSON objects instead of extracting embedded fragments
• Backward compatible - all existing tests pass

Files Changed

• npm/src/agent/schemaUtils.js: Added normalization and fixed extraction
• npm/tests/unit/single-quote-json-bug.test.js: Comprehensive test coverage

🤖 Generated with Claude Code

<full_diff>
--- npm/src/agent/schemaUtils.js
@@ -165,6 +165,74 @@ export function decodeHtmlEntities(text) {
return decoded;
}

+/**

• • Normalize JavaScript syntax to valid JSON syntax
• • Converts single quotes to double quotes for strings in JSON-like structures
• • @param {string} str - String that might contain JavaScript array/object syntax
• • @returns {string} - String with single quotes normalized to double quotes
• */
  +function normalizeJsonQuotes(str) {
• if (!str || typeof str !== 'string') {
• return str;
• }
• // Quick check: if there are no single quotes, no need to normalize
• if (!str.includes("'")) {
• return str;
• }
• let result = '';
• let inDoubleQuote = false;
• let inSingleQuote = false;
• let escaped = false;
• for (let i = 0; i < str.length; i++) {
• const char = str[i];
• const prevChar = i > 0 ? str[i - 1] : '';
• // Handle escape sequences
• if (escaped) {

•  result += char;
  

•  escaped = false;
  

•  continue;
  

• }
• if (char === '\') {

•  escaped = true;
  

•  result += char;
  

•  continue;
  

• }
• // Track when we're inside double-quoted strings
• if (char === '"' && !inSingleQuote) {

•  inDoubleQuote = !inDoubleQuote;
  

•  result += char;
  

•  continue;
  

• }
• // Convert single quotes to double quotes (when not inside double quotes)
• if (char === "'" && !inDoubleQuote) {

•  // Check if this is a single quote inside a string value (like "It's")
  

•  // If we're already in a single-quoted string, toggle the state
  

•  if (inSingleQuote) {
  

•    // Closing single quote - convert to double quote
  

•    result += '"';
  

•    inSingleQuote = false;
  

•  } else {
  

•    // Opening single quote - convert to double quote
  

•    result += '"';
  

•    inSingleQuote = true;
  

•  }
  

•  continue;
  

• }
• result += char;
• }
• return result;
  +}

/**

• Clean AI response by extracting JSON content when response contains JSON
• Only processes responses that contain JSON structures { or [
  @@ -189,29 +257,36 @@ export function cleanSchemaResponse(response) {
  // Try with json language specifier
  const jsonBlockMatch = trimmed.match(/json\s*\n([\s\S]*?)\n/);
  if (jsonBlockMatch) {

• return jsonBlockMatch[1].trim();

• return normalizeJsonQuotes(jsonBlockMatch[1].trim());
  }

// Try any code block with JSON content
const anyBlockMatch = trimmed.match(/\s*\n([{\[][\s\S]*?[}\]])\s*/);
if (anyBlockMatch) {

• return anyBlockMatch[1].trim();

• return normalizeJsonQuotes(anyBlockMatch[1].trim());
  }

// Legacy patterns for more specific matching
const codeBlockPatterns = [
/json\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/,

• /\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/,
• /([{\[][\s\S]*?[}\]])/

• /\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/
  ];

for (const pattern of codeBlockPatterns) {
const match = trimmed.match(pattern);
if (match) {

•  return match[1].trim();
  

•  return normalizeJsonQuotes(match[1].trim());
  

  }
  }

• // Single backtick pattern - ONLY if the entire input is just the code block

• // This prevents extracting inline code from within JSON objects (e.g., ['*'] from markdown text)

• const singleBacktickPattern = /^([{\[][\s\S]*?[}\]])$/;

• const singleBacktickMatch = trimmed.match(singleBacktickPattern);

• if (singleBacktickMatch) {

• return normalizeJsonQuotes(singleBacktickMatch[1].trim());

• }

• // Look for code block start followed immediately by JSON
  const codeBlockStartPattern = /```(?:json)?\s*\n?\s*([{[])/;
  const codeBlockMatch = trimmed.match(codeBlockStartPattern);
  @@ -236,7 +311,7 @@ export function cleanSchemaResponse(response) {
  }

  if (bracketCount === 0) {

•  return trimmed.substring(startIndex, endIndex);
  

•  return normalizeJsonQuotes(trimmed.substring(startIndex, endIndex));
  

  }
  }

@@ -261,7 +336,8 @@ export function cleanSchemaResponse(response) {
const isJsonArray = firstChar === '[' && lastChar === ']';

if (isJsonObject || isJsonArray) {

• return cleaned;

• // Normalize JavaScript syntax (single quotes) to valid JSON syntax (double quotes)
• return normalizeJsonQuotes(cleaned);
  }

return response; // Return original if no extractable JSON found

--- npm/tests/unit/single-quote-json-bug.test.js
@@ -0,0 +1,231 @@
+/**

• • Test for single-quote JSON bug
• • Bug Description:
• • When AI responses contain JavaScript array syntax with single quotes like ['*']
• • instead of valid JSON with double quotes ["*"], the JSON parser fails with:
• • "Unexpected token ''', "['*']" is not valid JSON"
• • This test replicates the bug seen in the debug logs where:
• • 1. AI returns response with JavaScript array syntax: ['*', '!bash']
• • 2. cleanSchemaResponse extracts it as-is (no syntax normalization)
• • 3. JSON.parse fails because single quotes are invalid in JSON
• */

+import { describe, test, expect } from '@jest/globals';
+import { cleanSchemaResponse, validateJsonResponse } from '../../src/agent/schemaUtils.js';
+
+describe('Single-quote JSON Bug', () => {

• describe('JavaScript array syntax with single quotes', () => {
• test('should fail to parse JavaScript array syntax with single quotes', () => {

•  // This is what the AI returns - JavaScript syntax, not JSON
  

•  const invalidJson = "['*']";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toBeDefined();
  

•  // Should contain error about single quote
  

•  expect(result.error.toLowerCase()).toMatch(/unexpected token|unexpected character|'|quote/);
  

• });
• test('should fail to parse array with single-quoted strings', () => {

•  // More complex example from the bug report
  

•  const invalidJson = "['*', '!bash']";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toBeDefined();
  

•  expect(result.errorContext).toBeDefined();
  

•  expect(result.errorContext.position).toBe(1); // Points to the first single quote
  

• });
• test('should succeed with double-quoted JSON array syntax', () => {

•  // This is what SHOULD be returned - valid JSON
  

•  const validJson = '["*"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should succeed with double-quoted JSON array with exclusions', () => {

•  // Valid JSON version of the bug example
  

•  const validJson = '["*", "!bash"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash']);
  

• });
• });
• describe('cleanSchemaResponse normalizes quote syntax (FIX)', () => {
• test('should not extract from javascript code blocks (only json/generic blocks)', () => {

•  // AI response with JavaScript array syntax in ```javascript block
  

•  const input = "```javascript\n['*', '!bash']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // cleanSchemaResponse doesn't extract from ```javascript blocks, only ```json and ```
  

•  // So it returns the original input unchanged
  

•  expect(cleaned).toBe(input);
  

•  // The uncleaned version with code block markers is not valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should extract JavaScript array from json code block AND normalize quotes', () => {

•  // Even in a ```json block, the AI might use single quotes
  

•  const input = "```json\n['*']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // FIX: Extraction happens AND quote normalization (single -> double quotes)
  

•  expect(cleaned).toBe('["*"]');
  

•  // Now it's valid JSON!
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should not modify valid JSON arrays with double quotes', () => {

•  const input = '```json\n["*", "!bash"]\n```';
  

•  const cleaned = cleanSchemaResponse(input);
  

•  expect(cleaned).toBe('["*", "!bash"]');
  

•  // This is valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash']);
  

• });
• test('should normalize single quotes in complex arrays', () => {

•  const input = "```json\n['*', '!bash', '!docker']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // Should normalize all single quotes to double quotes
  

•  expect(cleaned).toBe('["*", "!bash", "!docker"]');
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash', '!docker']);
  

• });
• test('should normalize single quotes in objects', () => {

•  const input = "```json\n{'key': 'value', 'num': 42}\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // Should normalize object keys and string values
  

•  expect(cleaned).toBe('{"key": "value", "num": 42}');
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual({ key: 'value', num: 42 });
  

• });
• });
• describe('Real-world examples from bug report', () => {
• test('should replicate exact error from debug log line 9', () => {

•  // From log line 9: [DEBUG] JSON validation: Preview: ['*', '!bash']
  

•  const buggyResponse = "['*', '!bash']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  // From log line 10: Parse failed with error: Unexpected token ''', "['*', '!bash']" is not valid JSON
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toContain("Unexpected token");
  

• });
• test('should replicate exact error from debug log line 48', () => {

•  // From log line 48: [DEBUG] JSON validation: Preview: ['*']
  

•  const buggyResponse = "['*']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  // From log line 49: Parse failed with error: Unexpected token ''', "['*']" is not valid JSON
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toContain("Unexpected token");
  

• });
• test('should show error context pointing to single quote', () => {

•  const buggyResponse = "['*']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.errorContext).toBeDefined();
  

•  // Error should point to position 1 (the first single quote after '[')
  

•  expect(result.errorContext.position).toBe(1);
  

•  expect(result.errorContext.snippet).toContain("['*']");
  

• });
• });
• describe('Edge cases with mixed quotes', () => {
• test('should fail with mixed single and double quotes', () => {

•  const invalidJson = '["foo", \'bar\']';
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should fail with object using single quotes', () => {

•  const invalidJson = "{'key': 'value'}";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should succeed with properly escaped single quotes inside double quotes', () => {

•  const validJson = '["It\'s valid"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(["It's valid"]);
  

• });
• });
• describe('Attempt completion context', () => {
• test('should succeed when attempt_completion result contains JavaScript array syntax (FIX)', () => {

•  // This simulates the exact scenario from the bug report where
  

•  // attempt_completion's result field contains ['*'] instead of ["*"]
  

•  const attemptCompletionResult = "['*']";
  

•  // First, cleanSchemaResponse is called (now it DOES normalize quotes - FIX!)
  

•  const cleaned = cleanSchemaResponse(attemptCompletionResult);
  

•  expect(cleaned).toBe('["*"]'); // Single quotes normalized to double quotes
  

•  // Then validateJsonResponse is called without schema
  

•  const result = validateJsonResponse(cleaned);
  

•  // With the fix, this should now succeed!
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should handle multi-line response with single-quote arrays', () => {

•  // AI might return explanation followed by array
  

•  const response = `Looking at the _parseAllowedTools method, when allowedTools is undefined,
  

+the system returns ['*'] which enables all tools.`;
+

•  // cleanSchemaResponse won't extract this (text before JSON)
  

•  const cleaned = cleanSchemaResponse(response);
  

•  expect(cleaned).toBe(response);
  

•  // Validation will fail because it's not valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(false);
  

• });
• });
  +});
  </full_diff>

<files_summary>

npm/src/agent/schemaUtils.js
modified
83
7


npm/tests/unit/single-quote-json-bug.test.js
added
231
0

</files_summary>
</pull_request>

### Raw AI Response
```json
[overview]
['*']

[external-label]
['*']

---

Powered by Visor from Probelabs

Last updated: 2025-11-16T17:32:44.664Z | Triggered by: opened | Commit: ed4eb93

💡 TIP: You can chat with Visor using /visor ask <your question>

[probelabs]

🔍 Code Analysis Results

Security Issues (5)

Severity	Location	Issue
🟡 Warning	npm/src/agent/schemaUtils.js:175-244	The normalizeJsonQuotes() function processes input character-by-character without length validation, making it vulnerable to denial of service attacks with very long strings. 💡 Suggestion Add input length validation to prevent processing extremely long strings that could cause resource exhaustion. Consider adding a maximum length limit (e.g., 1MB) and returning early if exceeded. 🔧 Suggested Fix function normalizeJsonQuotes(str) { if (!str || typeof str !== 'string') { return str; } // Prevent processing extremely long strings (DoS protection) if (str.length > 1048576) { // 1MB limit console.warn('[WARN] Input too long for JSON normalization, returning as-is'); return str; } // Quick check: if there are no single quotes, no need to normalize if (!str.includes("'")) { return str; }
🟡 Warning	npm/src/agent/schemaUtils.js:264	The regex pattern /``\s* ([\{\[][\s\S]*?[}\]])\s*```/ uses non-greedy quantifier [\s\S]*? which could be vulnerable to ReDoS attacks with maliciously crafted input. 💡 Suggestion Replace the non-greedy quantifier with a more specific pattern or add input length limits before regex processing to prevent catastrophic backtracking. 🔧 Suggested Fix // Try any code block with JSON content // Limit input length before regex processing to prevent ReDoS if (trimmed.length > 50000) { console.warn('[WARN] Input too long for regex processing, skipping extraction'); } else { const anyBlockMatch = trimmed.match(/```\s* ([\{\[][\s\S]{0,10000}[}\]])\s*```/); if (anyBlockMatch) { return normalizeJsonQuotes(anyBlockMatch[1].trim()); } }
🟡 Warning	npm/src/agent/schemaUtils.js:284	The single backtick pattern /^`([\{\[][\s\S]*?[}\]])`$/ uses non-greedy quantifier [\s\S]*? which could cause catastrophic backtracking with long input strings. 💡 Suggestion Add length limits and use more specific quantifiers to prevent ReDoS vulnerabilities in regex patterns. 🔧 Suggested Fix // Single backtick pattern - ONLY if the entire input is just the code block // This prevents extracting inline code from within JSON objects (e.g., ['*'] from markdown text) // Add length limit to prevent ReDoS if (trimmed.length <= 10000) { const singleBacktickPattern = /^`([\{\[][\s\S]{0,1000}[}\]])`$/; const singleBacktickMatch = trimmed.match(singleBacktickPattern); if (singleBacktickMatch) { return normalizeJsonQuotes(singleBacktickMatch[1].trim()); } }
🟡 Warning	npm/src/agent/schemaUtils.js:190	The normalizeJsonQuotes() function lacks proper validation for escape sequences, which could lead to unexpected behavior or potential injection vulnerabilities. 💡 Suggestion Add proper validation for escape sequences to prevent malformed input from causing unexpected behavior. 🔧 Suggested Fix // Handle escape sequences with validation if (escaped) { // Validate escape sequence if (char !== '"' && char !== '\\' && char !== '/' && char !== 'b' && char !== 'f' && char !== 'n' && char !== 'r' && char !== 't' && char !== 'u') { // Invalid escape sequence, treat as literal backslash result += '\\' + char; } else { result += char; } escaped = false; continue; }
🟡 Warning	npm/src/agent/schemaUtils.js:258	The regex pattern /```json\s* ([\s\S]*?) ```/ for JSON block matching uses non-greedy quantifier [\s\S]*? which could be vulnerable to ReDoS attacks. 💡 Suggestion Add input length validation and use more specific patterns to prevent catastrophic backtracking. 🔧 Suggested Fix // Try with json language specifier // Add length limit to prevent ReDoS if (trimmed.length > 50000) { console.warn('[WARN] Input too long for JSON block extraction, skipping'); } else { const jsonBlockMatch = trimmed.match(/```json\s* ([\s\S]{0,10000}) ```/); if (jsonBlockMatch) { return normalizeJsonQuotes(jsonBlockMatch[1].trim()); } }

Architecture Issues (4)

Severity	Location	Issue
🟠 Error	npm/src/agent/schemaUtils.js:175-234	The normalizeJsonQuotes() function duplicates quote parsing logic that already exists in bashCommandUtils.js. Both implement similar character state machines for handling quotes and escapes. 💡 Suggestion Extract the common quote parsing logic into a shared utility function in a separate module, then have both functions use it. This would eliminate duplication and make the logic reusable. 🔧 Suggested Fix Create a shared quote parsing utility and reuse it in both locations
🟡 Warning	npm/src/agent/schemaUtils.js:175-234	The normalizeJsonQuotes() function is overly complex for the specific problem being solved. A simple string replace would handle 90% of cases with much less complexity. 💡 Suggestion Replace the complex state machine with a simpler approach: str.replace(/'/g, '"') for basic cases, only using the complex logic for edge cases. Consider if the full complexity is actually needed. 🔧 Suggested Fix Use a simpler string replacement approach for the common case
🟡 Warning	npm/src/agent/schemaUtils.js:284-287	The single backtick pattern with anchors introduces a special case that could be generalized. This pattern specifically handles one edge case rather than being part of a consistent strategy. 💡 Suggestion Consider refactoring the extraction patterns into a more systematic approach that can handle various edge cases without requiring special-case patterns with anchors. 🔧 Suggested Fix Refactor to use a more general extraction strategy
🟡 Warning	npm/src/agent/schemaUtils.js:240-244	The cleanSchemaResponse() function handles both JSON extraction AND content stripping, which violates separation of concerns. The TODO comment acknowledges this architectural issue. 💡 Suggestion Split this function into separate functions: extractJsonContent() and stripNonJsonContent() as suggested in the TODO comment. This would improve testability and maintainability. 🔧 Suggested Fix Split into separate functions for extraction and stripping

Performance Issues (3)

Severity	Location	Issue
🟢 Info	npm/src/agent/schemaUtils.js:258-287	Multiple sequential regex.match() calls on the same string without early termination optimization, potentially causing unnecessary processing 💡 Suggestion Consider combining patterns or using a single regex with alternation to reduce multiple passes over the string 🔧 Suggested Fix // Combined pattern matching for better performance const combinedPatterns = [ { pattern: /```json\s* ([\s\S]*?) ```/, priority: 1 }, { pattern: /```\s* ([\{\[][\s\S]*?[}\]])\s*```/, priority: 2 }, { pattern: /^`([\{\[][\s\S]*?[}\])`$/, priority: 3 } ]; for (const { pattern } of combinedPatterns) { const match = trimmed.match(pattern); if (match) { return normalizeJsonQuotes(match[1].trim()); } }</code></pre>
🟡 Warning	npm/src/agent/schemaUtils.js:175-234	The normalizeJsonQuotes function uses O(n) character-by-character processing which could be optimized with regex for simple cases, potentially causing performance bottlenecks in hot paths with large JSON strings 💡 Suggestion Consider using regex replacement for simple cases (e.g., str.replace(/'/g, '"')) when no escaped quotes or complex string contexts are detected, falling back to character-by-character processing only when needed 🔧 Suggested Fix function normalizeJsonQuotes(str) { if (!str || typeof str !== 'string') { return str; } // Quick check: if there are no single quotes, no need to normalize if (!str.includes("'")) { return str; } // Fast path: use regex for simple cases without escaped quotes if (!str.includes('\') && !str.includes('"')) { return str.replace(/'/g, '"'); } // Complex case: fall back to character-by-character processing let result = ''; let inDoubleQuote = false; let inSingleQuote = false; let escaped = false; for (let i = 0; i < str.length; i++) { // ... existing character-by-character logic } return result; }
🟡 Warning	npm/src/agent/schemaUtils.js:270-273	The codeBlockPatterns array contains redundant regex patterns that overlap with patterns already checked above, causing unnecessary regex compilation and matching 💡 Suggestion Remove redundant patterns from codeBlockPatterns array since they duplicate checks already performed at lines 258 and 264 🔧 Suggested Fix // Legacy patterns for more specific matching (only patterns not already checked above) const codeBlockPatterns = [ // Remove redundant patterns as they're already handled above ];

Quality Issues (3)

Severity	Location	Issue
🟠 Error	npm/src/agent/schemaUtils.js:175	The normalizeJsonQuotes function has incomplete handling of edge cases with mixed quotes and escaped characters. The function may incorrectly normalize single quotes within double-quoted strings that contain escaped single quotes (e.g., "It\'s fine"), potentially breaking valid JSON. 💡 Suggestion Add proper handling for escaped single quotes within double-quoted strings and add comprehensive test cases for edge cases including mixed quotes, nested quotes, and various escape sequences. 🔧 Suggested Fix function normalizeJsonQuotes(str) { if (!str || typeof str !== 'string') { return str; } // Quick check: if there are no single quotes, no need to normalize if (!str.includes("'")) { return str; } let result = ''; let inDoubleQuote = false; let inSingleQuote = false; let escaped = false; for (let i = 0; i < str.length; i++) { const char = str[i]; const prevChar = i > 0 ? str[i - 1] : ''; // Handle escape sequences if (escaped) { result += char; escaped = false; continue; } if (char === &#39;\\&#39;) { escaped = true; result += char; continue; } // Track when we&#39;re inside double-quoted strings if (char === &#39;&#34;&#39; &amp;&amp; !inSingleQuote) { inDoubleQuote = !inDoubleQuote; result += char; continue; } // Convert single quotes to double quotes (when not inside double quotes) if (char === &#34;&#39;&#34; &amp;&amp; !inDoubleQuote) { // Check if this single quote is properly escaped within a double-quoted string // If we&#39;re already in a single-quoted string, toggle the state if (inSingleQuote) { // Closing single quote - convert to double quote result += &#39;&#34;&#39;; inSingleQuote = false; } else { // Opening single quote - convert to double quote result += &#39;&#34;&#39;; inSingleQuote = true; } continue; } result += char; } return result; }
🟡 Warning	npm/src/agent/schemaUtils.js:284	The single backtick pattern with anchors /^`([\{\[][\s\S]*?[}\]])`$/ may fail to match valid JSON objects that contain newlines or whitespace before/after the backticks, limiting its ability to extract JSON from certain response formats. 💡 Suggestion Add optional whitespace handling to the pattern: /^\s*`([\{\[][\s\S]*?[}\]])`\s*$/ to handle cases where there might be whitespace around the backticks. 🔧 Suggested Fix const singleBacktickPattern = /^\s*`([\{\[][\s\S]*?[}\]])`\s*$/;
🟡 Warning	npm/tests/unit/single-quote-json-bug.test.js:1	Test suite lacks comprehensive coverage for edge cases in normalizeJsonQuotes function, including nested quotes, escaped characters, and malformed input that could lead to incorrect normalization. 💡 Suggestion Add additional test cases for: 1) Mixed single and double quotes in complex nested structures, 2) Escaped single quotes within double-quoted strings, 3) Malformed input with unmatched quotes, 4) Performance with large JSON strings, 5) Unicode characters with quotes. 🔧 Suggested Fix Add comprehensive test suite covering edge cases for normalizeJsonQuotes function

🐛 Debug Information

Provider: anthropic
Model: glm-4.6
API Key Source: ANTHROPIC_API_KEY
Processing Time: 152986ms
Timestamp: 2025-11-16T17:32:42.723Z
Prompt Length: 36944 characters
Response Length: 10 characters
JSON Parse Success: ❌

AI Prompt

[overview]
<instructions>
PR Title: fix: Fix JSON parsing to handle single quotes and prevent incorrect extraction

You are generating PR overview, to help owners of the repository to understand what this PR is above, and help reviewer to point to the right parts of the code. First you should provide detailed but concise description, mentioning all the changes.

## Files Changed Analysis
After you need to summarize insights from `<files_summary>`: changed files, additions/deletions, notable patterns.

Next ensure you cover all below:

## Architecture & Impact Assessment
  - What this PR accomplishes
  - Key technical changes introduced
  - Affected system components
  - Include one or more mermaid diagrams when useful to visualize component relationships or flow.
  
## Scope Discovery & Context Expansion
- From the `<files_summary>` and code diffs, infer the broader scope of impact across modules, services, and boundaries.
- If your environment supports code search/extract tools, use them to peek at immediately-related files (tests, configs, entrypoints) for better context. If tools are not available, infer and list what you would search next.

You may also be asked to assign labels to PR; if so use this:
- `tags.review-effort`: integer 1–5 estimating review effort (1=trivial, 5=very high).
- `tags.label`: one of [bug, chore, documentation, enhancement, feature]. Choose the best fit.

Important:
- Propose `tags.review-effort` and `tags.label` only for the initial PR open event.
- Do not change or re-suggest labels on PR update events; the repository applies labels only on `pr_opened`.

Be concise, specific, and actionable. Avoid praise or celebration.

</instructions>

<context>
<pull_request>
  <!-- Core pull request metadata including identification, branches, and change statistics -->
  <metadata>
    <number>296</number>
    <title>fix: Fix JSON parsing to handle single quotes and prevent incorrect extraction</title>
    <author>buger</author>
    <base_branch>main</base_branch>
    <target_branch>json-parse-single-quotes</target_branch>
    <total_additions>314</total_additions>
    <total_deletions>7</total_deletions>
    <files_changed_count>2</files_changed_count>
  </metadata>
  <raw_diff_header>
diff --git a/npm/src/agent/schemaUtils.js b/npm/src/agent/schemaUtils.js
  </raw_diff_header>
  <!-- Full pull request description provided by the author -->
  <description>
## Summary

This PR fixes two related issues in JSON validation that were causing `attempt_completion` results to fail with "Unexpected token '''" errors:

1. **Single-quote normalization**: AI responses sometimes return JavaScript syntax with single quotes instead of valid JSON
2. **Incorrect fragment extraction**: Single-backtick pattern was extracting inline code from within JSON objects

## Problem

When AI returned `attempt_completion` with results like:
```json
{
  "text": "When `allowedTools` contains `['*']`, all tools are enabled",
  "intent": "comment_reply"
}

The system would incorrectly extract ['*'] (14 chars) from the inline code instead of preserving the full JSON object (1060+ chars), then fail validation with:

Unexpected token ''', "['*']" is not valid JSON

Solution

1. Added normalizeJsonQuotes() Function

Converts JavaScript array/object syntax to valid JSON by replacing single quotes with double quotes:

• ['*'] → ["*"]
• {'key': 'value'} → {"key": "value"}

2. Fixed Single-Backtick Pattern

Changed from /\([{[][\s\S]?[}]])`/to/^`([{[][\s\S]?[}]])`$/` with anchors to:

• ✅ Extract when entire input is `{"test": "value"}`
• ❌ Prevent extraction from {"text": "... `['*']` ..."}

Testing

New Test Suite

Added single-quote-json-bug.test.js with 17 comprehensive tests:

• Single-quote array/object normalization
• Code block extraction with normalization
• Real-world bug scenarios from error logs
• Edge cases with mixed quotes

Test Results

✅ New tests: 17/17 passed
✅ Existing schemaUtils tests: 96/96 passed
✅ Overall: 1114/1116 passed (2 unrelated failures)

Impact

• Fixes "Unexpected token '''" errors in attempt_completion validation
• Preserves full JSON objects instead of extracting embedded fragments
• Backward compatible - all existing tests pass

Files Changed

• npm/src/agent/schemaUtils.js: Added normalization and fixed extraction
• npm/tests/unit/single-quote-json-bug.test.js: Comprehensive test coverage

🤖 Generated with Claude Code

<full_diff>
--- npm/src/agent/schemaUtils.js
@@ -165,6 +165,74 @@ export function decodeHtmlEntities(text) {
return decoded;
}

+/**

• • Normalize JavaScript syntax to valid JSON syntax
• • Converts single quotes to double quotes for strings in JSON-like structures
• • @param {string} str - String that might contain JavaScript array/object syntax
• • @returns {string} - String with single quotes normalized to double quotes
• */
  +function normalizeJsonQuotes(str) {
• if (!str || typeof str !== 'string') {
• return str;
• }
• // Quick check: if there are no single quotes, no need to normalize
• if (!str.includes("'")) {
• return str;
• }
• let result = '';
• let inDoubleQuote = false;
• let inSingleQuote = false;
• let escaped = false;
• for (let i = 0; i < str.length; i++) {
• const char = str[i];
• const prevChar = i > 0 ? str[i - 1] : '';
• // Handle escape sequences
• if (escaped) {

•  result += char;
  

•  escaped = false;
  

•  continue;
  

• }
• if (char === '\') {

•  escaped = true;
  

•  result += char;
  

•  continue;
  

• }
• // Track when we're inside double-quoted strings
• if (char === '"' && !inSingleQuote) {

•  inDoubleQuote = !inDoubleQuote;
  

•  result += char;
  

•  continue;
  

• }
• // Convert single quotes to double quotes (when not inside double quotes)
• if (char === "'" && !inDoubleQuote) {

•  // Check if this is a single quote inside a string value (like "It's")
  

•  // If we're already in a single-quoted string, toggle the state
  

•  if (inSingleQuote) {
  

•    // Closing single quote - convert to double quote
  

•    result += '"';
  

•    inSingleQuote = false;
  

•  } else {
  

•    // Opening single quote - convert to double quote
  

•    result += '"';
  

•    inSingleQuote = true;
  

•  }
  

•  continue;
  

• }
• result += char;
• }
• return result;
  +}

/**

• Clean AI response by extracting JSON content when response contains JSON
• Only processes responses that contain JSON structures { or [
  @@ -189,29 +257,36 @@ export function cleanSchemaResponse(response) {
  // Try with json language specifier
  const jsonBlockMatch = trimmed.match(/json\s*\n([\s\S]*?)\n/);
  if (jsonBlockMatch) {

• return jsonBlockMatch[1].trim();

• return normalizeJsonQuotes(jsonBlockMatch[1].trim());
  }

// Try any code block with JSON content
const anyBlockMatch = trimmed.match(/\s*\n([{\[][\s\S]*?[}\]])\s*/);
if (anyBlockMatch) {

• return anyBlockMatch[1].trim();

• return normalizeJsonQuotes(anyBlockMatch[1].trim());
  }

// Legacy patterns for more specific matching
const codeBlockPatterns = [
/json\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/,

• /\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/,
• /([{\[][\s\S]*?[}\]])/

• /\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/
  ];

for (const pattern of codeBlockPatterns) {
const match = trimmed.match(pattern);
if (match) {

•  return match[1].trim();
  

•  return normalizeJsonQuotes(match[1].trim());
  

  }
  }

• // Single backtick pattern - ONLY if the entire input is just the code block

• // This prevents extracting inline code from within JSON objects (e.g., ['*'] from markdown text)

• const singleBacktickPattern = /^([{\[][\s\S]*?[}\]])$/;

• const singleBacktickMatch = trimmed.match(singleBacktickPattern);

• if (singleBacktickMatch) {

• return normalizeJsonQuotes(singleBacktickMatch[1].trim());

• }

• // Look for code block start followed immediately by JSON
  const codeBlockStartPattern = /```(?:json)?\s*\n?\s*([{[])/;
  const codeBlockMatch = trimmed.match(codeBlockStartPattern);
  @@ -236,7 +311,7 @@ export function cleanSchemaResponse(response) {
  }

  if (bracketCount === 0) {

•  return trimmed.substring(startIndex, endIndex);
  

•  return normalizeJsonQuotes(trimmed.substring(startIndex, endIndex));
  

  }
  }

@@ -261,7 +336,8 @@ export function cleanSchemaResponse(response) {
const isJsonArray = firstChar === '[' && lastChar === ']';

if (isJsonObject || isJsonArray) {

• return cleaned;

• // Normalize JavaScript syntax (single quotes) to valid JSON syntax (double quotes)
• return normalizeJsonQuotes(cleaned);
  }

return response; // Return original if no extractable JSON found

--- npm/tests/unit/single-quote-json-bug.test.js
@@ -0,0 +1,231 @@
+/**

• • Test for single-quote JSON bug
• • Bug Description:
• • When AI responses contain JavaScript array syntax with single quotes like ['*']
• • instead of valid JSON with double quotes ["*"], the JSON parser fails with:
• • "Unexpected token ''', "['*']" is not valid JSON"
• • This test replicates the bug seen in the debug logs where:
• • 1. AI returns response with JavaScript array syntax: ['*', '!bash']
• • 2. cleanSchemaResponse extracts it as-is (no syntax normalization)
• • 3. JSON.parse fails because single quotes are invalid in JSON
• */

+import { describe, test, expect } from '@jest/globals';
+import { cleanSchemaResponse, validateJsonResponse } from '../../src/agent/schemaUtils.js';
+
+describe('Single-quote JSON Bug', () => {

• describe('JavaScript array syntax with single quotes', () => {
• test('should fail to parse JavaScript array syntax with single quotes', () => {

•  // This is what the AI returns - JavaScript syntax, not JSON
  

•  const invalidJson = "['*']";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toBeDefined();
  

•  // Should contain error about single quote
  

•  expect(result.error.toLowerCase()).toMatch(/unexpected token|unexpected character|'|quote/);
  

• });
• test('should fail to parse array with single-quoted strings', () => {

•  // More complex example from the bug report
  

•  const invalidJson = "['*', '!bash']";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toBeDefined();
  

•  expect(result.errorContext).toBeDefined();
  

•  expect(result.errorContext.position).toBe(1); // Points to the first single quote
  

• });
• test('should succeed with double-quoted JSON array syntax', () => {

•  // This is what SHOULD be returned - valid JSON
  

•  const validJson = '["*"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should succeed with double-quoted JSON array with exclusions', () => {

•  // Valid JSON version of the bug example
  

•  const validJson = '["*", "!bash"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash']);
  

• });
• });
• describe('cleanSchemaResponse normalizes quote syntax (FIX)', () => {
• test('should not extract from javascript code blocks (only json/generic blocks)', () => {

•  // AI response with JavaScript array syntax in ```javascript block
  

•  const input = "```javascript\n['*', '!bash']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // cleanSchemaResponse doesn't extract from ```javascript blocks, only ```json and ```
  

•  // So it returns the original input unchanged
  

•  expect(cleaned).toBe(input);
  

•  // The uncleaned version with code block markers is not valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should extract JavaScript array from json code block AND normalize quotes', () => {

•  // Even in a ```json block, the AI might use single quotes
  

•  const input = "```json\n['*']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // FIX: Extraction happens AND quote normalization (single -> double quotes)
  

•  expect(cleaned).toBe('["*"]');
  

•  // Now it's valid JSON!
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should not modify valid JSON arrays with double quotes', () => {

•  const input = '```json\n["*", "!bash"]\n```';
  

•  const cleaned = cleanSchemaResponse(input);
  

•  expect(cleaned).toBe('["*", "!bash"]');
  

•  // This is valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash']);
  

• });
• test('should normalize single quotes in complex arrays', () => {

•  const input = "```json\n['*', '!bash', '!docker']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // Should normalize all single quotes to double quotes
  

•  expect(cleaned).toBe('["*", "!bash", "!docker"]');
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash', '!docker']);
  

• });
• test('should normalize single quotes in objects', () => {

•  const input = "```json\n{'key': 'value', 'num': 42}\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // Should normalize object keys and string values
  

•  expect(cleaned).toBe('{"key": "value", "num": 42}');
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual({ key: 'value', num: 42 });
  

• });
• });
• describe('Real-world examples from bug report', () => {
• test('should replicate exact error from debug log line 9', () => {

•  // From log line 9: [DEBUG] JSON validation: Preview: ['*', '!bash']
  

•  const buggyResponse = "['*', '!bash']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  // From log line 10: Parse failed with error: Unexpected token ''', "['*', '!bash']" is not valid JSON
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toContain("Unexpected token");
  

• });
• test('should replicate exact error from debug log line 48', () => {

•  // From log line 48: [DEBUG] JSON validation: Preview: ['*']
  

•  const buggyResponse = "['*']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  // From log line 49: Parse failed with error: Unexpected token ''', "['*']" is not valid JSON
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toContain("Unexpected token");
  

• });
• test('should show error context pointing to single quote', () => {

•  const buggyResponse = "['*']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.errorContext).toBeDefined();
  

•  // Error should point to position 1 (the first single quote after '[')
  

•  expect(result.errorContext.position).toBe(1);
  

•  expect(result.errorContext.snippet).toContain("['*']");
  

• });
• });
• describe('Edge cases with mixed quotes', () => {
• test('should fail with mixed single and double quotes', () => {

•  const invalidJson = '["foo", \'bar\']';
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should fail with object using single quotes', () => {

•  const invalidJson = "{'key': 'value'}";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should succeed with properly escaped single quotes inside double quotes', () => {

•  const validJson = '["It\'s valid"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(["It's valid"]);
  

• });
• });
• describe('Attempt completion context', () => {
• test('should succeed when attempt_completion result contains JavaScript array syntax (FIX)', () => {

•  // This simulates the exact scenario from the bug report where
  

•  // attempt_completion's result field contains ['*'] instead of ["*"]
  

•  const attemptCompletionResult = "['*']";
  

•  // First, cleanSchemaResponse is called (now it DOES normalize quotes - FIX!)
  

•  const cleaned = cleanSchemaResponse(attemptCompletionResult);
  

•  expect(cleaned).toBe('["*"]'); // Single quotes normalized to double quotes
  

•  // Then validateJsonResponse is called without schema
  

•  const result = validateJsonResponse(cleaned);
  

•  // With the fix, this should now succeed!
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should handle multi-line response with single-quote arrays', () => {

•  // AI might return explanation followed by array
  

•  const response = `Looking at the _parseAllowedTools method, when allowedTools is undefined,
  

+the system returns ['*'] which enables all tools.`;
+

•  // cleanSchemaResponse won't extract this (text before JSON)
  

•  const cleaned = cleanSchemaResponse(response);
  

•  expect(cleaned).toBe(response);
  

•  // Validation will fail because it's not valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(false);
  

• });
• });
  +});
  </full_diff>

<files_summary>

npm/src/agent/schemaUtils.js
modified
83
7


npm/tests/unit/single-quote-json-bug.test.js
added
231
0

</files_summary>
</pull_request>

[external-label]

PR Title: fix: Fix JSON parsing to handle single quotes and prevent incorrect extraction

You are generating PR overview, to help owners of the repository to understand what this PR is above, and help reviewer to point to the right parts of the code. First you should provide detailed but concise description, mentioning all the changes.

Files Changed Analysis

After you need to summarize insights from <files_summary>: changed files, additions/deletions, notable patterns.

Next ensure you cover all below:

Architecture & Impact Assessment

• What this PR accomplishes
• Key technical changes introduced
• Affected system components
• Include one or more mermaid diagrams when useful to visualize component relationships or flow.

Scope Discovery & Context Expansion

• From the <files_summary> and code diffs, infer the broader scope of impact across modules, services, and boundaries.
• If your environment supports code search/extract tools, use them to peek at immediately-related files (tests, configs, entrypoints) for better context. If tools are not available, infer and list what you would search next.

You may also be asked to assign labels to PR; if so use this:

• tags.review-effort: integer 1–5 estimating review effort (1=trivial, 5=very high).
• tags.label: one of [bug, chore, documentation, enhancement, feature]. Choose the best fit.

Important:

• Propose tags.review-effort and tags.label only for the initial PR open event.
• Do not change or re-suggest labels on PR update events; the repository applies labels only on pr_opened.

Be concise, specific, and actionable. Avoid praise or celebration.

296 <title>fix: Fix JSON parsing to handle single quotes and prevent incorrect extraction</title> buger main json-parse-single-quotes 314 7 2 diff --git a/npm/src/agent/schemaUtils.js b/npm/src/agent/schemaUtils.js ## Summary

This PR fixes two related issues in JSON validation that were causing attempt_completion results to fail with "Unexpected token '''" errors:

1. Single-quote normalization: AI responses sometimes return JavaScript syntax with single quotes instead of valid JSON
2. Incorrect fragment extraction: Single-backtick pattern was extracting inline code from within JSON objects

Problem

When AI returned attempt_completion with results like:

{
  "text": "When `allowedTools` contains `['*']`, all tools are enabled",
  "intent": "comment_reply"
}

The system would incorrectly extract ['*'] (14 chars) from the inline code instead of preserving the full JSON object (1060+ chars), then fail validation with:

Unexpected token ''', "['*']" is not valid JSON

Solution

1. Added normalizeJsonQuotes() Function

Converts JavaScript array/object syntax to valid JSON by replacing single quotes with double quotes:

• ['*'] → ["*"]
• {'key': 'value'} → {"key": "value"}

2. Fixed Single-Backtick Pattern

Changed from /\([{[][\s\S]?[}]])`/to/^`([{[][\s\S]?[}]])`$/` with anchors to:

• ✅ Extract when entire input is `{"test": "value"}`
• ❌ Prevent extraction from {"text": "... `['*']` ..."}

Testing

New Test Suite

Added single-quote-json-bug.test.js with 17 comprehensive tests:

• Single-quote array/object normalization
• Code block extraction with normalization
• Real-world bug scenarios from error logs
• Edge cases with mixed quotes

Test Results

✅ New tests: 17/17 passed
✅ Existing schemaUtils tests: 96/96 passed
✅ Overall: 1114/1116 passed (2 unrelated failures)

Impact

• Fixes "Unexpected token '''" errors in attempt_completion validation
• Preserves full JSON objects instead of extracting embedded fragments
• Backward compatible - all existing tests pass

Files Changed

• npm/src/agent/schemaUtils.js: Added normalization and fixed extraction
• npm/tests/unit/single-quote-json-bug.test.js: Comprehensive test coverage

🤖 Generated with Claude Code

<full_diff>
--- npm/src/agent/schemaUtils.js
@@ -165,6 +165,74 @@ export function decodeHtmlEntities(text) {
return decoded;
}

+/**

• • Normalize JavaScript syntax to valid JSON syntax
• • Converts single quotes to double quotes for strings in JSON-like structures
• • @param {string} str - String that might contain JavaScript array/object syntax
• • @returns {string} - String with single quotes normalized to double quotes
• */
  +function normalizeJsonQuotes(str) {
• if (!str || typeof str !== 'string') {
• return str;
• }
• // Quick check: if there are no single quotes, no need to normalize
• if (!str.includes("'")) {
• return str;
• }
• let result = '';
• let inDoubleQuote = false;
• let inSingleQuote = false;
• let escaped = false;
• for (let i = 0; i < str.length; i++) {
• const char = str[i];
• const prevChar = i > 0 ? str[i - 1] : '';
• // Handle escape sequences
• if (escaped) {

•  result += char;
  

•  escaped = false;
  

•  continue;
  

• }
• if (char === '\') {

•  escaped = true;
  

•  result += char;
  

•  continue;
  

• }
• // Track when we're inside double-quoted strings
• if (char === '"' && !inSingleQuote) {

•  inDoubleQuote = !inDoubleQuote;
  

•  result += char;
  

•  continue;
  

• }
• // Convert single quotes to double quotes (when not inside double quotes)
• if (char === "'" && !inDoubleQuote) {

•  // Check if this is a single quote inside a string value (like "It's")
  

•  // If we're already in a single-quoted string, toggle the state
  

•  if (inSingleQuote) {
  

•    // Closing single quote - convert to double quote
  

•    result += '"';
  

•    inSingleQuote = false;
  

•  } else {
  

•    // Opening single quote - convert to double quote
  

•    result += '"';
  

•    inSingleQuote = true;
  

•  }
  

•  continue;
  

• }
• result += char;
• }
• return result;
  +}

/**

• Clean AI response by extracting JSON content when response contains JSON
• Only processes responses that contain JSON structures { or [
  @@ -189,29 +257,36 @@ export function cleanSchemaResponse(response) {
  // Try with json language specifier
  const jsonBlockMatch = trimmed.match(/json\s*\n([\s\S]*?)\n/);
  if (jsonBlockMatch) {

• return jsonBlockMatch[1].trim();

• return normalizeJsonQuotes(jsonBlockMatch[1].trim());
  }

// Try any code block with JSON content
const anyBlockMatch = trimmed.match(/\s*\n([{\[][\s\S]*?[}\]])\s*/);
if (anyBlockMatch) {

• return anyBlockMatch[1].trim();

• return normalizeJsonQuotes(anyBlockMatch[1].trim());
  }

// Legacy patterns for more specific matching
const codeBlockPatterns = [
/json\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/,

• /\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/,
• /([{\[][\s\S]*?[}\]])/

• /\s*\n?([{\[][\s\S]*?[}\]])\s*\n?/
  ];

for (const pattern of codeBlockPatterns) {
const match = trimmed.match(pattern);
if (match) {

•  return match[1].trim();
  

•  return normalizeJsonQuotes(match[1].trim());
  

  }
  }

• // Single backtick pattern - ONLY if the entire input is just the code block

• // This prevents extracting inline code from within JSON objects (e.g., ['*'] from markdown text)

• const singleBacktickPattern = /^([{\[][\s\S]*?[}\]])$/;

• const singleBacktickMatch = trimmed.match(singleBacktickPattern);

• if (singleBacktickMatch) {

• return normalizeJsonQuotes(singleBacktickMatch[1].trim());

• }

• // Look for code block start followed immediately by JSON
  const codeBlockStartPattern = /```(?:json)?\s*\n?\s*([{[])/;
  const codeBlockMatch = trimmed.match(codeBlockStartPattern);
  @@ -236,7 +311,7 @@ export function cleanSchemaResponse(response) {
  }

  if (bracketCount === 0) {

•  return trimmed.substring(startIndex, endIndex);
  

•  return normalizeJsonQuotes(trimmed.substring(startIndex, endIndex));
  

  }
  }

@@ -261,7 +336,8 @@ export function cleanSchemaResponse(response) {
const isJsonArray = firstChar === '[' && lastChar === ']';

if (isJsonObject || isJsonArray) {

• return cleaned;

• // Normalize JavaScript syntax (single quotes) to valid JSON syntax (double quotes)
• return normalizeJsonQuotes(cleaned);
  }

return response; // Return original if no extractable JSON found

--- npm/tests/unit/single-quote-json-bug.test.js
@@ -0,0 +1,231 @@
+/**

• • Test for single-quote JSON bug
• • Bug Description:
• • When AI responses contain JavaScript array syntax with single quotes like ['*']
• • instead of valid JSON with double quotes ["*"], the JSON parser fails with:
• • "Unexpected token ''', "['*']" is not valid JSON"
• • This test replicates the bug seen in the debug logs where:
• • 1. AI returns response with JavaScript array syntax: ['*', '!bash']
• • 2. cleanSchemaResponse extracts it as-is (no syntax normalization)
• • 3. JSON.parse fails because single quotes are invalid in JSON
• */

+import { describe, test, expect } from '@jest/globals';
+import { cleanSchemaResponse, validateJsonResponse } from '../../src/agent/schemaUtils.js';
+
+describe('Single-quote JSON Bug', () => {

• describe('JavaScript array syntax with single quotes', () => {
• test('should fail to parse JavaScript array syntax with single quotes', () => {

•  // This is what the AI returns - JavaScript syntax, not JSON
  

•  const invalidJson = "['*']";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toBeDefined();
  

•  // Should contain error about single quote
  

•  expect(result.error.toLowerCase()).toMatch(/unexpected token|unexpected character|'|quote/);
  

• });
• test('should fail to parse array with single-quoted strings', () => {

•  // More complex example from the bug report
  

•  const invalidJson = "['*', '!bash']";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toBeDefined();
  

•  expect(result.errorContext).toBeDefined();
  

•  expect(result.errorContext.position).toBe(1); // Points to the first single quote
  

• });
• test('should succeed with double-quoted JSON array syntax', () => {

•  // This is what SHOULD be returned - valid JSON
  

•  const validJson = '["*"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should succeed with double-quoted JSON array with exclusions', () => {

•  // Valid JSON version of the bug example
  

•  const validJson = '["*", "!bash"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash']);
  

• });
• });
• describe('cleanSchemaResponse normalizes quote syntax (FIX)', () => {
• test('should not extract from javascript code blocks (only json/generic blocks)', () => {

•  // AI response with JavaScript array syntax in ```javascript block
  

•  const input = "```javascript\n['*', '!bash']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // cleanSchemaResponse doesn't extract from ```javascript blocks, only ```json and ```
  

•  // So it returns the original input unchanged
  

•  expect(cleaned).toBe(input);
  

•  // The uncleaned version with code block markers is not valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should extract JavaScript array from json code block AND normalize quotes', () => {

•  // Even in a ```json block, the AI might use single quotes
  

•  const input = "```json\n['*']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // FIX: Extraction happens AND quote normalization (single -> double quotes)
  

•  expect(cleaned).toBe('["*"]');
  

•  // Now it's valid JSON!
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should not modify valid JSON arrays with double quotes', () => {

•  const input = '```json\n["*", "!bash"]\n```';
  

•  const cleaned = cleanSchemaResponse(input);
  

•  expect(cleaned).toBe('["*", "!bash"]');
  

•  // This is valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash']);
  

• });
• test('should normalize single quotes in complex arrays', () => {

•  const input = "```json\n['*', '!bash', '!docker']\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // Should normalize all single quotes to double quotes
  

•  expect(cleaned).toBe('["*", "!bash", "!docker"]');
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*', '!bash', '!docker']);
  

• });
• test('should normalize single quotes in objects', () => {

•  const input = "```json\n{'key': 'value', 'num': 42}\n```";
  

•  const cleaned = cleanSchemaResponse(input);
  

•  // Should normalize object keys and string values
  

•  expect(cleaned).toBe('{"key": "value", "num": 42}');
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual({ key: 'value', num: 42 });
  

• });
• });
• describe('Real-world examples from bug report', () => {
• test('should replicate exact error from debug log line 9', () => {

•  // From log line 9: [DEBUG] JSON validation: Preview: ['*', '!bash']
  

•  const buggyResponse = "['*', '!bash']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  // From log line 10: Parse failed with error: Unexpected token ''', "['*', '!bash']" is not valid JSON
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toContain("Unexpected token");
  

• });
• test('should replicate exact error from debug log line 48', () => {

•  // From log line 48: [DEBUG] JSON validation: Preview: ['*']
  

•  const buggyResponse = "['*']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  // From log line 49: Parse failed with error: Unexpected token ''', "['*']" is not valid JSON
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.error).toContain("Unexpected token");
  

• });
• test('should show error context pointing to single quote', () => {

•  const buggyResponse = "['*']";
  

•  const result = validateJsonResponse(buggyResponse);
  

•  expect(result.isValid).toBe(false);
  

•  expect(result.errorContext).toBeDefined();
  

•  // Error should point to position 1 (the first single quote after '[')
  

•  expect(result.errorContext.position).toBe(1);
  

•  expect(result.errorContext.snippet).toContain("['*']");
  

• });
• });
• describe('Edge cases with mixed quotes', () => {
• test('should fail with mixed single and double quotes', () => {

•  const invalidJson = '["foo", \'bar\']';
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should fail with object using single quotes', () => {

•  const invalidJson = "{'key': 'value'}";
  

•  const result = validateJsonResponse(invalidJson);
  

•  expect(result.isValid).toBe(false);
  

• });
• test('should succeed with properly escaped single quotes inside double quotes', () => {

•  const validJson = '["It\'s valid"]';
  

•  const result = validateJsonResponse(validJson);
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(["It's valid"]);
  

• });
• });
• describe('Attempt completion context', () => {
• test('should succeed when attempt_completion result contains JavaScript array syntax (FIX)', () => {

•  // This simulates the exact scenario from the bug report where
  

•  // attempt_completion's result field contains ['*'] instead of ["*"]
  

•  const attemptCompletionResult = "['*']";
  

•  // First, cleanSchemaResponse is called (now it DOES normalize quotes - FIX!)
  

•  const cleaned = cleanSchemaResponse(attemptCompletionResult);
  

•  expect(cleaned).toBe('["*"]'); // Single quotes normalized to double quotes
  

•  // Then validateJsonResponse is called without schema
  

•  const result = validateJsonResponse(cleaned);
  

•  // With the fix, this should now succeed!
  

•  expect(result.isValid).toBe(true);
  

•  expect(result.parsed).toEqual(['*']);
  

• });
• test('should handle multi-line response with single-quote arrays', () => {

•  // AI might return explanation followed by array
  

•  const response = `Looking at the _parseAllowedTools method, when allowedTools is undefined,
  

+the system returns ['*'] which enables all tools.`;
+

•  // cleanSchemaResponse won't extract this (text before JSON)
  

•  const cleaned = cleanSchemaResponse(response);
  

•  expect(cleaned).toBe(response);
  

•  // Validation will fail because it's not valid JSON
  

•  const result = validateJsonResponse(cleaned);
  

•  expect(result.isValid).toBe(false);
  

• });
• });
  +});
  </full_diff>

<files_summary>

npm/src/agent/schemaUtils.js
modified
83
7


npm/tests/unit/single-quote-json-bug.test.js
added
231
0

</files_summary>
</pull_request>

### Raw AI Response
```json
[overview]
['*']

[external-label]
['*']

---

Powered by Visor from Probelabs

Last updated: 2025-11-16T17:32:45.545Z | Triggered by: opened | Commit: ed4eb93

💡 TIP: You can chat with Visor using /visor ask <your question>