JSON strings with quotes cause escaping issues

[konard]

🐛 Bug Description

JSON strings containing quotes and special characters get corrupted when passed through command-stream's shell interpolation. This makes it impossible to reliably work with JSON data using shell commands, affecting configuration files, API responses, and data processing scripts.

🔴 Impact

• Configuration management - Can't write JSON config files reliably
• Data processing - JSON data gets corrupted during shell operations
• API integrations - Can't pass JSON payloads through shell commands
• Build scripts - Package.json and similar files can't be manipulated

📝 Detailed Problem Analysis

JSON inherently contains quotes for string values, which conflict with shell quoting mechanisms:

1. Nested quotes in JSON values break shell parsing
2. Escaped quotes get double-escaped or mangled
3. Special characters like backticks and dollar signs cause additional issues
4. JSON structure can be broken, making it unparseable

🔄 Reproduction

const jsonData = {
  name: "Test Project",
  description: "A project with \\"quotes\\" and 'apostrophes'",
  scripts: {
    test: "echo \\"Running tests\\"",
    build: "node build.js --env='production'"
  },
  config: {
    special: "Value with \`backticks\` and $variables"
  }
};

const jsonString = JSON.stringify(jsonData, null, 2);

// ❌ This corrupts the JSON
await $\`echo '\${jsonString}' > package.json\`;

// JSON becomes invalid and can't be parsed back

✅ Expected Behavior

JSON should be written exactly as formatted, maintaining valid JSON syntax.

❌ Actual Behavior

• JSON structure gets corrupted
• Quotes are escaped incorrectly
• Resulting file contains invalid JSON
• JSON.parse() fails on the output

🔧 Workaround

// ✅ Use fs.writeFile for JSON data
await fs.writeFile('package.json', JSON.stringify(data, null, 2));

🎯 Root Cause

Shell interpretation of quotes within JSON strings creates multiple layers of escaping that conflict with each other.

🔗 References

• Full test case: https://github.com/deep-assistant/hive-mind/blob/main/command-stream-issues/issue-03-json-escaping.mjs

[konard]

🎉 Issue Fixed!

This issue has been resolved in PR #109.

📋 Summary of Fix

The problem was that JSON strings with nested quotes were being corrupted when passed through command-stream's shell interpolation. The fix involved:

1. Enhanced shell operator detection to recognize redirection operators like >, >>, etc.
2. Improved shell execution logic to force JSON commands with redirection to use real shell instead of virtual commands
3. Added bypass logic to prevent virtual echo from mangling JSON strings with redirection

✅ What Now Works

const jsonData = {
  name: "Test Project",
  description: "A project with \"quotes\" and 'apostrophes'",
  scripts: {
    test: "echo \"Running tests\"",
    build: "node build.js --env='production'"
  },
  config: {
    special: "Value with `backticks` and $variables"
  }
};

// This now works correctly! ✅
await $`echo ${JSON.stringify(jsonData, null, 2)} > package.json`;

The JSON is written exactly as formatted, maintaining valid JSON syntax that can be parsed back without corruption.

🧪 Testing

The fix includes comprehensive test coverage and has been verified to work with:

• ✅ Complex JSON with nested quotes and special characters
• ✅ File redirection scenarios
• ✅ Configuration management use cases
• ✅ API integration scenarios
• ✅ Build script operations

All existing functionality is preserved with no regressions.

🚀 Availability

The fix will be available in command-stream v0.7.2 once the PR is merged.

Thank you for reporting this issue! 🙏

[konard]

Additional Use Case: macOS Keychain Credentials Storage

I've encountered this same issue in a project that needs to store JSON credentials in macOS keychain. The issue affects the @deep-assistant/claude-profiles package where we're trying to use command-stream for a security command:

const credentials = { claudeAiOauth: { accessToken: "token", refreshToken: "refresh" } };
const jsonStr = JSON.stringify(credentials);
const escapedJson = jsonStr.replace(/"/g, '\\"');

// ❌ This fails due to the automatic quoting issue
await $`security add-generic-password -U -a $USER -s "Claude Code-credentials" -w "${escapedJson}"`;
// Results in keychain storing: '{"claudeAiOauth":{...}}'
// Instead of: {"claudeAiOauth":{...}}

Impact: This makes the stored JSON invalid for parsing by Claude Code, breaking credential restoration.

Current Workaround: Using Node.js execSync instead of command-stream for this operation.

Reproduction: I've created a minimal test case at: https://github.com/deep-assistant/claude-profiles/blob/issue-8-51a18762/examples/command-stream-json-issue-reproduction.mjs

This affects any shell command that needs to pass JSON data as arguments, particularly for system integration commands like keychain management, configuration file manipulation, and API calls requiring JSON payloads.