fix: return raw responses when requested even with 0 records

[aaronsteers]

Fix raw API responses bug and refactor for readability

Summary

Fixes a bug where execute_stream_test_read returned null for raw_api_responses when include_raw_responses_data: true was set, even when no records were found. This prevented users from debugging API response issues, particularly when dpath extractors were misconfigured.

Key changes:

• Bug fix: Ensure raw_api_responses always returns helpful data when requested, never null
• Enhanced debugging: Provide meaningful debug information when actual HTTP request/response data isn't captured by the CDK
• Code refactoring: Extract complex debugging logic into clean helper functions for better maintainability
• API behavior change: Set minimum values for max_pages_per_slice and max_slices to prevent CDK errors with max_records=0

Review & Testing Checklist for Human

Risk Level: 🟡 Medium - Changes core API behavior and return structure

• Test debugging scenarios: Verify that when include_raw_responses_data=true but no HTTP data is captured, the returned debugging information actually helps users understand and fix dpath extractor issues
• Validate API limits fix: Confirm that changing max(1, max_records) doesn't break legitimate use cases where max_records=0 should be supported, and that it properly fixes the "Record limit must be between 1 and 0" error
• End-to-end testing: Test both success and failure scenarios with include_raw_responses_data=true to ensure raw_api_responses is properly populated in all code paths
• Backward compatibility: Verify that existing consumers of execute_stream_test_read still work correctly with the modified return structure (now includes raw_api_responses in failure cases)

Notes

• The auxiliary_requests fallback logic is new behavior that may need monitoring in production
• Helper functions return debugging dictionaries instead of actual HTTP data when CDK doesn't capture slices - this is intentional but represents a change in data format
• Session requested by AJ Steers (@aaronsteers): https://app.devin.ai/sessions/98be4b20044e45f390a6036aa29f995b

[devin-ai-integration]

Original prompt from AJ Steers

Received message in Slack channel #ask-devin-ai:

@Devin - Can you investigate this bug? The raw responses were requested by the caller of the tool, but none were provided.
Thread URL: https://airbytehq-team.slack.com/archives/C08BHPUMEPJ/p1758566673352359?thread_ts=1758566673.352359

ATTACHMENT:"https://app.devin.ai/attachments/0f195a50-2163-4f0e-8556-3883d6e3ffa4/image.png"

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This Branch via MCP

To test the changes in this specific branch with an MCP client like Claude Desktop, use the following configuration:

{
  "mcpServers": {
    "connector-builder-mcp-dev": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/airbytehq/connector-builder-mcp.git@devin/1758568320-fix-raw-responses-bug", "connector-builder-mcp"]
    }
  }
}

Testing This Branch via CLI

You can test this version of the MCP Server using the following CLI snippet:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/connector-builder-mcp.git@devin/1758568320-fix-raw-responses-bug#egg=airbyte-connector-builder-mcp' --help

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poe <command> - Runs any poe command in the uv virtual environment
• /poe build-connector prompt="Star Wars API" - Run the connector builder using the Star Wars API.

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit 06974a3. ± Comparison against base commit b88e5d9.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit 06974a3. ± Comparison against base commit b88e5d9.

♻️ This comment has been updated with latest results.

[aaronsteers]

Devin, One comment inline. Also, your code is extremely ugly and hard to read. And I have a feeling you are still "doing too much". Move parsing into helper function (s) if you need to but please make this less ugly.

You added almost a hundred lines of code for what I expected in 2-10 lines