Skip to content

Bug: Auto-enable for raw_api_responses not working when zero records extracted #125

@devin-ai-integration

Description

@devin-ai-integration

Description

The auto-enable feature for raw_api_responses in execute_stream_test_read is not working as documented when zero records are extracted due to incorrect dpath configuration.

Expected Behavior

According to the code comment at line 437:

Toggle to include_raw_responses=True if we had an error or if we are returning no records

When include_raw_responses_data is set to false or omitted, it should automatically be enabled when zero records are returned, providing developers with debugging information.

Actual Behavior

The auto-enable only triggers when success=False, not when zero records are returned. The implementation at line 438 is:

include_raw_responses_data = include_raw_responses_data or not success

This does NOT check for zero records (len(records_data) == 0).

Impact

When a developer has an incorrect dpath configuration:

  1. API successfully returns data (HTTP 200 with slices)
  2. Wrong dpath extracts 0 records from the response
  3. success remains True (because slices exist)
  4. Auto-enable does NOT trigger
  5. Developer gets "raw_api_responses": null with no debugging information

This breaks a primary use case for raw responses: debugging dpath issues.

Reproduction

# Create manifest with wrong dpath
poe test-tool execute_stream_test_read '{
  "manifest": "@wrong_dpath_manifest",
  "stream_name": "users",
  "config": {},
  "max_records": 2,
  "include_raw_responses_data": false
}'

Result: Returns "records_read": 0 but "raw_api_responses": null

Root Cause

The success flag is only set to False when:

  1. No test read response record is returned (line 418)
  2. No slices are returned (line 423)

But when the API call succeeds and returns slices, success stays True even if dpath extraction results in zero records.

Proposed Fix

Update line 438 to also check for zero records:

include_raw_responses_data = include_raw_responses_data or not success or len(records_data) == 0

This would:

Context

This bug was discovered during verification testing for #121. See detailed analysis in this comment.

Environment

  • Repository: airbytehq/connector-builder-mcp
  • File: connector_builder_mcp/validation_testing.py
  • Function: execute_stream_test_read()
  • Lines: 437-438

Requested by: @aaronsteers

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions