Skip to content

Incompatible State Message Format Handling in tap-airbyte-wrapper for Incremental Extraction #17

@JichaoS

Description

@JichaoS

Description:
Certain airbyte taps are incompatible with incremental extraction with tap-airbyte-wrapper

For incremental to work, airbyte is expecting a state message in this form, a list of AirbyteStateMessage

  [
    {
      "type": "STREAM",
      "stream": {
        "stream_descriptor": {
          "name": "events",
          "namespace": null
        },
        "stream_state": {
          "server_upload_time": "2024-05-13 19:35:13.833000"
        }
      }
    }
  ]

The internal dict above is actually what gets sent by airbyte in a state message. But the tap-airbyte-wrapper unpack this and saves the unpacked version in the state table, which then gets feds back without modifications to airbyte binary as state.json. This prevents certain airbyte extractors from not working. (For example intercom’s one works totally fine, while amplitude does not)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions