Gl master - #1
Open
mukeshbhatt18gl wants to merge 28 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces (or largely rebuilds) the tap-qualtrics Singer tap implementation: an OAuth2-enabled Qualtrics API client, discovery/catalog generation (including access checks and dynamic-schema streams), stream implementations with JSON schemas, and a comprehensive test/CI setup.
Changes:
- Added core tap runtime (
client.py,discover.py,sync.py) and stream implementations (including dynamic-schema streams likeaudit_exportandsurvey_response_export). - Added static JSON schemas for streams under
tap_qualtrics/schemas/plus schema inference utilities. - Added unit tests and tap-tester integration tests, plus pre-commit and CircleCI configuration.
Reviewed changes
Copilot reviewed 131 out of 132 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unittests/test_sync.py | Unit tests for sync flow and currently_syncing handling |
| tests/unittests/test_sync_extended.py | Additional edge-case coverage for sync/write_schema/dynamic entries |
| tests/unittests/test_streams.py | Unit tests for pagination and selected stream behaviors |
| tests/unittests/test_schema.py | Unit tests for schema inference/loading utilities |
| tests/unittests/test_parent_child_bookmark.py | Tests for parent→child sync propagation behavior |
| tests/unittests/test_incremental_sync.py | Unit tests for incremental bookmark handling |
| tests/unittests/test_distributions_stream.py | Unit tests for distributions stream + child behaviors |
| tests/unittests/test_discovery.py | Unit tests for discovery access-check pruning and checks |
| tests/unittests/test_client.py | Unit tests for HTTP client behavior/error mapping |
| tests/unittests/test_client_extended.py | Extended unit tests for OAuth/cache/poll/backoff paths |
| tests/unittests/test_audit_events_stream.py | Unit tests for audit events pagination and sync |
| tests/test_start_date.py | Tap-tester start-date integration test |
| tests/test_pagination.py | Tap-tester pagination integration test |
| tests/test_interrupted_sync.py | Tap-tester interrupted sync integration test |
| tests/test_discovery.py | Tap-tester discovery integration test |
| tests/test_bookmark.py | Custom tap-tester bookmark integration test |
| tests/test_automatic_fields.py | Tap-tester automatic-fields integration test |
| tests/test_all_fields.py | Tap-tester all-fields integration test |
| tap_qualtrics/sync.py | Main sync orchestration + dynamic-schema handling |
| tap_qualtrics/streams/whatsapp_distributions.py | New WhatsApp distributions stream |
| tap_qualtrics/streams/users.py | Users parent stream definition |
| tap_qualtrics/streams/user.py | User detail child stream definition |
| tap_qualtrics/streams/transaction_batches.py | Transaction batches directory child stream |
| tap_qualtrics/streams/tickets.py | Tickets stream + child registration |
| tap_qualtrics/streams/ticket_teams.py | Ticket teams stream definition |
| tap_qualtrics/streams/ticket_statuses.py | Ticket statuses stream definition |
| tap_qualtrics/streams/ticket_root_causes.py | Ticket root causes child stream definition |
| tap_qualtrics/streams/ticket_retrieve_events.py | Ticket events child stream definition |
| tap_qualtrics/streams/ticket_groups.py | Ticket groups stream definition |
| tap_qualtrics/streams/surveys.py | Surveys incremental parent stream definition |
| tap_qualtrics/streams/survey.py | Survey detail child stream definition |
| tap_qualtrics/streams/survey_response_export.py | Dynamic-schema survey response export stream |
| tap_qualtrics/streams/survey_quotas.py | Survey quotas child stream definition |
| tap_qualtrics/streams/sms_distributions.py | SMS distributions incremental child stream definition |
| tap_qualtrics/streams/segments.py | Segments incremental directory child stream definition |
| tap_qualtrics/streams/segment_contacts.py | Segment contacts child stream definition |
| tap_qualtrics/streams/samples.py | Samples directory child stream definition |
| tap_qualtrics/streams/sample_definitions.py | Sample definitions directory child stream definition |
| tap_qualtrics/streams/sample_contacts.py | Sample contacts child stream definition |
| tap_qualtrics/streams/opted_out_contacts.py | Opted out contacts directory child stream |
| tap_qualtrics/streams/mailing_lists.py | Mailing lists incremental directory child stream |
| tap_qualtrics/streams/mailing_list_opted_out_contacts.py | Mailing list opted-out contacts child stream |
| tap_qualtrics/streams/mailing_list_contacts.py | Mailing list contacts child stream |
| tap_qualtrics/streams/mailing_list_bounced_contacts.py | Mailing list bounced contacts child stream |
| tap_qualtrics/streams/library_surveys.py | Library surveys child stream definition |
| tap_qualtrics/streams/library_blocks.py | Library blocks child stream definition |
| tap_qualtrics/streams/libraries.py | Libraries parent stream definition |
| tap_qualtrics/streams/libraries_survey_questions.py | Library survey questions child stream definition |
| tap_qualtrics/streams/libraries_messages.py | Library messages child stream definition |
| tap_qualtrics/streams/groups.py | Groups parent stream definition |
| tap_qualtrics/streams/group_users.py | Group users child stream definition |
| tap_qualtrics/streams/event_subscriptions.py | Event subscriptions stream definition |
| tap_qualtrics/streams/erasure_requests.py | Erasure requests incremental stream definition |
| tap_qualtrics/streams/distributions.py | Distributions incremental child stream definition |
| tap_qualtrics/streams/distribution_links.py | Distribution links child stream definition |
| tap_qualtrics/streams/distribution_history.py | Distribution history child stream definition |
| tap_qualtrics/streams/directories.py | Directories parent stream definition |
| tap_qualtrics/streams/directories_contacts.py | Directory contacts child stream definition |
| tap_qualtrics/streams/contact_transactions.py | Contact transactions child stream definition |
| tap_qualtrics/streams/contact_frequency_rules.py | Contact frequency rules child stream definition |
| tap_qualtrics/streams/audit_export.py | Dynamic-schema audit export stream |
| tap_qualtrics/streams/audit_export_event_types.py | Audit export event types parent stream |
| tap_qualtrics/streams/audit_events.py | Audit events child stream |
| tap_qualtrics/streams/audit_events_types.py | Audit event types parent stream |
| tap_qualtrics/streams/init.py | Central STREAMS registry wiring all streams |
| tap_qualtrics/schemas/whatsapp_distributions.json | Static schema for whatsapp_distributions |
| tap_qualtrics/schemas/users.json | Static schema for users |
| tap_qualtrics/schemas/user.json | Static schema for user |
| tap_qualtrics/schemas/transaction_batches.json | Static schema for transaction_batches |
| tap_qualtrics/schemas/tickets.json | Static schema for tickets |
| tap_qualtrics/schemas/ticket_teams.json | Static schema for ticket_teams |
| tap_qualtrics/schemas/ticket_statuses.json | Static schema for ticket_statuses |
| tap_qualtrics/schemas/ticket_root_causes.json | Static schema for ticket_root_causes |
| tap_qualtrics/schemas/ticket_retrieve_events.json | Static schema for ticket_retrieve_events |
| tap_qualtrics/schemas/ticket_groups.json | Static schema for ticket_groups |
| tap_qualtrics/schemas/surveys.json | Static schema for surveys |
| tap_qualtrics/schemas/survey.json | Static schema for survey |
| tap_qualtrics/schemas/survey_quotas.json | Static schema for survey_quotas |
| tap_qualtrics/schemas/survey_questions.json | Static schema for survey_questions |
| tap_qualtrics/schemas/survey_definitions.json | Static schema for survey_definitions |
| tap_qualtrics/schemas/sms_distributions.json | Static schema for sms_distributions |
| tap_qualtrics/schemas/segments.json | Static schema for segments |
| tap_qualtrics/schemas/segment_contacts.json | Static schema for segment_contacts |
| tap_qualtrics/schemas/samples.json | Static schema for samples |
| tap_qualtrics/schemas/sample_definitions.json | Static schema for sample_definitions |
| tap_qualtrics/schemas/sample_contacts.json | Static schema for sample_contacts |
| tap_qualtrics/schemas/opted_out_contacts.json | Static schema for opted_out_contacts |
| tap_qualtrics/schemas/mailing_lists.json | Static schema for mailing_lists |
| tap_qualtrics/schemas/mailing_list_opted_out_contacts.json | Static schema for mailing_list_opted_out_contacts |
| tap_qualtrics/schemas/mailing_list_contacts.json | Static schema for mailing_list_contacts |
| tap_qualtrics/schemas/mailing_list_bounced_contacts.json | Static schema for mailing_list_bounced_contacts |
| tap_qualtrics/schemas/library_surveys.json | Static schema for library_surveys |
| tap_qualtrics/schemas/library_messages.json | Static schema for library_messages |
| tap_qualtrics/schemas/library_blocks.json | Static schema for library_blocks |
| tap_qualtrics/schemas/libraries.json | Static schema for libraries |
| tap_qualtrics/schemas/libraries_survey_questions.json | Static schema for libraries_survey_questions |
| tap_qualtrics/schemas/groups.json | Static schema for groups |
| tap_qualtrics/schemas/group_users.json | Static schema for group_users |
| tap_qualtrics/schemas/event_subscriptions.json | Static schema for event_subscriptions |
| tap_qualtrics/schemas/erasure_requests.json | Static schema for erasure_requests |
| tap_qualtrics/schemas/distributions.json | Static schema for distributions |
| tap_qualtrics/schemas/distribution_links.json | Static schema for distribution_links |
| tap_qualtrics/schemas/distribution_history.json | Static schema for distribution_history |
| tap_qualtrics/schemas/directories.json | Static schema for directories |
| tap_qualtrics/schemas/directories_contacts.json | Static schema for directories_contacts |
| tap_qualtrics/schemas/contact_transactions.json | Static schema for contact_transactions |
| tap_qualtrics/schemas/contact_frequency_rules.json | Static schema for contact_frequency_rules |
| tap_qualtrics/schemas/audit_export_event_types.json | Static schema for audit_export_event_types |
| tap_qualtrics/schemas/audit_events.json | Static schema for audit_events |
| tap_qualtrics/schemas/audit_events_types.json | Static schema for audit_events_types |
| tap_qualtrics/schema.py | Schema loading + inference helpers |
| tap_qualtrics/exceptions.py | Qualtrics exception hierarchy + status mapping |
| tap_qualtrics/discover.py | Discovery implementation + access checks + dynamic entries |
| tap_qualtrics/client.py | OAuth2 client + retry/backoff + polling |
| tap_qualtrics/init.py | Tap entrypoint wiring discover/sync |
| spike/tap-qualtrics.json | Spike/reference spec-style definition for streams/config |
| setup.py | Packaging configuration |
| sample_config.json | Example config for running the tap |
| CHANGELOG.md | Initial changelog entry |
| .pre-commit-config.yaml | Pre-commit hooks (black/flake8/bandit/etc.) |
| .gitignore | Git ignore patterns |
| .github/copilot-instructions.md | Repo contribution guidelines for Singer taps |
| .circleci/config.yml | CI pipeline: lint, unit tests, coverage, integration tests |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+24
to
+31
| def manipulate_state(self): | ||
| return { | ||
| "currently_syncing": "surveys", | ||
| "bookmarks": { | ||
| "surveys": { "lastModified" : "2026-08-19T00:00:00Z"}, | ||
| "mailing_lists": {"lastModifiedDate": "2026-08-19T00:00:00.00Z"} | ||
| } | ||
| } |
Comment on lines
+13
to
+19
| distribution_id = (parent_id or {}).get("id") or parent_id | ||
| survey_id = (parent_id or {}).get("_survey_id", "") | ||
| if not distribution_id: | ||
| return | ||
| params = {"pageSize": self.page_size} | ||
| if survey_id: | ||
| params["surveyId"] = survey_id |
Comment on lines
+4
to
+6
| class DirectoryContacts(DirectoryChildStream): | ||
| """Contacts in a directory (page size 500); has contact_transactions child.""" | ||
| tap_stream_id = "directories_contacts" |
Comment on lines
+2
to
+18
| from typing import Dict | ||
| from singer import metadata | ||
| from tap_qualtrics.streams import STREAMS | ||
| from tap_qualtrics.client import Client | ||
|
|
||
| LOGGER = singer.get_logger() | ||
|
|
||
| # Base tap_stream_ids whose catalog entries are named <base>__<parent_key> at runtime. | ||
| _DYNAMIC_SCHEMA_BASES = {"audit_export", "survey_response_export"} | ||
|
|
||
|
|
||
| def update_currently_syncing(state: Dict, stream_name: str) -> None: | ||
| if not stream_name and singer.get_currently_syncing(state): | ||
| del state["currently_syncing"] | ||
| else: | ||
| singer.set_currently_syncing(state, stream_name) | ||
| singer.write_state(state) |
Comment on lines
+1
to
+3
| import unittest | ||
| from unittest.mock import patch, MagicMock, call | ||
| from tap_qualtrics.sync import sync, update_currently_syncing |
Comment on lines
+1
to
+5
| """Unit tests for stream pagination and async export flows.""" | ||
| import json | ||
| import unittest | ||
| from unittest.mock import MagicMock, patch, call | ||
|
|
Comment on lines
+2
to
+5
| import unittest | ||
| from unittest.mock import MagicMock, patch | ||
| from tap_qualtrics.streams.abstracts import FullTableStream, IncrementalStream | ||
|
|
Comment on lines
+4
to
+8
| from parameterized import parameterized | ||
| from requests.exceptions import Timeout, ConnectionError, ChunkedEncodingError | ||
| from tap_qualtrics.client import Client | ||
| from tap_qualtrics.exceptions import * | ||
|
|
Comment on lines
+11
to
+13
| classifiers=["Programming Language :: Python :: 3 :: Only"], | ||
| py_modules=["tap_qualtrics"], | ||
| install_requires=[ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of change
SAC-28585
Manual QA steps
Risks
Rollback steps