tap-sparkpost

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls raw data from the Sparkpost API.

• Extracts the following resources:

  • Events

  • Webhooks

  • Templates

  • SendingDomains

  • TrackingDomains

  • SuppressionList

  • RecipientLists

  • Account

  • Usage

  • Subaccounts

  • IpPools

  • MetricsRecipientDomain

  • MetricsSendingIp

  • MetricsIpPool

  • MetricsSendingDomain

  • MetricsSubaccount

  • MetricsCampaign

  • MetricsTemplate

  • MetricsSubjectCampaign

  • MetricsWatchedDomain

  • MetricsMailboxProvider

  • MetricsMailboxProviderRegion

  • MetricsTimeSeries

  • MetricsBounceReason

  • MetricsBounceReasonByDomain

  • MetricsBounceClassification

  • MetricsRejectionReason

  • MetricsRejectionReasonByDomain

  • MetricsDelayReason

  • MetricsDelayReasonByDomain

  • MetricsEngagementDetails

  • MetricsDeliveriesByAttempt

• Outputs the schema for each resource

• Incrementally pulls data based on the input state

Streams

events

• Data Key = results
• Primary keys: ['event_id']
• Replication strategy: INCREMENTAL

webhooks

• Data Key = results
• Primary keys: ['id']
• Replication strategy: INCREMENTAL

templates

• Data Key = results
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

sending_domains

• Data Key = results
• Primary keys: ['domain']
• Replication strategy: FULL_TABLE

tracking_domains

• Data Key = results
• Primary keys: ['domain']
• Replication strategy: FULL_TABLE

suppression_list

• Data Key = results
• Primary keys: ['recipient']
• Replication strategy: FULL_TABLE

recipient_lists

• Data Key = results
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

account

• Data Key = results
• Primary keys: ['customer_id']
• Replication strategy: FULL_TABLE

usage

• Data Key = results
• Primary keys: ['timestamp']
• Replication strategy: FULL_TABLE

subaccounts

• Data Key = results
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

ip_pools

• Data Key = results
• Primary keys: ['id']
• Replication strategy: FULL_TABLE

metrics_recipient_domain

• Data Key = results
• Primary keys: ['timestamp', 'domain']
• Replication strategy: INCREMENTAL

metrics_sending_ip

• Data Key = results
• Primary keys: ['timestamp', 'sending_ip']
• Replication strategy: INCREMENTAL

metrics_ip_pool

• Data Key = results
• Primary keys: ['timestamp', 'ip_pool']
• Replication strategy: INCREMENTAL

metrics_sending_domain

• Data Key = results
• Primary keys: ['timestamp', 'sending_domain']
• Replication strategy: INCREMENTAL

metrics_subaccount

• Data Key = results
• Primary keys: ['timestamp', 'subaccount_id']
• Replication strategy: INCREMENTAL

metrics_campaign

• Data Key = results
• Primary keys: ['timestamp', 'campaign_id']
• Replication strategy: INCREMENTAL

metrics_template

• Data Key = results
• Primary keys: ['timestamp', 'template_id']
• Replication strategy: INCREMENTAL

metrics_subject_campaign

• Data Key = results
• Primary keys: ['timestamp', 'subject_campaign']
• Replication strategy: INCREMENTAL

metrics_watched_domain

• Data Key = results
• Primary keys: ['timestamp', 'watched_domain']
• Replication strategy: INCREMENTAL

metrics_mailbox_provider

• Data Key = results
• Primary keys: ['timestamp', 'mailbox_provider']
• Replication strategy: INCREMENTAL

metrics_mailbox_provider_region

• Data Key = results
• Primary keys: ['timestamp', 'mailbox_provider_region']
• Replication strategy: INCREMENTAL

metrics_time_series

• Data Key = results
• Primary keys: ['timestamp']
• Replication strategy: INCREMENTAL
• Supports precision parameter: Controls aggregation level (1min, 5min, 15min, hour, 12hr, day, week, month)

metrics_bounce_reason

• Data Key = results
• Primary keys: ['timestamp', 'reason', 'classification_id']
• Replication strategy: INCREMENTAL

metrics_bounce_reason_by_domain

• Data Key = results
• Primary keys: ['timestamp', 'reason', 'domain', 'classification_id']
• Replication strategy: INCREMENTAL

metrics_bounce_classification

• Data Key = results
• Primary keys: ['timestamp', 'classification_id']
• Replication strategy: INCREMENTAL

metrics_rejection_reason

• Data Key = results
• Primary keys: ['timestamp', 'reason', 'rejection_category_id']
• Replication strategy: INCREMENTAL

metrics_rejection_reason_by_domain

• Data Key = results
• Primary keys: ['timestamp', 'reason', 'domain', 'rejection_category_id']
• Replication strategy: INCREMENTAL

metrics_delay_reason

• Data Key = results
• Primary keys: ['timestamp', 'reason']
• Replication strategy: INCREMENTAL

metrics_delay_reason_by_domain

• Data Key = results
• Primary keys: ['timestamp', 'reason', 'domain']
• Replication strategy: INCREMENTAL

metrics_engagement_details

• Data Key = results
• Primary keys: ['timestamp', 'link_name']
• Replication strategy: INCREMENTAL

metrics_deliveries_by_attempt

• Data Key = results
• Primary keys: ['timestamp', 'attempt']
• Replication strategy: INCREMENTAL

Authentication

Quick Start

1. Install

   Clone this repository, and then install using setup.py. We recommend using a virtualenv:

   > virtualenv -p python3 venv
   > source venv/bin/activate
   > python setup.py install
   OR
   > cd .../tap-sparkpost
   > pip install -e .

2. Dependent libraries. The following dependent libraries were installed.

   > pip install singer-python
   > pip install target-stitch
   > pip install target-json
   

   • singer-tools
   • target-stitch

3. Create your tap's config.json file. The tap config file for this tap should include these entries:

   • api_key (string, required): Your SparkPost API key
   • start_date (string, required): The default value to use if no bookmark exists for an endpoint (rfc3339 date string). Example: "2019-01-01T00:00:00Z"
   • request_timeout (integer, optional): Max time in seconds for request to wait for response. Default: 300
   • precision (string, optional): Time-series metrics only. Controls aggregation level for metrics_time_series endpoint. Default: "day"

   Precision Parameter Values:

   The precision parameter is only applicable to the metrics_time_series stream. It controls how data is aggregated across time:

   • "1min": 1-minute aggregation - Returns metrics aggregated in 1-minute intervals
   • "5min": 5-minute aggregation - Returns metrics aggregated in 5-minute intervals
   • "15min": 15-minute aggregation - Returns metrics aggregated in 15-minute intervals
   • "hour": Hourly aggregation - Returns metrics aggregated in 1-hour intervals
   • "12hr": 12-hour aggregation - Returns metrics aggregated in 12-hour intervals
   • "day": Daily aggregation (default) - Returns metrics aggregated per day
   • "week": Weekly aggregation - Returns metrics aggregated per week
   • "month": Monthly aggregation - Returns metrics aggregated per month

   Important Notes:

   • Precision parameter is NOT supported by other metrics endpoints (metrics_recipient_domain, metrics_sending_ip, etc.)
   • Once a sync begins with a specific precision, do not change it during the sync to avoid mixed aggregation levels
   • Smaller precision values (1min, 5min) will return more granular data but may impact API performance
   • Reference: SparkPost Time-Series Metrics API

   Example config.json:

   {
       "api_key": "your_sparkpost_api_key_here",
       "start_date": "2019-01-01T00:00:00Z",
       "request_timeout": 300,
       "precision": "day"
   }
   

   Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

   {
       "currently_syncing": "dummy_stream1",
       "bookmarks": {
           "dummy_stream1": "2019-09-27T22:34:39.000000Z",
           "dummy_stream2": "2019-09-28T15:30:26.000000Z",
           "dummy_stream3": "2019-09-28T18:23:53Z"
       }
   }

4. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

   tap-sparkpost --config config.json --discover > catalog.json

   See the Singer docs on discovery mode here.

5. Run the Tap in Sync Mode (with catalog) and write out to state file

   For Sync mode:

   > tap-sparkpost --config tap_config.json --catalog catalog.json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To load to json files to verify outputs:

   > tap-sparkpost --config tap_config.json --catalog catalog.json | target-json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To pseudo-load to Stitch Import API with dry run:

   > tap-sparkpost --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

6. Test the Tap While developing the sparkpost tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

   > pylint tap_sparkpost -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-arguments

   Pylint test resulted in the following score:

   Your code has been rated at 9.67/10

   To check the tap and verify working:

   > tap_sparkpost --config tap_config.json --catalog catalog.json | singer-check-tap > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   Unit Tests

   Unit tests may be run with the following.

   python -m pytest --verbose
   

   Note, you may need to install test dependencies.

   pip install -e .'[dev]'
   

---

Copyright © 2019 Stitch