clean commit

Author: miketaluc-bytecode

CHANGELOG.md

@@ -0,0 +1,4 @@
+# Changelog
+
+## 0.3.1
+  * Fixes a memory issue when syncing contacts [#4](https://github.com/singer-io/tap-emarsys/pull/4)

LICENSE

@@ -0,0 +1,2 @@
+include LICENSE
+include tap_frontapp/schemas/*.json

MANIFEST.in

@@ -0,0 +1,46 @@
+# tap-frontapp
+
+This is a [Singer](https://singer.io) tap that produces JSON-formatted data following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
+
+This tap:
+
+- Pulls raw data from FrontApp's [API](https://dev.frontapp.com/)
+- Extracts the following resources from FrontApp
+  - [Analytics](https://dev.emarsys.com/v2/email-campaigns/list-email-campaigns)
+      - Hourly/Daily analytics of metrics
+          - team_table
+- Outputs the schema for each resource
+
+## Setup
+
+Building follows the conventional Singer setup:
+
+python3 ./setup.py clean
+python3 ./setup.py build
+python3 ./setup.py install
+
+## Configuration
+
+This tap requires a `config.json` which specifies details regarding [API authentication](https://dev.frontapp.com/#authentication), a cutoff date for syncing historical data, and a time period range [daily,hourly] to control what incremental extract date ranges are. See [config.sample.json](config.sample.json) for an example.
+
+Create the catalog:
+
+```bash
+› tap-frontapp --config config.json --discover > catalog.json
+```
+
+Then to run the extract:
+
+```bash
+› tap-frontapp --config config.json --catalog catalog.json --state state.json 
+```
+
+Note that a typical state file looks like this:
+
+```json
+{"bookmarks": {"team_table": {"date_to_resume": "2018-08-01 00:00:00"}}}
+```
+
+---
+
+Copyright © 2018 Stitch

README.md

@@ -0,0 +1,6 @@
+{
+  "token": "<myapitoken>",
+  "start_date": "2018-01-01T00:00:00Z",
+  "metric": "team_table",
+  "incremental_range": "daily"
+}

example.config.json

@@ -0,0 +1,2 @@
+[metadata]
+description-file = README.md

setup.cfg

@@ -0,0 +1,28 @@
+#!/usr/bin/env python
+
+from setuptools import setup, find_packages
+
+setup(
+    name="tap-frontapp",
+    version="0.3.1",
+    description="Singer.io tap for extracting data from the FrontApp API",
+    author="bytcode.io",
+    url="http://singer.io",
+    classifiers=["Programming Language :: Python :: 3 :: Only"],
+    install_requires=[
+        "singer-python>=5.1.1",
+        "pendulum",
+        "ratelimit",
+        "backoff",
+        "requests",
+    ],
+    entry_points="""
+    [console_scripts]
+    tap-frontapp=tap_frontapp:main
+    """,
+    packages=find_packages(),
+    package_data = {
+        "schemas": ["tap_frontapp/schemas/*.json"]
+    },
+    include_package_data=True
+)

setup.py

@@ -0,0 +1,57 @@
+# FrontApp
+
+This tap is for pulling [Analytics](https://dev.frontapp.com/#analytics) data from the FrontApp API. Its current developed scope is limited to the teams table, but it is easily expandable to the other Analytics data sets.
+
+## Connecting FrontApp
+
+### FrontApp Setup Requirements
+
+To set up FrontApp in Stitch, you need to get your JSON web token directly from Front (go to > Plugins & API > API).
+
+### Setup FrontApp as a Stitch source
+
+1. [Sign into your Stitch account](https://app.stitchdata.com/)
+
+2. On the Stitch Dashboard page, click the **Add Integration** button.
+
+3. Click the **FrontApp** icon.
+
+4. Enter a name for the integration. This is the name that will display on the Stitch Dashboard for the integration; it’ll also be used to create the schema in your destination. For example, the name "Stitch FrontApp" would create a schema called `stitch_frontapp` in the destination. **Note**: Schema names cannot be changed after you save the integration.
+
+5. In the **Token** field, enter your FrontApp web token.
+
+6. In the **Metric** field, enter the Analytics metric needed.  The only schema supported in this tap right now is the team_table metric.
+
+7. In the **Incremental Range** field, enter the desired aggregation frame (daily or hourly).
+
+8. In the **Start Date** field, enter the minimum, beginning start date for FrontApp Analytics (e.g. 2017-01-1).
+
+---
+
+## FrontApp Replication
+
+With each run of the integration, the following data set is extracted and replicated to the data warehouse:
+
+- **Team Table**: Daily or hourly aggregated team member statistics since the last_update (last completed run of the integration) through the most recent day or hour respectively. On the first run, ALL increments since the **Start Date** will be replicated.
+
+---
+
+## FrontApp Table Schemas
+
+### team_table
+
+- Table name: team_table 
+- Description: A list of team members and their event statistics during the course of the day/hour starting from the analytics_date.
+- Primary key: analytics_date, analytics_range, teammate_id
+- Replicated incrementally
+- Bookmark column: analytics_date (written as resume_date in the state records)
+- API endpoint documentation: [Analytics](https://dev.frontapp.com/#analytics)
+
+---
+
+## Troubleshooting / Other Important Info
+
+- **Team_table Data**: The first record is for the teammate = "ALL" and so is an aggregated record across all team members.  Also, the API supports pulling specific teams by using a slightly different endpoint, but we have set it up to pull members from all teams.
+
+- **Timestamps**: All timestamp columns and resume_date state parameter are Unix timestamps.
+

stitch_setup_documentation.md

@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+import json
+
+import singer
+from singer import utils
+from singer.catalog import Catalog, CatalogEntry, Schema
+from . import streams
+from .context import Context
+from . import schemas
+
+REQUIRED_CONFIG_KEYS = ["token", "metric"]
+
+LOGGER = singer.get_logger()
+
+#def check_authorization(atx):
+#    atx.client.get('/settings')
+
+
+# with tap-emarsys, they do it this way where the catalog is read in from a call to the api
+#  but with the odd frontapp structure, we won't do that here
+# we never use atx in here since the schema is from file
+#  but we would use it if we pulled schema from the API
+# def discover(atx):
+def discover():
+    catalog = Catalog([])
+    for tap_stream_id in schemas.STATIC_SCHEMA_STREAM_IDS:
+        #print("tap stream id=",tap_stream_id)
+        schema = Schema.from_dict(schemas.load_schema(tap_stream_id))
+        metadata = []
+        if schema.selected is True:
+            metadata.append({
+                'metadata': {
+                    'selected': True
+                },
+                'breadcrumb': []
+            })
+        for field_name in schema.properties.keys():
+            #print("field name=",field_name)
+            if field_name in schemas.PK_FIELDS[tap_stream_id]:
+                inclusion = 'automatic'
+            else:
+                inclusion = 'available'
+            metadata.append({
+                'metadata': {
+                    'inclusion': inclusion
+                },
+                'breadcrumb': ['properties', field_name]
+            })
+        catalog.streams.append(CatalogEntry(
+            stream=tap_stream_id,
+            tap_stream_id=tap_stream_id,
+            key_properties=schemas.PK_FIELDS[tap_stream_id],
+            schema=schema,
+            metadata=metadata
+        ))
+    return catalog
+
+
+# this is already defined in schemas.py though w/o dependencies.  do we keep this for the sync?
+def load_schema(tap_stream_id):
+    path = "schemas/{}.json".format(tap_stream_id)
+    schema = utils.load_json(get_abs_path(path))
+    dependencies = schema.pop("tap_schema_dependencies", [])
+    refs = {}
+    for sub_stream_id in dependencies:
+        refs[sub_stream_id] = load_schema(sub_stream_id)
+    if refs:
+        singer.resolve_schema_references(schema, refs)
+    return schema
+
+
+def sync(atx):
+    for tap_stream_id in schemas.STATIC_SCHEMA_STREAM_IDS:
+        schemas.load_and_write_schema(tap_stream_id)
+
+    streams.sync_selected_streams(atx)
+
+
+@utils.handle_top_exception(LOGGER)
+def main():
+    args = utils.parse_args(REQUIRED_CONFIG_KEYS)
+    atx = Context(args.config, args.state)
+    if args.discover:
+        # the schema is static from file so we don't need to pass in atx for connection info.
+        catalog = discover()
+        json.dump(catalog.to_dict(), sys.stdout)
+    else:
+        atx.catalog = Catalog.from_dict(args.properties) \
+            if args.properties else discover()
+        sync(atx)
+
+if __name__ == "__main__":
+    main()

tap_frontapp/__init__.py

@@ -0,0 +1,59 @@
+from datetime import datetime, date
+
+import singer
+from singer import bookmarks as bks_, metadata
+
+from .http import Client
+
+class Context(object):
+    """Represents a collection of global objects necessary for performing
+    discovery or for running syncs. Notably, it contains
+
+    - config  - The JSON structure from the config.json argument
+    - state   - The mutable state dict that is shared among streams
+    - client  - An HTTP client object for interacting with the API
+    - catalog - A singer.catalog.Catalog. Note this will be None during
+                discovery.
+    """
+    def __init__(self, config, state):
+        self.config = config
+        self.state = state
+        self.client = Client(config)
+        self._catalog = None
+        self.selected_stream_ids = None
+        self.now = datetime.utcnow()
+
+    @property
+    def catalog(self):
+        return self._catalog
+
+    @catalog.setter
+    def catalog(self, catalog):
+        self._catalog = catalog
+        self.selected_stream_ids = set()
+        for stream in catalog.streams:
+            mdata = metadata.to_map(stream.metadata)
+            root_metadata = mdata.get(())
+            if root_metadata and root_metadata.get('selected') is True:
+                self.selected_stream_ids.add(stream.tap_stream_id)
+
+    def get_bookmark(self, path):
+        return bks_.get_bookmark(self.state, *path)
+
+    def set_bookmark(self, path, val):
+        if isinstance(val, date):
+            val = val.isoformat()
+        bks_.write_bookmark(self.state, path[0], path[1], val)
+
+    def get_offset(self, path):
+        off = bks_.get_offset(self.state, path[0])
+        return (off or {}).get(path[1])
+
+    def set_offset(self, path, val):
+        bks_.set_offset(self.state, path[0], path[1], val)
+
+    def clear_offsets(self, tap_stream_id):
+        bks_.clear_offset(self.state, tap_stream_id)
+
+    def write_state(self):
+        singer.write_state(self.state)