Skip to content

Commit af1d05e

Browse files
atttiwariatttiwari
authored
Revert "GL-Work (#242)" (#243)
This reverts commit 313e366.
1 parent 313e366 commit af1d05e

File tree

86 files changed

+79
-5732
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

86 files changed

+79
-5732
lines changed

‎CHANGELOG.md‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
11
# Changelog
22

3-
### 3.8.0
4-
* Introduced new streams [#242](https://github.com/singer-io/tap-shopify/pull/242)
5-
63
### 3.7.2
74
* New automatic_keys Support and Bulk State Persistence Logic [#237](https://github.com/singer-io/tap-shopify/pull/237)
85

‎README.md‎

Lines changed: 22 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
# tap-shopify
22

33
This is a [Singer](https://singer.io) tap that produces JSON-formatted data
4-
following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
4+
following the [Singer
5+
spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
56

67
This tap:
78

@@ -23,96 +24,34 @@ This tap:
2324
- [Locations](https://shopify.dev/docs/api/admin-graphql/latest/queries/locations)
2425
- [Inventory Levels](https://shopify.dev/docs/api/admin-graphql/latest/queries/inventorylevel)
2526
- [Inventory Item](https://shopify.dev/docs/api/admin-graphql/latest/queries/inventoryitems)
26-
- [Application Charges](https://shopify.dev/docs/api/admin-graphql/latest/queries/currentAppInstallation?example=retrieves-a-list-of-application-charges)
27-
- [Application Credits](https://shopify.dev/docs/api/admin-graphql/latest/queries/appInstallation?example=retrieves-all-application-credits)
28-
- [Article Authors](https://shopify.dev/docs/api/admin-rest/latest/resources/article#get-articles-authors)
29-
- [Article Tags](https://shopify.dev/docs/api/admin-rest/latest/resources/article#get-articles-tags)
30-
- [Blogs](https://shopify.dev/docs/api/admin-graphql/latest/queries/blogs?example=retrieve-a-list-of-all-blogs)
31-
- [Carrier Services](https://shopify.dev/docs/api/admin-rest/latest/resources/carrierservice#get-carrier-services)
32-
- [Comments](https://shopify.dev/docs/api/admin-graphql/latest/queries/comments?example=retrieves-a-list-of-comments)
33-
- [Currencies](https://shopify.dev/docs/api/admin-graphql/latest/queries/shop?example=retrieves-a-list-of-currencies-enabled-on-a-shop)
34-
- [Custom Collections](https://shopify.dev/docs/api/admin-graphql/latest/queries/collections?example=retrieve-a-list-of-custom-collections)
35-
- [Draft Orders](https://shopify.dev/docs/api/admin-graphql/latest/queries/draftOrders?example=retrieves-a-list-of-draft-orders)
36-
- [Fulfillment Services](https://shopify.dev/docs/api/admin-graphql/latest/queries/shop?example=receive-a-list-of-all-fulfillmentservices)
37-
- [Marketing Events](https://shopify.dev/docs/api/admin-graphql/latest/queries/marketingEvents?example=retrieves-a-list-of-all-marketing-events)
38-
- [Pages](https://shopify.dev/docs/api/admin-graphql/latest/queries/pages?example=retrieves-a-list-of-pages)
39-
- [Policies](https://shopify.dev/docs/api/admin-graphql/latest/queries/shop?example=retrieves-a-list-of-the-shops-policies)
40-
- [Price Rules](https://shopify.dev/docs/api/admin-graphql/latest/queries/discountNodes)
41-
- [Recurring Application Charges](https://shopify.dev/docs/api/admin-graphql/latest/queries/currentAppInstallationexample=retrieves-a-list-of-recurring-application-charges)
42-
- [Redirects](https://shopify.dev/docs/api/admin-graphql/latest/queries/urlRedirects?example=retrieves-a-list-of-url-redirects)
43-
- [Resource Feedback](https://shopify.dev/docs/api/admin-graphql/latest/queries/app?example=get-the-feedback-field)
44-
- [Script Tags](https://shopify.dev/docs/api/admin-graphql/latest/queries/scriptTags?example=retrieves-a-list-of-all-script-tags)
45-
- [Shipping Zones](https://shopify.dev/docs/api/admin-graphql/latest/queries/deliveryProfiles?example=receive-a-list-of-all-shippingzones)
46-
- [Shop](https://shopify.dev/docs/api/admin-rest/latest/resources/shop#get-shop)
47-
- [Smart Collections](https://shopify.dev/docs/api/admin-graphql/latest/queries/collections?example=retrieve-a-list-of-smart-collections)
48-
- [Tender Transactions](https://shopify.dev/docs/api/admin-graphql/latest/queries/tenderTransactions?example=retrieves-a-list-of-tender-transactions)
49-
- [Themes](https://shopify.dev/docs/api/admin-graphql/latest/queries/themes?example=retrieves-a-list-of-themes)
50-
- [Webhooks](https://shopify.dev/docs/api/admin-graphql/latest/queries/webhookSubscriptions)
5127
- Outputs the schema for each resource
5228
- Incrementally pulls data based on the input state
5329

5430
## Stream Details
5531

56-
| Stream Name | Replication Key | Key Properties|
57-
|------------------------|--------------------------|---------------|
58-
| abandoned_checkouts | updatedAt | id |
59-
| collections | updatedAt | id |
60-
| customers | updatedAt | id |
61-
| events | createdAt | id |
62-
| inventory_items | updatedAt | id |
63-
| inventory_levels | updatedAt | id |
64-
| locations | createdAt | id |
65-
| metafields_collections | updatedAt | id |
66-
| metafields_customers | updatedAt | id |
67-
| metafields_orders | updatedAt | id |
68-
| metafields_products | updatedAt | id |
69-
| order_refunds | updatedAt | id |
70-
| orders | updatedAt | id |
71-
| order_shipping_lines | updatedAt | id |
72-
| fulfillment_orders | updatedAt | id |
73-
| product_variants | updatedAt | id |
74-
| products | updatedAt | id |
75-
| transactions | createdAt | id |
76-
| blogs | updatedAt | id |
77-
| comments | updatedAt | id |
78-
| custom_collections | updatedAt | id |
79-
| draft_orders | updatedAt | id |
80-
| marketing_events | started_at | id |
81-
| pages | updatedAt | id |
82-
| policies | updatedAt | id |
83-
| redirects | createdAt | id |
84-
| resource_feedback | feedback_generated_at | app_id |
85-
| script_tags | updatedAt | id |
86-
| shop | updatedAt | id |
87-
| smart_collections | updatedAt | id |
88-
| tender_transactions | processedAt | id |
89-
| themes | updatedAt | id |
90-
| webhooks | updatedAt | id |
32+
| Stream Name | Replication Key | Key Properties |
33+
|------------------------|----------------|---------------|
34+
| abandoned_checkouts | updatedAt | id |
35+
| collections | updatedAt | id |
36+
| customers | updatedAt | id |
37+
| events | createdAt | id |
38+
| inventory_items | updatedAt | id |
39+
| inventory_levels | updatedAt | id |
40+
| locations | createdAt | id |
41+
| metafields_collections | updatedAt | id |
42+
| metafields_customers | updatedAt | id |
43+
| metafields_orders | updatedAt | id |
44+
| metafields_products | updatedAt | id |
45+
| order_refunds | updatedAt | id |
46+
| orders | updatedAt | id |
47+
| order_shipping_lines | updatedAt | id |
48+
| fulfillment_orders | updatedAt | id |
49+
| product_variants | updatedAt | id |
50+
| products | updatedAt | id |
51+
| transactions | createdAt | id |
9152

9253
Currently, `locations` graphql endpoint doesn't support querying on the `updatedAt`, therefore, `createdAt` is made the replication key.
9354

94-
## Required Account Access Scopes
95-
96-
1. CarrierServices, ShippingZones - shipping
97-
2. Comments, Pages, Redirect, Article, Blog - content
98-
3. Collections, CustomCollection , SmartCollection, ProductVariants, Products, ResourceFeedback - products
99-
5. DraftOrders - draft_orders
100-
6. TenderTransaction, Transactions, AbandonedCheckouts, OrderRefund, OrderShippingLines - orders
101-
7. FulfillmentService - fulfillments
102-
8. Marketing Events - marketing_events
103-
9. ResourceFeedback - resource_feedbacks
104-
10. ScriptTags - script_tags
105-
11. Themes - themes
106-
12. PriceRules - price_rules
107-
13. Customers - customers
108-
14. FulfillmentOrders - fulfillments_orders
109-
15. InventoryItems, InventoryLevels - inventory
110-
16. Locations - location
111-
17. MarketingEvents - marketing_events
112-
18. Policies - legal_policies
113-
19. PriceRules - discount
114-
20. Redirects - online_store_navigation
115-
11655
## Quick Start
11756

11857
1. Install

‎setup.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
setup(
66
name="tap-shopify",
7-
version="3.8.0",
7+
version="3.7.2",
88
description="Singer.io tap for extracting Shopify data",
99
author="Stitch",
1010
url="http://github.com/singer-io/tap-shopify",

‎tap_shopify/__init__.py‎

Lines changed: 5 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -59,19 +59,6 @@ def has_read_users_access():
5959
return False
6060
return True
6161

62-
def has_access_scope(stream, scopes):
63-
"""If the app does not have the required scope, return False"""
64-
app_access_scopes = fetch_app_scopes()
65-
66-
if not any(access_scope in scopes for access_scope in app_access_scopes):
67-
message = "The account credentials supplied do not have '{0}' access scope " \
68-
"to the following stream: {1}. The data for these streams would not be " \
69-
"collected due to lack of required permission.".format(", ".join(scopes), stream)
70-
LOGGER.warning(message)
71-
return False
72-
73-
return True
74-
7562
def get_abs_path(path):
7663
return os.path.join(os.path.dirname(os.path.realpath(__file__)), path)
7764

@@ -95,7 +82,7 @@ def get_discovery_metadata(stream, schema):
9582
mdata = metadata.write(mdata, (), 'table-key-properties', stream.key_properties)
9683
mdata = metadata.write(mdata, (), 'forced-replication-method', stream.replication_method)
9784

98-
if stream.replication_method=="INCREMENTAL" and stream.replication_key:
85+
if stream.replication_key:
9986
mdata = metadata.write(mdata, (), 'valid-replication-keys', [stream.replication_key])
10087

10188
for field_name in schema['properties'].keys():
@@ -119,15 +106,12 @@ def discover():
119106

120107
raw_schemas = load_schemas()
121108
streams = []
122-
error_list = []
123109

124110
for schema_name, schema in raw_schemas.items():
125111
if schema_name not in Context.stream_objects:
126112
continue
127113

128114
stream = Context.stream_objects[schema_name]()
129-
if stream.access_scope and not has_access_scope(stream.name, stream.access_scope):
130-
error_list.append({stream.name: stream.access_scope})
131115
catalog_schema = add_synthetic_key_to_schema(schema)
132116

133117
# create and add catalog entry
@@ -137,25 +121,11 @@ def discover():
137121
'schema': catalog_schema,
138122
'metadata': get_discovery_metadata(stream, schema),
139123
'key_properties': stream.key_properties,
124+
'replication_key': stream.replication_key,
125+
'replication_method': stream.replication_method
140126
}
141127
streams.append(catalog_entry)
142128

143-
if error_list:
144-
total_stream = len(raw_schemas.keys())
145-
missing_scopes_msg = [
146-
f"{stream}: {', '.join(scopes)}"
147-
for item in error_list
148-
for stream, scopes in item.items()
149-
]
150-
message = "The account credentials supplied do not have access to the following " \
151-
"stream(s): {}. The data for these streams would not be collected due to lack of " \
152-
"required permission.".format(", ".join(missing_scopes_msg))
153-
154-
if len(error_list) != total_stream:
155-
LOGGER.warning(message)
156-
else:
157-
raise ShopifyAPIError(message)
158-
159129
return {'streams': streams}
160130

161131
def shuffle_streams(stream_name):
@@ -188,7 +158,8 @@ def sync():
188158
if Context.is_selected(stream["tap_stream_id"]):
189159
singer.write_schema(stream["tap_stream_id"],
190160
stream["schema"],
191-
stream["key_properties"])
161+
stream["key_properties"],
162+
bookmark_properties=stream["replication_key"])
192163
Context.counts[stream["tap_stream_id"]] = 0
193164

194165
# Loop over streams in catalog

‎tap_shopify/schemas/application_charges.json‎

Lines changed: 0 additions & 56 deletions
This file was deleted.

‎tap_shopify/schemas/application_credits.json‎

Lines changed: 0 additions & 50 deletions
This file was deleted.

‎tap_shopify/schemas/article_authors.json‎

Lines changed: 0 additions & 11 deletions
This file was deleted.

‎tap_shopify/schemas/article_tags.json‎

Lines changed: 0 additions & 11 deletions
This file was deleted.

0 commit comments

Comments
 (0)