Connection broken via Stitch

[rrong]

Hey everyone,

On the Braintree integration on Stitch this tap sometimes fails with:

('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

I've also recently noticed that some transactions on Braintree are not being pushed to my destination on Stitch. Looking at the extraction logs it seems like the amount of records fetched by the tap does not match the amount of transactions I get when I query the same time range in Braintree.

For example, I get:
2020-02-25 14:35:45,367Z tap - INFO transactions: Fetched 5319 records from 2020-01-31 00:00:00+00:00 - 2020-02-01 00:00:00+00:00
But Braintree returns 5490 transactions in the same time period.

Can anyone shed light into this, and are the two issues related?

Thanks

[rrong]

I believe I've found the issue. Example below (all times are UTC):

2020-03-11 18:43:42,787Z    tap - INFO transactions: Written 535 records from 2020-03-11 00:00:00+00:00 - 2020-03-11 16:49:43.361745+00:00
2020-03-11 18:43:42,787Z    tap - INFO transactions: Skipped 2040 records from 2020-03-11 00:00:00+00:00 - 2020-03-11 16:49:43.361745+00:00
2020-03-11 18:43:42,787Z    tap - INFO transactions: Complete. Last updated record: 2020-03-11 18:27:12+00:00
2020-03-11 18:43:42,787Z    tap - INFO transactions: Complete. Last disbursement date: 2020-03-10 00:00:00+00:00
2020-03-11 18:43:42,787Z    tap - INFO Sync completed
2020-03-11 18:43:42,871Z target - INFO Sending batch with 541 messages for table transactions to https://api.stitchdata.com/v2/import/batch
2020-03-11 18:43:43,406Z   main - INFO State update: adding latest_disbursement_date = "2020-03-10T00:00:00.000000Z"
2020-03-11 18:43:43,406Z target - INFO Requests complete, stopping loop
2020-03-11 18:43:43,407Z   main - INFO State update: adding latest_updated_at = "2020-03-11T18:27:12.000000Z"
2020-03-11 18:43:43,407Z   main - INFO State update: adding transactions = "2020-03-11T16:49:43.361745Z"
2020-03-11 18:43:43,479Z   main - INFO Target exited normally with status 0

Since the tap took about 2 hours to run, there could be records that have a created_at value between the query interval (2020-03-11 00:00:00+00:00 - 2020-03-11 16:49:43.361745+00:00 in this case) but an updated_at value closer to the runtime (2020-03-11T18:27:12.000000Z). In this case, all values between 16:49:43 and 18:27:12 on March 11 will not be picked up.

I'll fix with a small PR.