feat: Add configuration options for PK chunking to help w/the initial sync of large/wide tables

[sunild]

I ran into two problems when syncing tables with the BULK api.

• If the job results are over 1GB or query takes more than 5 mins (described here in SF docs), the job fails w/a "retried 30 times" error
• Objects in the Note table in one of our SF accounts must be associated w/a large number of other objects, b/c we get this error: InvalidBatch : Failed to process query: OPERATION_TOO_LARGE: exceeded 20000 distinct ids

Addressing the first problem

We found it helpful to enable PK chunking for specific tables from the get go, rather than wait for it to fail over to PK chunking after a query timeout. The tables in question are fairly large and have lots of columns, and we have to use PK chunking w/them elsewhere.

The existing code is supposed to fail over to PK chunking when a query timeout occurs, but it didn't in our case. I believe the behavior of the API may have changed, based on what the the original tap-salesforce is doing now (and that seems outdated too, 15 vs 30 retries).

I'd like to submit a separate PR to address the fail over problem.

Addressing the second one

We got the "OPERATION_TOO_LARGE" error with and without PK chunking. One of the solutions is to do a smaller query, so a smaller chunk size avoided the error.

Configuration options for PK chunking

I'll happily submit PR's to update the docs. I added 4 config options:

config:
  pk_chunking:
    enabled: true
    chunk_size: 15000
    tables:
    - TableA
    - TableB
    polling_sleep_time: 20

Intent

This is intended to be used during an initial sync for specific problematic tables, and then disabled for subsequent runs.

[sunild]

I added this method b/c when a pk chunked job was complete, the bookmark still had the JobID and BatchIDs properties in it. The JobID needs to be cleared from the state, otherwise on the next syncing interval, it would incorrectly think it was resuming a failed batch (but the BatchIDs array was empty).

See:

tap-salesforce/tap_salesforce/__init__.py

Lines 453 to 455 in d802f68

	if job_id:
	with metrics.record_counter(stream) as counter:
	LOGGER.info("Found JobID from previous Bulk Query. Resuming sync for job: %s", job_id)