Have Option to Limit Number of Records per Sync

[AlxGlx]

What

Give option to limit sync by X number of records.

Why

Useful when you need to test the end-to-end or scheduling of and automated data feed, or need to limit data ingress per run.

How (potentially)

Adding in "Sync Limit" variable for limit by # records or (size in MB?) in the Connections -> [ConnectionName] -> Replication screen.

[marcosmarxm]

@AlxGlx one problem of having this is you ending up with a fraction of your total data. Maybe is something you're planning too but overall it looks a "dangerous" feature. For your use case it is for API or databases? If the second you can always create a view to manage the number of records from your side.

[AlxGlx]

That's a fair challenge @marcosmarxm, and I appreciate it relies on the user to then cleanup the database after the run (or use an incremental sync mode). However, I would perhaps expect any user setting up sync jobs to have a this level of understanding.

I suppose the use case in my head was if you wish to try an end-to-end run through of a connection, but not commit to a full run, so you could return just a handful of pages/records/MBs to reduce the potential hit to resource or quota constrained APIs or limit database ingress.

[tomlynchRNA]

Hi @marcosmarxm, this is currently something I would like to do with Airbyte.

Situation: We have a user with a database e.g. BigQuery and want them to connect to their datasource and sync some data to our destination.

Problem: We can't predict or control how much data will be pulled from the source and sent to our destination. We want to predict costs and obviously our infrastructure can only handle a certain amount of data or scale up so far.

For example, if our users have a quota and can only sync X number of records or Y amount of data, when the table the user requests to sync is 1000000000000 rows and TB of data, we can't handle that. We could write our own code to check the number of rows, but it's impractical to do this for each and every airbyte source and is the reason we use Airbyte for ELT in the first place.

One thing I've thought of trying is to listen to the "Successful syncs" webhook, extract the job ID from it, and attempt to cancel if the number of rows is too high. But this happens after the queries have already been made and data is already moving to the destination, and is an unreliable hack.

Thoughts?