Skip to content

Consider moving resumption state out of BigQuery #70

Open
@imor

Description

@imor

BigQuery rate limits number of DML statements against a table: https://cloud.google.com/bigquery/docs/data-manipulation-language#update_delete_merge_dml_concurrency. This might result in data writes to copied_tables and last_lsn tables failing under heavy load. Consider moving resumption state to either the API db or source db. Or maybe even hide resumption state logic behind a trait like ResumptionStateStore which allows users to pick a location of their own for the resumption state store (source db, API db, file etc.).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions