Hii GCP Bigtable team,
We are facing an issue when reading from Bigtable using the Spark Bigtable connector. Our Spark job fails with:
Caused by: com.google.cloud.spark.bigtable.repackaged.io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Error while reading table 'projects/***/instances/store/tables/data_lifecycle' : Read returned 269MiB from row '0RGWN8RPA2P3#PRODUCT_DUMP' which exceeds the limit of 256MiB.
Make sure you are setting an appropriate request filter to retrieve only recent versions and only the columns you want. If columns are accumulating more versions than you need to read, you can also create a garbage collection policy: https://cloud.google.com/bigtable/docs/configuring-garbage-collection#versions
Context
Spark version: 3.5
Connector version: 2.12-0.0.7
Workload: ETL job offloading Bigtable to GCS
Problem
Some rows in our table have accumulated many versions per cell, causing the connector to fetch huge row payloads (>256MiB) and fail.
In our ETL use case, we only need the latest version of each cell (or a configurable number of recent versions), not the entire history.
The current connector behavior doesn’t seem to expose a way to specify version filters.
Feature Request
It would be very helpful if the Spark Bigtable connector supported configuration to:
Limit versions per cell (e.g., only latest, or n recent versions).
Pass down Bigtable filters (versionsLimit, qualifierRegexFilter, etc.) directly from Spark read options.
Benefit
Prevents FAILED_PRECONDITION errors due to oversized rows.
Reduces data scanned, improving performance and lowering cost.
Aligns with common ETL/offload scenarios where only the latest state is required.
Request
Could you please confirm if there is already a way to set version limits when using the Spark connector?
If not, we’d like to request this as a feature enhancement.
Thanks for your help!
Regards
Maneesh K Bishnoi
LinkedIn: https://www.linkedin.com/in/maneesh-bishnoi/
Hii GCP Bigtable team,
We are facing an issue when reading from Bigtable using the Spark Bigtable connector. Our Spark job fails with:
Caused by: com.google.cloud.spark.bigtable.repackaged.io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Error while reading table 'projects/***/instances/store/tables/data_lifecycle' : Read returned 269MiB from row '0RGWN8RPA2P3#PRODUCT_DUMP' which exceeds the limit of 256MiB.
Make sure you are setting an appropriate request filter to retrieve only recent versions and only the columns you want. If columns are accumulating more versions than you need to read, you can also create a garbage collection policy: https://cloud.google.com/bigtable/docs/configuring-garbage-collection#versions
Context
Spark version: 3.5
Connector version: 2.12-0.0.7
Workload: ETL job offloading Bigtable to GCS
Problem
Some rows in our table have accumulated many versions per cell, causing the connector to fetch huge row payloads (>256MiB) and fail.
In our ETL use case, we only need the latest version of each cell (or a configurable number of recent versions), not the entire history.
The current connector behavior doesn’t seem to expose a way to specify version filters.
Feature Request
It would be very helpful if the Spark Bigtable connector supported configuration to:
Limit versions per cell (e.g., only latest, or n recent versions).
Pass down Bigtable filters (versionsLimit, qualifierRegexFilter, etc.) directly from Spark read options.
Benefit
Prevents FAILED_PRECONDITION errors due to oversized rows.
Reduces data scanned, improving performance and lowering cost.
Aligns with common ETL/offload scenarios where only the latest state is required.
Request
Could you please confirm if there is already a way to set version limits when using the Spark connector?
If not, we’d like to request this as a feature enhancement.
Thanks for your help!
Regards
Maneesh K Bishnoi
LinkedIn: https://www.linkedin.com/in/maneesh-bishnoi/