Description
What happened?
max retry is not applying to getDatasetLocation() and attempts to retry indefinitely.
Max retries is seen in the stack trace here at 9, but beam does not let go after the 9 retries, and the logs are filled with continuous attempts to getDatasetLocation (see below.) This is a singular instance of the stack trace that repeats indefinitely, hanging the job:
Error message from worker: java.lang.RuntimeException: unable to obtain dataset for dataset nonexistent_dataset in project <project_id>
org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.getDatasetLocation(BigQueryHelpers.java:598)
org.apache.beam.sdk.io.gcp.bigquery.WriteTables.startLoad(WriteTables.java:506)
org.apache.beam.sdk.io.gcp.bigquery.WriteTables.access$700(WriteTables.java:97)
org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:277)
Caused by: java.io.IOException: Unable to get dataset: nonexistent_dataset, aborting after 9 retries.
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.executeWithRetries(BigQueryServicesImpl.java:1476)
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner