Skip to content

[Bug]: Bigquery python streaming insertAll SSLError leads to stuck streaming job #34270

New issue
New issue
Open
Open
[Bug]: Bigquery python streaming insertAll SSLError leads to stuck streaming job#34270
Labels
P2awaiting triagebugdataflowiopython
@quentin-sommer

Description

@quentin-sommer
quentin-sommer
opened on Mar 13, 2025

What happened?

Since ~2 weeks my streaming job started getting this error regularly. It happened again today with latest beam (2.63.0) running on GCP Dataflow.

It makes the job retry the failed unit of work forever until it gives up and halt all processing, forcing me to re-deploy

My WriteToBigQuery config

WriteToBigQuery(table=self.table.table_id,
    project=self.table.project,
    dataset=self.table.dataset_id,
    insert_retry_strategy=RetryStrategy.RETRY_ON_TRANSIENT_ERROR,
    write_disposition=BigQueryDisposition.WRITE_APPEND,
    create_disposition=BigQueryDisposition.CREATE_NEVER,
    method=WriteToBigQuery.Method.STREAMING_INSERTS,
    batch_size=1900,
    triggering_frequency=10,
    with_auto_sharding=True,
).with_output_types(WriteResult)

Stacktrace:


Error message from worker: generic::unknown: urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:2427)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py\", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects/REDACTED/insertAll?prettyPrint=false (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2427)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py\", line 144, in retry_target
    result = target()
             ^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/cloud/_http/__init__.py\", line 482, in api_request
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/cloud/_http/__init__.py\", line 341, in _make_request
    return self._do_request(
           ^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/cloud/_http/__init__.py\", line 379, in _do_request
    return self.http.request(
           ^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/auth/transport/requests.py\", line 537, in request
    response = super(AuthorizedSession, self).request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects/REDACTED/insertAll?prettyPrint=false (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2427)')))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File \"apache_beam/runners/common.py\", line 1501, in apache_beam.runners.common.DoFnRunner.process
  File \"apache_beam/runners/common.py\", line 917, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File \"apache_beam/runners/common.py\", line 1061, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery.py\", line 1640, in process
    return self._flush_batch(destination)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery.py\", line 1683, in _flush_batch
    passed, errors = self.bigquery_wrapper.insert_rows(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py\", line 1293, in insert_rows
    result, errors = self._insert_all_rows(
                     ^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/utils/retry.py\", line 311, in wrapper
    raise exn.with_traceback(exn_traceback)
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/utils/retry.py\", line 298, in wrapper
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py\", line 744, in _insert_all_rows
    errors = self.gcp_bq_client.insert_rows_json(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/cloud/bigquery/client.py\", line 3889, in insert_rows_json
    response = self._call_api(
               ^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/cloud/bigquery/client.py\", line 837, in _call_api
    return call()
           ^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py\", line 293, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py\", line 153, in retry_target
    _retry_error_helper(
  File \"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_base.py\", line 221, in _retry_error_helper
    raise final_exc from source_exc
google.api_core.exceptions.RetryError: Timeout of 600.0s exceeded, last exception: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects/REDACTED/insertAll?prettyPrint=false (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2427)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py\", line 313, in _execute
    response = task()
               ^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py\", line 388, in <lambda>
    lambda: self.create_worker().do_instruction(request), request)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py\", line 658, in do_instruction
    return getattr(self, request_type)(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py\", line 696, in process_bundle
    bundle_processor.process_bundle(instruction_id))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py\", line 1271, in process_bundle
    self.ops[element.transform_id].process_timer(
  File \"apache_beam/runners/worker/operations.py\", line 974, in apache_beam.runners.worker.operations.DoOperation.process_timer
  File \"apache_beam/runners/common.py\", line 1553, in apache_beam.runners.common.DoFnRunner.process_user_timer
  File \"apache_beam/runners/common.py\", line 1591, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File \"apache_beam/runners/common.py\", line 1550, in apache_beam.runners.common.DoFnRunner.process_user_timer
  File \"apache_beam/runners/common.py\", line 645, in apache_beam.runners.common.DoFnInvoker.invoke_user_timer
  File \"apache_beam/runners/common.py\", line 1686, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File \"apache_beam/runners/common.py\", line 1799, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File \"apache_beam/runners/worker/operations.py\", line 263, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File \"apache_beam/runners/worker/operations.py\", line 950, in apache_beam.runners.worker.operations.DoOperation.process
  File \"apache_beam/runners/worker/operations.py\", line 951, in apache_beam.runners.worker.operations.DoOperation.process
  File \"apache_beam/runners/common.py\", line 1503, in apache_beam.runners.common.DoFnRunner.process
  File \"apache_beam/runners/common.py\", line 1591, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File \"apache_beam/runners/common.py\", line 1501, in apache_beam.runners.common.DoFnRunner.process
  File \"apache_beam/runners/common.py\", line 689, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File \"apache_beam/runners/common.py\", line 1686, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File \"apache_beam/runners/common.py\", line 1799, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File \"apache_beam/runners/worker/operations.py\", line 263, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File \"apache_beam/runners/worker/operations.py\", line 950, in apache_beam.runners.worker.operations.DoOperation.process
  File \"apache_beam/runners/worker/operations.py\", line 951, in apache_beam.runners.worker.operations.DoOperation.process
  File \"apache_beam/runners/common.py\", line 1503, in apache_beam.runners.common.DoFnRunner.process
  File \"apache_beam/runners/common.py\", line 1591, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File \"apache_beam/runners/common.py\", line 1501, in apache_beam.runners.common.DoFnRunner.process
  File \"apache_beam/runners/common.py\", line 689, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File \"apache_beam/runners/common.py\", line 1686, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File \"apache_beam/runners/common.py\", line 1799, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File \"apache_beam/runners/worker/operations.py\", line 263, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File \"apache_beam/runners/worker/operations.py\", line 950, in apache_beam.runners.worker.operations.DoOperation.process
  File \"apache_beam/runners/worker/operations.py\", line 951, in apache_beam.runners.worker.operations.DoOperation.process
  File \"apache_beam/runners/common.py\", line 1503, in apache_beam.runners.common.DoFnRunner.process
  File \"apache_beam/runners/common.py\", line 1612, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File \"apache_beam/runners/common.py\", line 1501, in apache_beam.runners.common.DoFnRunner.process
  File \"apache_beam/runners/common.py\", line 917, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File \"apache_beam/runners/common.py\", line 1061, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery.py\", line 1640, in process
    return self._flush_batch(destination)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery.py\", line 1683, in _flush_batch
    passed, errors = self.bigquery_wrapper.insert_rows(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py\", line 1293, in insert_rows
    result, errors = self._insert_all_rows(
                     ^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/utils/retry.py\", line 311, in wrapper
    raise exn.with_traceback(exn_traceback)
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/utils/retry.py\", line 298, in wrapper
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py\", line 744, in _insert_all_rows
    errors = self.gcp_bq_client.insert_rows_json(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/cloud/bigquery/client.py\", line 3889, in insert_rows_json
    response = self._call_api(
               ^^^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/cloud/bigquery/client.py\", line 837, in _call_api
    return call()
           ^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py\", line 293, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File \"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py\", line 153, in retry_target
    _retry_error_helper(
  File \"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_base.py\", line 221, in _retry_error_helper
    raise final_exc from source_exc
RuntimeError: google.api_core.exceptions.RetryError: Timeout of 600.0s exceeded, last exception: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects/REDACTED/insertAll?prettyPrint=false (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2427)'))) [while running 'sessions to bq/WriteToBQ/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn)-ptransform-82']

passed through:
==>
    dist_proc/dax/workflow/worker/fnapi_service_impl.cc:1331

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2awaiting triagebugdataflowiopython

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions