[Task]: Support non-ascii BigQuery field name for BigQueryIO STORAGE_WRITE_API

[Abacn]

What needs to happen?

mirrors #33920. While FILE_LOAD mode can write to table with non-ascii field name, using STORAGE_WRITE_API, it fails with

Error message from worker: org.apache.beam.sdk.util.UserCodeException: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: com.google.protobuf.Descriptors$DescriptorValidationException: Daeb91a54_360e_4da9_8037_ee40d18f7e43.âge: "âge" is not a valid identifier.
	org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
	org.apache.beam.sdk.io.gcp.bigquery.StorageApiConvertMessages$ConvertMessagesDoFn$DoFnInvoker.invokeProcessElement(Unknown Source)
	org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForParDo(FnApiDoFnRunner.java:810)
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: com.google.protobuf.Descriptors$DescriptorValidationException: D39b92aab_0bc0_41b0_b8d2_1ea713d9bfab.âge: "âge" is not a valid identifier.
 at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:592)
 at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:551)
...
 at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4922)
 at org.apache.beam.sdk.io.gcp.bigquery.TwoLevelMessageConverterCache.get(TwoLevelMessageConverterCache.java:66)
 at org.apache.beam.sdk.io.gcp.bigquery.StorageApiConvertMessages$ConvertMessagesDoFn.processElement(StorageApiConvertMessages.java:167)

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[mohamedawnallah]

.take-issue

[mohamedawnallah]

Hi @Abacn,

It seems this issue is a bit involved. The error com.google.protobuf.Descriptors$DescriptorValidationException occurred when using the non-ASCII field name âge. This happens because protobuf only allows ASCII field names, as seen in the protobuf source code.

Additionally, the Apache Beam table schema is converted accordingly to the proto file in the case of the STORAGE_WRITE_API method, as shown in StorageApiDynamicDestinationsTableRow.java.

What is the best way to handle this?

[Abacn]

I see, thanks for investigation, yeah this sounds non-trivial and I do not have immediate solution either without dive deep into it