Permission Error When Reading Bigtable from Spark.

[mkesici]

Scala Version: 2.12.16
Spark Version: 3.3.0
Connector Version: 0.4.0 (Also tried with 0.3.0 and 0.2.0)
Dataproc image version: 2.1.82-debian11

I am working on a dataproc cluster to run my spark jobs. I am currently not able to use ADC due to company policy but doing authentication by setting GOOGLE_APPLICATION_CREDENTIALS as an environment variable for executors and driver. Set service account has all available BigTable permissions that Reader, User, or Administrator.

Currently i can write my spark dataframe to the bigtable but while i try to read created table i am getting following error:

25/03/12 12:44:07 WARN BigtableChannelPrimer: Failed to prime channel: com.google.cloud.spark.bigtable.repackaged.com.google.api.gax.rpc.PermissionDeniedException: com.google.cloud.spark.bigtable.repackaged.io.grpc.StatusRuntimeException: PERMISSION_DENIED: Access denied. Missing IAM permission: bigtable.instances.ping.
25/03/12 12:44:07 ERROR BigtableTableScanRDD: Received error when creating partitions: com.google.cloud.spark.bigtable.repackaged.io.grpc.StatusRuntimeException: PERMISSION_DENIED: Access denied. Missing IAM permission: bigtable.tables.sampleRowKeys.

Part of stacktrace:

at java.base/java.lang.Thread.run(Thread.java:829)25/03/12 12:44:07 ERROR err:  at java.base/java.lang.Thread.run(Thread.java:829)

        Suppressed: com.google.cloud.spark.bigtable.repackaged.com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed25/03/12 12:44:07 ERROR err:  Suppressed: com.google.cloud.spark.bigtable.repackaged.com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed

                at com.google.cloud.spark.bigtable.repackaged.com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)25/03/12 12:44:07 ERROR err:              at com.google.cloud.spark.bigtable.repackaged.com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)

                at com.google.cloud.spark.bigtable.repackaged.com.google.cloud.bigtable.data.v2.BigtableDataClient.sampleRowKeys(BigtableDataClient.java:1342)25/03/12 12:44:07 ERROR err:              at com.google.cloud.spark.bigtable.repackaged.com.google.cloud.bigtable.data.v2.BigtableDataClient.sampleRowKeys(BigtableDataClient.java:1342)

                at com.google.cloud.spark.bigtable.datasources.BigtableTableScanRDD.getPartitions(BigtableTableScanRDD.scala:49)25/03/12 12:44:07 ERROR err:            at com.google.cloud.spark.bigtable.datasources.BigtableTableScanRDD.getPartitions(BigtableTableScanRDD.scala:49)

                at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:292)25/03/12 12:44:07 ERROR err:            at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:292)

[mkesici]

There is an update for this issue: When we temporarily give BigTable.User permission for attached service account to read bigtable table there was no issue. Setting GOOGLE_APPLICATION_CREDENTIALS as environment variable is not working while trying to read.