[Bug]: org.apache.beam.sdk.util.construction.BeamUrns.getUrn fails when trying to parse Protobuf messages

[dasphillipbrau]

What happened?

Hello.
We're using Scala 2.13.7
We have a set of PTransforms that

1. Creates a KafkaInputStream to read from a topic via KafkaIORead
2. reads serialized protobuf messages from said topic
3. deserializes the messages and instantiate them as case classes.

We recently decided to upgrade beam from 2.57.0 to 2.64.0, no other code changes were done. After doing so, the pipeline started crashing with the error message

java.lang.NoSuchMethodError: 'java.lang.String org.apache.beam.sdk.util.construction.BeamUrns.getUrn(org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ProtocolMessageEnum)'

After checking the Beam source code in the aforementioned class, I found this PR where the version of this GRPC dependency was updated. This change was first included in Beam v2.63.0

The version of this GRPC dependency went from 1.60.1 to 1.69.0. I thought the issue was due to a mismatch between this GRPC dependency in Beam and in our repository, so I updated the one in our repo (which was 1.60.1) to 1.69.0. The issue unfortunately persisted.
Upon downgrading to 2.62.0, however, everything works as expected.

There was a Protobuf issue reported with 2.63.0 and 2.64.0, however, no mention of a similar issue to this one, but I do think they're related

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[liferoad]

Is #34452 related here?

[damccorm]

Is #34452 related here?

Maybe. One thing that could be happening is the Kafka messages are serialized with an old protobuf version, and then there is an exception when the new version tries to deserialize them.

@dasphillipbrau could you share the full stack trace you get when you've upgraded to Beam 2.64 and GRPC 1.69.0?

[dasphillipbrau]

Is #34452 related here?

Maybe. One thing that could be happening is the Kafka messages are serialized with an old protobuf version, and then there is an exception when the new version tries to deserialize them.

@dasphillipbrau could you share the full stack trace you get when you've upgraded to Beam 2.64 and GRPC 1.69.0?

Well, that could be a reason. The messages in the topic are from proto2.
Unfortunately, that is the only message in the Stacktrace, the rest is being obscured by our runner.

[damccorm]

Unfortunately, that is the only message in the Stacktrace, the rest is being obscured by our runner.

Which runner are you using? I'd normally at least expect a code location where this is being thrown from, so that's a little surprising. I probably need a little more info to be able to diagnose this further unfortunately.

I'd expect the issue I described to be fixed with the next Beam upgrade - would you be able to try building Beam at head and using that to test whether or not you're still experiencing the issue? https://github.com/apache/beam/blob/master/contributor-docs/code-change-guide.md#run-your-pipeline-with-modified-beam-code

[dasphillipbrau]

It is the Datafow runner but we're trying to deploy this via flex template. When there's an exception, the only actual error that DF registers is the failure to launch of the template, and the last message in the stack trace (passed as an INFO level message) is the reason why the template failed to launch (in our case, this message about the getUrn method). We'll try to use the current head

[Abacn]

vendor grpc 1.69.0 was introduced in Beam 2.63.0, which explains 2.62.0 works fine

java.lang.NoSuchMethodError: 'java.lang.String org.apache.beam.sdk.util.construction.BeamUrns.getUrn(org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ProtocolMessageEnum)'

This is due to mixed dependency, specifically some of your dependency (the caller of BeamUrns.getUrn) is on old Beam version, why other dependencies (at least org.apache.beam.sdk.util.construction) is on newer Beam version. Please check your project settings to make sure all Beam dependencies are in the same verison

Let me close this issue for now - to resolve 2.64.0 milestone blocker