[KafkaIO] Remove beam_fn_api requirement for dynamic reads

[goutamadwant]

Fixes #29998.

KafkaIO dynamic reads currently fail during pipeline expansion unless the obsolete beam_fn_api experiment is enabled.

This change removes that expansion-time requirement. Dynamic reads remain SDF-only through the existing KafkaIOReadImplementationCompatibility selection, so runner fallback and compatibility behavior are unchanged.

The regression test verifies that:

• dynamic read is still classified as SDF-only
• the transform expands without enabling beam_fn_api

The change also updates the 2.77.0 release notes.

Testing

• ./gradlew :sdks:java:io:kafka:test :sdks:java:io:kafka:kafkaVersion392Test :sdks:java:io:kafka:spotlessCheck --no-daemon
• ./gradlew :sdks:java:io:kafka:test --tests org.apache.beam.sdk.io.kafka.KafkaIOReadImplementationCompatibilityTest.testDynamicReadUsesSdfWithoutBeamFnApiExperiment :sdks:java:io:kafka:check --no-build-cache --rerun-tasks --no-daemon

The full KafkaIO check includes Checkstyle, SpotBugs, Javadocs, dependency analysis, formatting, and unit tests.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

[github-actions]

Assigning reviewers:

R: @kennknowles for label java.
R: @Dippatel98 for label kafka.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[kennknowles]

@johnjcasey @Abacn

I think this is fine - but is there some nuance where it simply won't work? (the runner should really reject it, not the transform itself, but I know that KafkaIO does a bunch of logic around various flags and whatnot)

Is there a postsubmit with trigger file we should add here?

[Abacn]

@johnjcasey @Abacn

I think this is fine - but is there some nuance where it simply won't work? (the runner should really reject it, not the transform itself, but I know that KafkaIO does a bunch of logic around various flags and whatnot)

Is there a postsubmit with trigger file we should add here?

Agree, it's no longer needed. AI investigation (checked manually):

This check is outdated, inaccurate, and should be removed or updated.

1. Inaccurate for Dataflow Runner v2 users: In DataflowRunner.java, the portable / Runner v2 engine (useUnifiedWorker) can be enabled by any of several experiment flags:

static boolean useUnifiedWorker(DataflowPipelineOptions options) {
  return hasExperiment(options, "beam_fn_api")
      || hasExperiment(options, "use_runner_v2")
      || hasExperiment(options, "use_unified_worker")
      || hasExperiment(options, "use_portable_job_submission")
      || hasExperiment(options, "enable_portable_runner");
}

If a user launches a Dataflow pipeline with --experiments=use_runner_v2 (the standard and recommended flag for Runner v2) without also explicitly passing --experiments=beam_fn_api, the check in KafkaIO.java:L1663-1666 will throw an error:

IllegalArgumentException: Kafka Dynamic Read requires enabling experiment beam_fn_api.

even though the job is running on Runner v2.

2. Historical artifact from 2021: Git history (commit 45a2193) shows that this check was introduced in March 2021 when SDF support in Beam Java SDK was first being rolled out and required explicit activation via "beam_fn_api".

3. Already handled by the SDK implementation compatibility matrix: KafkaIO now uses KafkaIOReadImplementationCompatibility.java, where DYNAMIC_READ is properly mapped exclusively to the SDF implementation (ReadFromKafkaViaSDF).

Hardcoding a runner-specific experiment string inside KafkaIO.expand() breaks runner-agnostic SDK architecture. If a runner check is desired for Dataflow specifically, it should either rely on DataflowRunner.useUnifiedWorker(options) or let the runner transform override handle capability validation.