Task: Upgrade Apache Flink version from 1.13.6 to 1.15.2

[pallakartheekreddy]

We need to upgrade our Apache Flink jobs from version 1.13.6 to 1.15.2. This upgrade crosses two minor versions (1.14 and 1.15) and introduces several breaking changes, specifically regarding how Flink handles Scala dependencies, State Backends, and Checkpoint ownership.

Acceptance Criteria:

• Flink dependencies in the build configuration (pom.xml / build.sbt) are updated to 1.15.2.
• Project compiles successfully and unit tests pass.
• Dockerfiles and deployment configurations (Helm/Kubernetes) are updated to use the Flink 1.15.2 base image.
• Jobs successfully start from a 1.13.6 savepoint in the staging environment.

Implementation Steps & Technical Details for Jules:

1. Dependency Updates

• Version Bump: Change the Flink version property to 1.15.2.
• Scala Runtime Extraction (Breaking Change): Starting in Flink 1.15, Flink no longer includes Scala in the runtime. Because Sunbird jobs heavily rely on Scala, you must explicitly include the Scala version and other Scala dependencies in the project's fat JAR/zip.

Connectors: Ensure any connectors (e.g., Kafka, AWS) are updated to their respective 1.15.2 compatible versions.

2. API & Code Adjustments

• State Backend Refactoring: Flink 1.14+ restructured state backends. If the code explicitly references MemoryStateBackend, FsStateBackend, or RocksDBStateBackend, these must be replaced with the new HashMapStateBackend or EmbeddedRocksDBStateBackend combined with the CheckpointStorage API.
• Table API / SQL Planner: The legacy planner has been completely removed. Ensure the project relies on the Blink planner (which is now the default).
• Source/Sink API Deprecations: Note that the old SourceFunction and SinkFunction APIs are heavily deprecated in favor of the new Source (FLIP-27) and Sink (FLIP-143) APIs. Resolve any compilation errors or suppress warnings if we are deferring that specific refactor.

3. Deployment & Docker Updates

• Image Tags: Update the base Docker image. Since Scala was removed from the runtime, official Flink image tags no longer include the Scala version (e.g., you will use a tag like flink:1.15.2-java11 instead of flink:1.13.6-scala_2.12-java11).

4. State Migration & Snapshot Restore

• Savepoint Testing: We must guarantee state compatibility. Generate a savepoint from the current 1.13.6 job and verify the 1.15.2 job can restore from it.
• Snapshot Ownership: Be aware that Flink 1.15 introduces new restore modes (CLAIM, NO_CLAIM, LEGACY). The new default is NO_CLAIM, meaning Flink will create its own copy of the state and leave the existing savepoint entirely up to the user.