KAFKA-20839: Fix metadata.version stuck at MINIMUM_KRAFT_VERSION when…

[gp1314]

Description

When a KRaft controller is activated on a non-empty metadata log that lacks a FeatureLevelRecord for metadata.version, the ActivationRecordsGenerator.recordsForNonEmptyLog() method only logs a warning but does not write the missing record. This causes
FeatureControlManager to fall back to MINIMUM_KRAFT_VERSION (3.0-IV1), and the cluster's metadata.version remains permanently stuck at this value.

Root Cause

This happens when the controller loses a leadership race during initial bootstrap activation:

1. Controller formats metadata log with metadata.version = 3.9-IV0 (from BootstrapMetadata)
2. On first activation, recordsForEmptyLog() is called and writes bootstrap records (including the FeatureLevelRecord for metadata.version)
3. If the controller loses leadership mid-write, some records are committed but the FeatureLevelRecord may be lost
4. When the controller regains leadership, the log is non-empty but lacks the FeatureLevelRecord — recordsForNonEmptyLog() takes effect but previously only logged a warning without fixing the problem

Impact

• metadata.version permanently stuck at 3.0-IV1
• KIP-919 features broken: --bootstrap-controller returns UnsupportedVersionException: Direct-to-controller communication is not supported with the current MetadataVersion
• Dynamic controller registration unavailable
• All features gated behind metadata.version >= 3.7-IV0 are blocked

Reproduction

1. Deploy a fresh KRaft 3.9.x cluster
2. During initial controller activation, trigger a leadership re-election (e.g., network partition)
3. Once the controller stabilizes, check metadata.version:
   kafka-features.sh --bootstrap-server broker:9092 describe
4. metadata.version shows 3.0-IV1 instead of 3.9-IV0

Fix

In ActivationRecordsGenerator.recordsForNonEmptyLog(), when the current metadataVersion equals MINIMUM_KRAFT_VERSION and the log lacks a FeatureLevelRecord, write one using the version from BootstrapMetadata (which was correctly set during kafka-storage format).

Workaround

Users affected by this bug can manually upgrade metadata.version after the controller stabilizes:
kafka-features.sh --bootstrap-server broker:9092 upgrade --feature metadata.version=3.9