Remove deprecated Kafka topics: MetadataAuditEvent_v4, MetadataGraphEvent_v4, MetadataSearchEvent_v4

[askumar27]

Description

Three deprecated Kafka topics are still defined in the codebase but are no longer actively used:

• MetadataAuditEvent_v4
• MetadataGraphEvent_v4
• MetadataSearchEvent_v4

These topics were deprecated mid-2022 when DataHub transitioned to the more flexible MetadataChangeLog (MCL) and MetadataChangeProposal (MCP) event model. The legacy code remains only for backward compatibility, but it creates confusion for users and adds maintenance overhead.

Current State

Code References

• Topic constants defined in Topics.java
• Topic configuration in TopicConventionFactory.java
• Explicitly disabled in application.yaml: metadataAuditEvent.enabled: false
• Schema ordinals maintained in EventSchemaConstants.java and SchemaIdOrdinal.java for deserialization only

No Active Usage

• MetadataAuditEvent_v4: The produceMetadataAuditEvent method exists but has zero call sites in the codebase
• MetadataGraphEvent_v4 and MetadataSearchEvent_v4: Originally designed as CDC streams but never actively produced or consumed
• All three topics are marked as deprecated in documentation

Related Issues/PRs

• Original MAE deprecation (mid-2022)
• GitHub Issue What are MetadataSearchEvent and MetadataGraphEvent for? #1696 (referenced in code comments)
• GitHub PR docs(kafka): clarify Kafka topic retention requirements #15149 (MetadataGraphEvent_v4 schema removal)

Additional Context

Recent user question from Slack that prompted this issue:

"Why are MetadataGraphEvent_v4 and MetadataSearchEvent_v4 still preserved in the source code if they are not used anymore? Do I need that topic to be available while deploying datahub?"

This confusion will persist until we fully remove the deprecated code.

Slack link: https://datahubspace.slack.com/archives/CV2UVAPPG/p1765805181442149?thread_ts=1765804856.922539&cid=CV2UVAPPG