Add DataEntity subtype support for platform-level classification - #1812
Open
medhwu wants to merge 1 commit into
Open
Add DataEntity subtype support for platform-level classification#1812medhwu wants to merge 1 commit into
medhwu wants to merge 1 commit into
Conversation
Re-introduce a lightweight `subtype` VARCHAR column on `data_entity` so that entities of the same type can be classified by platform/engine/tool. Subtype is auto-detected from collector metadata (platform/engine/tool keys) during ingestion. New `DataEntitySubTypeDto` enum covers 60+ subtypes: - TABLE: STARROCKS_TABLE, DORIS_TABLE, TIDB_TABLE, POSTGRESQL_TABLE, ... - DASHBOARD: FINEBI_DASHBOARD, DATAEASE_DASHBOARD, TABLEAU_DASHBOARD, ... - KAFKA_TOPIC: ROCKETMQ_TOPIC, PULSAR_TOPIC, KAFKA_TOPIC_TYPE - JOB: AIRFLOW_DAG, DBT_JOB, SPARK_JOB, ... - DATABASE_SERVICE: STARROCKS_SERVICE, DORIS_SERVICE, SNOWFLAKE_SERVICE, ... - And VIEW, FILE, JOB_RUN New policy condition key `dataEntity:subtype` enables platform-specific RBAC. Changes: - Flyway V0_0_101: ALTER TABLE data_entity ADD COLUMN subtype VARCHAR(128) - IngestionMapperImpl.detectSubtype(): reads metadata, dispatches by type - DataEntitySubtypeFieldComparer: reflection-based comparer (avoids JOOQ dep) - IngestionServiceImpl.persistSubtypes(): batch update after ingestion - ReactiveDataEntityRepositoryImpl: raw DSL.field() for compile compat - policy_schema.json + PolicyConditionKeyDto + DataEntityConditionResolver
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Re-introduces a lightweight
subtypecolumn on thedata_entitytable so that data entities of the same type can be classified by their underlying platform/engine/tool.New enum
DataEntitySubTypeDtocovers 60+ subtypes across 8 entity types including StarRocks, Doris, TiDB, FineBI, DataEase, RocketMQ, Pulsar and more.Subtype is auto-detected from collector metadata -- the ingestion mapper reads
platform/engine/tool/service_typekeys.A new policy condition key
dataEntity:subtypeenables platform-specific RBAC rules.Why
All relational DB tables (PostgreSQL, MySQL, StarRocks, Doris, TiDB, ...) are classified as a single
TABLEtype. Similarly, all BI dashboards areDASHBOARD. This makes it impossible to filter by engine or apply platform-specific governance policies.How
V0_0_101:ALTER TABLE data_entity ADD COLUMN subtype VARCHAR(128)with partial indexDataEntitySubTypeDto: Java enum with 60+ valuesIngestionMapperImpl.detectSubtype(): reads collector metadata, dispatches by typeIngestionServiceImpl.persistSubtypes(): batch-updates subtypes after creation/updateReactiveDataEntityRepository.updateSubtypes(): rawDSL.field("subtype")to avoid JOOQ compile depdataEntity:subtypecondition key in schema + resolver + comparerBackward compatibility
subtypeis nullable -- all existing entities remainNULLChecklist
IF NOT EXISTSand partial indexEnrichedDataEntityIngestionDtoconstructor passessubtypeto superdataEntity:subtypecondition key