Skip to content

Add DataEntity subtype support for platform-level classification - #1812

Open
medhwu wants to merge 1 commit into
opendatadiscovery:mainfrom
medhwu:feature/data-entity-subtype-support
Open

Add DataEntity subtype support for platform-level classification#1812
medhwu wants to merge 1 commit into
opendatadiscovery:mainfrom
medhwu:feature/data-entity-subtype-support

Conversation

@medhwu

@medhwu medhwu commented Jun 26, 2026

Copy link
Copy Markdown

What

Re-introduces a lightweight subtype column on the data_entity table so that data entities of the same type can be classified by their underlying platform/engine/tool.

New enum DataEntitySubTypeDto covers 60+ subtypes across 8 entity types including StarRocks, Doris, TiDB, FineBI, DataEase, RocketMQ, Pulsar and more.

Subtype is auto-detected from collector metadata -- the ingestion mapper reads platform / engine / tool / service_type keys.

A new policy condition key dataEntity:subtype enables platform-specific RBAC rules.

Why

All relational DB tables (PostgreSQL, MySQL, StarRocks, Doris, TiDB, ...) are classified as a single TABLE type. Similarly, all BI dashboards are DASHBOARD. This makes it impossible to filter by engine or apply platform-specific governance policies.

How

  1. Flyway V0_0_101: ALTER TABLE data_entity ADD COLUMN subtype VARCHAR(128) with partial index
  2. DataEntitySubTypeDto: Java enum with 60+ values
  3. IngestionMapperImpl.detectSubtype(): reads collector metadata, dispatches by type
  4. IngestionServiceImpl.persistSubtypes(): batch-updates subtypes after creation/update
  5. ReactiveDataEntityRepository.updateSubtypes(): raw DSL.field("subtype") to avoid JOOQ compile dep
  6. Policy: dataEntity:subtype condition key in schema + resolver + comparer

Backward compatibility

  • subtype is nullable -- all existing entities remain NULL
  • Collectors without platform/engine metadata are unaffected
  • No breaking API changes

Checklist

  • Flyway migration uses IF NOT EXISTS and partial index
  • EnrichedDataEntityIngestionDto constructor passes subtype to super
  • Policy JSON schema extended with dataEntity:subtype condition key
  • No JOOQ code-generation dependency at compile time

Re-introduce a lightweight `subtype` VARCHAR column on `data_entity` so
that entities of the same type can be classified by platform/engine/tool.

Subtype is auto-detected from collector metadata (platform/engine/tool keys)
during ingestion. New `DataEntitySubTypeDto` enum covers 60+ subtypes:
- TABLE: STARROCKS_TABLE, DORIS_TABLE, TIDB_TABLE, POSTGRESQL_TABLE, ...
- DASHBOARD: FINEBI_DASHBOARD, DATAEASE_DASHBOARD, TABLEAU_DASHBOARD, ...
- KAFKA_TOPIC: ROCKETMQ_TOPIC, PULSAR_TOPIC, KAFKA_TOPIC_TYPE
- JOB: AIRFLOW_DAG, DBT_JOB, SPARK_JOB, ...
- DATABASE_SERVICE: STARROCKS_SERVICE, DORIS_SERVICE, SNOWFLAKE_SERVICE, ...
- And VIEW, FILE, JOB_RUN

New policy condition key `dataEntity:subtype` enables platform-specific RBAC.

Changes:
- Flyway V0_0_101: ALTER TABLE data_entity ADD COLUMN subtype VARCHAR(128)
- IngestionMapperImpl.detectSubtype(): reads metadata, dispatches by type
- DataEntitySubtypeFieldComparer: reflection-based comparer (avoids JOOQ dep)
- IngestionServiceImpl.persistSubtypes(): batch update after ingestion
- ReactiveDataEntityRepositoryImpl: raw DSL.field() for compile compat
- policy_schema.json + PolicyConditionKeyDto + DataEntityConditionResolver
@medhwu
medhwu requested a review from a team as a code owner June 26, 2026 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant