Skip to content

[Refactor] Split schema scanners into layered modules #76241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
alvin-phoenix-ai wants to merge 2 commits into from
Closed

[Refactor] Split schema scanners into layered modules #76241

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
69ae8b5
[Refactor] Extract schema scanner factory from Exec
alvin-phoenix-ai Jul 12, 2026
bf6d451
[Refactor] Split schema scanners into layered modules
alvin-phoenix-ai Jul 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 44 additions & 12 deletions be/AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,21 +292,53 @@ Default BE module bootstrap composition for built-in module registration, includ
- Core tests: `module_bootstrap_test`
- Remediation: Keep ModuleBootstrap as top-level default module composition; module implementations should expose registration hooks here instead of depending on service startup directly.

### ExecSchemaScannerCore (`execschemascannercore`)
Schema scanner base contract and shared mechanics without concrete scanner, pipeline, storage, service, or ExecEnv coupling.
- Targets: `ExecSchemaScannerCore`
- Allowed internal include prefixes: `exec/schema_scanner.h`, `exprs/`, `runtime/`, `column/`, `types/`, `common/`, `base/`, `gutil/`, `gen_cpp/`
### SchemaScannerCore (`schemascannercore`)
Schema scanner contract, factory interface, shared mechanics, and column filling without concrete scanner, storage, cache, service, pipeline, or ExecEnv coupling.
- Targets: `SchemaScannerCore`
- Allowed internal include prefixes: `exec/schema_scanner.h`, `exec/schema_scanner_factory.h`, `exec/schema_scanner/schema_column_filler.h`, `exprs/`, `runtime/`, `column/`, `types/`, `common/`, `base/`, `gutil/`, `gen_cpp/`
- Allowed target deps: `Expr`, `Runtime`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
- Core tests: `schema_scanner_core_test`
- Remediation: Keep SchemaScannerCore limited to the base SchemaScanner contract; move concrete scanner creation and service-specific logic into higher schema scanner modules.
- Remediation: Keep SchemaScannerCore limited to the scanner contract and dependency-neutral shared mechanics; move concrete scanners and composition into higher schema scanner layers.

### ExecSchemaScanners (`execschemascanners`)
Clean concrete schema scanners that do not depend on SchemaHelper, FE RPC/client helpers, storage, service, cache, pipeline, or ExecEnv.
- Targets: `ExecSchemaScanners`
### SchemaScannerLocal (`schemascannerlocal`)
Static and BE-local process schema scanners without SchemaHelper, platform clients, storage, cache, service, pipeline, or ExecEnv coupling.
- Targets: `SchemaScannerLocal`
- Allowed internal include prefixes: `exec/schema_scanner.h`, `exec/schema_scanner/schema_column_filler.h`, `runtime/`, `column/`, `types/`, `common/`, `base/`, `gutil/`, `gen_cpp/`
- Allowed target deps: `ExecSchemaScannerCore`, `Runtime`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
- Core tests: `exec_schema_scanners_test`
- Remediation: Keep this first schema scanner target limited to clean local/static scanners; leave SchemaHelper, storage, HTTP, cache, service, and ExecEnv users in higher compatibility modules until they get explicit boundaries.
- Allowed target deps: `SchemaScannerCore`, `Runtime`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
- Core tests: `schema_scanner_local_test`
- Remediation: Keep SchemaScannerLocal restricted to static and BE-local process data; move FE client, storage, cache, and composition behavior into their dedicated scanner layers.

### SchemaScannerFrontend (`schemascannerfrontend`)
Frontend-backed schema scanners and SchemaHelper above Platform and the local/core scanner layers, without storage, cache, service, pipeline, or ExecEnv coupling.
- Targets: `SchemaScannerFrontend`
- Allowed internal include prefixes: `exec/schema_scanner.h`, `exec/schema_scanner/schema_column_filler.h`, `platform/`, `exprs/`, `runtime/`, `column/`, `types/`, `common/`, `base/`, `gutil/`, `gen_cpp/`
- Allowed target deps: `SchemaScannerCore`, `SchemaScannerLocal`, `Platform`, `Expr`, `Runtime`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
- Core tests: `schema_scanner_frontend_test`
- Remediation: Keep SchemaScannerFrontend limited to FE RPC/HTTP-backed schema data over Platform and the local/core scanner layers; move storage and cache access into their dedicated layers.

### SchemaScannerStorage (`schemascannerstorage`)
Storage-backed schema scanners above concrete Storage and the frontend/local/core scanner layers, without cache, service, pipeline, or ExecEnv coupling.
- Targets: `SchemaScannerStorage`
- Allowed internal include prefixes: `exec/schema_scanner.h`, `exec/schema_scanner/schema_column_filler.h`, `storage/`, `storage_primitive/`, `runtime/`, `column/`, `types/`, `common/`, `base/`, `gutil/`, `gen_cpp/`
- Allowed target deps: `SchemaScannerCore`, `SchemaScannerFrontend`, `SchemaScannerLocal`, `Storage`, `StoragePrimitive`, `Runtime`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
- Core tests: `schema_scanner_storage_test`
- Remediation: Keep SchemaScannerStorage limited to tablet, transaction, compaction, and write-log schema data; route FE access through SchemaScannerFrontend and keep cache/composition behavior elsewhere.

### SchemaScannerCache (`schemascannercache`)
Data-cache schema scanner above Cache and SchemaScannerCore without storage, platform, service, pipeline, or ExecEnv coupling.
- Targets: `SchemaScannerCache`
- Allowed internal include prefixes: `exec/schema_scanner.h`, `cache/`, `runtime/`, `column/`, `types/`, `common/`, `base/`, `gutil/`, `gen_cpp/`
- Allowed target deps: `SchemaScannerCore`, `Cache`, `Runtime`, `ChunkCore`, `ColumnCore`, `Types`, `Common`, `Base`, `Gutil`, `StarRocksGen`
- Core tests: `schema_scanner_cache_test`
- Remediation: Keep SchemaScannerCache limited to data-cache schema data over Cache and SchemaScannerCore; keep storage, FE client, and composition behavior in their dedicated layers.

### SchemaScannerBuiltin (`schemascannerbuiltin`)
Top-level builtin schema scanner factory composition over the core, local, frontend, storage, and cache scanner layers without depending on Exec.
- Targets: `SchemaScannerBuiltin`
- Allowed internal include prefixes: `exec/builtin_schema_scanner_factory.h`, `exec/schema_scanner.h`, `exec/schema_scanner_factory.h`, `exec/schema_scanner/`, `gen_cpp/`
- Allowed target deps: `SchemaScannerCore`, `SchemaScannerLocal`, `SchemaScannerFrontend`, `SchemaScannerStorage`, `SchemaScannerCache`
- Core tests: `schema_scanner_builtin_test`
- Remediation: Keep builtin scanner selection as composition over the explicit scanner strata; do not add an Exec dependency or bypass the owning scanner layer.

### ExecJoinCore (`execjoincore`)
Reusable exec join hash table algorithms without join nodes, pipeline, storage, service, or util coupling.
Expand All @@ -328,7 +360,7 @@ Orchestration layer below Service for query, fragment, and ingestion lifecycle e
Diagnostic script execution and command dispatch layer below Service, HttpService, Tools, and AgentServer, and above the remaining reusable BE modules.
- Targets: `Script`
- Allowed internal include prefixes: `script/`, `base/`, `cache/`, `column/`, `common/`, `connector/`, `compute_env/`, `data_workflows/`, `exec/`, `exec_primitive/`, `exprs/`, `formats/`, `fs/`, `gen_cpp/`, `geo/`, `gutil/`, `io/`, `orchestration/`, `platform/`, `runtime/`, `storage/`, `storage_primitive/`, `types/`
- Allowed target deps: `Base`, `Gutil`, `Common`, `Cache`, `IO`, `FileSystem`, `Platform`, `Types`, `ColumnCore`, `ChunkCore`, `ColumnSortCore`, `Runtime`, `Runtime`, `Runtime`, `Formats`, `StoragePrimitive`, `StorageBase`, `Storage`, `ComputeEnv`, `DataWorkflows`, `Expr`, `ExprDict`, `ExprTableFunction`, `ExprUtility`, `ExecPrimitive`, `ExecRuntime`, `ExecSchemaScannerCore`, `ExecSchemaScanners`, `ExecJoinCore`, `Exec`, `ConnectorPrimitive`, `Connector`, `ConnectorBenchmark`, `ConnectorCacheStats`, `ConnectorElasticsearch`, `ConnectorMySQL`, `Orchestration`, `Geo`, `StarRocksGen`
- Allowed target deps: `Base`, `Gutil`, `Common`, `Cache`, `IO`, `FileSystem`, `Platform`, `Types`, `ColumnCore`, `ChunkCore`, `ColumnSortCore`, `Runtime`, `Runtime`, `Runtime`, `Formats`, `StoragePrimitive`, `StorageBase`, `Storage`, `ComputeEnv`, `DataWorkflows`, `Expr`, `ExprDict`, `ExprTableFunction`, `ExprUtility`, `ExecPrimitive`, `ExecRuntime`, `ExecJoinCore`, `Exec`, `ConnectorPrimitive`, `Connector`, `ConnectorBenchmark`, `ConnectorCacheStats`, `ConnectorElasticsearch`, `ConnectorMySQL`, `Orchestration`, `Geo`, `StarRocksGen`
- Core tests: `script_test`
- Remediation: Keep Script below Service, HttpService, Tools, and AgentServer; lower reusable behavior can live in lower BE modules that Script is allowed to depend on.

Expand Down
Loading
Loading