Skip to content

[refactor](fe) Merge fe-connector-api into fe-connector-spi - #66407

Merged
morningman merged 1 commit into
apache:masterfrom
morningman:correct-api
Aug 4, 2026
Merged

[refactor](fe) Merge fe-connector-api into fe-connector-spi#66407
morningman merged 1 commit into
apache:masterfrom
morningman:correct-api

Conversation

@morningman

@morningman morningman commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

The connector plugin contract was split across two modules whose names were
inverted relative to common usage: fe-connector-api held the types a plugin
IMPLEMENTS (Connector, ConnectorMetadata and its Ops sub-interfaces, the
scan / write / procedure providers, handles, pushdown and ddl value types) —
which is what "SPI" normally names — while fe-connector-spi held mostly the
engine services a plugin CONSUMES (ConnectorContext, ConnectorStorageContext,
ConnectorConf), which is what "API" normally names. Both modules' package-info
already documented the inversion and asked readers to "read the content, not the
name". The fe-connector-metastore-api / -metastore-spi pair right next to
them follows the opposite (standard) convention, so one directory carried two
contradictory naming rules.

Swapping the two names does not fix it. A strict split by "who implements" is
circular: ConnectorProvider.create takes a ConnectorContext (spi -> api),
while ConnectorContext.createSiblingConnector returns a Connector
(api -> spi). That cycle is exactly why Trino keeps its whole plugin surface in
a single trino-spi module, and this framework is modelled on Trino.

This commit does the same: fe-connector-api is merged into
fe-connector-spi, and org.apache.doris.connector.api becomes
org.apache.doris.connector.spi. The merge is dependency-neutral — every module
that depended on fe-connector-api (hive, hudi, maxcompute, paimon, trino,
fe-core) already depended on fe-connector-spi as well.

Two consequences worth calling out:

  • The connector plugin API version is bumped 2.0 -> 3.0. Every type on the
    contract changed its fully-qualified name, so a plugin built against 2.0 must
    be refused by ApiVersionGate at load time rather than fail later with
    NoClassDefFoundError. Both recorded baselines
    (connector-metadata-methods.txt, connector-plugin-surface.txt) and the
    version pinned in ConnectorPluginSurfaceTest are updated in this commit.
    ConnectorPluginManagerTest used to hardcode the compatible major as a
    literal; it now derives both the stale and the current major from the
    kernel's declared version, because what that test asserts is the gate
    (stale refused, current admitted), not which number is current.
    ConnectorPluginSurfaceTest remains the one deliberate speed bump.
  • ChildFirstClassLoader.DEFAULT_PARENT_FIRST_PACKAGES listed only
    org.apache.doris.connector.api., so ConnectorProvider and
    ConnectorContext were not parent-first. After the merge the single
    org.apache.doris.connector.spi. entry covers the whole contract.

The merged package-info.java keeps the existing Rule 1-7 design rules; Rule 4
now explains why both directions live in one module instead of disclaiming the
inverted names. README.md and AGENTS.md are updated accordingly.

Release note

None

Check List (For Author)

  • Test: Unit Test
    • Full FE build passes, including fe-core test compilation.
    • Every unit test under org.apache.doris.connector passes: 314 test
      classes / 1421 cases, 0 failures, 0 errors, across the 58-module
      reactor (fe-core included).
    • Checkstyle passes across the whole fe reactor.
    • Both architecture gates pass: check-fe-connector-imports.sh (plus its
      self-test) and check-fe-core-metadata-funnel.sh.
    • No regression test run: this is a pure rename/merge with no behavior
      change.
  • Behavior changed: No
  • Does this need documentation: No

🤖 Generated with Claude Code

@hello-stephen

hello-stephen commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

morningman commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

morningman commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman @claude
[refactor](fe) Merge fe-connector-api into fe-connector-spi
ae2fb5b 
### What problem does this PR solve?

Problem Summary:

The connector plugin contract was split across two modules whose names were
inverted relative to common usage: `fe-connector-api` held the types a plugin
IMPLEMENTS (`Connector`, `ConnectorMetadata` and its Ops sub-interfaces, the
scan / write / procedure providers, handles, pushdown and ddl value types) —
which is what "SPI" normally names — while `fe-connector-spi` held mostly the
engine services a plugin CONSUMES (`ConnectorContext`, `ConnectorStorageContext`,
`ConnectorConf`), which is what "API" normally names. Both modules' package-info
already documented the inversion and asked readers to "read the content, not the
name". The `fe-connector-metastore-api` / `-metastore-spi` pair right next to
them follows the opposite (standard) convention, so one directory carried two
contradictory naming rules.

Swapping the two names does not fix it. A strict split by "who implements" is
circular: `ConnectorProvider.create` takes a `ConnectorContext` (spi -> api),
while `ConnectorContext.createSiblingConnector` returns a `Connector`
(api -> spi). That cycle is exactly why Trino keeps its whole plugin surface in
a single `trino-spi` module, and this framework is modelled on Trino.

This commit does the same: `fe-connector-api` is merged into
`fe-connector-spi`, and `org.apache.doris.connector.api` becomes
`org.apache.doris.connector.spi`. The merge is dependency-neutral — every module
that depended on `fe-connector-api` (hive, hudi, maxcompute, paimon, trino,
fe-core) already depended on `fe-connector-spi` as well.

Two consequences worth calling out:

- The connector plugin API version is bumped 2.0 -> 3.0. Every type on the
  contract changed its fully-qualified name, so a plugin built against 2.0 must
  be refused by `ApiVersionGate` at load time rather than fail later with
  `NoClassDefFoundError`. Both recorded baselines
  (`connector-metadata-methods.txt`, `connector-plugin-surface.txt`) and the
  version pinned in `ConnectorPluginSurfaceTest` are updated in this commit.
  `ConnectorPluginManagerTest` used to hardcode the compatible major as a
  literal; it now derives both the stale and the current major from the
  kernel's declared version, because what that test asserts is the gate
  (stale refused, current admitted), not which number is current.
  `ConnectorPluginSurfaceTest` remains the one deliberate speed bump.
- `ChildFirstClassLoader.DEFAULT_PARENT_FIRST_PACKAGES` listed only
  `org.apache.doris.connector.api.`, so `ConnectorProvider` and
  `ConnectorContext` were not parent-first. After the merge the single
  `org.apache.doris.connector.spi.` entry covers the whole contract.

The merged `package-info.java` keeps the existing Rule 1-7 design rules; Rule 4
now explains why both directions live in one module instead of disclaiming the
inverted names. README.md and AGENTS.md are updated accordingly.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - Full FE build passes, including fe-core test compilation.
    - Every unit test under `org.apache.doris.connector` passes: 314 test
      classes / 1421 cases, 0 failures, 0 errors, across the 58-module
      reactor (fe-core included).
    - Checkstyle passes across the whole fe reactor.
    - Both architecture gates pass: `check-fe-connector-imports.sh` (plus its
      self-test) and `check-fe-core-metadata-funnel.sh`.
    - No regression test run: this is a pure rename/merge with no behavior
      change.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@morningman

morningman commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 4, 2026
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman

morningman commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

skip buildall

@morningman
morningman merged commit 13f5924 into apache:master Aug 4, 2026
32 of 33 checks passed
morningman added a commit to morningman/doris that referenced this pull request Aug 7, 2026
@morningman @claude
[fix](fluss) Follow the connector SPI into the module it was merged into
8b3dbe9 
Upstream apache#66407 merged fe-connector-api into fe-connector-spi, moving every
org.apache.doris.connector.api.* type to org.apache.doris.connector.spi.*
and deleting the module. The rebase reported no conflict for this connector:
its files are new on this branch, so there was no shared history for git to
find a conflict in -- they simply kept importing a package that no longer
exists, and only the compiler says so.

Mechanical: 112 references renamed across 27 files with the import group
re-sorted to keep checkstyle's order, the dead fe-connector-api dependency
dropped from the pom, and the matching exclude dropped from the plugin zip
descriptor (that artifact can no longer be produced, so excluding it from
lib/ excluded nothing). No behavior change -- every referenced type was
verified to exist at the new package before the rename, and the diff touches
no line that is not one of those references.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Aug 8, 2026
@morningman @claude
[fix](fluss) Follow the connector SPI into the module it was merged into
e7d7cab 
Upstream apache#66407 merged fe-connector-api into fe-connector-spi, moving every
org.apache.doris.connector.api.* type to org.apache.doris.connector.spi.*
and deleting the module. The rebase reported no conflict for this connector:
its files are new on this branch, so there was no shared history for git to
find a conflict in -- they simply kept importing a package that no longer
exists, and only the compiler says so.

Mechanical: 112 references renamed across 27 files with the import group
re-sorted to keep checkstyle's order, the dead fe-connector-api dependency
dropped from the pom, and the matching exclude dropped from the plugin zip
descriptor (that artifact can no longer be produced, so excluding it from
lib/ excluded nothing). No behavior change -- every referenced type was
verified to exist at the new package before the rename, and the diff touches
no line that is not one of those references.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@924060929 924060929 mentioned this pull request Aug 11, 2026
Merged
16 tasks
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Aug 11, 2026
@924060929
[refactor](fe) Clarify external metadata cache ownership
2b2a1b6 
### What problem does this PR solve?

Issue Number: close apache#66633

Related PR: apache#66407

Problem Summary: The unified metadata cache still exposed duplicate invalidation ownership through CachingHmsClient, retained an engine registry and route abstraction for two built-in FE cache types, and used FE-only names that were easy to confuse with the connector cache API. This change makes the connector CatalogMetaCache the sole invalidation owner, publishes partition collection and partition invalidation atomically, removes redundant Hive and Hudi client type branches, and renames FE-only cache adapters and catalog runtimes to describe their actual roles. Remote Doris schema and backend caching remain catalog-scoped, while statistics caches keep their independent asynchronous semantics.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - Connector cache, HMS, Hive, and Hudi Maven unit tests
    - Targeted FE unit tests for cache runtime, naming cache, Remote Doris, external catalog/database, and refresh replay
- Behavior changed: No. Internal cache ownership and invalidation publication are refactored without changing external SQL behavior.
- Does this need documentation: No
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Aug 11, 2026
@924060929
[refactor](fe) Clarify external metadata cache ownership
04e684b 
### What problem does this PR solve?

Issue Number: close apache#66633

Related PR: apache#66407

Problem Summary: The unified metadata cache still exposed duplicate invalidation ownership through CachingHmsClient, retained an engine registry and route abstraction for two built-in FE cache types, and used FE-only names that were easy to confuse with the connector cache API. This change makes the connector CatalogMetaCache the sole invalidation owner, publishes partition collection and partition invalidation atomically, removes redundant Hive and Hudi client type branches, and renames FE-only cache adapters and catalog runtimes to describe their actual roles. Remote Doris schema and backend caching remain catalog-scoped, while statistics caches keep their independent asynchronous semantics.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - Connector cache, HMS, Hive, and Hudi Maven unit tests
    - Targeted FE unit tests for cache runtime, naming cache, Remote Doris, external catalog/database, and refresh replay
- Behavior changed: No. Internal cache ownership and invalidation publication are refactored without changing external SQL behavior.
- Does this need documentation: No
@HappenLee HappenLee mentioned this pull request Aug 11, 2026
Open
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Aug 11, 2026
@924060929
[refactor](fe) Clarify external metadata cache ownership
97ef95e 
### What problem does this PR solve?

Issue Number: close apache#66633

Related PR: apache#66407

Problem Summary: The unified metadata cache still exposed duplicate invalidation ownership through CachingHmsClient, retained an engine registry and route abstraction for two built-in FE cache types, and used FE-only names that were easy to confuse with the connector cache API. This change makes the connector CatalogMetaCache the sole invalidation owner, publishes partition collection and partition invalidation atomically, removes redundant Hive and Hudi client type branches, and renames FE-only cache adapters and catalog runtimes to describe their actual roles. Remote Doris schema and backend caching remain catalog-scoped, while statistics caches keep their independent asynchronous semantics.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - Connector cache, HMS, Hive, and Hudi Maven unit tests
    - Targeted FE unit tests for cache runtime, naming cache, Remote Doris, external catalog/database, and refresh replay
- Behavior changed: No. Internal cache ownership and invalidation publication are refactored without changing external SQL behavior.
- Does this need documentation: No
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 17, 2026
@morningman @wyxxxcat
[refactor](fe) Merge fe-connector-api into fe-connector-spi (apache#6…
8dcda1c 
…6407)

### What problem does this PR solve?

Problem Summary:

The connector plugin contract was split across two modules whose names
were
inverted relative to common usage: `fe-connector-api` held the types a
plugin
IMPLEMENTS (`Connector`, `ConnectorMetadata` and its Ops sub-interfaces,
the
scan / write / procedure providers, handles, pushdown and ddl value
types) —
which is what "SPI" normally names — while `fe-connector-spi` held
mostly the
engine services a plugin CONSUMES (`ConnectorContext`,
`ConnectorStorageContext`,
`ConnectorConf`), which is what "API" normally names. Both modules'
package-info
already documented the inversion and asked readers to "read the content,
not the
name". The `fe-connector-metastore-api` / `-metastore-spi` pair right
next to
them follows the opposite (standard) convention, so one directory
carried two
contradictory naming rules.

Swapping the two names does not fix it. A strict split by "who
implements" is
circular: `ConnectorProvider.create` takes a `ConnectorContext` (spi ->
api),
while `ConnectorContext.createSiblingConnector` returns a `Connector`
(api -> spi). That cycle is exactly why Trino keeps its whole plugin
surface in
a single `trino-spi` module, and this framework is modelled on Trino.

This commit does the same: `fe-connector-api` is merged into
`fe-connector-spi`, and `org.apache.doris.connector.api` becomes
`org.apache.doris.connector.spi`. The merge is dependency-neutral —
every module
that depended on `fe-connector-api` (hive, hudi, maxcompute, paimon,
trino,
fe-core) already depended on `fe-connector-spi` as well.

Two consequences worth calling out:

- The connector plugin API version is bumped 2.0 -> 3.0. Every type on
the
contract changed its fully-qualified name, so a plugin built against 2.0
must
be refused by `ApiVersionGate` at load time rather than fail later with
  `NoClassDefFoundError`. Both recorded baselines
(`connector-metadata-methods.txt`, `connector-plugin-surface.txt`) and
the
version pinned in `ConnectorPluginSurfaceTest` are updated in this
commit.
`ConnectorPluginManagerTest` used to hardcode the compatible major as a
  literal; it now derives both the stale and the current major from the
  kernel's declared version, because what that test asserts is the gate
  (stale refused, current admitted), not which number is current.
  `ConnectorPluginSurfaceTest` remains the one deliberate speed bump.
- `ChildFirstClassLoader.DEFAULT_PARENT_FIRST_PACKAGES` listed only
  `org.apache.doris.connector.api.`, so `ConnectorProvider` and
  `ConnectorContext` were not parent-first. After the merge the single
  `org.apache.doris.connector.spi.` entry covers the whole contract.

The merged `package-info.java` keeps the existing Rule 1-7 design rules;
Rule 4
now explains why both directions live in one module instead of
disclaiming the
inverted names. README.md and AGENTS.md are updated accordingly.
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Aug 24, 2026
@924060929
[refactor](fe) Clarify external metadata cache ownership
0e81b33 
Issue Number: close apache#66633

Related PR: apache#66407

Problem Summary: The unified metadata cache still exposed duplicate invalidation ownership through CachingHmsClient, retained an engine registry and route abstraction for two built-in FE cache types, and used FE-only names that were easy to confuse with the connector cache API. This change makes the connector CatalogMetaCache the sole invalidation owner, publishes partition collection and partition invalidation atomically, removes redundant Hive and Hudi client type branches, and renames FE-only cache adapters and catalog runtimes to describe their actual roles. Remote Doris schema and backend caching remain catalog-scoped, while statistics caches keep their independent asynchronous semantics.

None

- Test: Unit Test
    - Connector cache, HMS, Hive, and Hudi Maven unit tests
    - Targeted FE unit tests for cache runtime, naming cache, Remote Doris, external catalog/database, and refresh replay
- Behavior changed: No. Internal cache ownership and invalidation publication are refactored without changing external SQL behavior.
- Does this need documentation: No
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Aug 26, 2026
@924060929
[refactor](fe) Clarify external metadata cache ownership
6ed3765 
Issue Number: close apache#66633

Related PR: apache#66407

Problem Summary: The unified metadata cache still exposed duplicate invalidation ownership through CachingHmsClient, retained an engine registry and route abstraction for two built-in FE cache types, and used FE-only names that were easy to confuse with the connector cache API. This change makes the connector CatalogMetaCache the sole invalidation owner, publishes partition collection and partition invalidation atomically, removes redundant Hive and Hudi client type branches, and renames FE-only cache adapters and catalog runtimes to describe their actual roles. Remote Doris schema and backend caching remain catalog-scoped, while statistics caches keep their independent asynchronous semantics.

None

- Test: Unit Test
    - Connector cache, HMS, Hive, and Hudi Maven unit tests
    - Targeted FE unit tests for cache runtime, naming cache, Remote Doris, external catalog/database, and refresh replay
- Behavior changed: No. Internal cache ownership and invalidation publication are refactored without changing external SQL behavior.
- Does this need documentation: No
924060929 added a commit that referenced this pull request Aug 31, 2026
@924060929
[refactor](catalog) Unify external metadata cache framework (#66633)
9a7174e 
### What problem does this PR solve?

Related PR: #66407

Problem Summary:

FE naming caches and connector metadata caches currently use duplicate
cache wrappers and independently maintained invalidation dependencies.
Each connector must remember every sibling cache affected by catalog,
database, table, or partition invalidation. Concurrent loads, refresh,
eviction callbacks, and catalog close also need consistent publication
and cleanup semantics across these implementations.

This PR makes `fe-connector-cache` the shared metadata-cache runtime
while preserving one cache owner per catalog. It introduces declarative
cache definitions and hierarchical scopes, then migrates ADBC, Hive,
HMS, Hudi, Iceberg, MaxCompute, and Paimon caches to the shared
framework. FE keeps a thin adapter for Doris-specific naming, object,
and `IdNameIndex` publication semantics.

The shared runtime provides:

- catalog/database/table/partition scoped invalidation without
connector-maintained sibling-cache lists;
- state-identity publication fencing so loads started before
invalidation cannot publish stale values afterward;
- exact-key and bulk-load invalidation fencing;
- load deduplication, refresh, eviction cleanup, metrics, and catalog
lifecycle management;
- expected-value physical cleanup so delayed removal callbacks cannot
remove newer publications;
- per-catalog isolation without registry-wide publication locks on
normal cache operations.

The change also removes duplicate `CacheSpec`, `CacheFactory`, and
legacy connector-cache entry implementations. FE-specific cache
publication retains the lock order `names -> object -> IdNameIndex`, and
validates identity before mutating cache state. After synchronizing the
latest master, Iceberg's snapshot-scoped equality-delete field-ID cache
is also managed by the same per-catalog owner, so catalog invalidation
and close cover both manifest and equality-delete metadata.

Performance measurements on the final commit show approximately 85.8
ns/op for the public `CatalogMetaCache -> MetaCache` hit path.
Eight-thread publication reached 2.85M ops/s for one cache and 2.51M
ops/s for separate caches in one registry. Invalidating and physically
cleaning 120,000 entries took approximately 32 ms on the local
benchmark.

### Release note

None

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
        - `fe-connector-cache`: 100 tests passed
        - FE targeted metadata-cache tests: 93 tests passed
        - Iceberg cache tests: 66 tests passed
- Latest-master conflict verification: 144 Iceberg tests passed
(`IcebergScanPlanProviderTest` and `IcebergManifestCacheTest`)
- `./build.sh --fe`: all 73 Maven reactor modules passed, including
Checkstyle; the outer script subsequently returned non-zero while
assembling already-built filesystem plugin archives into `output/`
    - [x] Manual test
- Re-profiled the final commit with CPU, allocation, and lock
async-profiler runs
- Ran hit, cold-load, eight-thread publication, logical invalidation,
and 120,000-entry cleanup benchmarks
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason

- Behavior changed:
- [x] No. This is an internal cache-framework refactor preserving
external metadata semantics.
    - [ ] Yes.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929 924060929 924060929 approved these changes
@CalvinKirs CalvinKirs CalvinKirs approved these changes
@morrySnow morrySnow Awaiting requested review from morrySnow morrySnow is a code owner
@englefly englefly Awaiting requested review from englefly englefly is a code owner
@starocean999 starocean999 Awaiting requested review from starocean999 starocean999 is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@morningman @hello-stephen @924060929 @CalvinKirs