[Enhancement] introduce fe extensions #67255
Conversation
StarRocks currently features a plugin mechanism that enables users to install and manage plugins via SQL, such as audit log plugins or data integration plugins. However, the existing plugin mechanism primarily focuses on plugin installation/uninstallation and single-function implementation. It lacks the capability for deep extension into internal system behaviors and cannot uniformly manage plugin lifecycles, dependencies, and resources. This makes it difficult to meet more complex business scenarios and expansion requirements. Signed-off-by: stdpain <[email protected]>
stdpain
force-pushed
the
fea_extensions
branch
from
24419f4 to
25e3917
Compare
|
|
Better to add docs to describe
|
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 490 / 535 (91.59%) file detail
|
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
see #67270 |
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package com.starrocks.extension; |
There was a problem hiding this comment.
Is it possible to move this extension to a new module like fe/fe-extension ?
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| } | ||
| } | ||
| return classes; | ||
| } |
There was a problem hiding this comment.
URLClassLoader closed before extension classes can load dependencies
The URLClassLoader in scanJar() is closed via try-with-resources at line 75, but the loaded classes are returned and later instantiated in ExtensionManager.loadExtensions(). When the extension's constructor or onLoad() method tries to use other classes from the same JAR that weren't already loaded during scanning, class loading will fail because the classloader is closed. The URLClassLoader needs to remain open for the lifetime of the extension, or all required classes must be loaded before closing.
Additional Locations (1)
| String normalized = | ||
| classFile.toString().replace(File.separatorChar, '/'); | ||
| return BASE_PACKAGES.stream().anyMatch(normalized::contains); | ||
| } |
There was a problem hiding this comment.
Package filter uses dots but path uses slashes
The isUnderBasePackage() method compares file paths (which use slashes like com/starrocks/extension) against BASE_PACKAGES which use dot notation (like com.starrocks.extension). The normalized.contains() check will never match because "path/com/starrocks/extension/Class.class".contains("com.starrocks.extension") returns false due to the slash vs dot mismatch. This causes the scanner to filter out all class files, preventing any extensions from being discovered.
3 participants
Why I'm doing:
StarRocks currently features a plugin mechanism that enables users to install and manage plugins via SQL, such as audit log plugins or data integration plugins.
However, the existing plugin mechanism primarily focuses on plugin installation/uninstallation and single-function implementation. It lacks the capability for deep extension into internal system behaviors and cannot uniformly manage plugin lifecycles, dependencies, and resources. This makes it difficult to meet more complex business scenarios and expansion requirements.
What I'm doing:
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check:
Note
Adds a lightweight extension system and decouples Gson configuration.
ExtensionManager,SRModule,StarRocksExtension, and scanners (JarScanner,DirectoryClassPathScanner) to auto-load extensions; FE now callsExtensionManager.loadExtensionsFromDir(Config.ext_dir)at startupConfigkeyext_dir(defaults to$STARROCKS_HOME/lib)DefaultExtensionContextproviding defaults (WarehouseManager,ResourceUsageMonitor,BaseSlotManagerviaSlotManager,IGsonBuilderFactory)GlobalStateMgrnow obtainsWarehouseManager,ResourceUsageMonitor, andBaseSlotManagerfromExtensionManagerinstead of direct instantiationIGsonBuilderFactoryandDefaultGsonBuilderFactory; moves adapters intopersist.gson.internal/*;GsonUtilsdelegates builder creation to the factoryUtFrameUtils) loads extensions from classpath (target/classes)Written by Cursor Bugbot for commit 25e3917. This will update automatically on new commits. Configure here.