NoClassDefFoundError: org/apache/hadoop/hdfs/HdfsConfiguration when using Paimon Lakehouse

[wuchong]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.7.0 (latest release)

Please describe the bug 🐞

Problem1

This is reported in https://apache-fluss.slack.com/archives/C08QXHUH3M1/p1752314994560869 by George Leonard (see more detailed information and files in the slack thread).

I copied the important configuration to here. The NoClassDefFoundError of HdfsConfiguration can be reproduced by following steps:

1. Configure datalake.format paimon with filesystem metastore and a local warehouse path on each Fluss server
2. Download fluss-fs-hadoop-0.7.0.jar, fluss-fs-s3-0.7.0.jar, fluss-lake-paimon-0.7.0.jar into <FLUSS_HOME>/lib
3. Start all the Fluss servers and Coordinator reports the error.

If remove step 2, all the things work. If adding flink-shaded-hadoop-2-uber-2.8.3-10.0.jar into fluss/lib, it also works.

          zookeeper.address: zookeeper:2181
          coordinator.host: coordinator-server
          remote.data.dir: hdfs://namenode:9000/fluss
          datalake.format: paimon
          datalake.paimon.metastore: filesystem
          datalake.paimon.warehouse: /tmp/paimon

coordinator-server  | 2025-07-12 10:00:26,777 ERROR com.alibaba.fluss.server.ServerBase                          [] - Could not start the CoordinatorServer.
coordinator-server  | java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/HdfsConfiguration
coordinator-server  | 	at org.apache.paimon.catalog.CatalogContext.<init>(CatalogContext.java:53) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at org.apache.paimon.catalog.CatalogContext.create(CatalogContext.java:65) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.lake.paimon.PaimonLakeCatalog.<init>(PaimonLakeCatalog.java:71) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.lake.paimon.PaimonLakeStorage.createLakeCatalog(PaimonLakeStorage.java:42) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.lake.paimon.PaimonLakeStorage.createLakeCatalog(PaimonLakeStorage.java:27) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.coordinator.CoordinatorServer.createLakeCatalog(CoordinatorServer.java:258) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.coordinator.CoordinatorServer.startServices(CoordinatorServer.java:195) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.start(ServerBase.java:118) [fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.startServer(ServerBase.java:88) [fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.coordinator.CoordinatorServer.main(CoordinatorServer.java:154) [fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.HdfsConfiguration
coordinator-server  | 	at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[?:?]
coordinator-server  | 	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) ~[?:?]
coordinator-server  | 	at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
coordinator-server  | 	... 10 more
coordinator-server  | 2025-07-12 10:00:26,789 ERROR com.alibaba.fluss.server.ServerBase                          [] - Could not start CoordinatorServer.
coordinator-server  | com.alibaba.fluss.exception.FlussException: Failed to start the CoordinatorServer.
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.start(ServerBase.java:131) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.startServer(ServerBase.java:88) [fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.coordinator.CoordinatorServer.main(CoordinatorServer.java:154) [fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/HdfsConfiguration
coordinator-server  | 	at org.apache.paimon.catalog.CatalogContext.<init>(CatalogContext.java:53) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at org.apache.paimon.catalog.CatalogContext.create(CatalogContext.java:65) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.lake.paimon.PaimonLakeCatalog.<init>(PaimonLakeCatalog.java:71) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.lake.paimon.PaimonLakeStorage.createLakeCatalog(PaimonLakeStorage.java:42) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.lake.paimon.PaimonLakeStorage.createLakeCatalog(PaimonLakeStorage.java:27) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.coordinator.CoordinatorServer.createLakeCatalog(CoordinatorServer.java:258) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.coordinator.CoordinatorServer.startServices(CoordinatorServer.java:195) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.start(ServerBase.java:118) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	... 2 more
coordinator-server  | Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.HdfsConfiguration
coordinator-server  | 	at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[?:?]
coordinator-server  | 	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source) ~[?:?]
coordinator-server  | 	at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
coordinator-server  | 	at org.apache.paimon.catalog.CatalogContext.<init>(CatalogContext.java:53) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at org.apache.paimon.catalog.CatalogContext.create(CatalogContext.java:65) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.lake.paimon.PaimonLakeCatalog.<init>(PaimonLakeCatalog.java:71) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.lake.paimon.PaimonLakeStorage.createLakeCatalog(PaimonLakeStorage.java:42) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.lake.paimon.PaimonLakeStorage.createLakeCatalog(PaimonLakeStorage.java:27) ~[fluss-lake-paimon-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.coordinator.CoordinatorServer.createLakeCatalog(CoordinatorServer.java:258) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.coordinator.CoordinatorServer.startServices(CoordinatorServer.java:195) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.start(ServerBase.java:118) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	... 2 more

Problem2

Reverting to fluss/fluss:0.7.0 with a dockerfile tha references lakehouse storage tiering/paimon/hdfs with the same configuration,

          zookeeper.address: zookeeper:2181
          coordinator.host: coordinator-server
          remote.data.dir: hdfs://namenode:9000/fluss
          datalake.format: paimon
          datalake.paimon.metastore: filesystem
          datalake.paimon.warehouse: /tmp/paimon

it throws the exception during initialize fluss HDFS fs.

 | 2025-07-12 13:25:56,802 ERROR com.alibaba.fluss.server.ServerBase                          [] - Could not start CoordinatorServer.
coordinator-server  | com.alibaba.fluss.exception.FlussException: Failed to start the CoordinatorServer.
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.start(ServerBase.java:131) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.startServer(ServerBase.java:88) [fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.coordinator.CoordinatorServer.main(CoordinatorServer.java:154) [fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | Caused by: com.alibaba.fluss.fs.UnsupportedFileSystemSchemeException: Cannot support file system for 'hdfs' via Hadoop, because Hadoop is not in the classpath, or some classes are missing from the classpath.
coordinator-server  | 	at com.alibaba.fluss.fs.hdfs.HadoopFsPlugin.create(HadoopFsPlugin.java:160) ~[?:?]
coordinator-server  | 	at com.alibaba.fluss.fs.PluginFileSystemWrapper.create(PluginFileSystemWrapper.java:58) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FileSystem.lambda$getUnguardedFileSystem$3(FileSystem.java:400) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.utils.concurrent.LockUtils.inLock(LockUtils.java:41) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:305) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FileSystem.get(FileSystem.java:300) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FsPath.getFileSystem(FsPath.java:264) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.start(ServerBase.java:116) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	... 2 more
coordinator-server  | Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/HdfsConfiguration
coordinator-server  | 	at com.alibaba.fluss.fs.hdfs.HadoopFsPlugin.create(HadoopFsPlugin.java:70) ~[?:?]
coordinator-server  | 	at com.alibaba.fluss.fs.PluginFileSystemWrapper.create(PluginFileSystemWrapper.java:58) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FileSystem.lambda$getUnguardedFileSystem$3(FileSystem.java:400) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.utils.concurrent.LockUtils.inLock(LockUtils.java:41) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:305) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FileSystem.get(FileSystem.java:300) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FsPath.getFileSystem(FsPath.java:264) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.start(ServerBase.java:116) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	... 2 more
coordinator-server  | Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.HdfsConfiguration
coordinator-server  | 	at java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]
coordinator-server  | 	at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
coordinator-server  | 	at com.alibaba.fluss.classloading.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:152) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.classloading.ComponentClassLoader.loadClass(ComponentClassLoader.java:115) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
coordinator-server  | 	at com.alibaba.fluss.fs.hdfs.HadoopFsPlugin.create(HadoopFsPlugin.java:70) ~[?:?]
coordinator-server  | 	at com.alibaba.fluss.fs.PluginFileSystemWrapper.create(PluginFileSystemWrapper.java:58) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FileSystem.lambda$getUnguardedFileSystem$3(FileSystem.java:400) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.utils.concurrent.LockUtils.inLock(LockUtils.java:41) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:305) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FileSystem.get(FileSystem.java:300) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.fs.FsPath.getFileSystem(FsPath.java:264) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	at com.alibaba.fluss.server.ServerBase.start(ServerBase.java:116) ~[fluss-server-0.7.0.jar:0.7.0]
coordinator-server  | 	... 2 more

Solution

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[wuchong]

🧪 Problem 1

After some debugging, I believe the issue stems from several root causes:

1. Placing fluss-lake-paimon-0.7.0.jar directly into the fluss/lib directory causes the Paimon plugin to be loaded from there instead of from the intended plugin directory (fluss/plugins/paimon). However, fluss/lib does not include Hadoop dependencies like HdfsConfiguration, leading to a ClassNotFoundException.

2. The fluss-fs-hadoop-0.7.0.jar does not include bundled Hadoop dependencies, so placing it into fluss/lib doesn't resolve the missing dependency issue.

3. Adding flink-shaded-hadoop-2-uber-2.8.3-10.0.jar to fluss/lib resolves the problem because it contains the necessary Hadoop dependencies.

✅ Conclusion: The main issue was the incorrect placement of fluss-lake-paimon-0.7.0.jar into fluss/lib. When using the standard plugin installation mechanism (i.e., placing plugins under fluss/plugins), everything works as expected.

---

🧪 Problem 2

This issue occurs because fluss-fs-s3-0.7.0.jar, fluss-fs-oss-0.7.0.jar, and fluss-fs-gs-0.7.0.jar all contain the class HadoopFsPlugin.class, but do not bundle Hadoop dependencies.

As a result, Fluss’s plugin system may load HadoopFsPlugin from any of these JARs — including s3, oss, or gs plugins — (depending on OS or file ordering?). If it's loaded from one of the non-HDFS plugins, a ClassNotFoundException is thrown due to missing Hadoop dependencies.

🔧 Verification: Removing the s3, oss, and gs plugins from fluss/plugins resolved the issue.

---

✅ Proposed Solutions

I think we can make several improvements here:

1. Enhance Plugin Loading Logic:
   Add an explicit error message when the same plugin is loaded from multiple locations, e.g.:

   "Plugin 'HadoopFsPlugin' loaded from both fluss/plugins/hadoop and fluss/plugins/s3. This may lead to class loading issues."

   May also include links to dependency resolution documentation as a part of the error message to make it more actionable. (from @MehulBatra)

2. Fix Download Page Link:
   Update the Fluss download page to point to fluss-fs-hadoop-0.7.0-with-dependencies.jar instead of the current fluss-fs-hadoop-0.7.0.jar.

3. Publish Bundled HDFS Artifact:
   In the next release, publish a new artifact named fluss-fs-hdfs-0.7.0.jar that includes all required Hadoop dependencies, replacing the current fluss-fs-hadoop-*.jar which lacks them.

4. Remove Redundant Classes from Non-HDFS Plugins:
   Remove HadoopFsPlugin.class from fluss-fs-s3, fluss-fs-oss, fluss-fs-gs, and other new filesystem plugins (e.g., azure blob, huawei obs, tencent cos) to avoid accidental class loading from the wrong plugin.

5. Add End-To-End Tests (from @MehulBatra)
   We need to add more tests, especially end-to-end tests that set up a distributed Fluss cluster for verification. This is crucial because nearly all of the issues we encounter are related to class loading.

6. Troubleshooting Guide (from @MehulBatra)
   Create a troubleshooting guide for Lakehouse that can contain examples of lib vs plugins directory usage or Common dependency error scenarios (if one doesn't exist)

7. Any Other Suggestions?
   Open to more ideas or alternative approaches.

What are your thoughts on this @luoyuxia @michaelkoepf @polyzos?

Looking forward to your feedback!

[georgelza]

in the end... making it as simple as possible for the end user is important.

might make sense to include the correct jar's in the correct locations by default. Make the Fluss container complete and only list what needs to be added to the Flink container/lib directory to complete the integration.

[polyzos]

Thanks Jark, the proposed solution sound good to me

As discussed in the thread, it’s mainly the absence of Hadoop classes and thus the uber jar is currently required. - this is something I have been experiencing with Paimon since the early days.

I really like the idea of throwing a descriptive error when loading from multiple locations. 👍

[MehulBatra]

Thank you, Jark, for the excellent breakdown! It made the problem much easier to understand and digest.

To build on your analysis, I'd like to suggest:

• Testing Coverage: Should we add integration tests that specifically validate:

  • Multiple filesystem plugins loaded simultaneously
  • Plugin dependency isolation scenarios
  • The lib vs plugins directory behavior

• Enhanced Plugin Loading Logic:

  • Include links to dependency resolution documentation as a part of the error message to make it more actionable.
  • Create a troubleshooting guide for Lakehouse that can contain examples of lib vs plugins directory usage or Common dependency error scenarios (if one doesn't exist)

These additions could help prevent similar issues in the future and make debugging easier for users.

[michaelkoepf]

@wuchong i will have a detailed look in the next days.

[georgelza]

• Multiple filesystem plugins loaded simultaneously
  Think thesis important as a client might be moving data from one storage provider to another due to legacy or whatever.

• The lib vs plugins directory behaviour
  might be idea to include whatever needs to be included from a project perspective via the plugin directory and allow the user to use the lib, but... at the bare minimum.

also did allot of reading over night, we have some opportunity to improve the docs also... eventually found notes of also what needs to be added to Flink home/lib. Jark think that is the cause of my error when I execute the lakehouse process.
Need to check/confirm... just slightly stuck at the moment, went out with family to the local Corona Surf competition thats in town and lost my glasses so the world is really fuzzy atm.

[wuchong]

Hi @MehulBatra , yes, I think we need to add tests, especially end-to-end tests that set up a distributed fluss cluster for verifying, because almost all the problems are related to the classloading issues and are difficult to test in UT/IT.

I added your idea to the above proposals.

[luoyuxia]

@wuchong Thanks for the detail trouble shooting. A real greate work.

As a result, Fluss’s plugin system may load HadoopFsPlugin from any of these JARs — including s3, oss, or gs plugins — (depending on OS or file ordering?). If it's loaded from one of the non-HDFS plugins, a ClassNotFoundException is thrown due to missing Hadoop dependencies.

It looks strange to me. But it does happens, had never been awared of that it should load the hdfs plugins from s3 plugin. I think maybe we can remove com.alibaba.fluss.fs.hdfs.HadoopFsPlugin from META-INF while building. So that the class HadoopFsPlugin in s3 plugin won't be loaded as a Plugin for hdfs.

Try with

                                <filter>
                                    <artifact>com.alibaba.fluss:fluss-fs-hadoop</artifact>
                                    <excludes>
                                        <exclude>META-INF/**</exclude>
                                    </excludes>
                                </filter>

It can exclude com.alibaba.fluss.fs.hdfs.HadoopFsPlugin in s3 META-INF

[wuchong]

@luoyuxia yes, I think this should work. Could you help to create the sub-issues mentioned above?

And welcome anyone to take the issues.