Missing org.openjdk.jol from imported dependencies in xtable-utilities makes it to fail with NoClassDefFoundError

[ssandona]

Search before asking

• I had searched in the issues and found no similar issues.

Please describe the bug 🐞

I cloned the repo then moved to 0.3.0-incubating:

git checkout 0.3.0-incubating

then moved to the xtable-utilities folder

cd xtable-utilities

and compiled it

mvn clean package -DskipTests

Then run the tool with proper conf files

java -jar xtable-utilities/target/xtable-utilities_2.12-0.3.0-incubating-bundled.jar --datasetConfig datasetConfig.yaml --icebergCatalogConfig glueDataCatalog.yaml

it failed with

Exception in thread "main" java.lang.NoClassDefFoundError: org/openjdk/jol/info/GraphLayout at org.apache.hudi.common.util.ObjectSizeCalculator.getObjectSize(ObjectSizeCalculator.java:57) at org.apache.hudi.common.util.HoodieRecordSizeEstimator.(HoodieRecordSizeEstimator.java:40) at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:109) at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:76) at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:466) at org.apache.hudi.metadata.HoodieMetadataLogRecordReader$Builder.build(HoodieMetadataLogRecordReader.java:219) at org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:498) at org.apache.hudi.metadata.HoodieBackedTableMetadata.openReaders(HoodieBackedTableMetadata.java:429) at org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$getOrCreateReaders$10(HoodieBackedTableMetadata.java:412) at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) at org.apache.hudi.metadata.HoodieBackedTableMetadata.getOrCreateReaders(HoodieBackedTableMetadata.java:412) at org.apache.hudi.metadata.HoodieBackedTableMetadata.lookupKeysFromFileSlice(HoodieBackedTableMetadata.java:291) at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:255) at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:145) at org.apache.hudi.metadata.BaseTableMetadata.fetchAllPartitionPaths(BaseTableMetadata.java:316) at org.apache.hudi.metadata.BaseTableMetadata.getAllPartitionPaths(BaseTableMetadata.java:125) at org.apache.xtable.hudi.HudiDataFileExtractor.getFilesCurrentState(HudiDataFileExtractor.java:116) at org.apache.xtable.hudi.HudiConversionSource.getCurrentSnapshot(HudiConversionSource.java:110) at org.apache.xtable.spi.extractor.ExtractFromSource.extractSnapshot(ExtractFromSource.java:40) at org.apache.xtable.conversion.ConversionController.syncSnapshot(ConversionController.java:281) at org.apache.xtable.conversion.ConversionController.syncTableFormats(ConversionController.java:203) at org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:106) at org.apache.xtable.utilities.RunSync.syncTableMetdata(RunSync.java:186) at org.apache.xtable.utilities.RunSync.runSync(RunSync.java:297) at org.apache.xtable.utilities.RunSync.main(RunSync.java:282) Caused by: java.lang.ClassNotFoundException: org.openjdk.jol.info.GraphLayout at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) ... 25 more

To fix I modified xtable-utilities/pom.xml and changed this:

<dependency>
  <groupId>org.openjdk.jol</groupId>
  <artifactId>jol-core</artifactId>
  <scope>test</scope>
</dependency>

to:

<dependency>
  <groupId>org.openjdk.jol</groupId>
  <artifactId>jol-core</artifactId>
  <version>0.16</version>
</dependency>

then recompiled with:

mvn clean package -DskipTests

this time running the tool everything is ok.

Are you willing to submit PR?

• I am willing to submit a PR!
• I am willing to submit a PR but need help getting started!

Code of Conduct

• I agree to follow this project's Code of Conduct

[the-other-tim-brown]

@vinishjail97 was this removed due to license issues?

[ssandona]

@the-other-tim-brown @vinishjail97 any update on this? Can this be fixed?