Search before asking
Fluss version
main (development)
Please describe the bug 🐞
Describe the bug
The fluss-fs-s3 uber-jar bundles hadoop-common and aws-java-sdk-*
dependencies without any shade relocations. As a result, 1,114 unshaded
jackson class files leak into the jar at com/fasterxml/* paths:
| Package |
Approx. count |
Source |
com/fasterxml/jackson/databind/** |
~800 |
hadoop-common |
com/fasterxml/jackson/core/** |
~200 |
hadoop-common |
com/fasterxml/jackson/annotation/** |
~20 |
hadoop-common |
com/fasterxml/jackson/dataformat/cbor/** |
~50 |
aws-java-sdk-core |
META-INF/versions/{11,17,19}/com/fasterxml/** (MRJ) |
~25 |
jackson-core MRJ |
When a downstream application puts fluss-fs-s3 on its classpath alongside its
own jackson-core (e.g. 2.16+), the leaked classes can shadow the application's
jackson and cause runtime failures such as NoSuchMethodError — the same
failure mode reported in #3553 for the MRJ entries, but here affecting the full
jackson-databind/core/annotations/cbor surface, not just MRJ entries.
In contrast, fluss-fs-hadoop-shaded already relocates these same packages
(com.fasterxml, org.codehaus, com.ctc, com.google.re2j,
org.apache.htrace) into org.apache.fluss.fs.shaded.hadoop3.*.
To Reproduce
./mvnw -pl fluss-filesystems/fluss-fs-s3 -am -DskipTests package
unzip -l fluss-filesystems/fluss-fs-s3/target/fluss-fs-s3-*.jar | grep -c " com/fasterxml"
# -> 1114
Expected behavior
All bundled third-party dependencies should be relocated under Fluss's shaded
namespace (mirroring the fluss-fs-hadoop-shaded pattern), so the uber-jar
never exposes com/fasterxml/* classes on the classpath.
Additional context
Solution
Mirror the fluss-fs-hadoop-shaded approach: add maven-shade-plugin
relocations for com.fasterxml.jackson.*, com.fasterxml.jackson.databind.*,
com.fasterxml.jackson.dataformat.cbor.* to org.apache.fluss.fs.shaded.s3.*,
and add a <filter> to exclude META-INF/versions/*/com/fasterxml/** (MRJ
entries are not relocated by the shade plugin). A draft PR is in progress.
Are you willing to submit a PR?
Search before asking
Fluss version
main (development)
Please describe the bug 🐞
Describe the bug
The
fluss-fs-s3uber-jar bundleshadoop-commonandaws-java-sdk-*dependencies without any shade relocations. As a result, 1,114 unshaded
jackson class files leak into the jar at
com/fasterxml/*paths:com/fasterxml/jackson/databind/**com/fasterxml/jackson/core/**com/fasterxml/jackson/annotation/**com/fasterxml/jackson/dataformat/cbor/**META-INF/versions/{11,17,19}/com/fasterxml/**(MRJ)When a downstream application puts
fluss-fs-s3on its classpath alongside itsown jackson-core (e.g. 2.16+), the leaked classes can shadow the application's
jackson and cause runtime failures such as
NoSuchMethodError— the samefailure mode reported in #3553 for the MRJ entries, but here affecting the full
jackson-databind/core/annotations/cbor surface, not just MRJ entries.
In contrast,
fluss-fs-hadoop-shadedalready relocates these same packages(
com.fasterxml,org.codehaus,com.ctc,com.google.re2j,org.apache.htrace) intoorg.apache.fluss.fs.shaded.hadoop3.*.To Reproduce
Expected behavior
All bundled third-party dependencies should be relocated under Fluss's shaded
namespace (mirroring the
fluss-fs-hadoop-shadedpattern), so the uber-jarnever exposes
com/fasterxml/*classes on the classpath.Additional context
larger surface: the entire jackson family is unshaded, not just MRJ entries.
fluss-fs-oss,fluss-fs-gs,fluss-fs-azure,fluss-fs-obs,fluss-fs-cos,fluss-fs-hdfs) may have the same gap; canbe checked in follow-ups.
Solution
Mirror the
fluss-fs-hadoop-shadedapproach: addmaven-shade-pluginrelocations for
com.fasterxml.jackson.*,com.fasterxml.jackson.databind.*,com.fasterxml.jackson.dataformat.cbor.*toorg.apache.fluss.fs.shaded.s3.*,and add a
<filter>to excludeMETA-INF/versions/*/com/fasterxml/**(MRJentries are not relocated by the shade plugin). A draft PR is in progress.
Are you willing to submit a PR?