Upgrade hadoop and spark (#343)

* New version

* Unify links

* New hadoop version

* Drop exclude

* Drop deprecation

* New deps

* PK fix

* S3 integration test port

* Fix

* Fix created bucket

* Fix unit methods

* Fix unit methods

* S3Setup fixes

* PK fix

* Upgrade common-beanutils

* PK fix

* Scala linting

* Scala formatting

* Changes

* Changes

* Ignore export parallelism IT

* Scalafix

* Drop guava dep

* PK fix

* Pom cleanup

* PK fix

* Deps to fix CVEs

* S3A dependencies

* More CVE upgrades

* Alluxio deps

* Bring back grpc-core

* Final tweaks

* Trying to enable ExportParallelismIT

* Add detailed list of CVEs fixed

Author: Shmuma

.github/workflows/ci-build.yml

@@ -0,0 +1,69 @@
+# Cloud Storage Extension 2.9.0, released 2025-09-01
+
+Code name: Upgrade of hadoop libraries
+
+## Summary
+This version upgrades hadoop from 3.3.6 to the latest 3.4.1, which fixes several CVEs in transient dependencies and
+leverages all the improvements the recent hadoop libs have.
+
+Security fixes which were fixed:
+
+### CVE-2025-48924: org.apache.commons:commons-lang3:jar:3.17.0:compile
+
+Uncontrolled Recursion vulnerability in Apache Commons Lang.
+
+This issue affects Apache Commons Lang: Starting with commons-lang:commons-lang 2.0 to 2.6, and, from org.apache.commons:commons-lang3 3.0 before 3.18.0.
+
+The methods ClassUtils.getClass(...) can throw StackOverflowError on very long inputs. Because an Error is usually not handled by applications and libraries, a
+StackOverflowError could cause an application to stop.
+
+#### References
+* https://ossindex.sonatype.org/vulnerability/CVE-2025-48924?component-type=maven&component-name=org.apache.commons%2Fcommons-lang3&utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1
+* http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2025-48924
+* GHSA-j288-q9x7-2f5v
+
+### CVE-2025-53864: com.nimbusds:nimbus-jose-jwt:jar:9.47:compile
+
+Connect2id Nimbus JOSE + JWT before 10.0.2 allows a remote attacker to cause a denial of service via a deeply nested JSON object supplied in a JWT claim set, because of uncontrolled recursion. NOTE: this is independent of the Gson 2.11.0 issue because the Connect2id product could have checked the JSON object nesting depth, regardless of what limits (if any) were imposed by Gson.
+
+#### References
+* https://ossindex.sonatype.org/vulnerability/CVE-2025-53864?component-type=maven&component-name=com.nimbusds%2Fnimbus-jose-jwt&utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1
+* http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2025-53864
+* https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/583/stackoverflowerror-due-to-deeply-nested
+
+### CVE-2025-55163: io.netty:netty-codec-http2:jar:4.1.119.Final:compile
+
+Netty is an asynchronous, event-driven network application framework. Prior to versions 4.1.124.Final and 4.2.4.Final, Netty is vulnerable to MadeYouReset DDoS. This is a logical vulnerability in the HTTP/2 protocol, that uses malformed HTTP/2 control frames in order to break the max concurrent streams limit - which results in resource exhaustion and distributed denial of service. This issue has been patched in versions 4.1.124.Final and 4.2.4.Final.
+
+#### References
+* https://ossindex.sonatype.org/vulnerability/CVE-2025-55163?component-type=maven&component-name=io.netty%2Fnetty-codec-http2&utm_source=ossindex-client&utm_medium=integration&utm_content=1.8.1
+* http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2025-55163
+* GHSA-prj3-ccx8-p6x4
+
+## Features
+
+* #310: Upgrade spark and hadoop versions
+
+## Dependency Updates
+
+### Cloud Storage Extension
+
+#### Compile Dependency Updates
+
+* Added `com.google.code.gson:gson:2.13.1`
+* Updated `io.netty:netty-codec-http2:4.1.119.Final` to `4.1.124.Final`
+* Updated `org.apache.commons:commons-lang3:3.17.0` to `3.18.0`
+* Updated `org.apache.hadoop:hadoop-aws:3.3.6` to `3.4.1`
+* Updated `org.apache.hadoop:hadoop-azure-datalake:3.3.6` to `3.4.1`
+* Updated `org.apache.hadoop:hadoop-azure:3.3.6` to `3.4.1`
+* Updated `org.apache.hadoop:hadoop-common:3.3.6` to `3.4.1`
+* Updated `org.apache.hadoop:hadoop-hdfs-client:3.3.6` to `3.4.1`
+* Updated `org.apache.hadoop:hadoop-hdfs:3.3.6` to `3.4.1`
+* Updated `org.apache.orc:orc-core:1.9.6` to `1.9.5`
+* Added `software.amazon.awssdk:s3-transfer-manager:2.32.31`
+* Added `software.amazon.awssdk:s3:2.32.31`
+
+#### Plugin Dependency Updates
+
+* Updated `com.exasol:error-code-crawler-maven-plugin:2.0.3` to `2.0.4`
+* Updated `com.exasol:project-keeper-maven-plugin:5.1.0` to `5.2.3`

.github/workflows/dependencies_check.yml

@@ -150,7 +150,7 @@ downloaded jar file is the same as the checksum provided in the releases.
 To check the SHA256 result of the local jar, run the command:
 
 ```sh
-sha256sum exasol-cloud-storage-extension-2.8.8.jar
+sha256sum exasol-cloud-storage-extension-2.9.0.jar
 ```
 
 ### Building From Source
@@ -180,7 +180,7 @@ mvn clean package -DskipTests=true
 ```
 
 The assembled jar file should be located at
-`target/exasol-cloud-storage-extension-2.8.8.jar`.
+`target/exasol-cloud-storage-extension-2.9.0.jar`.
 
 ### Create an Exasol Bucket
 
@@ -202,7 +202,7 @@ for the HTTP protocol.
 Upload the jar file using curl command:
 
 ```sh
-curl -X PUT -T exasol-cloud-storage-extension-2.8.8.jar \
+curl -X PUT -T exasol-cloud-storage-extension-2.9.0.jar \
   http://w:<WRITE_PASSWORD>@exasol.datanode.domain.com:2580/<BUCKET>/
 ```
 
@@ -237,7 +237,7 @@ OPEN SCHEMA CLOUD_STORAGE_EXTENSION;
 
 CREATE OR REPLACE JAVA SET SCRIPT IMPORT_PATH(...) EMITS (...) AS
   %scriptclass com.exasol.cloudetl.scriptclasses.FilesImportQueryGenerator;
-  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.8.8.jar;
+  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.9.0.jar;
 /
 
 CREATE OR REPLACE JAVA SCALAR SCRIPT IMPORT_METADATA(...) EMITS (
@@ -247,12 +247,12 @@ CREATE OR REPLACE JAVA SCALAR SCRIPT IMPORT_METADATA(...) EMITS (
   end_index DECIMAL(36, 0)
 ) AS
   %scriptclass com.exasol.cloudetl.scriptclasses.FilesMetadataReader;
-  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.8.8.jar;
+  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.9.0.jar;
 /
 
 CREATE OR REPLACE JAVA SET SCRIPT IMPORT_FILES(...) EMITS (...) AS
   %scriptclass com.exasol.cloudetl.scriptclasses.FilesDataImporter;
-  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.8.8.jar;
+  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.9.0.jar;
 /
 ```
 
@@ -271,12 +271,12 @@ OPEN SCHEMA CLOUD_STORAGE_EXTENSION;
 
 CREATE OR REPLACE JAVA SET SCRIPT EXPORT_PATH(...) EMITS (...) AS
   %scriptclass com.exasol.cloudetl.scriptclasses.TableExportQueryGenerator;
-  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.8.8.jar;
+  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.9.0.jar;
 /
 
 CREATE OR REPLACE JAVA SET SCRIPT EXPORT_TABLE(...) EMITS (ROWS_AFFECTED INT) AS
   %scriptclass com.exasol.cloudetl.scriptclasses.TableDataExporter;
-  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.8.8.jar;
+  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.9.0.jar;
 /
 ```
 
@@ -410,13 +410,13 @@ CREATE OR REPLACE JAVA SCALAR SCRIPT IMPORT_METADATA(...) EMITS (
 ) AS
   %jvmoption -DHTTPS_PROXY=http://username:password@10.10.1.10:1180
   %scriptclass com.exasol.cloudetl.scriptclasses.FilesMetadataReader;
-  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.8.8.jar;
+  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.9.0.jar;
 /
 
 CREATE OR REPLACE JAVA SET SCRIPT IMPORT_FILES(...) EMITS (...) AS
   %jvmoption -DHTTPS_PROXY=http://username:password@10.10.1.10:1180
   %scriptclass com.exasol.cloudetl.scriptclasses.FilesDataImporter;
-  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.8.8.jar;
+  %jar /buckets/bfsdefault/<BUCKET>/exasol-cloud-storage-extension-2.9.0.jar;
 /
 ```