Unable to run a job with main file point to s3 bucket

[nownikhil]

What question do you want to ask?

• [ Y ] ✋ I have searched the open/closed issues and my issue is not listed.
  Similar issue found: Unable to use S3 File as mainApplicationFile #996

Error

│ Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found                          │
│   at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)                                                                              │
│   at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)                                                                            │
│   at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)                                                                              │
│   at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)                                                                                     │
│   at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)                                                                             │
│   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)                                                                                     │
│   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)                                                                                            │
│   at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1831)                                                                                  │
│   at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:727)                                                                                           │
│   at org.apache.spark.util.DependencyUtils$.downloadFile(DependencyUtils.scala:264)                                                                      │
│   at org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:103)                                                         │
│   ... 18 more                                                                                                                                            │
│ Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found                                                      │
│   at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)                                                                        │
│   at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)                                                                              │

Additional context

This is happening because spark-operator image doesn't have hadoop-aws jar. Is there a recommended way to pull jars from S3?

No response

Have the same question?

Give it a 👍 We prioritize the question with most 👍

[ujjawal-khare]

Facing the same issue, the only solution that I found was to bake the hadoop jar inside the operator but not sure if this is the right way as we would be having dynamic jars coming in.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ChenYi015]

/lifecycle frozen

[aagumin]

Hi! @nownikhil
Pull the main application file from S3 using Apache Spark's native mechanism.
The S3 pull operation runs on the Apache Spark driver.

Add the hadoop-aws jars to your Spark application image (not the controller) and try this:

...
  driver:
    env:
    - name: AWS_SECRET_ACCESS_KEY
      value: secret-data
    - name: AWS_ACCESS_KEY_ID
      value: secret-data
 ...
hadoopConf:
    fs.s3a.aws.credentials.provider: com.amazonaws.auth.EnvironmentVariableCredentialsProvider
 

[Safarveisi]

@nownikhil The steps I followed to get everything running are outlined below:

1. I built a custom Docker image on top of the official Spark 3.5.2 base image that bundles every JAR dependency required to download the main application artifact directly from S3, and write a Spark DataFrame back to S3 (e.g., as Parquet or CSV). The versions of all bundled JARs must correspond exactly to the Spark version used at runtime. I would say this is the tricky part.

FROM docker.io/library/spark:3.5.2 

RUN curl -L -O http://search.maven.org/remotecontent?filepath=org/apache/ivy/ivy/2.5.3/ivy-2.5.3.jar

RUN java -Divy.cache.dir=/tmp -Divy.home=/tmp -jar ivy-2.5.3.jar -notransitive \
-dependency org.apache.hadoop hadoop-client-api 3.3.4 \
-retrieve "/opt/spark/jars/[artifact]-[revision](-[classifier]).[ext]"

RUN java -Divy.cache.dir=/tmp -Divy.home=/tmp -jar ivy-2.5.3.jar -confs compile \
-dependency org.apache.hadoop hadoop-client-api 3.3.4 \
-retrieve "/opt/spark/jars/[artifact]-[revision](-[classifier]).[ext]"

RUN java -Divy.cache.dir=/tmp -Divy.home=/tmp -jar ivy-2.5.3.jar -notransitive \
-dependency org.apache.hadoop hadoop-client-runtime 3.3.4 \
-retrieve "/opt/spark/jars/[artifact]-[revision](-[classifier]).[ext]"

RUN java -Divy.cache.dir=/tmp -Divy.home=/tmp -jar ivy-2.5.3.jar -confs compile \
-dependency org.apache.hadoop hadoop-client-runtime 3.3.4 \
-retrieve "/opt/spark/jars/[artifact]-[revision](-[classifier]).[ext]"

RUN java -Divy.cache.dir=/tmp -Divy.home=/tmp -jar ivy-2.5.3.jar -notransitive \
-dependency org.apache.hadoop hadoop-aws 3.3.4 \
-retrieve "/opt/spark/jars/[artifact]-[revision](-[classifier]).[ext]"

RUN java -Divy.cache.dir=/tmp -Divy.home=/tmp -jar ivy-2.5.3.jar -confs compile \
-dependency org.apache.hadoop hadoop-aws 3.3.4 \
-retrieve "/opt/spark/jars/[artifact]-[revision](-[classifier]).[ext]"

RUN java -Divy.cache.dir=/tmp -Divy.home=/tmp -jar ivy-2.5.3.jar -notransitive \
-dependency org.apache.hadoop hadoop-common 3.3.4 \
-retrieve "/opt/spark/jars/[artifact]-[revision](-[classifier]).[ext]"

RUN java -Divy.cache.dir=/tmp -Divy.home=/tmp -jar ivy-2.5.3.jar -confs compile \
-dependency org.apache.hadoop hadoop-common 3.3.4 \
-retrieve "/opt/spark/jars/[artifact]-[revision](-[classifier]).[ext]"

RUN java -Divy.cache.dir=/tmp -Divy.home=/tmp -jar ivy-2.5.3.jar -notransitive \
-dependency com.amazonaws aws-java-sdk-bundle 1.12.162 \
-retrieve "/opt/spark/jars/[artifact]-[revision](-[classifier]).[ext]"

RUN java -Divy.cache.dir=/tmp -Divy.home=/tmp -jar ivy-2.5.3.jar -confs compile \
-dependency com.amazonaws aws-java-sdk-bundle 1.12.162 \
-retrieve "/opt/spark/jars/[artifact]-[revision](-[classifier]).[ext]"

2. Here is the manifest file

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
   name: spark-python
   namespace: playground
spec:
   type: Python
   pythonVersion: "3"
   mode: cluster
   image: {image_created_above}
   imagePullPolicy: Always
   mainApplicationFile: s3a://bucket/prefix/spark_app.py
   sparkConf:
     spark.hadoop.fs.s3a.aws.credentials.provider: "com.amazonaws.auth.EnvironmentVariableCredentialsProvider"
     spark.hadoop.fs.s3a.path.style.access: "true"
     spark.hadoop.fs.s3a.endpoint: "{custom_endpoint}"
     spark.hadoop.fs.s3a.connection.ssl.enabled: "false"
     spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
     spark.hadoop.fs.s3a.endpoint.region: "{custom_region}"
sparkVersion: 3.5.2
driver:
  cores: 1
  memory: 512m
  serviceAccount: spark-operator-spark
  securityContext:
    capabilities:
      drop:
      - ALL
    runAsGroup: 185
    runAsUser: 185
    runAsNonRoot: true
    allowPrivilegeEscalation: false
    seccompProfile:
      type: RuntimeDefault
  env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: s3-credentials
          key: accessKey
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: s3-credentials
          key: secretKey
executor:
  instances: 1
  cores: 1
  memory: 512m
  securityContext:
    capabilities:
      drop:
      - ALL
    runAsGroup: 185
    runAsUser: 185
    runAsNonRoot: true
    allowPrivilegeEscalation: false
    seccompProfile:
      type: RuntimeDefault
  env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: s3-credentials
          key: accessKey
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: s3-credentials
          key: secretKey