Skip to content

7066: Move static properties to instance properties#7135

Draft
ca61688 wants to merge 11 commits intodevelopfrom
7066-configure-spark-nodes-on-eks
Draft

7066: Move static properties to instance properties#7135
ca61688 wants to merge 11 commits intodevelopfrom
7066-configure-spark-nodes-on-eks

Conversation

@ca61688
Copy link
Copy Markdown
Collaborator

@ca61688 ca61688 commented May 7, 2026

Make sure you have checked all steps below.

Issue

  • My PR fully resolves the following issues. I've referenced an issue in the PR title, for example "Issue 1234 - My
    Feature". Note that before an issue is finished, you can still make a pull request by raising a separate issue
    for your progress.

Tests

  • My PR adds the following tests based on our test strategy OR does not need testing for this extremely good reason:
    • WremockITs
    • Ran EMRBulkImportST & EKSBulkImportST which both pased

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it, or I have linked to a
    separate issue for that below.
  • If I have added new Java code, I have added Javadoc that explains it following our conventions and style.
  • If I have added or removed any dependencies from the project, I have updated the NOTICES file.

@ca61688 ca61688 linked an issue May 7, 2026 that may be closed by this pull request
@ca61688
Copy link
Copy Markdown
Collaborator Author

ca61688 commented May 7, 2026

Core change: separate EKS Spark configuration from EMR
The branch introduces dedicated Spark configuration support for bulk import on EKS, separating it from the existing EMR properties.

New/removed property classes (java/core/.../properties/instance/)
EKSProperty.java (new) - defines EKS-specific Spark properties: executor instances, driver/executor memory, memory overhead, executor cores, and default parallelism. All with EKS-appropriate defaults (e.g. 12 g memory, tuned for Fargate).
BulkImportProperty.java (new) - extracts properties shared between EMR and EKS modes: BULK_IMPORT_CLASS_NAME, speculation settings, shuffle codec, executor Java options, etc. Previously these lived in EMRProperty.
EMRProperty.java (deleted) - properties split between BulkImportProperty (shared) and the remaining EMR-specific classes.

StateMachinePlatformExecutor.java
Simplified - Spark config construction moved into ConfigurationUtils, reducing duplication.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Configure Spark nodes on EKS to match EMR

1 participant