fix: install the raydp Python package (not just the jars) by jiaoew1991 · Pull Request #92 · lumalabs/nurion

jiaoew1991 · 2026-06-10T02:38:46Z

No description provided.

Build engine/Dockerfile (ubuntu-based: py3.12, JDK17, Ray, RayDP spark4.1/scala2.13, lance-spark, pylance) and push to ghcr.io/lumalabs/nurion-engine so downstream images can FROM it without cross-account AWS ECR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

lance-spark 0.5.1 is the latest (DataFrame write support); pin pylance to 7.0.0 to match lance-spark 0.5.1's lance-core (unified 7.0.0 version line). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…bug)

The RayDP fork's protobuf-maven-plugin reads its gRPC proto from ${project.basedir}/../../../anvil-rs/proto (the sibling lib/anvil-rs/proto). The base-image Dockerfile flattened the COPY layout (/app/build/anvil-rs, /app/build/java) so that relative path missed, and it deleted anvil-rs before the RayDP maven build. With no proto, protobuf-maven-plugin generated nothing and scalac failed with 6 "not found: PushRequest/PushResponse/AnvilGrpc" errors. Copy the sources into /app/build/lib/{anvil-rs,raydp/java} to preserve the relative proto path, and merge the anvil-rs + RayDP builds into one layer so the proto survives through the maven build. Reverts the -e debug flag added while diagnosing. Verified locally: mvn -P scala-2.13 clean package -> BUILD SUCCESS, AnvilGrpc.java generated under target/generated-sources, all three raydp jars produced. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The base built the RayDP Scala jars but never installed the raydp Python package, so `import raydp` / raydp.init_spark() failed for consumers (e.g. data-api query_processor builds on this base). Stage the raydp Python source plus the already-built Scala-2.13 jars into site-packages (raydp/utils.py resolves jars at <pkg>/jars/); runtime deps (ray, pyspark, pandas) are already installed above. No Maven rebuild, and downstream images need no nurion git access to get `import raydp`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

cursor · 2026-06-10T02:38:51Z

PR Summary

Medium Risk
Touches container build/publish and pins runtime versions (pylance, lance-spark); incorrect proto/JAR layout could break Spark/RayDP in downstream images, but scope is CI and base image only.

Overview
Adds a GitHub Actions workflow that builds engine/Dockerfile and pushes ghcr.io/lumalabs/nurion-engine with :develop and :${{ github.sha }} tags (triggered on branch push or manual dispatch), so downstream images can base on GHCR instead of cross-account ECR.

engine/Dockerfile changes make the image a fuller Spark-on-Ray base: lib/ is copied under /app/build/lib/ so RayDP’s Maven build can resolve anvil-rs protos; anvil-rs wheel build and RayDP JAR packaging run in one layer with shared cleanup. Runtime deps bump lance-spark-bundle to 0.5.1 and pin pylance to 7.0.0. The raydp Python package (sources + built JARs under site-packages/raydp/jars) is installed so consumers get import raydp without extra nurion git access.

^{Reviewed by Cursor Bugbot for commit e0a80cf. Bugbot is set up for automated code reviews on this repo. Configure here.}

cursor

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Bugbot Autofix prepared fixes for both issues found in the latest run.

✅ Fixed: Invalid pylance version pin
- Changed pylance==7.0.0 to pylance>=4.0.0 to match the version used throughout the project and available on PyPI.
✅ Fixed: Develop tag from feature branch
- Updated workflow trigger from feature branch [enwei/publish-engine-ghcr] to [develop, main] so the :develop tag tracks the actual develop branch.

Or push these changes by commenting:

@cursor push f2218a9a8a

Preview (f2218a9a8a)

diff --git a/.github/workflows/publish-engine-base.yaml b/.github/workflows/publish-engine-base.yaml
--- a/.github/workflows/publish-engine-base.yaml
+++ b/.github/workflows/publish-engine-base.yaml
@@ -7,7 +7,7 @@
 
 on:
     push:
-        branches: [enwei/publish-engine-ghcr]
+        branches: [develop, main]
     workflow_dispatch:
 
 permissions:

diff --git a/engine/Dockerfile b/engine/Dockerfile
--- a/engine/Dockerfile
+++ b/engine/Dockerfile
@@ -92,7 +92,7 @@
     pandas>=2.0.0 \
     click>=8.1.7 \
     "fsspec[s3]>=2024.6.0" \
-    pylance==7.0.0 \
+    pylance>=4.0.0 \
     s3fs>=2024.6.0 \
     grpcio>=1.68.0

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit e0a80cf. Configure here.}

cursor · 2026-06-10T02:40:25Z

    click>=8.1.7 \
    "fsspec[s3]>=2024.6.0" \
-    pylance>=0.38.0 \
+    pylance==7.0.0 \


Invalid pylance version pin

High Severity

The Dockerfile pins pylance==7.0.0, but that release is not published on PyPI (the Lance SDK tops out at 6.0.1). The uv pip install step in the engine base image build should fail, so the new GHCR publish workflow cannot produce a working image until the version is corrected.

^{Reviewed by Cursor Bugbot for commit e0a80cf. Configure here.}

cursor · 2026-06-10T02:40:25Z

+                  push: true
+                  tags: |
+                      ghcr.io/lumalabs/nurion-engine:develop
+                      ghcr.io/lumalabs/nurion-engine:${{ github.sha }}


Develop tag from feature branch

Medium Severity

The workflow runs only on pushes to enwei/publish-engine-ghcr, yet it always pushes ghcr.io/lumalabs/nurion-engine:develop. Downstream images that FROM that tag will pull builds from this feature branch, not from the repo’s develop branch, and merges to develop/main will not refresh the tag automatically.

^{Reviewed by Cursor Bugbot for commit e0a80cf. Configure here.}

github-actions · 2026-06-10T03:03:04Z

Coverage Report

Overall: 56%

Diff Coverage (changed files only)

Diff Coverage

Diff: origin/develop...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

jiaoew1991 and others added 5 commits June 9, 2026 12:44

engine: bump lance-spark 4.1_2.13 0.4.0->0.5.1 + pin pylance==7.0.0

dd9248f

lance-spark 0.5.1 is the latest (DataFrame write support); pin pylance to 7.0.0 to match lance-spark 0.5.1's lance-core (unified 7.0.0 version line). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

engine: drop -q on raydp mvn build to surface scala-compile error (de…

3b05556

…bug)

jiaoew1991 changed the title ~~Enwei/publish engine ghcr~~ fix: install the raydp Python package (not just the jars) Jun 10, 2026

cursor Bot reviewed Jun 10, 2026

View reviewed changes

jiaoew1991 merged commit 1c30e2a into develop Jun 10, 2026
18 of 19 checks passed

jiaoew1991 mentioned this pull request Jun 11, 2026

feat(engine): build lance-spark bundle from main (lance-core 8.x) + pyspark 4.1.2 #94

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: install the raydp Python package (not just the jars) - #92

fix: install the raydp Python package (not just the jars)#92
jiaoew1991 merged 5 commits into
developfrom
enwei/publish-engine-ghcr

jiaoew1991 commented Jun 10, 2026

Uh oh!

cursor Bot commented Jun 10, 2026 •

edited

Loading

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

cursor Bot Jun 10, 2026

Uh oh!

cursor Bot Jun 10, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 10, 2026

Diff Coverage

Diff: origin/develop...HEAD, staged and unstaged changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jiaoew1991 commented Jun 10, 2026

Uh oh!

cursor Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 10, 2026

Choose a reason for hiding this comment

Invalid pylance version pin

Uh oh!

cursor Bot Jun 10, 2026

Choose a reason for hiding this comment

Develop tag from feature branch

Uh oh!

Uh oh!

github-actions Bot commented Jun 10, 2026

Coverage Report

Diff Coverage

Diff: origin/develop...HEAD, staged and unstaged changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cursor Bot commented Jun 10, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading