[ZEPPELIN-6195] Fix Selenium tests by switching from Chrome to Firefox and upgrading outdated Spark by ParkGyeongTae · Pull Request #4941 · apache/zeppelin

ParkGyeongTae · 2025-06-07T03:21:59Z

What is this PR for?

The test-selenium-with-spark-module-for-spark-3-4 job in GitHub Actions was consistently failing with the error:

from unknown command: 'Runtime.evaluate' wasn't found

This issue was caused by ChromeDriver not recognizing the Runtime.evaluate command.
To resolve it, the browser used in the test was switched from Chrome to Firefox.

During the migration to Firefox, another issue occurred:

Unable to download from https://dlcdn.apache.org/spark/spark-3.4.2/spark-3.4.2-bin-hadoop3.tgz
java.io.FileNotFoundException: https://dlcdn.apache.org/spark/spark-3.4.2/spark-3.4.2-bin-hadoop3.tgz

This was due to the Spark 3.4.2 binaries no longer being available from the Apache mirror.
The test environment was therefore updated to use Spark 3.5.6, which resolved the download problem.

Summary of changes:

Switched Selenium tests from Chrome to Firefox (non-snap) using geckodriver.
Updated Spark version in the test environment from 3.4.2 to 3.5.6 to fix missing binary downloads.

Before: test-selenium-with-spark-module-for-spark-3-4 (Chrome, Spark 3.4.2)
After: test-selenium-with-spark-module-for-spark-3-5 (Firefox, Spark 3.5.6)

What type of PR is it?

Bug Fix

Todos

- Verified that other integration tests (if any) are not affected
- Confirmed long-term compatibility with the Chrome for Testing approach

What is the Jira issue?

Jira: https://issues.apache.org/jira/browse/ZEPPELIN-6195

How should this be tested?

Run the test-selenium-with-spark-module-for-spark-3-4 job in GitHub Actions.
Verify that AuthenticationIT and any other affected Selenium tests pass without encountering Runtime.evaluate errors.

Screenshots (if appropriate)

N/A

Questions:

Does the license files need to update? No.
Is there breaking changes for older versions? No.
Does this needs documentation? No.

…rome/ChromeDriver to 125 and upgrading selenium-java to 4.20.0

Reamer · 2025-06-10T07:01:19Z

Thank you for wanting to correct the Selenium tests. I see that your solution works. However, I find the downgrade very problematic, as it is not future-proof.

I had also looked at the problem some time ago and found a problem with the maximize method.

zeppelin/zeppelin-integration/src/test/java/org/apache/zeppelin/WebDriverManager.java

Line 159 in 94ae845

driver.manage().window().maximize();

The problem with Selenium is analyzed in more detail here.
SeleniumHQ/selenium#15358

I would suggest a fixed height and width. Unfortunately, my initial tests were not successful. Perhaps you can find suitable values.

I like the update of the Selenium version, but not the downgrade of Chrome.

ParkGyeongTae · 2025-06-10T09:40:11Z

@Reamer
I always appreciate your feedback. I understand your point, and I’ve reviewed the discussion in the link you shared.
I’ll look into a way to keep Chrome up to date while ensuring the tests run without errors.

pan3793 · 2025-08-01T14:54:07Z

The issue looks to be there for a long time, if it is specific to Chrome, how about switching the test to Firefox?

Reamer · 2025-08-02T11:58:41Z

Firefox has other bugs as far as I remember

… maximize window issue

ParkGyeongTae · 2025-08-02T15:56:12Z

@pan3793 @Reamer
Thanks for the suggestion! I switched the tests to Firefox, and they are now passing successfully. Please take a look and let me know if you have any further concerns.

Reamer

Thank you for looking at the issue again. In my opinion, too much useful code for other platforms has been removed.

…fox in integration tests

…to improve test stability

pan3793 · 2025-08-06T10:34:27Z

I see the failed run-e2e-tests-in-zeppelin-web still prints

[INFO] [09:56:05] I/launcher - Running 1 instances of WebDriver
[INFO] [09:56:05] I/direct - Using ChromeDriver directly...

…n-e2e-tests-in-zeppelin-web job

Reamer

Please completely revert your changes to the run-e2e-tests-in-zeppelin-web test. We should stick with one frontend test for this pull request and not touch both.

…state and scope Firefox setting to spark-3-5 job only

Reamer

LGTM

Reamer

The question still needs to be clarified.

….4.2

pan3793 · 2025-08-08T08:19:03Z

@ParkGyeongTae thanks for fixing this, could you please update the PR title and description to reflect the final solution?

ParkGyeongTae · 2025-08-08T08:38:45Z

@pan3793
I’ve updated the PR title and description, as well as the Jira issue title and description.

…x and upgrading outdated Spark ### What is this PR for? The `test-selenium-with-spark-module-for-spark-3-4` job in GitHub Actions was consistently failing with the error: ```bash from unknown command: 'Runtime.evaluate' wasn't found ``` This issue was caused by ChromeDriver not recognizing the `Runtime.evaluate` command. To resolve it, the browser used in the test was switched from Chrome to Firefox. During the migration to Firefox, another issue occurred: ```bash Unable to download from https://dlcdn.apache.org/spark/spark-3.4.2/spark-3.4.2-bin-hadoop3.tgz java.io.FileNotFoundException: https://dlcdn.apache.org/spark/spark-3.4.2/spark-3.4.2-bin-hadoop3.tgz ``` This was due to the Spark 3.4.2 binaries no longer being available from the Apache mirror. The test environment was therefore updated to use Spark 3.5.6, which resolved the download problem. Summary of changes: - Switched Selenium tests from Chrome to Firefox (non-snap) using geckodriver. - Updated Spark version in the test environment from 3.4.2 to 3.5.6 to fix missing binary downloads. Before: test-selenium-with-spark-module-for-spark-3-4 (Chrome, Spark 3.4.2) After: test-selenium-with-spark-module-for-spark-3-5 (Firefox, Spark 3.5.6) ### What type of PR is it? Bug Fix ### Todos * [x] - Verified that other integration tests (if any) are not affected * [x] - Confirmed long-term compatibility with the Chrome for Testing approach ### What is the Jira issue? * Jira: https://issues.apache.org/jira/browse/ZEPPELIN-6195 ### How should this be tested? - Run the `test-selenium-with-spark-module-for-spark-3-4` job in GitHub Actions. - Verify that `AuthenticationIT` and any other affected Selenium tests pass without encountering `Runtime.evaluate` errors. ### Screenshots (if appropriate) N/A ### Questions: * Does the license files need to update? No. * Is there breaking changes for older versions? No. * Does this needs documentation? No. Closes #4941 from ParkGyeongTae/fix-selenium-runtime-evaluate-error. Signed-off-by: Philipp Dallig <philipp.dallig@gmail.com> (cherry picked from commit 8ec487f) Signed-off-by: Philipp Dallig <philipp.dallig@gmail.com>

Reamer · 2025-08-08T09:50:46Z

Merged into master and branch-0.12

### What is this PR for? This PR stabilizes the `test-selenium...` job by switching the Selenium browser from Firefox to Microsoft Edge. In #4941, we worked around a Linux ChromeDriver issue (SeleniumHQ/selenium#15358) by moving from Chrome to Firefox, but that workaround also seems to be not working well. I use Edge driver instead, since it does not have flakiness like Firefox driver and is more aligned with Chrome since it is also based on Chromium. Limitation: The root cause of Firefox browser failures has not been fully analyzed. ### What type of PR is it? Bug Fix ### What is the Jira issue? [[ZEPPELIN-6350]](https://issues.apache.org/jira/browse/ZEPPELIN-6350) ### How should this be tested? - Check `test-selenium...` job in `frontend` ### Questions: * Does the license files need to update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Closes #5089 from tbonelee/fix-selenium-test. Signed-off-by: Chan Lee <chanho0325@gmail.com>

### What is this PR for? This PR stabilizes the `test-selenium...` job by switching the Selenium browser from Firefox to Microsoft Edge. In #4941, we worked around a Linux ChromeDriver issue (SeleniumHQ/selenium#15358) by moving from Chrome to Firefox, but that workaround also seems to be not working well. I use Edge driver instead, since it does not have flakiness like Firefox driver and is more aligned with Chrome since it is also based on Chromium. Limitation: The root cause of Firefox browser failures has not been fully analyzed. ### What type of PR is it? Bug Fix ### What is the Jira issue? [[ZEPPELIN-6350]](https://issues.apache.org/jira/browse/ZEPPELIN-6350) ### How should this be tested? - Check `test-selenium...` job in `frontend` ### Questions: * Does the license files need to update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Closes #5089 from tbonelee/fix-selenium-test. Signed-off-by: Chan Lee <chanho0325@gmail.com> (cherry picked from commit 3d1ee7c) Signed-off-by: Chan Lee <chanho0325@gmail.com>

[ZEPPELIN-6195] Fix Selenium Runtime.evaluate error by downgrading Ch…

cdec556

…rome/ChromeDriver to 125 and upgrading selenium-java to 4.20.0

Reamer self-assigned this Jun 13, 2025

[ZEPPELIN-6195] Switch Selenium tests from Chrome to Firefox to avoid…

ab8ead0

… maximize window issue

Reamer requested changes Aug 4, 2025

View reviewed changes

Comment thread .github/workflows/frontend.yml

Comment thread zeppelin-integration/src/test/java/org/apache/zeppelin/WebDriverManager.java

Comment thread zeppelin-integration/src/test/java/org/apache/zeppelin/WebDriverManager.java Outdated

ParkGyeongTae added 2 commits August 4, 2025 20:25

[ZEPPELIN-6195] Add support for browser selection and default to Fire…

2b1d8bd

…fox in integration tests

[ZEPPELIN-6195] Increase WebDriverWait timeout from 30 to 60 seconds …

d32e62d

…to improve test stability

pan3793 reviewed Aug 6, 2025

View reviewed changes

Comment thread zeppelin-integration/src/test/java/org/apache/zeppelin/WebDriverManager.java Outdated

ParkGyeongTae added 3 commits August 6, 2025 22:40

[ZEPPELIN-6195] Revert indentation changes

0896a85

[ZEPPELIN-6195] Use Firefox instead of Chrome for web E2E tests in ru…

1657373

…n-e2e-tests-in-zeppelin-web job

[ZEPPELIN-6195] Revert indentation changes

3b7ca0d

Reamer requested changes Aug 7, 2025

View reviewed changes

Comment thread .github/workflows/frontend.yml Outdated

[ZEPPELIN-6195] Revert run-e2e-tests-in-zeppelin-web job to original …

48524ac

…state and scope Firefox setting to spark-3-5 job only

Reamer approved these changes Aug 7, 2025

View reviewed changes

Reamer requested changes Aug 7, 2025

View reviewed changes

Comment thread zeppelin-web/package-lock.json Outdated

Comment thread zeppelin-web/package-lock.json Outdated

[ZEPPELIN-6195] Update duplexify dependency version from ^3.5.6 to ^3…

2b488d8

….4.2

Reamer approved these changes Aug 8, 2025

View reviewed changes

pan3793 changed the title ~~[ZEPPELIN-6195] Fix test-selenium-with-spark-module-for-spark-3-4 job failure caused by “from unknown command: 'Runtime.evaluate' wasn't found” error~~ [ZEPPELIN-6195] Fix Selenium tests by switching from Chrome to Firefox and upgrading outdated Spark Aug 8, 2025

pan3793 approved these changes Aug 8, 2025

View reviewed changes

Reamer merged commit 8ec487f into apache:master Aug 8, 2025
29 of 33 checks passed

tbonelee mentioned this pull request Sep 30, 2025

[ZEPPELIN-6350] Fix Selenium test by Replacing Firefox with Edge #5089

Merged

Conversation

ParkGyeongTae commented Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

Reamer commented Jun 10, 2025

Uh oh!

ParkGyeongTae commented Jun 10, 2025

Uh oh!

pan3793 commented Aug 1, 2025

Uh oh!

Reamer commented Aug 2, 2025

Uh oh!

ParkGyeongTae commented Aug 2, 2025

Uh oh!

Reamer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pan3793 commented Aug 6, 2025

Uh oh!

Uh oh!

Reamer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reamer left a comment

Choose a reason for hiding this comment

Uh oh!

Reamer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pan3793 commented Aug 8, 2025

Uh oh!

ParkGyeongTae commented Aug 8, 2025

Uh oh!

Uh oh!

Reamer commented Aug 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ParkGyeongTae commented Jun 7, 2025 •

edited

Loading