JS tests are flaky

[mgeisler]

          Oh, this fails because of an unrelated timeout:

[0-1] Error in "Playground.executes the hello world code and prints the hello message"
Error: Expect  to have text

Expected: StringContaining "🌍"
Received: "The operation timed out: deadline has elapsed"
    at Context.<anonymous> (/home/runner/work/comprehensive-rust/comprehensive-rust/tests/src/playground.test.ts:[30](https://github.com/google/comprehensive-rust/actions/runs/12767680980/job/35586546795?pr=2548#step:15:31):37)
[0-1] Error in "Playground.shows error messages in stderr if the code is broken"
Error: Expect $(`code.result.stderr`) to be displayed

The new JS test is from #2513 (cc @michael-kerscher). We might need to disable it for now if it turns out to be flaky.

Originally posted by @mgeisler in #2548 (comment)

[max-heller]

Getting the same error on my latest commit merged to main

[mgeisler]

Thanks @max-heller, let's disable this test for now!

[michael-kerscher]

I'm not entirely sure yet how to proceed from here. I would like to test the playground, as this is an important bit of the course book - but I also don't want to have flaky tests that don't have anything to do with regular commits.
There would be some options though that I wanted to document and bring up for discussion:

• increase timeouts for tests and/or some retry behavior (in production not test code)
• detect if relevant files have been changed (e.g. filter for templates and javascript only). A change in the markdown part or translations should not modify the behavior of the html files. This limits impact of broken tests to relevant changes only.
• mock the playground server https://webdriver.io/docs/mocksandspies/ in such a way that we don't need to rely on external dependencies
  Example on how to do this:

    // mock the response so this test does not rely on the playground backend to be working
    const playground_mock = await browser.mock(
      "https://play.rust-lang.org/execute",
      { method: "post" }
    );

    // be aware that there is a still a preflight OPTIONS request
    playground_mock.respond(
      {
        success: true,
        exitDetail: "Exited with status 0",
        stdout: "Hello 🌍!\n",
        stderr:
          "   Compiling playground v0.0.1 (/playground)\n    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.62s\n     Running `target/debug/playground`\n",
      },
      {
        headers: {
          "Access-Control-Allow-Origin": "*",
        },
        fetchResponse: false,
      }
    );

[mgeisler]

• increase timeouts for tests and/or some retry behavior (in production not test code)

That would be my first suggestion too. If there are some timeouts we can adjust across the board, then we should do that. If there are retries we can enable easily, then we should do that too.

[michael-kerscher]

Regarding timeouts, there is our own 15 second timeout:

comprehensive-rust/theme/book.js

Line 31 in 3cca473

function fetch_with_timeout(url, options, timeout = 15000) {

But there are also HTTP 500 results from the playground server with this body:

{"error":"The operation timed out: deadline has elapsed"}

I also noticed that if the second variant occurs, this can occur multiple times - so retrying is not likely to succeed...

We could try to determine if we get any of the above error situations in the tests and ignore them explicitly - these results are most likely not due to issues on our side so our test infrastructure should not fail (but log warnings)

[djmitche]

Is it possible to adjust the browser automation to intercept the fetch request to the playground and return a value without actually interacting with the playground server?

[michael-kerscher]

that is the mocking approach that I mentioned above. Unfortunately I noticed that fetchResponse: false is documented but has no effect. I created a bug in webdriverio for that - but nevertheless it could overwrite the HTTP 500 - it does wait for the actual request to be finished currently. There is also a bug regarding using unicode that is currently worked on. So yes, something like that will be possible and useful quite soon.