Regression testing for Nuitka website

[kayhayen]

I would like to be able to make changes to the Website in good conscience. In the past, I have kept the output of the build in a repo and made diffs to convince myself that it's only better. This is more complicated these days.

There are frameworks for Python that are capable of running a local browser, chromium, firefox at least and take images of the rendered page. These then being determistic, could be kept and compared. Every time or when we consider it matters.

We should capture several pages with this, and by virtue of selecting the right one, with tables, etc. know that layouts are not broken and stay the same. Recently the Nuitka logo was the only thing broken by inlining the SVGs (or so we are thinking). It would be nice to be able to know, so we can act more agile. Specifically the PurgeCSS change feels scary.

Ideally we could cover the macOS browser too, but I don't see that happening in CI.

Updating the captures should be an option to run from update.py and then I could change CI to stop from deployment if the images are changed in anyway. Images of course must not have time stamps for that. We can later decide if we want to put those images into a separate repository, potentially a sub-module, but those also cause issues from past experience with Nuitka and the CPython test suites. Verifying would be a task to execute.

Let me know if I am missing something idea wise.

[dauryjey]

@kayhayen That sounds like a good idea. I’m not sure how open you are to including another node module, but in node we have this in Playwright: https://playwright.dev/docs/test-snapshots.

It’s really easy to implement and covers everything we need. We could write a few tests (nothing difficult to maintain long-term) and run them from update.py just like we do with the other tools. Then, as you suggested, we can have a way to update screenshots, we could even create a VS Code task for that.

[kayhayen]

Playwright hat python findings too. Nuitka supports those. Selenium is another candidate. Not sure how it compares. I used selenium in the past with python to automate browser games as it automatically exposed DOM which could allow for internal checks.

[kayhayen]

I was thinking it could write to cache files with one option and other files to compare to, then we call and image comparison tool of some sorts.

If we have access to Dom we could output also stats like tag counts, I don't know. Whatever gives us non regression testing for performance features. Or if a page e.g. does or doesn't load asciicinema.

[kayhayen]

Tag counts are of course non-sense, we could take those from the HTML, so outside of checking what JS is loaded, I guess there is not too much benefit of internally scripting the thing. I would appreciate it though, if it was possible one day to add a test for the stripe purchase process, but honestly I have fared well without it so far, and it's an entirely separate thing, so selenium would only come into play if we think it's easier to use, which it may be. I thought it's designed for testing whereas playwright might be more for real applications wanting to display HTML.

[kayhayen]

You haven't taken this one operational yet, with golden snapshots yet to be made I take. But I guess, we can open more issues and close this one, once we agree on golden ones.

I think for M4, you shall try and add those for PR Action runs (to create new ones and diffs), and for main to compare them and fail over if they are different without a commit message granting that. I would then at merge time put a [New Look] or such tag into the message, and golden artifacts would be stored under the commit ID of the last one with that tag.

[dauryjey]

Yeah, we can do that. I already have another branch to submit what we are missing

[kayhayen]

I think you can just add github workflows and have them execute on PR branches, can you not, or does this need some kind of allow listing from me, not sure now. Just try it, and if I have to allow it I will.