Description
Problem
I have been using https://github.com/meinaart/cypress-plugin-snapshots for a good while now. When I started to use it I quickly realized that it had a big flaw: using it on locals images were generated with subtle differences which would always cause failures. These differences were due to the fact that initial images were generated on X OS and on Y browser. Now everyone who was using a different combination of OS or browser than what was used initially would bump into this issue. This was a deal breaker. From a quick look I do not see any solution for it in this plugin either.
Semi solution
I have managed to create a solution that works for us:
- Do not run visual checks when in GUI mode.
- Run visual checks only in RUN mode and only in a specific docker container, which is also shared with the CI pipeline. Thus keeping the same OS+browser always when screenshots are generated.
- For local setup a simple shellscript that would slightly improve development with the visual tests: allow quickly selecting which test to run in RUN mode and to decide whether to accept or not the changes.
But this has certain inconveniences:
- Visual tests don't run in GUI mode. This means differences cannot quickly be seen and accepted in GUI. Devs need to manually run the separate command and to manually open files to check for differences.
- It breaks up the development cycle: you use GUI for general cypress assertion and another command for visual checks.
Theoretical solution
I've been thinking that maybe this whole process with docker could be automated and even connected with the GUI so that previously mentioned issues/inconveniences would be solved. The high level idea is something like this:
- Optionally with configuration enable taking screenshots through docker, for consistent images in any host. This would atleast imply a docker image configured.
- On visual assertion command spin up a container and generate the screenshot inside it (using cypress), instead of generating it in the host.
For this to work the main concerns are:
- How can we efficiently move the current DOM from host cypress into docker cypress? As we don't want to rerun all previous steps from a test given we want it to happen as fast as possible. Here I've been thinking that maybe we could save the current DOM from under the iframe into a simple html file. We would then inject this into the container, and the cypress from within would be specially configured to open this file. The other thing that needs to be supported here is loading of any outside resource of the DOM, for which we could connect the docker container directly to host network.
- How would we actually take the screenshot? In host cypress we could just register the parameters for the screenshot command and send it to the docker cypress to actually run it.
- Even if all this works wouldn't it be too slow? Indeed I would expect that spinning up the container, booting cypress and then taking the screenshot could be 3-5 seconds, or more. Infact starting a container is not that slow, that is quite fast. The most time that is spent in addition is on booting cypress. We could maybe have a very bare-bone configuration on this instance (like not using bundlers / typescript). An additional idea is that maybe we can have only one container doing all the screenshots during the whole lifecycle of the host cypress, this way we'd only need to boot once and we could probably do it before any tests start. The thing to figure out here would be how we can run cypress in RUN mode in an infinite loop manner, so it can continually accept new DOMs and make new screenshots.
- Would we need to share any cypress configs between host and docker instance? I am really unsure about this. Only thing that pops in my mind is maybe the default resolution.
Let me hear what you think, or whether you have different ideas. Also I am open to collaborate on the implementation for this.