Add regression tests

[pathawks]

It would be cool to be able to test two refs and warn of any differences between the two builds.

1. Build each site in sites with Jekyll version A
2. Build each site in sites with Jekyll version B
3. Diff the output of version B with the output of version A

I'm not sure there would be an automated way to decide if the differences were minimal enough to be acceptable. Perhaps the diff could just be posted as is, so that it could be reviewed by a person. It should be obvious if eg. a tag or plugin has stopped working.

[ashmaroli]

In theory, you could build a bare-bones site with all tags needed to be observed, included in the index.md which in turn, is using a minimal layout (or simply layout: none). Then its just a matter of reading the generated index.html, and piping the two contents into a third page that can be manually screened by a person.

[pathawks]

That's mostly what our integration tests already do. The goal here is to specifically use real world sites. That way, we can also test popular plugins, and also check some of the weird ways that users use (and abuse) Jekyll in ways we haven't considered.

[ashmaroli]

That way, we can also test popular plugins, and also check some of the weird ways that users use (and abuse) Jekyll in ways we haven't considered.

👍 👍

[DirtyF]

Could a visual diff with Pupeteer headless browser help us spot differences in the rendering?

https://meowni.ca/posts/2017-puppeteer-tests/

[pathawks]

Perhaps. Hopefully there would be zero differences the vast majority of the time.

If there are zero differences in the output, this check should pass ✅
If there are differences in the generated output, the check should not fail but give a neutral status with maybe a link to the diff, and perhaps Puppeteer screenshots of the before and after?