Provide guidelines on validation testing of vega charts

[ohltyler]

Currently vega charts are all rendered using vega view, which when rendered on a page, just produces a single canvas html element. This limits being able to validate what is actually being rendered and if certain datapoints are visible as expected or not. vislib-implemented charts do not have this problem as each datapoint / legend / axis are separate elements and can be validated by a test runner (cypress).

A few potential options are:

1. introduce an image comparison testing framework in the functional-test repo to compare generated chart canvas images with persisted ones and find any mismatches
2. explore if validation on the source datatable can be relied upon as enough sufficient validation
3. explore different ways to render vega charts and if the individual datapoints can be surfaced in html

Ideally we can come up with a final plan and have it documented, then begin adding consistent validation testing on all existing tests that are generating vega-implemented charts.

[ashwin-pc]

Right now canvas based charts are validated using visual regresion testing. Unfortunately these tests only exist in the selenium environment and not in the cypress test runner. We have two options here, I'll start with the one i prefer, addng visual regression testing to the functional test repo and adding tests there, the other is to use the existing selenium tests to create these tests. Here is an example of the selenium test that performs a visual regression test on the chart alone: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/test/functional/apps/dashboard/dashboard_snapshots.js

[wbeckler]

Since Selenium is on its way out. We need to come up with a cypress solution. Found this but I'm not sure if it's best practice: https://github.com/cypress-visual-regression/cypress-visual-regression

Can we consider chucking Vega and using a library with easier testability? For what purpose is this test?

[joshuarrrr]

Can we consider chucking Vega and using a library with easier testability?

This isn't really a Vega/Vega-lite problem, it's a limitation of the rendering technique and test frameworks. SVG, Canvas, and WebGL visualization rendering all have their pros and cons, but SVG-based rendering is not inherently better. The bigger question is about the general test strategies we should use for validating visualization rendering.

I agree with @ashwin-pc's suggestion to add visual regression testing to the cypress tests as the preferred way forward here.

[ohltyler]

I'm currently experimenting with cypress-image-snapshot built off of jest-image-snapshot, and I'm able to get snapshots of the individual charts to show. There's some nice settings to tune how exact the matching should be, which can help account for small differences such as timestamp labels, colors, etc.

[ohltyler]

@ashwin-pc / @joshuarrrr please see this FTRepo PR which adds cypress-image-snapshot + some tests that use it :)