Testing conformance to the kitty TGP (terminal graphics protocol)

[ctrlcctrlv]

I have recently decided to begin writing some conformance tests as typescripts for the Kitty TGP.

kitty-TGP-001.tar.gz

Here is the first one. The result is not altogether good for non-kitty TGP implementers.

kitty (baseline) (perfect pass)

2023-06-03.14-37-44.mp4

konsole (near-perfect pass)

konsole fails to composite as kitty does. Note how the alpha around the cartoon characters remains constant rather than gets progressively harsher.

2023-06-03.14-38-52.mp4

wezterm (catastrophic failure)

@wez's term fails in every way conceivable.

2023-06-03.14-39-52.mp4

[kovidgoyal]

wezterm as I recall has a bug that if you dont provide distinct image ids it overwrites the previous image. IIRC there is already an open issue about it.

[ctrlcctrlv]

Good idea, I'll make a test for that too. For typescript distributed conformance tests I can't really test the memory or file features as they won't work outside my environment, but not all conformance tests must be typescript's, they're just easiest to run as scriptreplay is part of util-linux.

I think a conformance test suite for TGP is needed. Thought about it while writing TFP, which I'm still working on.

[wez]

A couple of thoughts on the use of typescript:

• It is unfortunately not portable enough. macOS and perhaps other BSD systems have an incompatible implementation of script. In wezterm, I added a record/replay implementation that uses asciicast as the serialization format. The recording/replaying machinery in wezterm as well as the official tools from https://asciinema.org/ work on all platforms; I would suggest looking at using those to hold your test scenarios.
• Please consider making individual recordings for the different cases that you verify, and most importantly, accompanying them with a description of the conforming behavior. Ideally the conforming behavior would be able to be determined by machine, but that would require an additional set of protocols to be agreed upon and correctly implemented across terminals! The easier you can make it to verify conformance, the more valuable this collection will be to implementors! More concretely: the description should not be "do what you see when you run this in kitty", but more along the lines of describing what should and should not appear on screen for any given test scenario.
• The vttest utility occupies a similar space as this style of conformance suite; it has a few problems:
  • it requires running xterm with a specific configuration (that is difficult to run ad-hoc) in order to see 100% conformant behavior, making it difficult to use xterm as the visual baseline for comparison purposes
  • Doesn't explain what sequences are being tested in any given screen (but does tell you what you should expect to see on the screen)
  • There isn't an easy way to figure out what sequences are being sent by a given screen
  • Not possible to jump directly to a specific screen, and many screens look similar and run in sequence, making it very easy for human error to manifest when iterating on and verifying the implementation under-test.

[kovidgoyal]

When I considered making the existing grapics tests in kitty portable, I was planning to just write a simple DSL that would specify the graphics escape code to send to the terminal at specific size and then specify invariants for the terminal state machine such as cursor position, image position, z-index, etc. as well as expected responses from the terminal. This allows for automatic machine based testing, where an implementor just needs to write a few lines of code to read the DSL and convert the invariants to a form suitable for their terminal. It's not as simple as running vttest but then vttest while being simple on the surface to run is pretty useless for more powerful use cases such as automated test suites.

[kovidgoyal]

On Sat, Jun 03, 2023 at 09:17:17PM -0700, Fredrick Brennan wrote: Good idea, I'll make a test for that too. For `typescript` distributed conformance tests I can't really test the memory or file features as they won't work outside my environment, but not all conformance tests must be `typescript`'s, they're just easiest to run as `scriptreplay` is part of `util-linux`. I think a conformance test suite for TGP is needed. Thought about it while writing TFP, which I'm still working on.

Yes, definitely, someday, especially if I write a PTY server in kitten.

[drzraf]

I think it would be appreciated. Whether in form of raw PNG window captures (for a given set of commands/assets/terminal size/prompt, fg/bg color and timeout) or scriptreplay sequences, it would allow conformance evaluation on a (not-so-simple) spec.