Recommended ways to test completion generation, or generation stability

[nc7s]

I've been packaging a few applications which use clap_complete to generate shell completions, and some of them use snapshot tests to cover them. However, minor changes happen all the time across minor or even patch level versions of clap_complete, and even a single character change fails the tests because the snapshots are just strings, and "snapshot tests" are just string comparisons. Some even go as far as maintaining a list of strings to replace in the output, which of course fail when there are changes, even if tiny.

Thus I'm wondering, is there some kind of stability in the generation? Is clap_complete free to change its output? What are the recommended ways to test the results? Simply testing for string equality already fails a lot as the above shows.

FWIW, I know shell languages are rather less formal, so a "no" is expected. But it's still nice if there are ways to do it ;)

[epage]

There are no stability guarantees about the generated text, only in how it behaves.

It can still be reasonable to have snapshot tests for content you care about as long as it outweighs the update costs. For example, I regularly have snapshot tests for error output, help output, etc.

You could take the approach of saying "its tested in clap". Another approach would be to use completest to create tests. This can be very slow so you might only want some basic smoke tests.

With our new completion system, more of the logic is in Rust and the quality of tests is higher, so its only the shell glue that needs testing which we have tests for, so I wouldn't bother testing that (we don't in cargo).