Export Test Artifacts to a persistent Directory

[twobiers]

Currently, all the Test artifacts live in a temporary directory which makes it difficult to debug tests.
I think it would be great if xprin could export all test artifacts to a directory relative to the working directory.
This would allow to upload artifacts from a pipeline run, inspect the rendered output, and so on.

[tampakrap]

Thanks for the report, indeed this is a useful feature that is in my todo as I need it as well. The current workaround for this is to create a post-test hook and copy the files that you need, eg:

tests:
  - name: mytest
    inputs:
      ...
    hooks:
      post-test:
        - name: "export outputs to a permanent dir"
           run: |
             cp {{ .Outputs.Render }} mypermdir
             cp {{ .Outputs.Validate }} mypermdir

If the destination dir is a relative path, then it will be relative to the testsuite file, not to the current working dir.

Regarding the exporting feature, I'm thinking of implementing a flag xprin test --export=path/to/mydir:

• Takes as value the destination dir, and it exports all the tests that have an id set.
• It is important to have the id set, as it has restrictions on both the number and the allowed characters.
• The artifacts are exported under a dir with a predefined name, eg path/to/mydir/${testsuite_filename}_artifacts.
• We could also allow empty/no value to the --export flag, which will export the artifacts in a subdir with a pre-defined name (eg ${testsuite_filename}_artifacts) under the same dir that the testsuite file is.

Pros:

• all test cases exported at once, regardless of how many files are loaded
• artifacts are exported on request

Cons:

• Test cases need to have the id field set in order to have their artifacts exported

What do you think about this?

[twobiers]

I've started experimenting with xprin only today, so I might not get the full picture but I believe having an implicit dependency on the id increases complexity.

Currently the outputs already operate under a testcase-scoped directory e.g. /path/to/tmp/xprin-testcase-011147233/outputs/rendered.yaml. Wouldn't it be possible to export that directory to the defined path/to/mydir/xprin-testcase-011147233 for every test - even without an ID specified?

I think, we could even improve the behavior of this tool in general so we don't need an --export option if that would be feasible for you:

1. Define a standard directory like .xprin-test-artifacts
2. Instead of a temporary directory, xprin operates on the standard directory in the working dir
3. Instead of deleting the files after a test run, the directory is nuked before the test run

This would be in line to how most of the test frameworks I worked with in the past and thus hopefully match the expectations of users.