add GitHub action to test examples with VUnit action

[marcoieni]

Examples should be valid after each push or pull request on this repository.
In order to do this, VUnit action should be run against each example.

[eine]

Hi Marco! On the one hand, examples are executed as part of the acceptance test suite: https://github.com/VUnit/vunit/blob/master/tests/acceptance/test_external_run_scripts.py. On the other hand, the docker image used in the VUnit Action is ghdl/vunit:llvm, which includes latest stable VUnit.

Hence, I think we should not use the Action to test the core of the project itself. Otherwise, we might run into issues when incompatible changes are done to the codebase and the examples at the same time, because the Action would not match.

Nevertheless, I think that the Action can be used in other repositories of the org, such as the verification component repos. Currently https://github.com/VUnit/vc_axi and https://github.com/VUnit/vc_uart are placeholders. But we expect to use them soon (see VUnit/vunit#520 and VUnit/vunit#511).

Regarding enhancements related to Actions in this repository, I think we should replace the installation on Windows (https://github.com/VUnit/vunit/blob/master/.github/workflows/push.yml#L102-L109) and maybe on Linux (https://github.com/VUnit/vunit/blob/master/.github/workflows/push.yml#L54-L75) with setup-ghdl-ci. See, for example, https://github.com/ghdl/ghdl-cosim/blob/master/.github/workflows/test.yml#L42-L79. Unfortunately, there are some performance issues yet: msys2/setup-msys2#23.

---

About VUnit Action, a possible enhancement would be to allow users to select between ghdl/vunit:llvm or ghdl/vunit:llvm-master. However, I don't think selecting the image is supported. Alternatively, master VUnit can be optionally installed in the Dockerfile.

Apart from that, in the longer term I'd like to extend VUnit Action with a JavaScript/TypeScript version. Unfortunately, only JavaScript Actions can be executed on the three host virtual environments (ubuntu-latest, windows-latest and macos-latest). Docker/Container Actions can only be used in Linux jobs. Ideally, it would be expected to be used after setup-ghdl-ci and, since Python is available by default (or users can use Action setup-python), it would just decide which VUnit version to install, before running the user-provided run.py script. Note that this would/will be a complement to the current Docker/Container Action, because there are use cases where containers suffice and it's the solution with the lowest startup time.

[marcoieni]

examples are executed as part of the acceptance test suite

Is the acceptance test suite part of the CI?

the docker image used in the VUnit Action is ghdl/vunit:llvm, which includes latest stable VUnit.

Oh, you are right ^^'

[eine]

Is the acceptance test suite part of the CI?

Yes. There are basically three groups of tests (unit, acceptance and vcomponents): https://github.com/VUnit/vunit/blob/master/tox.ini#L31-L37. unit are Python-only tests. acceptance do require a simulator, and core VHDL features and some optional VHDL libs are tested. vcomponents tests verification components library.

On Linux:

https://github.com/VUnit/vunit/blob/209e27d28cf9abb93b2d4f7ccc9e26882df11859/.github/workflows/push.yml#L36-L38

On Docker:

https://github.com/VUnit/vunit/blob/209e27d28cf9abb93b2d4f7ccc9e26882df11859/.github/workflows/push.yml#L60-L61

On Windows:

https://github.com/VUnit/vunit/blob/209e27d28cf9abb93b2d4f7ccc9e26882df11859/.github/workflows/push.yml#L83-L87

[marcoieni]

Ok, thank you, after a little walk-through of the actions it was not clear to me at first.

However, I don't think selecting the image is supported.

Maybe with a typescript action in some way you can read the tag of the dockerfile from an optional action parameter that defaults to "llvm"? Then you just execute "docker run" with the selected image.
Maybe this is a relevant example. I haven't seen it in detail.

Anyway if this is OT here, this could be a topic for another issue (in the vunit_action repository?) and this issue may be closed.

[eine]

However, I don't think selecting the image is supported.

Maybe with a typescript action in some way you can read the tag of the dockerfile from an optional action parameter that defaults to "llvm"? Then you just execute "docker run" with the selected image.
Maybe this is a relevant example. I haven't seen it in detail.

I think it might be easier to use two Dockerfiles and the features in action.yml. Something such as the following might work:

inputs:
  version:
    description: 'Use VUnit master instead of stable'
    default: 'stable'
  run_file:
    description: 'python VUnit run file'
    default: 'run.py'
runs:
  using: 'docker'
  image: 'Dockerfile.${{ inputs.version }}'
  args:
    - ${{ inputs.run_file }}

The current Dockerfile would be renamed to Dockerfile.stable and a new Dockerfile.master would be added.

OTOH, inputs are exposed as environment variables in the container. I know it works when the container is executed, but I don't know if the same variables are available when it is built. So, the following might or might not work:

inputs:
  image:
    description: 'Use another image'
    default: 'ghdl/vunit:llvm'
  run_file:
    description: 'python VUnit run file'
    default: 'run.py'
runs:
  using: 'docker'
  image: 'Dockerfile'

FROM $INPUT_IMAGE

# Copies your code file from your action repository to the filesystem path `/` of the container
COPY entrypoint.sh /root/entrypoint.sh

# Code file to execute when the docker container starts up (`entrypoint.sh`)
ENTRYPOINT ["/root/entrypoint.sh"]
CMD [$INPUT_RUN_FILE]

Anyway if this is OT here, this could be a topic for another issue (in the vunit_action repository?) and this issue may be closed.

Yes, I think we should transfer this issue. However, I don't have enough permissions to do it. I can move it to other repos in the org, but not to vunit_action. @LarsAsplund, can you do it?