[FEATURE REQUEST] Improve timing report when `run_all_in_same_sim` is used

[nselvara]

Behaviour Clarification

Hi,
This is about -- vunit: run_all_in_same_sim.

If you don't use this pragma, each test runs in a separate simulation, which is fine for CI but inconvenient with a GUI. With it, all tests run in the same simulation, which is much better for waveform debugging.

The only downside is that the reported sub-times for each test case are not accurate, they are just divided equally by the total run time. The overall time is still correct.

It would be nice if the logs could make this clearer, or if there were an option to skip sub-times or improve the estimates when using this mode.

Thanks for considering!

[LarsAsplund]

Since we report on individual test cases, we need to report an execution time for each of them. The sum of those times should be the total execution time. For that reason, I don't think skipping time is an option.

A better approach is to improve the accuracy of time estimates for tests executed within the same simulation. However, this is difficult due to the lack of a reliable timing sources in VHDL. VHDL-2019 introduced a way to get the current simulation time but the standard only requires a resolution of one second which is too coarse for our needs. VHDL-2019 is not widely supported yet. I've seen this feature in Aldec's simulators and they are using the 1-second resolution.

For example, a testbench containing multiple fast test cases may sum to a total of one second. This second is dominated by the simulator startup time. The test cases themselves takes much less then a second.

In your case I suggest that you activate the run_all_in_same_sim attribute from the run script depending on if you're in GUI mode or not. Something like this:

from vunit import VUnit, VUnitCLI

cli = VUnitCLI()
args = cli.parse_args()
vu = VUnit.from_args(args=args)
vu.add_vhdl_builtins()

# Add your source code

if args.gui:
    for lib in vu.get_libraries():
        for tb in lib.get_test_benches(allow_empty=True):
            tb.set_attribute("run_all_in_same_sim", None)

vu.main()

[nselvara]

Hi @LarsAsplund,

Thanks for the detailed explanation that makes perfect sense. I understand that splitting tests across separate simulations gives more accurate per-test timing because of simulator start-up overhead and the lack of high-resolution simulation time in VHDL.

I see the benefit of using run_all_in_same_sim in GUI mode, but I’ll make sure my team is aware that the summed sub-times can be misleading for very fast tests.

Would you be open to ideas for improving this? For example, could there be a way for the VUnit runner to fork a separate process or add hooks that capture real-world wall-clock time per test case, even if they run in the same sim? Or maybe some cycle counting if that data is accessible?

I realise this is probably not trivial, but it could be useful where the simulator's resolution isn’t fine enough or where start-up overhead dominates tiny tests.

Thanks again for clarifying and for all the great work on VUnit!

[LarsAsplund]

I see only two options. Either the simulators provide a high resolution timer that beats the requirements in the standard or it's solved using a foreign language interface. We have an experimental development branch for calling Python from within a simulation, see https://www.linkedin.com/posts/lars-asplund-40b7395_vunit-embeddedpython-vhdl-activity-7164390843779346432-zyYO. That can be used to get a high-resolution timestamp. Option 2 is likely to happen before option 1. The drawback is that it requires more than the standard simulator implementation so it will only be available for those working with such a setup

[nselvara]

Thanks for clarifying the two paths forward and for sharing the experimental branch. I appreciate the transparency.

Using the embedded Python option for a higher-resolution timestamp sounds like a promising workaround for now, even if it is limited to setups that can handle the extra requirements.

Do you have any rough estimate for when the embedded Python approach might be available as a development feature?
I will check out the LinkedIn post and discuss this with the team to see if it could fit into our flow.

Thanks again!