Migrate tests to use cocotb

[amykyta3]

Currently there is a significant amount of duplication between CPU interface testbench library components

Specifically:

• tb_inst.sv templates contain a lot of common boilerplate
• *_driver.sv files all implement the exact same assert_read methods, among other things

For the tb_inst.sv template, see if it makes sense to autogenerate more of it. Or possibly use a hierarchical Jinja template using blocks & inheritance to make it less verbose. Important: Be careful to not sacrifice clarity. This TB infrastructure is already pretty non-traditional and can be confusing to newcomers.

For drivers, migrate the driver API to live inside SV classes so that they can use proper inheritance.

Or a different wild idea: migrate ALL testbench infrastructure to something like cocotb. SystemVerilog is nice, but much of the verilator compatibility issues are due to fancy things being done in the testbench code and not the generated RTL.

[darsor]

I'll put in my vote for cocotb for a couple of reasons.

• I've used it extensively and I've found it makes testbenches easy to write and reuse.
• It's simulator agnostic. Switching to a different simulator generally requires only telling it which simulator to use. For more advanced use cases it may require custom flags for different tools.
• There's an ecosystem of extensions for various bus interfaces already written (AXI-lite, APB, etc.).
• It ties into pytest very well.
• It could be leveraged by the downstream VHDL exporter. Right now that exporter uses the same Systemverilog testbenches and auto-generates a test adapter between SV and VHDL. Testbenches written in python could work for either language.

[amykyta3]

Yeah the more I read about cocotb, the more I like it. I've been meaning to learn it for a while and this may just be the perfect excuse to dive into it.

[davekeeshan]

@amykyta3 @darsor

I have rewritten virtually all the tests from regblock into cocotb for my project etana. Part of my verification flow is that my tests still work on the output from regblock.

Because it is open, i can use verilator on github and run actions against every checkin, you can see a cocotb run against the output from regblock using verilator in this action:

https://github.com/daxzio/PeakRDL-etana/actions/runs/18811852950

Cocotb has a problem with structs at the top level in verilator, they should appear as flattened signals that cocotb can just attach to, that is not the case right now (not sure if it is a cocotb or verilator issue) to bypass this i wrote a python script that generates a system verilog wrapper file that provides a mapping of flatten signals to struct signals, hopefully if and when the above issue get addressed the wrapper can be dropped out.

Not every test is there, some tests are more python tests of regblock, rather than simulation tests, so I have as of yet not found an comaprison test to them that cocotb makes sense to do (test_user_cpuif)

[davekeeshan]

* It could be leveraged by the downstream VHDL exporter. Right now that exporter uses the same Systemverilog testbenches and auto-generates a test adapter between SV and VHDL. Testbenches written in python could work for either language.

@darsor

I have also started looking at can this be used for vhdl, yes it can, however the current output of registers that are broken up into fields does not match what verilog does, I think it only supplies the register value at the top level, rather than them broken up into fields, or at least that was as far as i got into the probelm. For tests that did not have the issue they passed staright away against the cocotb test written for the verilog, it was beautiful to see.

[darsor]

@davekeeshan

For tests that did not have the issue they passed staright away against the cocotb test written for the verilog, it was beautiful to see.

That's great! I expect there will need to be some changes to the tests to work with the VHDL exporter, but they should be fairly minor and easy to track relative to upstream. Thanks for testing it!

Having GitHub actions run full simulator verification would be a big win.

[amykyta3]

@davekeeshan I started dabbling in cocotb yesterday and pretty quickly encountered the struct issue as well with Verilator.

From what I gather, this is a limitation in Verilator, not cocotb. Internally, Verilator doesn't actually support structs at all - instead it mangles structs into flattened arrays as you point out. In a closed design this ends up not making any difference, however when manipulating the design via VPI (what cocotb does) the true representation of the structs shows up - as flat vectors.

This is also why Verilator will throw a warning by default if it encounters an unpacked struct: Verilator is warning the user that the semantics of an un-packed struct is no longer being enforced since it is effectively treating it as packed. This is also apparent when viewing waveforms: VCD dumps only show the flattened structs. Fortunately Verilator is capable of re-building this by passing in the --trace-structs flag.

I'll explore a similar workaround, albeit on the Python side. Rather than flattening in Verilog wrapper, I'd rather auto-generate an interface in Python that re-constructs the struct as a cocotb-equivalent object. That way we preserve an identical user-experience on the TB side so that testbenches can ideally be used across all simulator vendors (ok, except maybe icarus due to very limited SV support...)

Possibly related Verilator issues:

• Support unpacked structs in DPI-C calls verilator/verilator#4679 (discusses DPI, so not sure if this also applies to VPI)

[davekeeshan]

@amykyta3

I am not sure how one can connect anything into cocotb if the hooks into verilator are broken, I went looking for anything I could find just instating the SV code from regblock, if you do find something please tell me, I'll feel like an idiot for missing it I am sure.

For the work I needed to do, however, the wrapper was the perfect bridge from where you were to where i needed to be, and got me there pretty fast.

As an observation, I in someways think the most productive and most reusable version of this code is something that looks like almost the straight verilog version of code that I have ended up producing and then it is only as a wrapper around that work that you end up adding in the struct at the top-level, effectively, producing your work. In a code reuse world this hits so many more places and tools. Also when one is producing an open source tool, i personally think one has a mandate to make sure my code works on the free tools first, where possible (icarus, verilator, yosys, ghdl), that is my aspriational goal, but I don't expect anyone to share it.

I have been coding for a long long time, but it is only in the last 10 years I have been writing code and trying to bring that same code through multi tools from different vendors, I am still blown away by how, even some top shelf providers don't do something that has been in the LRM going on 15 to 20 years, the streaming operator issue in xcelium, #165, is a classic example of this. For reference I also found this problem in icarus and fixed it there for that. Last year I was working on some ARM projects, and was surprised (or maybe not surprised when you think about it) the code was written very simply, nothing exotic, pretty much verilog without any system verilog enhancements. I am guessing that comes from the same place, confidence it will work in more places.

[davekeeshan]

@amykyta3

Sorry I meant to say this in the previous post (don't try an post comments on a Sunday afternoon with screaming kids running around), I hope you can use some of my cocotb work, i think it is aboyt 80-85% of what you need right now. I have been actively using cocotb for the past 4 years, and I have had to switch simulation providers a few times, it has been a dream just passing in a different SIM=and it just ... works.

[amykyta3]

This would be an adapter layer in Python that, if Verilator is being used, would transparently map dut.hwif_in.some.struct.member --> dut.hwif_in[SLICE]. If Verilator ends up fixing this in the future, the shim can be removed and we're back to running 'normally'

Still just an idea, but should be workable. I generally want to avoid flattening since that would add another semantic hurdle when writing testcases. Would prefer the structure to remain unchanged, and instead insert a shim to satisfy Verilator. Agree that flattening would also work, but it seems like an anti-pattern to do that for all simulators just to satisfy Verilator's quirk.

[davekeeshan]

Ah man, all simulator work seems to have some anti pattern problems, hacks upon hacks. When you need to get something out the door you do it with the stated goal of "well get back to it" or "next time", but you don't and then it's sitting there waiting for you on the next project, perpetual groundhog day.

My hack is to stay verilog as much as I can, design is usually 10% of my workload, so it can be tedious upfront it pays out on the backend. There are some nice sv features, but never seem to be worth the pain.

[davekeeshan]

@darsor

OK i have incorporated regblock-vhdl into my cocotb testbench for etana.

https://github.com/daxzio/PeakRDL-etana/actions/runs/19045553460/job/54392747879

it mostly passes, except the regblock testing now has the ability external write-buffered register syntax fixed, #167, this bug still exists in regblock-vhdl and has not been updated yet.

I will raise a issue against that project to capture it.

EDIT: You reported that bug, i think

[RasmusGOlsen]

I may be late to the discussion, but I highly recommend creating a Verilog testbench to wrap the DUT and connecting that testbench to Cocotb. The main reason for this approach is performance. Generating clocks in the Verilog testbench is much more efficient than generating clocks in Cocotb since it eliminates the need to switch between the simulator and Python for every clock cycle.

Additionally, you can write reusable tests using PyUVM, an UVM implementation in Python designed to work with Cocotb. You can generate the UVM register model with peakrdl-pyuvm and utilize the BFMs from the cocotbext-axi package.