What do you use for test suite orchestration?

[Richard Head]

Hello

Here, I have all testbenches set up running OSVVM all good. But where Im at a little bit of a crossroads is getting opinions on what people use for test orchestration.

Let me give an example. We use ActiveHDL, which I run in batch mode because these tests run as part of CI on gitlab. I have a project with a FIFO that has 3 possible input widths and 2 output widths. Each test case can be run from the command line with in house tcl, where the test file is passed in as a config file with all the testbench generics needed for that test (via the -f option of vsim):
vsimsa -tcl run_my_test.tcl ~t some_testcase

So to run a regression, I have another tcl file listing all the testcases (or generating all the test cases from a template in one case where there are 90 testcases!). This would be fine, except that:
1. If I simply loop them, on the first testcase fail, the job ends, so I can only see the first failed case
2. I could try catching the error code from the TCL, except that the ActiveHDL batch mode tcl interpreter returns a null string from catch (ticket raised!) so all tests appear to pass (when they might have actually failed).

So Im looking at other runner options. I have had a quick play with PyTest which seems really easy to set up, and it is being used by others for physical unit tests.
I know VUnit could do this kind of thing, but from my understanding its really meant to hook in at the VHDL level. Given I already have OSVVM doing everything I need in the VHDL, I am only really after the top level runner.

Nose is mentioned, but apparently its lack of updates puts people off.

So does anyone have experience with orchestration? which systems do people use to run OSVVM through their many testcases?
Or is this really out of the scope for here?

Thanks for any help
Does anyone else

[Jim Lewis]

Hi Richard,
This is definitely in scope of a good discussion.

In 2020.08, OSVVM upgraded our scripting. Currently everything is TCL based, but the intent is to also have a BASH based executor of the scripting environment. One of the goals of the current approach is to create a simulation execution environment that is tool independent – ie: scripts for common simulators. Currently we support Aldec ActiveHDL/RivieraPro/VsimSA (for both Active and Riviera) as well as Mentor ModelSim/QuestaSim. I am hoping to have some time in the first half of next year to add support for GHDL, Synopsys and Cadence. GHDL works to some degree by using tclsh and the current scripts, but I am not comfortable calling it a good solution. I think GHDL would be happier with a BASH executor (set of procedures like was done for TCL). One of the key concepts is that everything in the scripts (.pro extension) is executable TCL when the environment is loaded (procedures).

It is in its early phases – it needs a number of additions – such as a way to turn on tool coverage in a tool independent manner. Currently it compiles all code as 2008, but it would be nice to enhance it to be able to switch modes – although I am not sure why even for RTL code I would want to use anything other than 2008 for simulation. This is just to name a few.

It too currently has issues with stopping when a compile/simulate fail. Previous scripts have not done this so I need to look into the settings. However, for the development work I am doing, this is ok.

It is also missing a path into CI. I am thinking that GitLab and BASH might be one possible path.

Currently all OSVVM IP is delivered with OSVVM compile scripts.

Documentation is in Documentation Repository as well as the README.md in the Scripts directory.

Do you use git? Have you looked at GitLab?

Best Regards,
Jim

[Richard Head]

Hi Jim, sorry for the late reply (I got no notifications and I dont check here often)
We currently use Gitlab (local) as our main repo and Firmware pipelines. I have it running well, encouraging people to add their testbenches to the pipeline when complete, so that we find broken testbenches sooner rather than 2 years after the fact! It is currently 4 stages – checks, sims, build, release. Checks are a combination of syntax checks and elaborations, simulations are all the self checking testbenches (set to only appear when code they rely on changes). There is talk of migrating back to Jenkins, but we will see. It has grown from an initial couple of check jobs and <10 simulations to 7 checks jobs and ~30 testbenches in just over a year.

Ive looked into pytest because while TCL scripts are fine at a basic level, when you have several testcases for a single UUT, the CI just fails at the first error unless you build a TCL test framework. PyTest has an easy setup so that it will always run all tests and report the errors at the end, and was very easy to set up and wrap around existing TCL (plus the current bug with Aldec not throwing errors properly to Catch). Plus it is really easy to set up and generate the testcases from python itself. My current UUT has 73 test cases, and its really useful to see what passed and failed without working through the bugs one by one. I have looked breifly into VUnit for this, but the test setups appear to be neede in VHDL, which then doesnt allow generics as different test paramters (eg, bus widths). And another plus, its also used for our unit tests.

I have in house TCL scripts for compiling and running testbenches, then the Python needed to run PyTest on my current UUT is only 90 lines, and that generates all of the config files for all of the tests – and it would be fairly straightforward to put most of the code in a generic library. I doubt the TCL could be so compact.

[Author]

Posts