What do you use for test suite orchestration?
October 23, 2020 at 16:06 #1735Richard HeadMember
Here, I have all testbenches set up running OSVVM all good. But where Im at a little bit of a crossroads is getting opinions on what people use for test orchestration.
Let me give an example. We use ActiveHDL, which I run in batch mode because these tests run as part of CI on gitlab. I have a project with a FIFO that has 3 possible input widths and 2 output widths. Each test case can be run from the command line with in house tcl, where the test file is passed in as a config file with all the testbench generics needed for that test (via the -f option of vsim):
vsimsa -tcl run_my_test.tcl ~t some_testcase
So to run a regression, I have another tcl file listing all the testcases (or generating all the test cases from a template in one case where there are 90 testcases!). This would be fine, except that:
1. If I simply loop them, on the first testcase fail, the job ends, so I can only see the first failed case
2. I could try catching the error code from the TCL, except that the ActiveHDL batch mode tcl interpreter returns a null string from catch (ticket raised!) so all tests appear to pass (when they might have actually failed).
So Im looking at other runner options. I have had a quick play with PyTest which seems really easy to set up, and it is being used by others for physical unit tests.
I know VUnit could do this kind of thing, but from my understanding its really meant to hook in at the VHDL level. Given I already have OSVVM doing everything I need in the VHDL, I am only really after the top level runner.
Nose is mentioned, but apparently its lack of updates puts people off.
So does anyone have experience with orchestration? which systems do people use to run OSVVM through their many testcases?
Or is this really out of the scope for here?
Thanks for any help
Does anyone elseNovember 20, 2020 at 16:27 #1739Jim LewisMember
This is definitely in scope of a good discussion.
In 2020.08, OSVVM upgraded our scripting. Currently everything is TCL based, but the intent is to also have a BASH based executor of the scripting environment. One of the goals of the current approach is to create a simulation execution environment that is tool independent – ie: scripts for common simulators. Currently we support Aldec ActiveHDL/RivieraPro/VsimSA (for both Active and Riviera) as well as Mentor ModelSim/QuestaSim. I am hoping to have some time in the first half of next year to add support for GHDL, Synopsys and Cadence. GHDL works to some degree by using tclsh and the current scripts, but I am not comfortable calling it a good solution. I think GHDL would be happier with a BASH executor (set of procedures like was done for TCL). One of the key concepts is that everything in the scripts (.pro extension) is executable TCL when the environment is loaded (procedures).
It is in its early phases – it needs a number of additions – such as a way to turn on tool coverage in a tool independent manner. Currently it compiles all code as 2008, but it would be nice to enhance it to be able to switch modes – although I am not sure why even for RTL code I would want to use anything other than 2008 for simulation. This is just to name a few.
It too currently has issues with stopping when a compile/simulate fail. Previous scripts have not done this so I need to look into the settings. However, for the development work I am doing, this is ok.
It is also missing a path into CI. I am thinking that GitLab and BASH might be one possible path.
Currently all OSVVM IP is delivered with OSVVM compile scripts.
Documentation is in Documentation Repository as well as the README.md in the Scripts directory.
Do you use git? Have you looked at GitLab?
- You must be logged in to reply to this topic.