Reply To: What do you use for test suite orchestration?

[Richard Head]

Hi Jim, sorry for the late reply (I got no notifications and I dont check here often)
We currently use Gitlab (local) as our main repo and Firmware pipelines. I have it running well, encouraging people to add their testbenches to the pipeline when complete, so that we find broken testbenches sooner rather than 2 years after the fact! It is currently 4 stages – checks, sims, build, release. Checks are a combination of syntax checks and elaborations, simulations are all the self checking testbenches (set to only appear when code they rely on changes). There is talk of migrating back to Jenkins, but we will see. It has grown from an initial couple of check jobs and <10 simulations to 7 checks jobs and ~30 testbenches in just over a year.

Ive looked into pytest because while TCL scripts are fine at a basic level, when you have several testcases for a single UUT, the CI just fails at the first error unless you build a TCL test framework. PyTest has an easy setup so that it will always run all tests and report the errors at the end, and was very easy to set up and wrap around existing TCL (plus the current bug with Aldec not throwing errors properly to Catch). Plus it is really easy to set up and generate the testcases from python itself. My current UUT has 73 test cases, and its really useful to see what passed and failed without working through the bugs one by one. I have looked breifly into VUnit for this, but the test setups appear to be neede in VHDL, which then doesnt allow generics as different test paramters (eg, bus widths). And another plus, its also used for our unit tests.

I have in house TCL scripts for compiling and running testbenches, then the Python needed to run PyTest on my current UUT is only 90 lines, and that generates all of the config files for all of the tests – and it would be fairly straightforward to put most of the code in a generic library. I doubt the TCL could be so compact.