OSVVM 2021.10: Functional Coverage Reports

OSVVM release 2021.10 focused on improving OSVVM’s reporting capability. In my last post, “OSVVM 2021.10: Build Summary Reports“, I talked about our HTML and JUnit CI reports that allow us to assess whether a set of tests have passed or quickly access which test cases failed.

This post features OSVVM’s 2021.10 HTML based Functional Coverage reports. This has been in my mind ever since OSVVM was disparaged by an EDA sales drone who correctly pointed out that the SystemVerilog Functional Coverage reporting was much better. Sorry SystemVerilog, that is no longer the case.

The OSVVM functional coverage reports, shown below, provide details of the “Coverage Settings” (hidden by default) and the “Coverage Bins” (shown by default). In a similar fashion to SystemVerilog, the OSVVM reports allow details to either be shown or hidden – in our case by simply clicking on a triangle.

Now the big difference between VHDL+OSVVM vs SystemVerilog+UVM functional coverage reporting is the cost.  OSVVM is free open source software (FOSS) and only requires a basic VHDL-2008 simulation license to run.

Early Work

Early OSVVM coverage reports were all text based. No effort was made to justify the columns of the reports, so perhaps the disparagement we received was deserved.

In late 2016, in work led by Aldec, we introduced a vendor API to the functional coverage models. This API transfers OSVVM functional coverage to a simulator as it is captured and allows the simulator to display the results. Although this interface is available to all vendors, only Aldec’s Riviera-PRO and Active-HDL support it.

In preparation for generating OSVVM functional coverage reports, I looked into using Accellera’s UCIS standard. This turned into a dead end. SystemVerilog and UCIS focus on building cross coverage models from cartesian crosses of point (aka item) bins. This is a severe limitation. OSVVM focuses on creating cross coverage models organically. While OSVVM can do cartesian crosses just as concisely as SystemVerilog, OSVVM can also construct an ad-hoc cross coverage model, piece by piece. This allows us to collect just the relevant functional coverage information without also capturing extraneous information.

VHDL Aspects of Generating OSVVM Functional Coverage Reports

Generation of OSVVM Functional Coverage Reports is a natural consequence of creating functional coverage, ending the test by calling EndOfTestReports, and running the test with OSVVM scripts.

To generate functional coverage reports, you need to have the following OSVVM elements in your VHDL testbench. More details of this are in OSVVM Test Writers User Guide in the documentation repository.

-- Reference the OSVVM Utility Library
library OSVVM ;
context OSVVM.OsvvmContext ;

architecture Random_1 of TestCtrl is
  signal RxCov    : CoverageIDType ;                 -- coverage object
  use work.ScoreboardPkg_uart.all ; 
  signal SB       : work.Scoreboardpkg_uart.ScoreboardIDType;
  signal TestDone : integer_barrier ; 
  . . . 
begin
  ControlProc : process
  begin
    SetAlertLogName("TbUart_Random_1");
    SB   <= NewID("SB") ; 
    . . .
    WaitForBarrier(TestDone, 5 ms) ; 
    . . .
    -- Generate Reports with EndOfTestReports (replaces ReportAlerts)
    EndOfTestReports ;           -- Create reports - including functional coverage
    std.env.stop(GetAlertCount) ;
  end process ControlProc ;

  TxProc : process
    variable RV   : RandomPType; 
  begin
    for I in 1 to 10000 loop
      case RV.DistInt((. . .)) is
         . . . 
       end case ; 
      Push(SB,  (TxD, ErrorMode));
      Send(TRec, TxD, ErrorMode);
    end loop ;
    WaitForBarrier(TestDone) ; 
    wait ; 
  end process TxProc ; 

  RxProc : process
    . . .
  begin
    RxCov <= NewID("RxCov") ;                         -- Get a Coverage ID
    wait for 0 ns ;
    -- Define the functional coverage model
    AddItemBinNames(RxCov, "MODE") ;
    AddBins(RxCov, "NORMAL",       GenBin(1) ) ;        -- Normal
    AddBins(RxCov, "PARITY",       GenBin(3) ) ;        -- Parity Error
    AddBins(RxCov, "STOP ",        GenBin(5) ) ;        -- Stop Error
    AddBins(RxCov, "PARITY_STOP",  GenBin(7) ) ;        -- Parity + Stop
    AddBins(RxCov, "BREAK",        GenBin(9, 15, 1) ) ; -- Break
    for I in 1 to 10000 loop
      Get(RRec, RxD, RxErrorMode); 
      Check(SB, (RxD, RxErrorMode));
      ICover(RxCov, RxErrorMode) ;      -- Collect functional coverage
    end loop ;
    WaitForBarrier(TestDone) ; 
    wait ; 
  end process RxProc ; 

Run Your Test with OSVVM’s Scripts

The VHDL code above will generate a coverage report in YAML format. The conversion to HTML is handled in the OSVVM scripts when the “simulate” API command finishes.

You don’t need a complicated script. For a simple test, where the design named is Dut, the testbench is TbDut, and both are in a file of the form “name”.vhd, then we can run the testbench with the following script.

# File name: TbDut.pro
analyze Dut.vhd
analyze TbDut.vhd
simulate TbDut


If we run this test with using “build Dut.pro”, Dut and TbDut will be compiled into the library named default. After the build completes, the functional coverage report will be in the bottom portion of the file ./reports/Default/TbDut.html. A link to the functional coverage report will be in the Build Summary Report.

Getting OSVVM

You can get OSVVM by either downloading from osvvm.org or cloning the repository using git:
git clone --recursive https://github.com/osvvm/OsvvmLibraries

More About OSVVM Scripting

For more about OSVVM Scripting see The OSVVM Script User Guide. This guide includes detailed directions on how to compile the OSVVM sources and run the OSVVM regression suite. It also includes details on running test cases in more sophisticated environments than discussed here.

More About OSVVM

A great place to start learning OSVVM is to start with the README.md that is displayed at the bottom of the OSVVM Documentation Repository page.

If you are ready for the next step in learning OSVVM, you might consider taking SynthWorks’ “Advanced VHDL Testbenches and Verification – OSVVM Boot Camp“. For class dates see our class schedule.

Why VHDL?

Time and time again, the Wilson Verification Survey has shown that VHDL is the preferred FPGA design and verification language.

SynthWorks believes that VHDL is the right way to do both design and verification. We have invested 1000’s of hours in participating in the VHDL standards to make sure we have the next set of features we need for design and verification. We have also invested 1000’s of hours into OSVVM to make sure VHDL has simple, powerful, and effective verification libraries.

At this point, we are really good at VHDL and I think you will see this reflected in OSVVM.

Summary

With the 2021.10 reporting, OSVVM has moved forward to match the SystemVerilog Functional Coverage reporting capability.

OSVVM has been a leading VHDL Verification Methodology for some time and just continues to get better. With the advances in reporting (in 2021.10), the advances in verification data structures (2021.06 and 2021.07), and simulator support (2021.05 and 2021.09), OSVVM remains the obvious choice for the VHDL verification community.