Safer Check if Simulation passed or Not

[Mikael]

Today the outcome of the simulation is solely dependent on the OSVVM report server and if you have triggered an error or not.
But if you have for example an external library cell/ip that uses assertions, it can still look as the simulation passed while it indeed failed.

I added “assert false” in a testcase to illustrate the problem. This is the output log:

# %%         3510 ns    Log    PASSED    in manager_1: ReadResp SB,      Received: 0   Item Number: 150
# %%         3510 ns    Log    PASSED    in Default,                     slvBurstVector = D001, D003, D005, D007, D009
# ** Error: Assertion violation.
#    Time: 3510 ns  Iteration: 2  Instance: /tbaxi4memory/TestCtrl_1
# ** Note: %%         3530 ns    DONE   PASSED   TbAxi4_DemoMemoryReadWrite1  Passed: 334  Affirmations Checked: 334
#    Time: 3530 ns  Iteration: 3  Instance: /tbaxi4memory/TestCtrl_1

And if you check the generated html file, it also states that the simulation passed. Which is clearly wrong!

The safe way is to check the simulator status, which can be done with a simple tcl procedure, when the simulation is done:
Here is how that would look:

# %%         3510 ns    Log    PASSED    in Default,                     slvBurstVector = D001, D003, D005, D007, D009
# ** Error: Assertion violation.
#    Time: 3510 ns  Iteration: 2  Instance: /tbaxi4memory/TestCtrl_1
# ** Note: %%         3530 ns    DONE   PASSED   TbAxi4_DemoMemoryReadWrite1  Passed: 334  Affirmations Checked: 334
#    Time: 3530 ns  Iteration: 3  Instance: /tbaxi4memory/TestCtrl_1
# Break in Process ControlProc at ../AXI4/Axi4/TestCases/TbAxi4_DemoMemoryReadWrite1.vhd line 83
# Stopped at ../AXI4/Axi4/TestCases/TbAxi4_DemoMemoryReadWrite1.vhd line 83
# process_teststatus
# ** Error:#########################################################################
# ** Error:UCDB Teststatus:FAILED  TESTSTATUS > 1 ACTUAL=2
# **       TSTAT_REASON=
# ** Error:#########################################################################

To make it easy to handle, you just want to check in one place if the simulation passed or failed. Therefor it is also a good idea to trigger the teststatus in the AlertLogPkg.vhd, procedure PrintTopAlerts

   if TestFailed then
        report buf.all severity error ;
      else
        report buf.all severity note ;
      end if ;

      --WriteLine(buf) ;
    end procedure PrintTopAlerts ;

So if we remove the assert and trigger an error by OSSVM, now it looks like this:

# %%      ReadBurstFifo    Failures: 0  Errors: 0  Warnings: 0  Passed: 0
# Break in Process ControlProc at ../AXI4/Axi4/TestCases/TbAxi4_DemoMemoryReadWrite1.vhd line 83
# Stopped at ../AXI4/Axi4/TestCases/TbAxi4_DemoMemoryReadWrite1.vhd line 83
# process_teststatus
# ** Error:#########################################################################
# ** Error:UCDB Teststatus:FAILED  TESTSTATUS > 1 ACTUAL=2
# **       TSTAT_REASON=%         3530 ns    DONE   FAILED   TbAxi4_DemoMemoryReadWrite1  Total Error(s) = 1  Failures: 0  Errors: 1  Warnings: 0  Passed: 333  Affirmations Checked: 334
# ** Error:#########################################################################

This is how Questa works, I’m sure that other simulators have the same mechanism.
BR
Mikael Andersson, Siemens EDA

[Mikael]

By saving the UCDB file for each testcase (<testname>.ucdb , you can also check the teststatus afterwards:

>vcover attribute OsvvmTemp_Questa/TbAxi4_DemoMemoryReadWrite1.ucdb

The tcl procedure I use to check the test status,just contact me and I will send it. Seems like the code is blocked.

[Jim Lewis]

Hi Mikael,
This is a good point. In VHDL-2019, we put things in that will allow OSVVM to collect error information from VHDL and PSL assertions. Sounds like a great addition for the next release.

Do you know which version of Questa started supporting the following subprograms from std.env:
For VHDL Asserts:
GetVhdlAssertCount, IsVhdlAssertFailed, ClearVhdlAssert, IsVhdlAssertFailed,
SetVhdlAssertEnable, GetVhdlAssertEnable
For PSL:
PslAssertFailed, PslIsCovered, ClearPslState

With these OSVVM should be able to report these directly in our reports.

In the VHDL-2019 presentation, I am lead to believe these are at least supported in the current version.

Best Regards,
Jim

[Mikael]

The Report APIs should be there in Questa sim 2025.2.
The latest Questasim version is now 2025.3

Remember that today, mixed language designs are very common. Some IPs are only available in Verilog.
So it is not just VHDL assertions that can be a source of errors.
There could be SVA assertions, both immediate and concurrent.

IPs or library models written in verilog/systemverilog might have code like this:

if (false) $error("FAKE ERROR"):

Or VHDL library models can also use:

if ... then
report "FAKE ERROR" severity ERROR;

Or you might even have a UVM passive environment that monitors part of the testbench, and that is using
uvm_error(…);`

I would believe that the safe way is always to check the simulator status in the end of the simulation.
Because the simulator status is picking up all the severities of report in VHDL, Assertion fails, $error,$warning, $fatal (from verilog).

The simulator status is what we use when we run regressions using Questa Verification Run Manager.

[Author]

Posts