Requirements Tracking with OSVVM, Part 2

OSVVM release 2023.07 updates OSVVM’s Requirement Tracking (aka Specification Tracking) capability so that HTML and CSV based merged requirements reports are produced automatically.

Requirements Tracking (Part 1) started with the 2020.08 release. In the 2020.08 release, each test could write out a CSV requirements report. The expectation was that external requirements tools imported these and merged them to track requirements.

The remainder of this post focuses on how OSVVM requirements tracking works.

Before we get there, if you need additional, different, or better capability from OSVVM requirements tracking, please provide feedback either through github issues, here as a forum discussion, or direct email to me.

How-To for Requirements Tracking

OSVVM models requirements as a special case of the AffirmIf capability as requirements can either pass or fail. Hence, if you are already using affirmations to track errors in your tests, adding requirements tracking is easy.

Step 1: Set the Test Case name using SetTestName (a normal part of OSVVM methodology)

SetTestName("AxiLite_Uart1") ;

Step 2: Create an AlertLogID for requirements using GetReqID.

signal ReqID1 : AlertLogIDType ;
. . .
Initialize : process
begin
  ReqID1 <= GetReqID("R1.1", PassedGoal => 1, ParentID => REQUIREMENT_ALERTLOG_ID) ;
  wait ;
end process Initialize ;

Step 3: Check Requirements Using AffirmIf

TestbenchProc : process
begin
  . . .
  -- Any variation of AffirmIf will work with requirements
  AffirmIfEqual(ReqID1, D, X"A5", "Data") ;
  ...
  -- Alternately, the Requirement name can be used in place of an ID
  AffirmIf("R1.1", D = X"A5", "Data Actual: " & to_hstring(D), " /= Expected: A5") ;
  ...
  -- Checking off a requirement as Passed or Failed
  AffirmPassed(ReqID1) ;
  AffirmFailed(ReqID1) ;
  ...
  -- Require that a test case passes
  AffirmIf(ReqID2, GetAlertCount = 0, GetTestName & "Test Passed") ;
  ...
  -- Require that all coverage models reach 100%
  AffirmIf(ReqID3, GetCov = 100.0, GetTestName & "Test Covered") ;
  wait ;
end process TestbenchProc ;

Step 4: Report Requirements in a YAML data structure using EndOfTestReports

EndOfTestReports ;

Step 5: View Requirement Reports
OSVVM’s scripts convert the YAML files (created during EndOfTestReports) to HTML files.

Currently Test Case requirements are displayed as part of the Alert report (shown below).

For the test suite and build, OSVVM scripts merge the individual YAML files and produces an HTML requirements report (shown below). If two test produce results for the same requirement name, their results are merged together. In doing this, the value for the requirements goal is the maximum of the goal values and the value for the requirements passed is the sum of the passed values (saturated at the requirements goal for that test).

For a build, OSVVM scripts produce a CSV requirements report (shown below). The order of the CSV list is “RequirementsName, RequirementsGoal, RequirementsPassed, ChecksFailed, AlertFailures, AlertErrors, AlertWarnings, ChecksTotal”.

Summary

OSVVM has made requirements tracking easy as we use the same capability we use for error handling (alerts) with small variations and generating HTML and CSV Requirements reports is automatic when using EndOfTestReports and OSVVM scripts.