Webinar: OSVVM’s Test Reports and Simulator Independent Scripting

Abstract

According to the 2020 Wilson Verification Survey FPGA verification engineers spend 46% of their time debugging. As a result, we need good scripting to simplify running tests and good reports to simplify debug and help find problems quickly.

Scripting can be complicated no matter what language – particularly with EDA tools that need to stay rooted in one directory while it is advantageous co-locate the user scripts with the verification IP they support. As a result, the scripts must manage the file source locations relative to the simulator directory. Further complicating the scripts is that each simulator API has a different way of specifying commands and command options. No wonder it is frustrating and messy.

With OSVVM scripting, it is ok to hate TCL as it is unlikely you will be use it directly. OSVVM creates a procedure-based API on top of TCL. User scripts are based on the OSVVM simple simulator command API that uses paths that are relative to the script’s location. The messy TCL stuff is handled internally by the OSVVM command API. The result is that the scripts have just a little more complexity than a source file list – but not much. Generally, the most TCL that OSVVM user scripts need is a simple if statement – but even this is rare (and there are examples in the OSVVM library).

Not meaning to name drop, but OSVVM scripting supports Aldec’s Active-HDL and Riviera-PRO, Siemens’ ModelSim and QuestaSim, GHDL, Synopsys’ VCS, and Cadence’s Xcelium.

With respect to reports, when we run a set of tests, we need to be able to assess whether all test cases passed or quickly identify which test cases failed. Once we have determined which test case failed, we need to have detailed information for each test case in a separate report that helps reveal the source of the issue.

OSVVM’s test reporting capability adds another reason as to why OSVVM should be your VHDL Verification Methodology. Our test reporting includes:

• An HTML Build Summary Report for human inspection that summarizes the completion status of each test case in a test suite
• A JUnit XML Build Summary Report for use with continuous integration (CI/CD) tools.
• A separate HTML Test Case Detailed report for each test case with Alert, Functional Coverage, and Scoreboard reports.
• An HTML based simulator transcript/log files (simulator output)
• A text-based test case transcript file (from OSVVM’s TranscriptOpen)
• Links to tool generated code coverage reports

 
Why do we go to all this trouble? When OSVVM runs its full regression suite, our build includes 22 test suites with a total of 524 test cases. The log file is 170K lines long. Without good tools we could not easily run our regressions and quickly assess whether it passed or failed.

How well does OSVVM work with continuous integration tools? OSVVM uses our scripts and JUnit XML output when running our verification component regression suite on GitHub using GHDL. See https://github.com/OSVVM/OsvvmLibraries/actions.

About OSVVM

OSVVM is an advanced verification methodology that defines a VHDL verification framework, verification utility library, verification component library, and a scripting flow that simplifies your FPGA or ASIC verification project from start to finish. Using these libraries, you can create a simple, readable, and powerful testbench that is suitable for either a simple FPGA block or a complex ASIC.

OSVVM is developed by the same VHDL experts who have helped develop VHDL standards. We have used our expert VHDL skills to create advanced verification capabilities that:

• Are simple to use and work like built-in language features.
• Maximize reuse and reduce project schedule.
• Improve readability and reviewability by the whole team including software and system engineers.
• Facilitate debug with HTML based test suite and test case reporting.
• Support continuous integration (CI/CD) with JUnit XML test suite reporting.
• Provide buzz word features including Constrained Random, Functional Coverage, Scoreboards, FIFOs, Memory Models, error logging and reporting, and message filtering.
• Rival the verification capabilities of SystemVerilog + UVM.

OSVVM is a competitive solution with SystemVerilog + UVM for FPGA Verification. World-wide, 18% of the FPGA market uses OSVVM [1]. In Europe, OSVVM (with 34%) leads SystemVerilog+UVM (with 26%). Based on the growth in our training, we expect to see improved numbers in the next survey.

Register

Pre-Webinar Homework

Want to try out OSVVM before the webinar? See our OSVVM Script User Guide to run our demos. See: https://github.com/osvvm/OSVVM-Scripts#readme. If you prefer pdf (it has section numbers) see: https://github.com/OSVVM/Documentation/blob/main/Script_user_guide.pdf.

Presenter BIO

Jim Lewis is an innovator and leader in the VHDL community. He has 30 plus years of design and teaching experience. He is the Chair of the IEEE 1076 VHDL Standards Working Group. He is a co-founder of the Open Source VHDL Verification Methodology (OSVVM) and the chief architect of the packages and methodology. He is an expert VHDL trainer for SynthWorks Design Inc. In his design practice, he has created designs for print servers, IMA E1/T1 networking, fighter jets, video phones, and space craft.

Whether teaching, developing OSVVM, consulting on VHDL design and verification projects, or working on the IEEE VHDL standard, Mr Lewis brings a deep understanding of VHDL to architect solutions that solve difficult problems in simple ways.

Agenda

• 50 min presentation/live demo
• 10 min Q&A