Writing a Test Plan, Identifying Transactions, and Documenting BFMs/MIT

[David]

Hello fellow forum members,

First time posting here! I’m currently early in the adoption of OSVVM. I work for a company that develops safety critical FPGAs and we put lots of emphasis (and rely) on planning and documentation artefacts. I’ve encountered a few questions and would greatly appreciate your guidance and insights.

1. Writing a Test Plan:
I’m eager to learn more about the process of crafting a robust test plan using transactions and randomizations. Our conventional approach involves describing stimulus and expected results, but I’m uncertain about incorporating transactions within this framework. If any of you could provide practical examples or share effective strategies for writing comprehensive test plans that embrace transactions, I would be grateful.

2. Identifying Transactions:
In order to ensure sufficient coverage of the desired functional features in my FPGA design, I’m seeking best practices and guidelines for identifying transactions. Are there any specific techniques or approaches you’ve found effective? I’m keen to learn how to systematically define transactions that encompass a range of scenarios and adequately stress the design.

3. Documenting BFMs (Bus Functional Models) / MIT (Model Independent Tests):
When it comes to documenting and planning BFMs or MITs, I’m interested in learning about best practices for documenting BFMs or MITs. What information should I include in the documentation to ensure clarity and comprehensiveness?

If any of you could recommend any insightful reading materials or resources that delve into test planning or provide examples of how to structure a test plan effectively, it would be perfect.

Thank you kindly for your time and support. I look forward to hearing your thoughts!

Warm regards,
Dave

[Jim Lewis]

Hi Dave,
I suspect a proper response to this may end up being a couple of blog posts in length. Lets start with a couple of references:

Comprehensive Functional Verification by Bruce Wile, John Goss, and Wolfgang Roesner covers test plans. I have not read the whole book. It is on my some day maybe list.

Peet James Snug 2000 paper on Verification plans
https://www.rose-hulman.edu/class/ee/doering/From_Novell/EC533/qualis/verifplan.pdf

Verification Plans by Peet James. More words than the Snug paper. I was not fond of his writing style in the book though. I liked the paper better.

[Jim Lewis]

Lets address, 2. Identifying Transactions:

A transaction is an abstract representation of an atomic action on an interface. For a UART transmitter, it is sending one word on the interface. For an Axi4Lite interface, it is Write and Read operations on the interface.

For the simple interface we look at in our Comprehensive VHDL Introduction, it is drive the device select line active for one clock cycle and put a value on the command interface. There also needs to be some time between driving a values, so we added a wait time before accepting the next cycle.

From a signal wiggling perspective, this is our sequence:

LedFlasherSel  <= 
    '1' after tpd_Clk_LedFlasherSel, 
    '0' after tperiod_Clk + tpd_Clk_LedFlasherSel ;
Addr           <= FLASH_ONE_CMD after tpd_Clk_Addr ;
wait for (3 * TIME_BASE) - 1 ns ; 
wait until Clk = '1' ;

I call tests written using signal wiggling, brute force testing. Abstractly a test for this device looks like the following code. When we write tests this way, we are challenged to understand what each step does, so comments focus on exactly that.

TestProc : process
begin
  wait until nReset = '1' ;
  wait until Clk = '1' ;

  -- Cmd = FLASH_ONE_CMD  WAIT = 1 TIME_BASE
  ... -- block of code that does the above

  -- Cmd = FLASH_ONE_CMD  WAIT = 5 TIME_BASE
  ... -- block of code that does the above

  -- Cmd = FLASH_ONE_CMD  WAIT = 1 TIME_BASE
  . . . 
  -- Cmd = FLASH_ONE_CMD  WAIT = 4 TIME_BASE
  -- Cmd = FLASH_ONE_CMD  WAIT = 3 TIME_BASE
  -- Cmd = FLASH_CONT_CMD WAIT = 7 TIME_BASE
  -- Cmd = FLASH_OFF_CMD  WAIT = 5 TIME_BASE
  -- Cmd = FLASH_CONT_CMD WAIT = 7 TIME_BASE
  -- Cmd = FLASH_OFF_CMD  WAIT = 2 TIME_BASE

Transactions abstract the behavior. With transactions our test sequence becomes the following. Note that by using a good set of constants, all of the comments of the brute force approach were replaced by transaction calls. Since we are still inclined to comment, we write appropriate constants that illuminate the intent of the sequences.

TestProc : process
begin
  wait until nReset = '1' ;
  wait until Clk = '1' ;

  -- verify basic Flash One functionality
  SendCmd(FLASH_ONE_CMD, 1 * TIME_BASE, Clk, LedFlasherSel, Addr) ;
  SendCmd(FLASH_ONE_CMD, 5 * TIME_BASE, Clk, LedFlasherSel, Addr) ;
  SendCmd(FLASH_ONE_CMD, 1 * TIME_BASE, Clk, LedFlasherSel, Addr) ;
  SendCmd(FLASH_ONE_CMD, 4 * TIME_BASE, Clk, LedFlasherSel, Addr) ;
  SendCmd(FLASH_ONE_CMD, 3 * TIME_BASE, Clk, LedFlasherSel, Addr) ;

  -- verify Flash Continuously and Flash Off functionality
  SendCmd(FLASH_CONT_CMD, 7 * TIME_BASE, Clk, LedFlasherSel, Addr) ;
  SendCmd(FLASH_OFF_CMD,  5 * TIME_BASE, Clk, LedFlasherSel, Addr) ;
  SendCmd(FLASH_CONT_CMD, 7 * TIME_BASE, Clk, LedFlasherSel, Addr) ;
  SendCmd(FLASH_OFF_CMD,  5 * TIME_BASE, Clk, LedFlasherSel, Addr) ;

When we work on the IEEE standard, our proposals include the presentation of a new language feature and use models that show how the feature will be used. The use model gives us more insight into the proposed language feature.

In the same way, writing the test cases gives us more insight into the test plan – and perhaps what is missing. For example, in the design above, the FlashOne functionality was intended to be implemented separately from the FlashContinuous functionality. So testing took this perspective and tested the two features independently. What happens if the hardware was implemented in another way? What do we expect to happen if FLASH_ONE is issued while the Led is already flashing continuously. Should it continue flashing continuously or should it stop after flashing once? We should know. Therefore, we should verify.

By using transactions, we are able to get a deeper understanding of what the test is doing and see its potential limitations. Also by explaining what the transactions do, people other than hardware design and verification engineers can read the test (at least somewhat).

Is it always this easy? No. The interesting thing about testing this way is that we must have a transaction that supports everything that is gracefully handled by the interface. So for UART, this includes generating errors such as break, stop, and parity.

[Jim Lewis]

Implementing Transactions
The behavior of a transaction can either be implemented in a verification component (VC) or in the subprogram body (as SendCmd above suggests). Note the acronym BFM means bus functional model and can refer to either a VC or subprogram body. BFM means the model implements only the interface behavior and not internal behavior (such as of a UART board part).

OSVVM created a library of transactions called Model Independent Transactions (MIT). In that library we defined two classes of interfaces: Address Bus (also called Memory Mapped) and Stream. Address Bus transactions are for devices like Axi4Manager, Avalon, Wishbone. Stream transactions are for devices such as UART, xMII, or AxiStream.

When you can use the OSVVM MIT library, writing a verification component takes no more effort than writing a procedure. We call this approach then our “Faster than Lite” approach. This allows us to write a simple blocking verification component to get our testing started and then expand the capability of the VC over time as we need more advanced features.

Hence, it is also our intent to expand the capability of the currently defined classes of transactions and add additional classes of transactions – as needed.

[Jim Lewis]

Why Randomize
Our goal is to thoroughly test a design. For some designs, directed testing will get the job done.

For some designs, there are too many variations to be able to test every variation in the system. Variations can be a multiplying factor in testing a design. If the design can limit the variations in the system, the system can be tested more thoroughly – however this may sacrifice capability (flexibility) or performance of the system.

The depth of state information in the design also impacts verification. Is it sufficient to test relationships between the current and next transaction (such as read followed by a write), or do we need to test deeper relationships than that (read, read, write).

When we need to explore a larger space, randomization does a good job. Such as varying packet types, sizes, and cycle delays work well with randomization.

Exploring interactions of different interfaces interacting with the same resource. Simple example of this is where the DUT is an AxiStreamFifo and we randomize the data and delays in writing to and reading from the Fifo.

A more interesting example would be creating traffic to a network switch. Our first thing we might try is doing balanced traffic between all nodes in the switch. A more interesting test might be to designate a node as a file server and another as the WAN. Then randomize to have each node send most of its traffic to either the file server or the WAN and very little (or none) from node to node.

With randomization, we need to confirm we did something useful. This is where functional coverage comes into play. With functional coverage we look to correlate events (or values) on one or more independent objects generally happening at the same time that confirm something useful happened.

[Jim Lewis]

Documenting a Verification Component needs to address

• What does the VC need to do
• How do I use the VC
• What transactions does the VC support

I note that the Axi4_VC_users_guide in OsvvmLibraries/Documentation primarily addresses the last two. Rather than addressing the first one, it assumes that is must support all operations on the AXI4 interface. It probably should be addressing the first one so people understand what it does and does not do – which is an evolving topic.

[Jim Lewis]

Hopefully this gets the topic started. Hopefully others will comment too. It is a long topic and I probably missed some things. I may have an entirely different perspective than others.

[David]

Hello Jim, I just wanted to say thank you for a detailed reply!
I’m churning my way through the concepts and I’m sure that I’ll be back to ask more questions, thanks again for your help!

[Author]

Posts