Combining Results from Multiple Runs into One HTML Report

Why OSVVM™? › Forums › OSVVM › Combining Results from Multiple Runs into One HTML Report

Tagged: , ,

Viewing 4 posts - 1 through 4 (of 4 total)
  • Author
    Posts
  • January 14, 2025 at 13:35 #2602
    David
    Member

    Hi everyone,

    I’m hoping to be able to run my test cases on different machines and then combine everything into one HTML report (transcripts, code coverage, and functional coverage).

    Is there a way in OSVVM to gather all these separate results into a single HTML report? I’d like to merge logs, coverage data, and any pass/fail info for the test case summary report.

    Are there scripts or best practices for this?

    Thanks for any pointers!

    January 14, 2025 at 16:44 #2603
    Jim Lewis
    Member

    Hi David,
    Not yet. There are some planned changes coming later this year (2nd half) that should allow us to do this.

    If you are thinking code coverage data, you should be able to merge the results from each build together using the tool provided methods. It would be easy to add that capability to OSVVM.

    Where we currently have one big linear log file is where it gets interesting. Do you think of your separate runs as multiple separate things and the log file gets concatenated or are the log files small and need to be integrated together.

    The big picture we would like to address with this change is the ability to take advantage of open source simulators and an infinite number of simulator licenses – and hence, parallelize the running of our separate simulations – because OSVVM currently has 852 to run in a regression.

    Best Regards,
    Jim

    January 14, 2025 at 17:20 #2604
    David
    Member

    Hi Jim,

    Thanks for the update and the clarification.

    Exactly as you say, my main desire is parallelising the test execution. If I have 24 test cases and 24 simulator licenses, I’d like to complete the entire suite in one hour by running everything in parallel, rather than taking 24 hours to complete sequentially. Its common for us to have hundreds of test cases.

    For code coverage, I’m comfortable with merging results across runs, as we already do this with Questa for other testbenches.

    Do you think of your separate runs as multiple separate things and the log file gets concatenated or are the log files small and need to be integrated together.

    I wouldn’t expect the logs to be concatenated into one large transcript. Instead, I picture a test case summary in the HTML report where each test case links to its own log file, similar to how alert reports are currently handled.

    As for the combined HTML report, it would be great to have a single summary report that includes pass/fail results, code coverage, and functional coverage for the entire suite, with links to the test cases so you can drill in to get more detail. Right now, I can get a separate HTML reports per test case, but having a combined report would be awesome.

    Looking forward to seeing how this evolves later in the year!

    Thanks again for your quick replies!

    Best Regards,
    David

    January 14, 2025 at 17:52 #2605
    Jim Lewis
    Member

    Hi David,
    There may be some things you could do now to create what you want. I would probably merge the YAML files and then look at how build finishes as that is the step where the code coverage is merged.

    Best Regards,
    Jim

  • Author
    Posts
Viewing 4 posts - 1 through 4 (of 4 total)
  • You must be logged in to reply to this topic.