full functional coverage

Hi Al,

Functional coverage (the one you write - such as with OSVVM) tells you whet her you have exercised all of the items in your test plan. Your test may s et out to achieve some objectives - the functional coverage observes these happening and validates that your directed test really did what it said it was going to do or that your random test did something useful. For more o n this, see osvvm dot org and SynthWorks' OSVVM blog.

Code coverage (what the tool captures for you) tells you that you have exec uted all lines of code. If your functional coverage is 100%, but your code coverage is not, it indicates there is logic there that is not covered by your test plan - either extra logic (which is bad) or your test plan is inc omplete (which is also bad).

Functional coverage at 100% and Code coverage at 100% indicates that you ha ve completed validation of the design.

Best Regards, Jim

Hi Al, Looking at your second question:

For some ideas on this, see my paper titled, "Accelerating Verification Th rough Pre-Use of System-Level Testbench Components" that is posted on Synth Works' papers page.

If you would like this in a class room setting, make sure to catch our VHDL TLM + OSVVM World Tour (our class VHDL Testbenches and Verification). N ext stop is in Sweden on May 5-9 and Germany in July 14-18.

Best Regards, Jim

let's take a bus arbiter for example. It will provide access to the bus to multiple 'masters' with a certain priority and/or schedule. The problem arises when the 'slave' does not reply or finish at the expected time. Let's assume the bus protocol allows the arbiter to 'put on hold' the current master and allow others to access the bus.

In my unit testing I have full controllability and I can simulate a non responsive slave, therfore verifying the correct behavior of the arbiter, but when I integrate the arbiter in a system with several slaves which have been properly verified, I do not have the freedom to force a slave in a non responsive state, therefore at system level I'm not able to cover that functionality (neither that code).

What happen then with my functional coverage? Should it be a collection of system level and unit level reports?

What if my code has been exercised by a unit level test? Theoretically it has been exercised, but not in the system that it is supposed to work in. Does that coverage count?

There might be functionality that is not practically verifiable because it requires a too long simulation, whether it can be easily validated on the bench with hardware running at full speed. What about those cases?

Old engineering maxim: you can't test quality into a product. (You have to design it into the product)

Don't confuse verification and validation. A crude distinction is that one ensures you have the right design, the other ensures you have implemented the design right. (I always forget which is which, sigh!)

One designer's unit is another designer's system.

Unit tests are helpful but insufficient; you also need system integration tests.

Hi Jim, Jim Lewis wrote: []

I know that paper and it served me well in the past, but I have two issues with the proposed approach:

1. Following the example on the paper with the MemIO, the CpuIF has 'two' interfaces, one towards the CPU, and another one toward the inner part of the entity. If we do not hook up the internal interface somehow we are not verifying the full unit. While in the example the internal interface might be very simple (like a list of registers), it might not be always the case.
2. As subblocks are available, together with testcases through the system level testbench, a complex configuration system has to be maintained in order to instantiate only what is needed. This overhead might be trivial if we have 4 subblocks, but it may pose several problems when the amount of them increases drastically.

I do not quite understand the reason to split testcases in separate architectures, I use to envelop the TbMemIO in what I call 'harness' and instantiate it in each of my testcases. The harness grows as BFMs become available and it never breaks an earlier testcase since the newly inserted BFM was not used in earlier testcases.

Unfortunately cost and time are not yet available. I'd love to follow that course but as of now I need to rely on my own trials and guidance like yours ;-)

Hi Al,

See next answer.

Many do integrate the functional coverage of the core (unit) and system. The risk is that the system is connected differently than the core level testbench. However in your case you are testing for correctness and you get this at the system level by accessing the working slaves.

However, VHDL does give you a couple of good ways to test non-responsive slave model in the system. First you can use about VHDL configuration to swap in the non-responsive slave model for one of the correct models.

Alternately (not my preference), you can use VHDL-2008 external names and VHDL-2008 force commands to drive the necessary signals to make a correct model non-responsive.

For ASICs, the general solution to this is to use an emulator or FPGA board to test it. In most cases, your lab board is as good as an emulator. The only case where emulators would be better is if they collect coverage and report it back in a form that can be integrated with your other tests.

Best Regards, Jim

Hi Jim. Jim Lewis wrote: []

that is indeed yet another reason for 'fearing' the core level testbench approach indeed. Thanks for pointing that out.

As easy as it sounds it looks to be the most elegant solution. A simple beh. model of a non responsive module may trigger various types of scenarios.

I've found a third option which might be quite interesting especially when dealing with standard buses:

klabs.org/richcontent/software_content/vhdl/force_errors.pdf ???

The interesting thing about this technique is how you can randomly generate errors on a bus without the need to model them. You do need to model the coverage though.

In that case we have a pile of docs (verification reports, compliance matrices, coverage results, ...) which feels the gap (and the day).

Al

Hi Al,

It is a problem. If the interface is simple enough, you test it when integrating the next block. If the interface is complex enough, it could writing behavioral model to test it.

The alternative of course is to instantiate them all and deal with the run time penalty of having extra models present that are not being used. Depending on your system, this may be ok.

The other alternative is to develop separate testbenches for each of the different sets of blocks being tested - I suspect the configurations will always be easier than this.

You are right though as too many configurable items results in a proliferation of configurations that I call a configuration expolsion. :)

This separation is important for reuse at different level of testing. Separate that test case from that models that implement the exact interface behavior. The models can be implemented with either a procedure in a package (for simple behaviors) or an entity and architecture (for more complex models - this is what the paper I referenced shows).

Lets say you are testing a UART. Most UART tests can be done both at the core-level and system-level. If you have the test cases separated in this manner, then by using different behavior models in the system, you can use the same test cases to test both.

Do you work for free? If not, I suspect the cost of you learning by reading, trial and error, and searching the internet when you run into issues is going to cost much more than a class. You seem to be progressing ok though.

Jim

Hi Al,

This is ok at a testbench level, but if you wanted to use it inside your system, you will need to check and see if your synthesis tool is tolerant of user defined resolution functions.

You will want to make sure that the error injector (on the generation side) communicates with the checker (on the receiver side) so that the checker knows it is expecting a particular type of error. In addition, you will want your test case generator to be able to initiate errors in a non-random fashion.

For my testbenches, I like each BFM to be able to generate any type of error that the DUT can gracefully handle. Then I set up the transactions so they can communicate this information. To randomly generate stimulus and inject errors, I use the OSVVM randomization methods to do this at the test case generation level (TestCtrl).

Cheers, Jim

P.S. Like @Tom Gardner hinted at, I avoid the word "UNIT" as it means different things to different organizations. I have seen authors use UNIT to mean anywhere from Design Unit (Entity/Architecture) to a Core to a Subsystem (a box of boards that plugs into an airplane). As a result, I use Core/Core Tests as most people seem to relate that to being a reusable piece of intellectual property - like a UART.

IMNSHO, the "unit" is the thing to which the stimulus is applied and the response measured. Hence a "unit" can be a capacitor, opamp, bandpass filter, register, adder, xor gate, ALU, CPU, whatever is hidden inside an FPGA, PCBs containing one or more of the above, crate of PCBs, a single statement a=b+c, a statement invoking library functions anArrayOfPeople.sortByName() aMessage.email()

What too many people don't understand is that in many of those cases they aren't "simple unit tests", rather they are integration tests - even if it is only testing that your single statement works with the library functions.

Another point, which is too often missed, is to ask "what am I trying to prove with this test". Classically in hardware you need one type of test to prove the design is correct, and an /entirely different/ set of tests to prove that each and every manufactured item has been manufactured correctly.

But I'm sure you are fully aware of all that!

Hi Jim,

Jim Lewis wrote: []

I guess that's a kind of a lesson learned the hard way. I do not have as of now any working example that follows the option above mentioned, but if I manage to give it a try I'll post some results/impressions.

Right. My idea was to use the 'error injector' as a /slave/ but I do not yet know how to connect it to my transaction generator (or whatever is the appropriate name for it). I may potentially use global signals, but I'm not a big fan of them since they tend to increase universe entropy drastically!

In my framework I use two global records to communicate with the 'server' and I could in principle extend them to pass the information further along within the defective module. It is not entirely clean though, in fact the server cannot handshake with the defective module since it does not know it has been so configured. Well, we can imagine that if the global records elements are filled it is because the client wants to inject those errors, so the server can assume the defective module is in place and will handshake with it. Uhm, this is getting interesting...

And indeed the mechanism would work even if the BFM are encapsulated within internal components of the DUT and implement the error injection as suggested above.

BTW I just put my hands on 'Comprehensive functional verification' Wile, Goss, Roesner, it's a 700 pages volume on the verification cycle, I guess it'll keep me busy for some time ;-)

Al

Hi Jim, Jim Lewis wrote: []

I think I did not correctly phrased what I meant. I do understand the importance of testcases to be separate. I have my 'client/server' model in place, where my transaction happens between them and then the server implements each call with a specific low level transaction, involving the BFM for each interface. In this way my client may operate as long as the server interface is kept the same.

What I did not understand in your paper was the 'need' of a separate architecture instead of a separate file with a new entity/architecture.

The harness I referred to earlier is what instantiate the DUT and all the necessary BFMs, the client/server transaction happens with a pair of global records (to_srv_ctrl/fr_srv_ctrl) which handles the 'handshake' in the transaction. The harness grows as more BFMs become available and more interfaces are being implemented.

I'm used to learn by reading (and asking). On top of this, my actual job description does not require these skills...another reason for not being able to ask for such a course! But I'm not giving up ;-)

Al

Hi Al, Maybe this time I will answer the right question. :)

There are two issues. The obvious one is having multiple copies of the sam e entity can lead to maintenance issues if the interface changes.

The not so obvious is the impact of compile rules. Dependencies in VHDL ar e on primary units (entity, package declaration, configuration declaration) and not secondary design units (architecture or package body).

Lets assume the TestCtrl entity and architecture are in separate files and I am not using configurations. I can compile my design and test hierarchy and run test1. Now to run test2, all I need to do is compile test2 archit ecture and I can restart the testbench and it runs (because the most recent ly compiled architecture is selected).

Going further, when I add configurations, I can compile my design, test hie rarchy, test1 architecture, and configuration1, and then run configuration1 . Then I can compile test2 architecture and configuration2, and then run c onfiguration2. To re-run configuration1, all I need to do is select run co nfiguration1.

On the other hand, if we put the entity in the same file as the architectur e, then the process gets much harder. If you compile test1, since the enti ty was recompiled, the testbench hierarchy above TestCtrl must also be reco mpiled, then you can start a simulation. To run test2, again compile test2 and the testbench hierarchy above TestCtrl, and then start a simulation. Configuration declarations would require this same compile extensive proce ss.

I note some simulators work around some of this stuff with switches and aut omatically. With switches works well, but is tedious. Automatically works most of the time, but from time to time it does not and that makes for deb ug sessions in which you are chasing a nonexistent problem.

Hope I answered the right question this time.

Cheers, Jim