processor core validation

Ooh! What a good application for test driven design! You're too late, though.

Testing software for quality assurance is like closing your eyes and waving a butterfly net around -- if you find a bug, you know there was one, but if you don't find a bug, you can't be sure there aren't a lot out there still.

I'd advocate for an app that's custom-designed for the core that makes sure that every bit of the thing gets exercised. This is going to mean a lot of assembly code, and some of it oddball.

--
www.wescottdesign.com

Hi Al,

I assume you know that Code Coverage has nothing to do with formal Verification and/or tells you if your processor is bug free, it just measure of the quality of your testbench.

If you want to formally test your processor then AFAIK your only option is to start writing lots of assertion and spend a years salary on a formal tool (solidify/Questa formal/etc). You can also run assertions in your simulator (like Modelsim DE/Riviera Pro) but this will not be exhaustive.

If your processor is small you might get away of writing top level assertions but most likely (due to the cone of logic) you will have to formally prove sub-blocks and use good old testvectors for the interconnects.

What do you think ;-)

If you have a golden reference model then I would suggest a regression suite with lots of constrained random generated instructions, interrupt test programs, OS load, etc and comparing the output to your reference model.

I would also suggest you look up the (old) Viper processor which was a hot topic when I was working in the space industry.

Good luck,

Regards, Hans.

What happened to Viper? Haven't heard anything about it since the early

90's.

--

******************************************************************** Paul E. Bennett IEng MIET..... Forth based HIDECS Consultancy............. Mob: +44 (0)7811-639972 Tel: +44 TBA (due to re-location) Going Forth Safely ..... EBA. ********************************************************************

Hi Paul,

I have no idea, from what I remember Marconi made an SOS rad-hardened version but I don't believe it ever went commercially,

Regards, Hans.

IIRC there was a lawsuit. I don't know the real story, but I remember hearing something to do with the chain of proof not quite being complete from top to bottom.

I have no idea what happened to Viper's associated language, NewSpeak (in which it was supposed to be impossible to express an incorrect thought!).

Hi Tim,

Tim Wescott wrote: []

[]

Sure, taking a piece of stuff and wanting to validate it without thinking it verifiable from day one is certainly a big effort, but maybe less important than having to redesign it from scratch.

Instrumenting the code with appropriate 'test points' is certainly awkward but still doable.

That's the very basic principle of verification, you cannot guarantee anything else except what you verified it working. Testing has it's own limits and that is why software development in hirel apps is not as straight forward as for commercial/industrial applications.

I believe this is only the trivial stuff, verifying instruction correctness should be not that complex. More critical would be interrupt handling according to code execution. Is the interrupt always served? Are there any specific instances where the interrupt can be masked by a certain combination of instructions in the pipeline?

Still I have more questions than answers at this point! :-/

Al

Hi Hans,

HT-Lab wrote: []

sure, functional verification is certainly something different from code coverage.

That's an interesting tool, I didn't know about that. Surely it will speed up functional coverage since it analyses the reachable/unreachable states from a defined stimulus.

I believe though that is out of budget for the intended purpose.

It looks like ABV might be one path but it looks to me extremely tedious, especially if it needs to go at block level.

Yeah, that was a silly question!

We should be able to set up the environment with one of the models provided by OVP, then throw a lot of transactions to both of them and see how they behave.

I suspect though that the model itself might be not so bug-free in the end. Still it would be a good starting point.

Didn't know about that. Thanks for the pointer.

Al

You could have a combination hardware/software test harness, where you have a full test app (or perhaps an app that loops through the processor states that are of most interest) while at the same time the processor is getting interrupted by an asynchronous source that either has a different period from the processor's, or which is intentionally random.

Just going through the thing step-by-step and, positing problem modes, and examining the code to ask "how am I going to test for this?" will force you to think in depth about everything -- one of the frequent ways that TDD helps me to find bugs is by forcing me to discover them in the test design, before I even get to the point of throwing software at the test.

--

Tim Wescott 
Wescott Design Services 
http://www.wescottdesign.com

Hi Tim,

Tim Wescott wrote: []

This test harness could effectively be running on a simulator, where I could use a golden model and crosscheck that throwing random interrupts does not cause any deviation from the golden model behavior.

The main difficulty I see would be the functional coverage model I need to put in place to tell me I'm done.

There might be corner cases where the only solution would be crafting a special purpose test program to generate stimuli (or sequences of stimuli) which might be very difficult to achieve otherwise (some sort of direct testing and random testing).

I can simply get the spec of the processor and start to classify the requirements and then go one by one trying to build my functional coverage model. It is possible that going through the code would not be needed, even though it might help us discovering corner cases.

Assuming the golden model is provided (I've found one on OVP, yet not sure whether it is the right one!) I'll just verify compliance to the golden model and maybe present some usecases where some high level software is able to run and work without crashing.

The black box approach has some advantages:

1. no need to know the details (unless we need a fix)
2. bfm for processor's interface are /available/ (write_mem, read_mem)
3. heuristic approach (let's see if it breaks!)

OTOH we might face the need to open the box and go deeper with our verification, where interactions between internal elements have to be analyzed.

With the hypothesis that the processor 'state' does not depend on previous instructions (there's no cache), the order of the instructions does not necessarily matter. Should this be the case I'll probably can simply live with throwing random instructions at it.

Al

I wouldn't put much faith in using random interrupts other than to help your thinking to see if they cause any problems you don't think of. I would say it would much more important to analyze the circuitry and generate stimuli that test the corner cases.

The problem with random stimuli is just that, it is random... well pseudo random and so you don't really know how well it tested things. Testing can't prove something works other than the specific things that are being tested.

What about other aspects of the processor? Surely it has registers and very likely the processor *does* have state, flags, status registers, control registers, pipelining...

--

Rick