ANNOUNCE: Maia 0.8.2: module-level HDL verification tool

Not to be too much of a slug, but I'm kinda missing the point I think of this tool....

Why would I want to write test vectors (Hint: I don't). Using a language with control structures is far more concise and easier to maintain.

If I did have to write test vectors I would probably use something to generate those test vectors as an output file artifact from a simulation run.

Having flashbacks to generating test vectors in ABEL...something from

20 years ago.

KJ

KJ wrote:

Maybe the intended audience is those unmotivated to learn vhdl or verilog. Sort of a testbench 'wizard' to go with wizard generated code.

Not my cup of tea, but it might get someone started.

-- Mike Treseler

If you look at tools like Mentor's inFact they do exactly that although they don't create a vector file but an intelligent testbench instead. The inputs are high-level rules and actions. These high-level rules are compiled into testbench graphs which are then use to generate optimised vector sets.

For example to test a UART you write different rules for all the different options like word size, parity type, number of stop bits, interrupt sequence, read/write sequence etc. The compile will then use this info to generate an optimised vector set that test all valid combinations (not 100% sure on this one). I believe this is one step up from writing assertions and feeding your design with constrained random data.

This is a trivial example but you can understand that with more rules this becomes more and more powerful. From what I understand these tools are very complex since they not only need to find the smartest set of testvectors but also handle all the different constraints and constraint solvers is one area which is still heavily being researched.

I would say kudos to Evan for attempting to develop a product like this and making it freely available to the rest of us,

Hans

Having flashbacks to generating test vectors in ABEL...something from

20 years ago.

KJ

ey

s

There is a huge difference between specifying high level rules and specifying vectors as the thing that the test creator needs to input. In fact it's hard to imagine a lower level form of rules than vectors. As a general rule, most skilled people are likely to be much more productive working with a higher level of abstraction than with a lower one.

%

nd

y

ut

ea

Sounds like a pitch for the Mentor stuff, not Evan's.

d

I agree, I wasn't trying to disparage the effort, just questioning the rationale for creating a tool that requires the user to input vector sets which is a form of test creation popular 20 years ago.

KJ

Could be, but I hope not. I've noticed a couple of interesting things over the years:

- Most EEs aren't programmers; electronic design and programming are two completely different things. Verification is programming; the techniques, tools, and mindset required are different from those required to do electronic design. And yet, bizzarely, we generally expect the same people to use the same language to do both things. The reasons for this are, I think, historical and market-related, and have nothing to do with the abilities (or otherwise) of Verilog and VHDL.

- Most EEs don't actually verify their code, mainly for the reason above. I know people who don't verify at all, and others who use waveform displays. You can get away with this in the FPGA world because it's fairly simple (if time-consuming) to fix things, and in the ASIC world because you've got a verification department to find the problems (or you've actually got your personal verification engineer assigned to you; I've been there).

Given that lot, the target audience is actually competent EEs who don't have the time or inclination to start writing complex programs in an inappropriate language to test their RTL code. Imagine you've spent a day writing your FSM, or bus interface, or whatever. It gets to 5 o'clock, and your choice is either (a) to go home and spend the next day writing a testbench for it, when you've already forgotten what it does, or (b) spend the next half-hour writing "test vectors" (yes, bad phrase) for it. I know which one I'd go for.

There's at least one other target audience. If you can read a spec for someone else's module, then you write vectors for it, even if you know absolutely no VHDL or Verilog. So, next time your intern's got nothing to do, you can get them doing something constructive without having to send them on a training course first.

-Evan

I wouldn't get too carried away by the 'test vector' terminology. When you get down to it, most TBs involve only 3 activities:

1 - generate a set of input data 2 - make sure that the input data gets applied to the DUT at the right time 3 - wait a bit, check that the output data is as expected, repeat

That's how the RTL works, so it would be difficult to do it any other way. I call that lot a 'test vector'. It's convenient to put everything in a "[,,] -> [,,]" format, because it's then obvious exactly what expressions have to be evaluated and applied to the DUT, and what exactly the DUT outputs have to be tested against. It's actually a lot more sophisticated than ABEL - all the inputs and outputs can be arbitrary expressions, for example, and there are full C-like control structures, so you get reactive TBs, and all the rest of it. If you'd prefer, they're not actually "test vectors" - they're just groups of input and output expressions, but I couldn't think of a word for that. The great thing (if I do say so myself :)) is that you have to write absolutely no multi-process code to do this. You don't need to know any of the TB-related intricacies of VHDL or Verilog to do this; it's trivial.

It's not a magic bullet, as I say on the website. There are basically

2 reasons that you can't use it for all your everyday verification needs: 4 - the real smarts in a TB is generating the input data; ie. step 1 above (I'm ignoring here the additional smarts required for coverage stats collection, temporal assertions, and so on). Currently, your only options in Maia are essentially table- and algorithm-based. I should also have file input before too long. I've already done constrained randomisation for another tool, and that should make its way in at some point in the future. 5 - There's currently no way to add additional behavioural code to talk to your DUT (a complex bus interface fronting a memory, for example). But, again, I've already done this code elsewhere, so it should make its way in at some point.

-Evan

...

That's a good point. I'm always confused when someone asks me if I have "test vectors" for a design. No; I have a testbench. The whole concept of test vectors doesn't even seem to apply for any design with feedback. I'm having flashbacks to testbenches composed of simulator-specific "force" commands. -Kevin

Different concept. This is not a traditional device "test vector"; it's more of a software "test vector". The phrase "test vector" does seem to have pressed all the wrong buttons and I've changed it on the website, and added an explanation on the front page.

In short, these "vectors" may contain arbitrary expressions, and can be enclosed in standard control constructs (my first posting actually showed a simple example of both). You can therefore write arbitrary reactive ("feedback") testbenches; it's nothing like a device test vector.

However, those are just details. The fundamental idea behind a "vector" is something different; it's a step up in abstraction levels. The idea is that you don't have to worry about the details of writing explicit multi-process timed code; no clock or reset processes, no reader and checker processes, no driving and sampling, no pass/fail determination, no error reporting, no process communication, and so on. It's all wrapped up, automatically, in the "[,,] -> [,,]" ("vector") statement.

As an engineer testing your module, why would you care about any of that stuff? It's just irrelevant busywork, because Verilog and VHDL aren't smart enough to handle the details for you. I've expanded all this on the front page of maia-eda.net, under "what is a vector?".

-Evan

Any engineer who can't write simple code in his target language to test the RTL he wrote, shouldn't be writing either one.

Secondly, if the vectors can be written in a half-hour, the RTL design code shouldn't have taken all day, unless you're testing to what you designed, instead of testing to the requirements. My point is that a designer should approach developing the test from the same point of view as developing the design. What's the likelihood that a designer will have spent all day and forgotten a required feature, and a half- hour later will have remembered that same feature when writing the vectors?

Andy

That may be true, but some get by with trial and error synthesis on designs that glue together IP and "known good" modules. An fpga guy who is good with logic analyzers and test fixtures can debug some fpga designs on the bench. Not my cup of tea, but I've seen it done on some successful products.

For new synthesis code, most of my simulation time is testing the design. Working demo units are required to sign off a business case, and specifications start out fuzzy. Quick turn-around is the key, and a clean, closed loop testbench is an advantage.

I agree. I design and test interactively. I can't imagine doing otherwise.

Once I have a testable procedure with inputs and outputs, I add stim procedures to the testbench and at least some "printf" style verification before moving on to the next lump of logic.

-- Mike Treseler