Test Driven Design?

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
Anyone doing any test driven design for FPGA work?

I've gone over to doing it almost universally for C++ development,  
because It Just Works -- you lengthen the time to integration a bit, but  
vastly shorten the actual integration time.

I did a web search and didn't find it mentioned -- the traditional "make  
a test bench" is part way there, but as presented in my textbook* doesn't  
impose a comprehensive suite of tests on each module.

So is no one doing it, or does it have another name, or an equivalent  
design process with a different name, or what?

* "The Verilog Hardware Description Language", Thomas & Moorby, Kluwer,  
1998.

--  
www.wescottdesign.com

Re: Test Driven Design?
Quoted text here. Click to load it

We do it.  We have an equivalence checker that fuzzes random inputs to
both the system and an executable 'golden model' of the system, looking for
discrepancies.  If found, it'll then reduce down to a minimal example.

In particular this is very handy because running the test cases is then
synthesisable: so we can run the tests on FPGA rather than on a simulator.

Our paper has more details and the code is open source:
https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/201509-memocode2015-bluecheck.pdf

Theo

Re: Test Driven Design?
On Wed, 17 May 2017 00:47:39 +0100, Theo Markettos wrote:

Quoted text here. Click to load it
memocode2015-bluecheck.pdf
Quoted text here. Click to load it

So, you have two separate implementations of the system -- how do you  
know that they aren't both identically buggy?

Or is it that one is carefully constructed to be clear and easy to  
understand (and therefor review) while the other is constructed to  
optimize over whatever constraints you want (size, speed, etc.)?

--  
www.wescottdesign.com

Re: Test Driven Design?
Quoted text here. Click to load it

Is that the problem with any testing framework?
Quis custodiet ipsos custodes?
Who tests the tests?

Quoted text here. Click to load it

Essentially that.  You can write a functionally correct but slow
implementation (completely unpipelined, for instance).  You can write an
implementation that relies on things that aren't available in hardware
(a+b*c is easy for the simulator to check, but the hardware implementation
in IEEE floating point is somewhat more complex).  You can also write high
level checks that don't know about implementation (if I enqueue E times and
dequeue D times to this FIFO, the current fill should always be E-D)

It helps if they're written by different people - eg we have 3
implementations of the ISA (hardware, emulator, formal model, plus the spec
and the test suite) that are used to shake out ambiguities: specify first,
write tests, three people implement without having seen the tests, see if
they differ.  Fix the problems, write tests to cover the corner cases.  
Rinse and repeat.

Theo

Re: Test Driven Design?
On Thu, 18 May 2017 14:48:12 +0100, Theo Markettos wrote:

Quoted text here. Click to load it

It's a bit different on the software side -- there's a lot more of "poke  
it THIS way, see if it squeaks THAT way".  Possibly the biggest value is  
that (in software at least, but I suspect in hardware) it encourages you  
to keep any stateful information simple, just to make the tests simple --  
and pure functions are, of course, the easiest.

I need to think about how this applies to my baby-steps project I'm  
working on, if at all.

--  
www.wescottdesign.com

Re: Test Driven Design?
On 18/05/17 15:22, Tim Wescott wrote:
Quoted text here. Click to load it

Interesting questions with FSMs implemented in software...

Which of the many implementation patterns should
you choose?

My preference is anything that avoids deeply nested
if/the/else/switch statements, since they rapidly
become a maintenance nightmare. (I've seen nesting
10 deep!).

Also, design patterns that enable logging of events
and states should be encouraged and left in the code
at runtime. I've found them /excellent/ techniques for
correctly deflecting blame onto the other party :)

Should you design in a proper FSM style/language
and autogenerate the executable source code, or code
directly in the source language? Difficult, but there
are very useful OOP design patterns that make it easy.

And w.r.t. TDD, should your tests demonstrate the
FSM's design is correct or that the implementation
artefacts are correct?

Naive unit tests often end up testing the individual
low-level implementation artefacts, not the design.
Those are useful when refactoring, but otherwise
are not sufficient.

Re: Test Driven Design?
On 5/18/2017 12:14 PM, Tom Gardner wrote:
Quoted text here. Click to load it

Personally, I custom design FSM code without worrying about what it  
would be called.  There really are only two issues.  The first is  
whether you can afford a clock delay in the output and how that impacts  
your output assignments.  The second is the complexity of the code  
(maintenance).


Quoted text here. Click to load it

Such deep layering likely indicates a poor problem decomposition, but it  
is hard to say without looking at the code.

Normally there is a switch for the state variable and conditionals  
within each case to evaluate inputs.  Typically this is not so complex.


Quoted text here. Click to load it

Designing in anything other than the HDL you are using increases the  
complexity of backing up your tools.  In addition to source code, it can  
be important to be able to restore the development environment.  I don't  
bother with FSM tools other than tools that help me think.


Quoted text here. Click to load it

I'll have to say that is a new term to me, "implementation  
artefacts[sic]".  Can you explain?

I test behavior.  Behavior is what is specified for a design, so why  
would you test anything else?


Quoted text here. Click to load it


--  

Rick C

Re: Test Driven Design?
On 18/05/17 18:01, rickman wrote:
Quoted text here. Click to load it

It was a combination of technical and personnel factors.
The overriding business imperative was, at each stage,
to make the smallest and /incrementally/ cheapest modification.

The road to hell is paved with good intentions.


Quoted text here. Click to load it

This was an inherently complex task that was ineptly
implemented. I'm not going to define how ineptly,
because you wouldn't believe it. I only believe it
because I saw it, and boggled.


Quoted text here. Click to load it

Very true. I use that argument, and more, to caution
people against inventing Domain Specific Languages
when they should be inventing Domain Specific Libraries.

Guess which happened in the case I alluded to above.


Quoted text here. Click to load it

Nothing non-obvious. An implementation artefact is
something that is part of /a/ specific design implementation,
as opposed to something that is an inherent part of
/the/ problem.


Quoted text here. Click to load it

Clearly you haven't practiced XP/Agile/Lean development
practices.

You sound like a 20th century hardware engineer, rather
than a 21st century software "engineer". You must learn
to accept that all new things are, in every way, better
than the old ways.

Excuse me while I go and wash my mouth out with soap.


Quoted text here. Click to load it


Re: Test Driven Design?
On 5/18/2017 6:06 PM, Tom Gardner wrote:
Quoted text here. Click to load it

If we are bandying about platitudes I will say, penny wise, pound foolish.


Quoted text here. Click to load it

Good design is about simplifying the complex.  Ineptitude is a separate  
issue and can ruin even simple designs.


Quoted text here. Click to load it

An exception to that rule is programming in Forth.  It is a language  
where programming *is* extending the language.  There are many  
situations where the process ends up with programs written what appears  
to be a domain specific language, but working quite well.  So don't  
throw the baby out with the bath when trying to save designers from  
themselves.


Quoted text here. Click to load it

Why would I want to test design artifacts?  The tests in TDD are  
developed from the requirements, not the design, right?


Quoted text here. Click to load it

Lol

--  

Rick C

Re: Test Driven Design?
On 19/05/17 01:53, rickman wrote:
Quoted text here. Click to load it

I see why you are saying that, but I disagree. The
Forth /language/ is pleasantly simple. The myriad
Forth words (e.g. cmove, catch, canonical etc) in most
Forth environments are part of the "standard library",
not the language per se.

Forth words are more-or-less equivalent to functions
in a trad language. Defining new words is therefore
like defining a new function.

Just as defining new words "looks like" defining
a DSL, so - at the "application level" - defining
new functions also looks like defining a new DSL.

Most importantly, both new functions and new words
automatically have the invaluable tools support without
having to do anything. With a new DSL, all the tools
(from parsers to browsers) also have to be built.



Quoted text here. Click to load it

Ideally, but only to some extent. TDD frequently used
at a much lower level, where it is usually divorced
from specs.

TDD is also frequently used with - and implemented in
the form of - unit tests, which are definitely divorced
from the spec.

Hence, in the real world, there is bountiful opportunity
for diversion from the obvious pure sane course. And
Murphy's Law definitely applies.

Having said that, both TDD and Unit Testing are valuable
additions to a the designer's toolchest. But they must
be used intelligently[1], and are merely codifications of
things most of us have been doing for decades.

No change there, then.

[1] be careful of external consultants proselytising
the teaching courses they are selling. They have a
hammer, and everything /does/ look like a nail.

Re: Test Driven Design?
On 5/19/2017 4:59 AM, Tom Gardner wrote:
Quoted text here. Click to load it

I can't find a definition for "trad language".


Quoted text here. Click to load it

I have no idea what distinction you are trying to make.  Why is making  
new tools a necessary part of defining a domain specific language?

If it walks like a duck...

FRONT LED ON TURN

That could be the domain specific language under Forth for turning on  
the front LED of some device.  Sure looks like a language to me.

I have considered writing a parser for a type of XML file simply by  
defining the syntax as Forth words.  So rather than "process" the file  
with an application program, the Forth compiler would "compile" the  
file.  I'd call that a domain specific language.


Quoted text here. Click to load it

There is a failure in the specification process.  The projects I have  
worked on which required a formal requirements development process  
applied it to every level.  So every piece of code that would be tested  
had requirements which defined the tests.


Quoted text here. Click to load it

They are?  How then are the tests generated?


Quoted text here. Click to load it


--  

Rick C

Re: Test Driven Design?
Den torsdag den 18. maj 2017 kl. 15.48.19 UTC+2 skrev Theo Markettos:
Quoted text here. Click to load it

the test?

if two different implementations agree, it adds a bit more confidence that an  
implementation agreeing with itself.  




Re: Test Driven Design?
On 5/18/2017 12:08 PM, snipped-for-privacy@gmail.com wrote:
Quoted text here. Click to load it

The point is if both designs were built with the same misunderstanding  
of the requirements, they could both be wrong.  While not common, this  
is not unheard of.  It could be caused by cultural biases (each company  
is a culture) or a poorly written specification.

--  

Rick C

Re: Test Driven Design?
On Thu, 18 May 2017 13:05:40 -0400, rickman wrote:

Quoted text here. Click to load it

Yup.  Although testing the real, obscure and complicated thing against  
the fake, easy to read and understand thing does sound like a viable  
test, too.

Prolly should both hit the thing with known test vectors written against  
the spec, and do the behavioral vs. actual sim, too.

--  

Tim Wescott
Wescott Design Services
We've slightly trimmed the long signature. Click to see the full one.
Re: Test Driven Design?
On 18/05/17 18:05, rickman wrote:
Quoted text here. Click to load it

The prior question is whether the specification is correct.

Or more realistically, to what extent it is/isn't correct,
and the best set of techniques and processes for reducing
the imperfection.

And that leads to XP/Agile concepts, to deal with the suboptimal
aspects of Waterfall Development.

Unfortunately the zealots can't accept that what you gain
on the swings you lose on the roundabouts.


Re: Test Driven Design?
On 5/18/2017 6:10 PM, Tom Gardner wrote:
Quoted text here. Click to load it

I'm sure you know exactly what you meant.  :)

--  

Rick C

Re: Test Driven Design?
I do a sloppy version of it.  
Sometime I allow myself not to do tests for some simple and small modules which will be tested on a higher hierarchy level anyway. (Because I also make tests for modules of different hierarchy levels)
Tests are randomized and they are launched with different seeds every time. If there is a problem, I also can launch the faulty test with a specific seed to repeat the problem.

Re: Test Driven Design?
On 5/16/2017 4:21 PM, Tim Wescott wrote:
Quoted text here. Click to load it

I'm not clear on all of the details of what defines "test driven  
design", but I believe I've been using that all along.  I've thought of  
this as bottom up development where the lower level code is written  
first *and thoroughly tested* before writing the next level of code.

How does "test driven design" differ from this significantly?

--  

Rick C

Re: Test Driven Design?
On Wed, 17 May 2017 11:47:10 -0400, rickman wrote:

Quoted text here. Click to load it

The big difference in the software world is that the tests are automated  
and never retired.  There are generally test suites to make the mechanics  
of testing easier.  Ideally, whenever you do a build you run the entire  
unit-test suite fresh.  This means that when you tweak some low-level  
function, it still gets tested.

The other big difference, that's hard for one guy to do, is that if  
you're going Full Agile you have one guy writing tests and another guy  
writing "real" code.  Ideally they're equally good, and they switch off.  
The idea is basically that more brains on the problem is better.

If you look at the full description of TDD it looks like it'd be hard,  
slow, and clunky, because the recommendation is to do things at a very  
fine-grained level.  However, I've done it, and the process of adding  
features to a function as you add tests to the bench goes very quickly.  
The actual development of the bottom layer is a bit slower, but when you  
go to put the pieces together they just fall into place.

--  

Tim Wescott
Wescott Design Services
We've slightly trimmed the long signature. Click to see the full one.
Re: Test Driven Design?
On 5/17/2017 1:17 PM, Tim Wescott wrote:
Quoted text here. Click to load it

I guess I'm still not picturing it.  I think the part I don't get is  
"adding features to a function".  To me the features would *be*  
functions that are written, tested and then added to next higher level  
code.  So I assume what you wrote applies to that next higher level.

I program in two languages, Forth and VHDL.  In Forth functions (called  
"words") are written at *very* low levels, often a word is a single line  
of code and nearly all the time no more than five.  Being very small a  
word is much easier to write although the organization can be tough to  
settle on.

In VHDL I typically don't decompose the code into such fine grains.  It  
is easy to write the code for the pieces, registers and logic.  The hard  
part is how they interconnect/interrelate.  Fine decomposition tends to  
obscure that rather than enhancing it.  So I write large blocks of code  
to be tested.   I guess in those cases features would be "added" rather  
than new modules being written for the new functionality.

I still write test benches for each module in VHDL.  Because there is a  
lot more work in writing a using a VHDL test bench than a Forth test  
word this also encourages larger (and fewer) modules.

Needless to say, I don't find much synergy between the two languages.

--  

Rick C

Site Timeline