Requesting critique of a C unit test environment - Page 2

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
Re: Requesting critique of a C unit test environment
Ian Collins wrote, On 01/09/07 08:21:
Quoted text here. Click to load it

This, however, is not always the case. I've written a function of about
20 lines that IIRC required something line 100-200 tests. If test took a
similar amount of time to run as the code took to compile. It was doing
maths and there where a *lot* of cases to consider.

For an audit of the entire piece of SW (rather than just that one
function) the customer insisted that we print out all of the module test
specs. The stack of A4 paper produced was a couple of feet tall! Running
that set of tests would take rather more than overnight.

On another project, doing a build of our piece of the SW took 8 hours.
Doing a build of all of the SW for the processor took 48 hours. Add
testing to that for each build...

Some projects are a lot harder than yours.
--
Flash Gordon

Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it

Um, I've never seen one like that before, probably because TDD doesn't
yield that type of code.

Quoted text here. Click to load it

Sounds like the US DOD, I'm sure they just weigh or measure
documentation rather than read it!

Quoted text here. Click to load it
Those were the days.  Thank goodness for fast CPUs and distributed building.

--
Ian Collins.

Re: Requesting critique of a C unit test environment
Ian Collins wrote, On 02/09/07 01:35:
Quoted text here. Click to load it

That is only true if it produces more complex code. The 20 odd lines of
code resulted from a requirement to implement two simple looking
equations and one simple statement in English. The reason it was so many
test cases was that it was dealing with one angle in the range +/- 270
degrees, one in the range +/-170 degrees and two in the range +/1 6
degrees. The testing had to verify behaviour with every angle in each
quadrant, every angle at 0, 90 etc, every angle just either side etc. It
was the *maths* together with the chances of selecting the wrong
solution from the trig that meant a lot of test cases, not the
complexity of the code.

Quoted text here. Click to load it

You guessed right, but the important thing is the quantity of tests and
therefore the time it would take to run them all.

Quoted text here. Click to load it

Also the larger projects which tie up the processors for just as long
because they are so much more complex.

I know there is still SW that takes hours to build because within the
last few years I have done builds that have taken hours.
--
Flash Gordon

Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it

Must be huge, the biggest think I build regularly is the OpenSolaris
code base, which takes about 40 minutes on my box.

If a build takes too long, throw more cores at it.  If the tools don't
support distributed building, change the tools.

--
Ian Collins.

Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it

If every unit test has to check every possible value over a large
range, then yes, things could take a while.  I just wrote a program
that iterated over a range of 2**24 in under a second, but if a
function takes three 32-bit arguments an exhaustive test starts to be
impractical.

But presumably in that case you'd just test a carefully chosen subset
of the possible argument values.

--
Keith Thompson (The_Other_Keith) snipped-for-privacy@mib.org  <http://www.ghoti.net/~kst
San Diego Supercomputer Center             <*>  <http://users.sdsc.edu/~kst
We've slightly trimmed the long signature. Click to see the full one.
Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it



Under TDD, you might only need enough tests to establish a linear function
along that int24_t (two tests), and enough to establish its extrema (two
more tests). These run fast.

If you then need more tests, you add them. There is no TDD "or" a formal
test campaign. You are allowed to use "unit test" techniques. For example,
you might calculate a sweep that covers a representative subset of all
possible integers.

If your tests run too long to help development, you push the slow ones out
into a slow suite, and run this on a test server. You can use one of many
Continuous Integration tools to trigger that batch run each time developers
commit. (And they should commit every 5 to 15 minutes.)

You leave every TDD test in the developers' suite.

If the long suites fail, you treat the failure the same as a bug reported by
users. You determine the fix, then write a trivial TDD test which fails
because the fix is not there. Note the test does not exhaustively prove the
fix is correct; it's just enough to pin the fix down. You leave this test
case with the developers' suite.

A program that uses a wide range of an int24_t's possible states is a
program with a high dimensional space. All programs have such a space! The
exhaustive test for such spaces would take forever to run. However, high
dimension spaces are inherently sparse. Tests only need to constrain
specific points and lines within that space.

One way to determine these points and lines is to analyze that space to
determine the minimum set of sweeps of inputs that will cover the whole
space. You test a carefully choses subset of the possible input values.

Another way is to write the tests first. Then you have a test for two points
on every line, and each endpoint to each line, and so on.

If you write tests first, you also get to avoid a lot of debugging. If your
tests fail unexpectedly, you have the option to revert back to the last
state where all tests passed, and try again.

This leads to some curious effects. Firstly, you can make much more savage
changes between each test run. You can even change code that someone else
wrote! a long time ago!! that everything else uses!!!

Next, if your tests are cheap and sloppy, but the code can't exist without
them, you get the ultimate in Design for Testing. You get Design BY Testing.
That means your tests might fail even if your code had no bug.

Again: Your tests might fail even if your code had no bug.

That means your code occupies a high-dimension space that is easy for your
tests to cover. So instead of analyzing your code and wrapping your tests
around it, you used Adaptive Planning with both the code and tests to
simplify that high-dimension space.

--
  Phlip
  http://www.oreilly.com/catalog/9780596510657 /
We've slightly trimmed the long signature. Click to see the full one.
Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it

Is it really necessary to test all 2**24 values?  It would seem that
testing the minimum, maximum, zero, and some representative values in
between would suffice.  The "representative values" should be numerically
irrational (pi and e are good, for instance) so as to catch cases of
certain bits not being handled properly; 1 and 2 are not good choices
although they need to work properly as well.

In the area of branch testing, one has to test loops for proper
termination.  [I just found some bugs last evening that involved
some simple counting loops that didn't terminate due to doing a
check for <0 on an unsigned value -- oops.]

Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it

The example of "gullibility measurement and conversion" in
http://www.macroexpressions.com/dl/C%20code%20unit%20testing%20on%20a%20shoestring.pdf
  may be reasonably convincing

Quoted text here. Click to load it

IMHO, unsigned<0 condition doesn't rise to testing: Lint will find it
before you compile

Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

I think Ian refers to "developer tests". Giving them different definitions
helps. They have overlapping effects but distinct motivations.

The failure of a unit test implicates only one unit in the system, so the
search for a bug should be very easy. The failure of a developer test
implicates the last edit - not that it inserted a bug, but only that it
failed the test suite! Finding and reverting that edit is easier than
debugging.

Quoted text here. Click to load it

They are more cost effective than endless debugging!!!

--
  Phlip
  http://www.oreilly.com/catalog/9780596510657 /
We've slightly trimmed the long signature. Click to see the full one.
Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

Perhaps because this is an example of a medical product where that is
felt to be required.  There is a list of various kinds of "structural
coverages," those selected commensurate with the level of risk posed
by the software.  (Saying something has "coverage," I think, always
implies 100% coverage, too.  Not partial.  So you either have coverage
or you don't.)

Borrowing from one of the US CDRH PDFs I have laying about:

 · Statement Coverage – This criteria requires sufficient test cases
   for each program statement to be executed at least once; however,
   its achievement is insufficient to provide confidence in a
   software product's behavior.

 · Decision (Branch) Coverage – This criteria requires sufficient test
   cases for each program decision or branch to be executed so that
   each possible outcome occurs at least once. It is considered to be
   a minimum level of coverage for most software products, but
   decision coverage alone is insufficient for high-integrity
   applications.

 · Condition Coverage – This criteria requires sufficient test cases
   for each condition in a program decision to take on all possible
   outcomes at least once. It differs from branch coverage only when
   multiple conditions must be evaluated to reach a decision.

 · Multi-Condition Coverage – This criteria requires sufficient test
   cases to exercise all possible combinations of conditions in a
   program decision.

 · Loop Coverage – This criteria requires sufficient test cases for
   all program loops to be executed for zero, one, two, and many
   iterations covering initialization, typical running and termination
   (boundary) conditions.

 · Path Coverage – This criteria requires sufficient test cases for
   each feasible path, basis path, etc., from start to exit of a
   defined program segment, to be executed at least once. Because of
   the very large number of possible paths through a software program,
   path coverage is generally not achievable. The amount of path
   coverage is normally established based on the risk or criticality
   of the software under test.

 · Data Flow Coverage – This criteria requires sufficient test cases
   for each feasible data flow to be executed at least once. A number
   of data flow testing strategies are available.

For potentially high risk software, you may not just use a different
compiler or a different operating system environment or change even
the optimization options.  As the OP mentioned, it's probably going to
enough just justifying an instruction simulator.

I can easily see a desire for an automated way of demonstrating that
structural testing has achieved one or more of these cases.  If I read
the OP right about this, anyway.

Jon

Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it
The why was prompted by the posting subject "critique of a C unit test
environment".  To my way of thinking (TDD), unit tests are developer
tool, not formal product tests.


<interesting stuff snipped>


--
Ian Collins.

Re: Requesting critique of a C unit test environment

|--------------------------------------------------------------------------------------|
|"[..]                                                                          
      |
|                                                                              
      |
|Unit testing is an integral component of [..] "formal" [..]                    
      |
|models of development. [..]                                                    
      |
|                                                                              
      |
|[..]"                                                                          
      |
|--------------------------------------------------------------------------------------|

Testing is not an intgeral component of formal methods intended to
reduce testing.

Regards,
Colin Paul Gloster

Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

Why would a formal method intend to reduce a Good Thing??

--
 Phlip
 http://www.oreilly.com/catalog/9780596510657 /
We've slightly trimmed the long signature. Click to see the full one.
Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it

Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can prove
that there are no errors why test for them?

--
Erik Wikström

Re: Requesting critique of a C unit test environment
Erik Wikström said:

<snip>
 
Quoted text here. Click to load it

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

--
Richard Heathfield <http://www.cpax.org.uk
Email: -www. +rjh@
We've slightly trimmed the long signature. Click to see the full one.
Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

Proof of Correctness depends very much on how and when "correct" is
used. It is a crock of shit in most day to day SW development.

Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

But this was a "by hand" proof in 1977.  A machine assisted proof of
the actual code could be expected to inspire a little more confidence.

--
Ben.

Re: Requesting critique of a C unit test environment
Ben Bacarisse said:

Quoted text here. Click to load it

Why? Presumably the machine that is doing the assisting is itself a
computer program. What makes you think the assistance program is
correct?

--
Richard Heathfield <http://www.cpax.org.uk
Email: -www. +rjh@
We've slightly trimmed the long signature. Click to see the full one.
Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

What do you test your software with if not more software?

If you think that a machine assisted proof would not inspire "a little
more" confidence than a hand proof, then I won't try to persuade you
(it was a modest enough claim) but the fact that a proof system is
software does not invalidate the method any more than testing is
invalidated by being done in software.

--
Ben.

Re: Requesting critique of a C unit test environment
Ben Bacarisse said:

Quoted text here. Click to load it
<snip>
Quoted text here. Click to load it

A rolling pin. Any software that can withstand the pastry test is likely
to be able to withstand anything else too.

Quoted text here. Click to load it

Yes, on reflection I see that I'm guilty of (accidentally) extending
your claim, which was indeed modest enough.

<snip>

--
Richard Heathfield <http://www.cpax.org.uk
Email: -www. +rjh@
We've slightly trimmed the long signature. Click to see the full one.

Site Timeline