Requesting critique of a C unit test environment - Page 3

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

Sed quis custodiet ipsos custodes?

Richard

Re: Requesting critique of a C unit test environment

|------------------------------------------------------------------------|
|"Ben Bacarisse said:                                                    |
|                                                                        |
|>                                                                       |
|>> Erik Wikström said:                                                  |
|>>                                                                      |
|>> <snip>                                                               |
|>>                                                                      |
|>>> Testing is used to find errors, while formal methods are used to    |
|>>> prove that there are no errors, at least that's the goal. So if you |
|>>> can prove that there are no errors why test for them?               |
|>>                                                                      |
|>> "Beware of bugs in the above code; I have only proved it correct, not|
|>> tried it." - Donald E Knuth.                                         |
|>                                                                       |
|> But this was a "by hand" proof in 1977.  A machine assisted proof of  |
|> the actual code could be expected to inspire a little more confidence.|
|                                                                        |
|Why? Presumably the machine that is doing the assisting is itself a     |
|computer program. What makes you think the assistance program is        |
|correct?"                                                               |
|------------------------------------------------------------------------|

Full points to Mister Heathfield.

Re: Requesting critique of a C unit test environment
I will second that, well put.

The Achilles heel for either Testing or formal methods is
contaminating the evaluation process with information
from the implementation.

I have seen unit tests contaminated from just knowing
the application area the code was going to be used.

w..


Colin Paul Gloster wrote:

Quoted text here. Click to load it


Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

YMMV of course, but if I could get Donald Knuth to prove my
programs correct "by hand", I'd feel no need for additional
confidence.
--
Ben Pfaff
http://benpfaff.org

Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

No, MMIS[1].  I did not intend to disparage Prof. Knuth's "hand
proofs" (what a thought!) but rather to say that the problem he is
referring to is as likely to be that one proves something other than
the program one has written (or later writes) as it is to be that ones
proof is (internally) flawed.

I suspect that he is not entirely happy with the way that quip is used
so often to suggest the pointlessness of proofs[2] (after all, what
did he choose to do with his "Notes on van Emde Boas constriction of
priority deques" -- a proof rather than a test implementation!).

[1] "My mileage is similar".
[2] This not one of those times -- RH was just countering the much
stronger assertion that proof => no need to test.

--
Ben.

Re: Requesting critique of a C unit test environment
[snip]
Quoted text here. Click to load it

Aside:
Working vEB tree implementation here:
http://www.itu.dk/people/kokholm/veb /


Re: Requesting critique of a C unit test environment
Ben Bacarisse said:

<snip>
Quoted text here. Click to load it

<snip>

Quoted text here. Click to load it

Right. One problem is that they don't always prove what you asked them
to prove. What you actually want to know is "does this program properly
do what I need it to do?", but what a prover actually tells you is
whether program X conforms to a particular expression of specification
Y. It makes no comment whatsoever on whether specification Y
corresponds to wishlist Z. And, very often, such correspondence is far
from perfect.

--
Richard Heathfield <http://www.cpax.org.uk
Email: -www. +rjh@
We've slightly trimmed the long signature. Click to see the full one.
Re: Requesting critique of a C unit test environment

|--------------------------------------------------------------------------|
|"[..]                                                                     |
|                                                                          |
|YMMV of course, but if I could get Donald Knuth to prove my               |
|programs correct "by hand", I'd feel no need for additional               |
|confidence."                                                              |
|--------------------------------------------------------------------------|

Such as the way Donald E. Knuth told Leslie Lamport that TeX would
hardly change at all? From
HTTP://research.Microsoft.com/users/Lamport/pubs/lamport-latex-interview.pdf
:"[..]
[..] When Don
was writing TEX80, he announced that it would be a
reimplementation of TEX78, but he was not going to
add new features. I took him seriously and asked for
almost no changes to TEX itself. [..] However, there were many other
im-
provements that I could have suggested but didn't. In
the end, Don wound up making very big changes to
TEX78. But they were all incremental, and there was
never a point where he admitted that he was willing
to make major changes. Had I known at the begin-
ning how many changes he would be making, I would
have tried to participate in the redesign. [..]
[..]"

Regards,
Colin Paul Gloster

Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it
HTTP://research.Microsoft.com/users/Lamport/pubs/lamport-latex-interview.pdf
Quoted text here. Click to load it

"Principle 4

" * Level out the workload (heijunka). (Work like the tortoise, not the
hare).

"This helps achieve the goal of minimizing waste (muda), not overburdening
people or the equipment (muri), and not creating uneven production levels
(mura)."

http://en.wikipedia.org/wiki/The_Toyota_Way

--
 Phlip

Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

I use testing as a formal method, to prevent errors. I have not tried
the "proof" systems (and please don't try to tell the mathematicians I used
to hang out with that they are really "proofs").

You are describing writing and running tests in isolation from the
development process. Don't do that.

--
 Phlip
 http://www.oreilly.com/catalog/9780596510657 /
We've slightly trimmed the long signature. Click to see the full one.
Re: Requesting critique of a C unit test environment
Thank you folks for the fruitful discussion :)

Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it

What was your conclusion?

--
Ian Collins.

Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it
Since you asked... I expected a more substantial feedback (perhaps,
naively).

I've been heavily involved in a SIL 3 project with ARM and IAR EWARM 4.x
Our consultant-on-safety-stuff company spelled out requirements for unit
testing /documentation/ so that it could be accepted by TUumlautV - a
certifying agency. So we started looking for test automation tools that
would help /demonstrate/ test results including code/branch coverage.
IPL Cantata++ didn't (at least at the time) support IAR. Period.
LDRA Testbed was (at least at the time) outright buggy; we had to
abandon it.
We ended up doing unit tests ad hoc with manual documentation, asking
for code coverage proof from the debugger. In the process, I found a way
of instrumenting the code by abusing the C preprocessor. It was not what
I asked to critique but the ideas were similar. I used it for regression
testing to make sure the execution trace was the same - or it changed
for a purpose and I had to set a new regression base.
It occurred to me later that the same instrumentation ideas could be
used to demonstrate code coverage, and that the framework could be
commonized with a very small footprint. That's what I asked to critique.
Personally, I love this thingie. It can get me through SIL business for
free (as opposed 10K per license or so), and additional coding policy
items should not be an issue for those already restricted by (a subset
of) MISRA.
Note that this stuff is applicable whether you do TDD or a formal
testing campaign.

--
Ark




Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it

Well TDD done correctly will give you the required coverage, might be
interesting proving it to the certifying agency though (a branch won't
be there unless a test requires it).

--
Ian Collins.

Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it
The mantra is, if a branch is to never be executed, it shall not be in
the production code. If it is executable, I must demonstrate how I
exercised it.
BTW, I tend to doubt TDD can be followed in semi-research environment
where you churn out piles of code to see what works vs. how the plant
behaves etc. Of course there is some lightweight informal T but none
documented. Once you found a decent solution, you got a fair amount of
code that needs, aside from bug finding/fixing, only error conditions
handling for productizing. At this point, T is already way behind D.
Or I must be missing something here...

--
Ark

Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it
Well there you would be wrong.  I even use it for quick one offs,
because it helps me go faster.  The time saved not having to debug more
than justifies the process.

--
Ian Collins.

Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it


Sounds like TDD.

Quoted text here. Click to load it

Ian, you are responding to the straw-person argument, "Projects that use TDD
only ever write any tests before the tested code."

They don't. Now let's hear why these "research" environments are _allowed_
to write code without a failing test, first!

--
  Phlip
  http://www.oreilly.com/catalog/9780596510657 /
We've slightly trimmed the long signature. Click to see the full one.
Re: Requesting critique of a C unit test environment
Quoted text here. Click to load it
[If we agree that a test is a contraption to check if the code works as
expected:]
If we don't know what to expect ("research"), we cannot write a test.
[Or again I am missing something]

E.g. if I'm writing a version control system, I know exactly what _has_
to happen, and I can write the tests.
If e.g. I'm writing a monitoring code for the car's wheel speed sensors,
I may have a rock solid idea that e.g. whatever the speeds, the wheels
always remain in the vertices of a rectangle of the original size. Enter
sensor noise, wheel spinning, tire inflation and what not. I need lots
of code just to study what's going on before I can arrive at a sensible
algorithm.
--
Ark

Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it

The weakest possible such contraption - yes.

Quoted text here. Click to load it

If you can think of the next line of code to write, you must perforce be
able to think of a test case that will fail because the line is not there.

Next, if you are talking about research to generate algorithms for some
situation, then you aren't talking about production code. Disposable code
doesn't need TDD. Once you have a good algorithm, it will have details that
lead to simple test cases.

Quoted text here. Click to load it

That's an acceptance test. TDD tests don't give a crap if your code is
acceptable - if it targets wheels or wings. It's just a system to match
lines of code to trivial, nearly useless test cases.

--
  Phlip
  http://www.oreilly.com/catalog/9780596510657 /
We've slightly trimmed the long signature. Click to see the full one.
Re: Requesting critique of a C unit test environment

Quoted text here. Click to load it
That's the whole point. I end up with some working prototype code for
which I need to create tests post factum.

Site Timeline