Requesting critique of a C unit test environment

Why not just use one of the free frameworks such as CppUnit?

It works well with both C (with a little fiddling like you do in your paper for "static") and C++. I'm sure the same applies for other frameworks.

Ian, Thank you for your response.

Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a code execution trace, so it's pretty darn hard to prove code coverage. [There must be reasons why testing tools vendors command big money.]

Also, if I use C in non-C++ compatible way (e.g. tentative definitions), my source won't even compile for CppUnit.

And finally there is a port issue (it's an embedded type talking :)). I am proposing something that requires only the compiler.

Regards, Ark

If you develop your software test first, you get all the code coverage you need.

If you mean K&R style prototypes, don't use them. Write and compile your tests in C++ and your code in C. Don't attempt to compile your C with a C++ compiler.

Shouldn't matter for unit testing, develop and test on a hosted system. If you require bits of the target environment, mock (simulate) them.

Test first is a nice model but not of a universal applicability. Besides, I need to demonstrate test coverage to the certifying/auditing entity. OTOH, I wonder if the proposed instrumentation can be made a part of CppUnit. I think, there is nothing in either that would prohibit it.

Right. It just didn't occur to me :(

The farthest I can go away from the target is a software simulator of the instruction set. Same compiler, same version, perhaps, more "memory". I think I am not alone in this...

-- Ark

It sounds like learning what their requirements are and meeting them is more important than guessing if TDD will incidentally meet their requirements.

You can keep the TDD thing a secret...

Why would a formal method intend to reduce a Good Thing??

You are not alone in that, I'd suggest you take this to a TDD list for advice.

Why?

|--------------------------------------------------------------------------------------| |"[..] | | | |Unit testing is an integral component of [..] "formal" [..] | |models of development. [..] | | | |[..]" | |--------------------------------------------------------------------------------------|

Testing is not an intgeral component of formal methods intended to reduce testing.

Regards, Colin Paul Gloster

Testing is used to find errors, while formal methods are used to prove that there are no errors, at least that's the goal. So if you can prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not tried it." - Donald E Knuth.

I use testing as a formal method, to prevent errors. I have not tried the "proof" systems (and please don't try to tell the mathematicians I used to hang out with that they are really "proofs").

You are describing writing and running tests in isolation from the development process. Don't do that.

Proof of Correctness depends very much on how and when "correct" is used. It is a crock of shit in most day to day SW development.

There are many possible *valid* reasons for this. One is that if you are not using the same version of the same compiler with the same switches then the code you are testing is not the same as the code that will be run. Since compilers *do* have bugs it is possible that the bug will be triggered in the real environment but not in the test environment unless you ensure that they are the same.

If I was doing QA for a product I would insist than you either use the same version of the same compiler or you provide testing to the same level that the deliverable SW requires of *both* the compiler used for test *and* the compiler used for the final SW. The more critical the SW, the more insistent I would be on this, and the more testing you would have to do on the compilers, for safety critical SW this would probably kill the project dead if you did not use identical SW to build for test and build for delivery. BTW, I *have* rejected SW and documentation at review, and even told developers that there was no point in putting it in for review because I would fail it.

But this was a "by hand" proof in 1977. A machine assisted proof of the actual code could be expected to inspire a little more confidence.

Why? Presumably the machine that is doing the assisting is itself a computer program. What makes you think the assistance program is correct?

But one has to differentiate between developer unit testing (the subject of this post) and QA (customer) acceptance testing. The former can be performed on any environment the developer chooses, the later must be run on the target.

Again, this is different from developer unit testing, I don't think anyone would be daft enough to release a product that hadn't been through acceptance testing on the target platform.

What do you test your software with if not more software?

If you think that a machine assisted proof would not inspire "a little more" confidence than a hand proof, then I won't try to persuade you (it was a modest enough claim) but the fact that a proof system is software does not invalidate the method any more than testing is invalidated by being done in software.

You have missed out internal formal testing which in many environments is far more complete than acceptance testing. For example, I've worked on projects where a formal test literally takes a week to complete but the customer acceptance testing takes only a few hours.

Unit test can also be formal, and in a lot of environments, including the afore mentioned safety critical projects, you are *required* to perform formal unit tests.

Informal testing can be run in any environment the developer has available. Formal testing, which is the only sort of testing that you can guarantee will be available and working for those maintaining later, is another matter.

All formal testing, whether unit testing or testing at a higher level has to be run on code compiled with the correct compiler, although not always on an identical target.

Acceptance testing has very little to do with proving whether the system works, it is just to give the customer some confidence. The real worth while formal testing has to be completed *before* doing customer acceptance testing and done with the correct compiler. At least, this is the case in many environments, including all the projects where I have been involved in QA, and on the safety critical project I was involved in.

If your customer acceptance testing is sufficient to prove the SW is sufficiently correct then your customer has either very little trust in your company or a lot of time to waste. If your customer acceptance testing is the only testing done with the correct compiler and it is not sufficient to prove your SW is sufficiently correct then your SW is not tested properly. At least, not according to any standard of testing I have come across.

A rolling pin. Any software that can withstand the pastry test is likely to be able to withstand anything else too.

Yes, on reflection I see that I'm guilty of (accidentally) extending your claim, which was indeed modest enough.