Unit Testing Embedded Code

Tightly coupled set of functions / data / classes with little and well defined interface loosly coupled with the outer world

I do unit testing in the host with compiler A. I do system testing in the target with compiler B. Different compilers should not be a problem, as long as both compilers adhere strictly to the standard, or their bugs are well known, and you don't allow any compiler warnings to remain unaddressed.

Do your code testable. One way of achieving this is to make your test code along, or even before, your target code.

I use Cantata for coverage. I do my own drivers, but import the test vectors from, and export the test results to, spreadsheets. To instrument code is to insert calls to coverage libraries in it in order to record execution metrics. Instrumented code is then compiled as normal code.

In my experience, if you spend T to write target code, then you'll spend 1,5*T to implement a basic test, and another T to complete the test cases.

No.

-- Ignacio G.T.

Hi

style,

code.

Your employer might consider hiring me!!! Hey, just kiddin'... However, I am looking for a job...

Your questions are very general in nature. Basically, it all comes down to rigid project control, quality control and -assurance and a very, very, very strict change control.

And now to the point: First and foremost, don't let a programmer test his/her own code for the obvious reasons, and secondly, conduct "structured walkthroughs" with specifications and code to be scrutinized. Take a peek at e.g.

for a fairly complete overview and from the man himself:

I perceive a "unit" as a functional entity as described in your functional specification.

using

Eventually, yes. That's what I'd call integration testing... When everything is put together on a target machine for the ultimate test. As to your second question: If the *same* code produces different results on different hardware then it's only reasonable to assume that there's something wrong with a compiler. You may have stumbled over a bug.

unit

You don't! NEVER EVER!!! Design and write your code with optimal performance in mind. Even if you're cross-developing. Simulate I/O, interrupts and whatnots. Do as much hardware abstraction as possible. I'm not oversimplifying, too aware of the fact that the ultimate test is your code running on the target machine. Note that a real time environment will complicate things one way or the other. A simulated environment may or will exhibit a much different behavior than the hotwired target machine. Be aware that testing is a means, not a goal! Incorporating stuff like:

#ifdef DEBUGENVIRONMENT #if (DEBUGENVIRONMENT==SIMULATEDHARDWARE) // place debug code here when running with simulated hardware #elif (DEBUGENVIRONMENT==LIVESYSTEM) // place debug code here when running on a live system #else #error DEBUGENVIRONMENT should be either SIMULATEDHARDWARE or LIVESYSTEM #endif // if(DEB...) #endif // ifdef

makes perfect sense. In fact, it should be encouraged, albeit that above code snippet implies a rather elaborate debugging strategy. But keep in mind that stuff like this should interfere as little as possible with the functioning of your system.

I'm not much of a fan when it comes to test tools, unless they unburden you of repetetive tasks. From what I gather "Instrumenting the code" means something like "the practice of monitoring and managing the behavior of the "instruments" of IT infrastructure

-at multiple levels-to allow for more efficient administration and orchestration." as you would play a musical instrument... What's in a name? ;-)

Unit tests are part of your specification (See below: "It all boils down to..."). At least the textual part should be ready before you start writing any code. A test bed for your black box should be as simple as possible. I prefer a command line driven program (if the application allows it, of course) as to easily redirect output to a file for inspection later on. If a command line program is not possible for whatever reasons... Yeah, well, then your systems designer needs to come up with a viable solution... Note that you get your test tools for free when writing your test bed code. A point of interest is to invest some time beforehand to find out if and how your test bed code can be incorporated in some sort of regression test system.

Nope, I was able to avoid issues, conflicts, etc. by conducting my structured walkthroughs. Stuff like that may be very counterproductive and certainly doesn't add to one's motivation.

Common sense, tenacity, and some organizational skills.

You're welcome.

It all boils down to this: your functional specification describes what the various functions are supposed to do. Hence you know the outcome of your unit tests. Tests should be organized as follows:

-Test numer and name

-Short description

-Prerequisites, like e.g. tables, files, and parameters and ranges

-Expected results (from the functional specification)

-Side effects

-Observed results It will be obvious that there's a problem (read challenge) if the latter two do not match, i.e. your test failed.

Have some (testing) fun

Waldemar

two

What I meant was: your test failed if the observed result differ from the expected result. "Side effects" should've moved up a line.

Waldemar

According to my understanding, that would be dictated by your software's internal structure. With half a million lines of code, your software is bound to have some well-defined internal interfaces. These interfaces would be candidates for unit boundaries.

Yes and yes. Host-based testing is used as long as it's feasible, but from some point on, target testing is unavoidable (e.g. need to access specific hardware, etc.). It is good to have as many well-tested (on the host) units at that point.

Compiler considerations are the same as when designing code for portability:

- stick to the standardized (e.g. ANSI-) subset of the language you use.

- be aware of any issues with different endianess/structure padding. etc.

Modular structure with well-defined interfaces between the modules,

Yes...

.. no.

Defining tests solely with test coverage in mind is usually not a good idea.

Tests cases are to be derived from the software's specification. *Iff* the specification was complete, and if the testcases cover each and every item from the specification, then the test is also complete. Coverage can help to sort of verify these assumptions, i.e. if a coverage test unveils a section of code that is never touched by any of the tests, then either:

1. the specification is not complete (i.e. the code does something that is necessary, but is not described by the spec), or 2. it _is_ in in the spec, but the tester forgot to write a test case for it, or 3. the code is in fact superfluous and consequently it can (has to) be removed.

Usually this means to run your program through a special "coverage compiler" that will transparently add code at certain points (i.e. every opening brace, every decision statement, etc.) to trace the execution paths that are taken during the test run. The instrumented program (ideally) behaves exactly like its uninstrumented equivalent, except that, when run, it leaves behind some form of trace data that is later analyzed by the tool.

Usually you get information about what lines of code where executed how many times, which branches where taken or not taken how many times, etc.

Note that a coverage compiler is a compiler in its own right, and it often comes with its own parser. That is another reason to write code as portable as possible.

Rob

--
Robert Kaiser                     email: rkaiser AT sysgo DOT com
SYSGO AG                          http://www.elinos.com
Klein-Winternheim / Germany       http://www.sysgo.com

Huh? The programmer should be the FIRST to test his code. Having a separate QA / system test is good, but the programmer shouldn't deliver anything for integration or release until he has confidence that it works.

Good suggestion!

Usually it is a case of misusing the language, such as assuming that type char is signed, ints are little endian, multiple updates between C sequence points, etc.

I disagree. The goal is to deliver code that meets the specification in a reasonable amount of time. Making the code easy to verify would certainly help achieve that goal. If a particular section is time critical, implied by "optimal performance", then it may indeed be harder to test. Usually that is a very small portion of the overall code. The non-critical code benefits from being easy to test.

Good suggestion.

Thad

I would hope so too. From the functional specification you should have been able to conceive a structure that matches the ap[plication and is graded into "Keystone Functions" and "Supporting Functions (apparently a 50/50 split usually - see AMUSE - "Architectural Modelling to Understand System Evolution"

The structure of the application and the structure of the software should have a great deal of similarity and the units would be quite self evident at a number of levels from subroutines, functions and macros upto complete sub-systems. You should be doing unit testing at each and every level from the smallest up to just under the level where complete system testing begins.

Others who have responded earlier have given quite good advice in general but I will add that:-

As units fail tests (innevitable) you will need to implement corrective changes. Please ensure that you continue to put those changes through as rigourous a change management process as you do the rest of the development (yo do run change management don't you?). I know companies that have lost the plot when they started testing because they find an error, fix it there and then, and forget to properly record the problem and the changes made to correct it.

--
********************************************************************
Paul E. Bennett ....................
Forth based HIDECS Consultancy .....
Mob: +44 (0)7811-639972 .........NOW AVAILABLE:- HIDECS COURSE......
Tel: +44 (0)1235-811095 .... see http://www.feabhas.com for details.
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************

True, but someone else should do the formal testing and that's what I meant with this remark. Of course a programmer tests, but a programmer becomes blind (however much experience) for the obvious. That's a proven fact.

Don't misuse the language. Make sure that any code produced conforms to the coding standards. You'll probably weed out swearing in code during the structured walkthroughs. One should not assume, ever... Especically not in this type of work ;-)

make

performance

Perhaps I should have elaborated a little on the term "optimal performance"... What I meant is that the code performs as defined in the functional specification, i.e. no more, no less. Furthermore, it'll be evident that the result of your design is efficient code. In your project plan you determine the amount of time you allot to this particular piece of code, which should be realistic, taking the (perceived) complexity into account.

Love the discussion

Cheers

Waldemar

Many years ago, programmers at IBM worked according to the "cleanroom" method, which meant that a programmer only had an editor. The actual compiling was done by other people, who reported errors and bugs to the programmers. If programmers were caught with a compiler, they suffered severe penalties.

Meindert

Thank you for all the great responses! They do help.

One tricky thing will be figuring out where to start. Since the code wasn't written with unit tests in mind, there are more interdependencies than would be convenient. Another hat trick will be trying to figure out how to fit it all in to our tight, customer-driven schedules, which are concerned with little deliverables along the whole way and not just the end.

Waldemar, my employer is actually hiring. Drop me a line! Replace the whole email adress prefix with my first name, and remove the remaining capital letter section too.

Jeanne

My companies primary business is software verification and validation for safety-critical applications. The general rule of thumb is that units are C functions. One function = one unit. The same generally holds true for assembly language. One subroutine (function) = one unit.

We use one of the many commercial tools to perform unit testing, Cantata from IPL. We develop and run the unit tests on a PC. The Cantata tool includes a set of libraries that are linked with the code under test. When the tests are all stable on the PC environment we rebuild the libraries using the target compiler and then link it with the code under test to run on the target. Results data collection is done through a serial port, or an emulator console, etc. depending on what you have available.

The simple answer is use a coding standard, have a design document (requirements), and subject everything to peer and/or formal review.

The Cantata tool I mentioned above does coverage and instrumentation based on a set of test scripts that you write. See

for details on the tools.

That is a real tough question with an answer that prevents many from doing an adequate job of documentation and testing. A full fledged validation effort, assuming that essentially no requirements and design documentation exists probably takes 3 times as much effort as the actual design. Simple unit testing can take one half to over two person days per unit depending on the code complexity, code quality, and function length. We typically consider 50 lines of actual C code to be a function. This needs to be adjusted based on code quality. If the code looks like ten different people worked on it and everyone used their own style, then it will more difficult to test.

Gotchas happen all the time. We see "finished" code headed for highly critical operations where lives depend on it that absolutely scares us. I have found numerous hard bugs in code simply be reviewing it to develop and estimate on the testing effort.

We or others can sub-contract the entire project for you to get you to a point where you can take over without being overwhelmed, or we can come train your engineers how to do the job themselves and leave you with a complete process to follow for the future.

Scott Validated Software Corp.

I trust that there is a specification for this humungous peice spaghetti (which, until you have proven it isn't, I shall consider it to be based on your own comments). If you have the specification, then I would consider identifying the "Keystone Functions", usually closely associated with interfaces, and begin mapping ut the structure of the application. Then identify which software functions fulfil these Keystone Functions. Tidy up the structural view and use it as a map to the rest of the software. Fill in details of the rest of the functions as you find them.

From having this map you can then formulate a strategy for unit testing first the Keystone functions then the other supporting functions. As I stated befoire, ensure that you employ a sound, rigorous change management process throughout the whole process and continue to review every change that is made.

I have enough to do with my own day job and private projects thanks.

--
********************************************************************
Paul E. Bennett ....................
Forth based HIDECS Consultancy .....
Mob: +44 (0)7811-639972 .........NOW AVAILABLE:- HIDECS COURSE......
Tel: +44 (0)1235-811095 .... see http://www.feabhas.com for details.
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************