I'm looking for comments on automated unit testing of embedded C software where the automated testing is run on a PC, not in the target.
Any guesses as to what percentage of embedded C apps are tested with automated testing?
Any guesses as to what percentage of embedded C apps are tested with a well-defined,documented test spec and procedure as opposed to undocumented ad-hoc testing?
For a new application, does the use of automated testing typically mean a longer development time up to release one of the product and thereafter save time, or is development time up to release one typically reduced or not much different?
Are the tests performed by the automated testing normally not repeated by manual testing in the target?
If substantial automated testing is done, is there normally also a lot of manual unit testing required in the target, to test things that couldn't be tested in the host environment?
How do you verify that the automated testing is doing its job correctly
- is there any automated means of doing this?
Is it common to spend a lot of time writing code to emulate the target in the host?
I used to test functions written in C (Whitesmiths C compiler) for a
68HC11 target on a PC using Borland C. All my functions were tested that way.
For the sake of discussion, I will define your "automated testing" to be what I call unit testing: testing individual functions isolated from other code using some sort of test program, possibly written on another platform. The testing program calls the function under test and checks the results, automatically.
For me it means a longer code development time to first release. Typically, it might take twice as long compared to not unit testing at all. Unit testing also means fewer bugs, especially the hard-to-find ones. For me unit testing is not an option. The time it saves in later debugging easily pays for it.
Correct. Some of the testing cannot be performed by final testing. For example, you could have a function that formats a number for a display. With unit testing, you could run the function through literally all possibilities and have the test program check the results. That cannot be done in the final product because it might take forever to get all values for display. Then you have the human factor where a tester might miss a display format problem.
But if you are tempted to skip the unit testing, you could see a bug show up at a customers location, maybe months later, when you have thousands of units in the field. Not fun.
Typically, if all the units (maybe C functions) are unit tested, then there needs to be an integration test where all the units are tested together. If your system is relatively small, you can combine integration testing and functional testing into one test.
At some point, humans must take charge! :-) When unit tests are written, the program performing the unit test knows the deisred result of a test. The test either passes or fails. The test program records the failures and reports them.
Yes, there are ways of having the test results automatically scanned to verify all tests passed correctly. I used to write test scripts that would run test programs. The output of the programs would be saved in a file. The final step of the test script was to compare the unit test output (file) to an expected output (file). I used diff in the Korn shell to do that. I also used that testing process with make such that if any source code file changed, its unit test would be re-run and verified automatically.
I never had to do this. It might all depend on what programming language you use. If your firmware is written in assembly language, unit testing will be much more difficult. In that case, a target emulator or instruction simulator might be necessary to run the tests. That is why I like writing code in a higher leve language like C - it's more portable and more easily tested.
I have done this. It depends on what your target MCU is and if there is an OS.
A lot of them are now tested with automated testing. Either on a sim or on target with an ICE. I have done both.
A lot more than there used to be. There are several automated testing tools for doing just this. Several outfits I know are getting up to 95% test coverage doing unit testing of an application
On the first pass it appears that it is not much longer (rule of thumb here as I have no accurate figures for this) as the testing is more complete and the bugs are picked up more easily. This cuts down the amount of time spent debugging or bug hunting. This is with Unit testing so the bugs are in a relatively small area of code.
Of course regression testing is very much faster. Usually it will run on its own.
Yes and no. I have done testing of embedded apps in a SIM on a PC and them we did additional tests on the target. However depending on target and the ICE you have you can do all the unit and system testing on the target in hard real time. Depending on the target you can do some testing with the ICE and no target.
As the automated testing CAN be done in both the sim and on the target usually some testing is done in the Sim until the HW is stable and them the testing is done on the target.
How do you mean? test the tests?
NONE AT ALL. Use a commercial simulator or an ICE. No one in their right mind writes their own sim it is not cost effective. It takes far too long and it needs testing in it'self.
The other thing you can't do in most cases is use a different compiler and run it on the PC as a native app. That way lies madness :-)
See my (rather old) paper on testing on
formatting link
It is due for a re-write so any comments welcome.
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ \/\/\/\/\ Chris Hills Staffs England /\/\/\/\/\ /\/\/ snipped-for-privacy@phaedsys.org
I'm sorry I can't resist. My guess is that less than 2 percent of C apps are tested with automated testing. Probably only about 5 percent gets any formal testing at all. It's a sad thing.
yep, that's what I meant. I was just wondering if any effort has to be made to check that the automated testing is doing what you think it's doing i.e. that it is calling the function you think it's calling, with the data you intended and that if the function actually performs incorrectly, the automated testing successfully reports the error.
I have an idea this is what's being planned for us but I've yet to find out the details.
The extremely short timescales make it difficult to get a proper overview for strategic planning (which would save any organisation money and effort). If more companies sat back and applied some proper thought to their structure, strategies and processes they could improve the lot for their employees as well as improve the bottom line.
One staff position that can make a real difference here is the document registrar (who leads the document registry efforts). The person who fills this post looks after all the documentation, source code listings, version control, configuration management, records of letters, faxes and contracts, in fact any piece of information that is of value to the company's well being. I am amazed that so many companies do not think it worthwhile to look after such valuable assets properly. The most successful companies take extremely good care of it.
Chris Hills, in one of his ESE columns raised the subject of certification of software engineers and technicians. I guess it is going to come in eventually and it should be led by the BCS/IEE/C&G (in the UK) before the insurance companies get their mits on it and force the too rapid implementation of something that is less than suitable.
I know that some tools, nice though they might be, are way too expensive and require hardware platforms to run on that more than double the tool budget. However, do you really need that particular tool. It pays to look around quite a bit and see what else is out there.
--
********************************************************************
Paul E. Bennett ....................
Forth based HIDECS Consultancy .....
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-811095
Going Forth Safely ....EBA. http://www.electric-boat-association.org.uk/********************************************************************
Cygnus wrote a software emulator for then-unavailable ASICs in the PS2 so that the software folks could get busy instead of waiting on the hardware.
If you check out the MAME project
formatting link
you'll see that there's a rather impressive set of software emulators for real-world (albeit somewhat dated) video game and small computer hardware. Not bad for free :)
Atmel's AVR Studio does a nice job of simulating not just their CPU but also some simple hardware that one might typically use during early development. Not bad for free :)
I think these two trends (faster and cheaper desktop PCs, bigger and more complicated systems to test) will easily converge in the next few years into full up simulation environments.
Actually it does have an effect in some cases. If the target5 is an 8 bit system and the PC is a 23 bit system then the size of ints may vary and therefore the way the get promoted and converted.
Your C has to be VERY neutral for the platform to have no effect. Many do not realise this and I have seen code that was "good" when it was run on a PC hide many subtle bugs when it was run on a target system.
As it interacts with the memory any program falls into that category.
Not so easy when data is stored in discrete part of the memory map. However it does depend on your target. Some are easier to isolate than others.
Fair enough,
Of course.
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ \/\/\/\/\ Chris Hills Staffs England /\/\/\/\/\ /\/\/ snipped-for-privacy@phaedsys.org
TO clear up the terms:- SIMULATOR is a software tool that simulates the target on another host. It is not real time but usually cycle accurate within itself.
EMULATOR (or ICE) is a hardware tool that replicated the chip in hardware often with a bond out part. These are usually hard real time accurate. Often the can (with a bond out ) run stand alone with out target hardware..
Simulators can be produced before the silicon for the target is ready. However this is dangerous as the silicon can change. As I know to my cost when working with a silicon company and a compiler/simulator company.
Emulators and ICE tend to arrive after the silicon has stabilised.
Looks good.
The point about the "commercial" is that there is (usually) some real time support for these and they have to be maintained to a reasonable quality or the company sinks.
This was commercially developed by Atmel to sell silicon. If the cost wasn't offset by silicon sales it would not be "free."
This happened years ago. Where have you been? There are good quality simulators for most MCU targets. Not cheap for some but still very good.
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ \/\/\/\/\ Chris Hills Staffs England /\/\/\/\/\ /\/\/ snipped-for-privacy@phaedsys.org
In my firm we only do it in safety-critical systems.
Idem.
Release one: double the coding time, halve the system testing time, more than halve the re-coding time due to errores found in testing time.
Release two: we are still on it :-)
We perform unit testing only out of target, and system testing in target. But we use the same compiler out of target that in target.
In special cases, we run semi-automated testing: perform automated testing under the control of a debugger, setting breakpoints when needed and introducing modifications in the data by hand when the breakpoints are reached, then continuing.
Code that is "impossible" to test in target is tested as part of the system integration, with manual methods. We try to keep this code as small as possible. It is typically low-level hardware access code.
Yes. We use scripts that translate test data and test results from Excel sheets to and from the automated testing programs. A quick examination of the Excel results tell as if and where are the errors.
I used to work for a company that produces systems for automated testing of embedded software. See
formatting link
Most of the early customers were medical device manufacturers. These guys do a lot of formal testing, and are interested in automating as much as possible.
A lot of their business now is cell phone, palmtop, and other portable wireless device manufactuers. That shift came after I left, so I'm not sure if those guys always did a lot of formal testing, or are just lately getting religion...
My first job out of college was developing software for commercial avionics. A lot of very formal testing there.
It has been over 30 years since college, but from what I see, much of the problem starts in the University. There is far too little emphasis on software test, including self-testing by the developer, in the classroom. One of my associates is teaching the ONLY class on software testing at the University of Colorado. This is a continuing education class run at night and currently has six students. In a school that will likely graduate a couple of thousand engineers next month this is truly sad.
I agree with Paul's post that tools are often much too expensive, but that is as much an indication of the volume and lack of need as anything else.
Something that came my way in an email to work today from HITEX and mentioned the following link in connection to this subject. They are aiming at the IEC61508/DO178-B/MISRA-C markets for those coding in C.
formatting link
--
********************************************************************
Paul E. Bennett ....................
Forth based HIDECS Consultancy .....
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-811095
Going Forth Safely ....EBA. http://www.electric-boat-association.org.uk/********************************************************************
Thanks for that very interesting link. They are targeting high end applications with safety constraints. That might be useful also for more modest applications, but I wonder what the price is and what the real experiences are. you are sometimes disappointed even when the marketing guys did their job well.
I haven't looked at the aspect for this. Seeing that you seem to be in France, and not so far away, I guess that you could always look to getting into the up and coming free seminar (12th May).
--
********************************************************************
Paul E. Bennett ....................
Forth based HIDECS Consultancy .....
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-811095
Going Forth Safely ....EBA. http://www.electric-boat-association.org.uk/********************************************************************
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.