Test framework for automated FW testing

Hi!

I'm planning to automatically test a reasonably simple board in an embedded system. The tests cover functional requirements of the firmware. Unit testing is not needed (it's already done).

The module under test talks to a 3 other boards via SPI and serial. It has a few sensors and relays connected. All external boards and sensors will be simulated by a software test bench running on a PC.

I'll create small programs that simulate external components using some language that can access DLL-s or Active X components (SPI and I/O adapters provide those). The details largely depend on the test framework which must control those simulations.

This brings me to the actual problem - which testing framework to use? There are shiny expensive commercial solutions (such as TestComplete and NI TestStand). There are FOSS programs (dejagnu). There is a pile of unit testing frameworks, 2-3 per every language ever conceived.

Most of these emphasize PC testing - pushing buttons, looking for text responses etc. I assume testing an embedded system is a bit different - lots of stuff going on at any given time, it is less determined, there are real-time considerations etc. Then again, maybe I assume wrong.

I suspect experience can sort out this mess. What frameworks have you used for testing firmware and how did it work out?

-- Kind regards, Tarmo Kuuse

Reply to
Tarmo Kuuse
Loading thread data ...

[...]

[...]

This is a well known problem. No one knows your system better then you are. Hence, unless your board belongs to a category that they already have in their framework, you will do better with the test framework of your own, created in house from the scratch. Otherwise you will have to fight with the bugs and constraints of somebody else's instead of working on your target system.

Since there can be a wild variety of the different conditions and situations, the test system should target only the relevant case or the specific set of cases. Making the universal test system === bloat, bugs and limitations. The right way to do it is developing the test suite as the common program in the general purpose programming language such as C++.

Vladimir Vassilevsky DSP and Mixed Signal Design Consultant

formatting link

Reply to
Vladimir Vassilevsky

[snip]

You describe something slightly different. I probably used the term "test framework" too loosely. I'll just call it a "test tool". I certainly don't expect it to know my system. I implement all required knowledge of the system myself - it is embedded in test cases and HW simulation.

What I am looking for is a tool that allows me to define individual test cases in a high-level language. The language can access our I/O devices (DLL-s or Active X components). It is real-time aware (can use delays, timeouts). Test cases can be grouped, run, paused, stopped. Tool outputs a formatted test report (usable in a review with managers).

I did not get the point. It's the test engineers job to design tests so relevant stuff is covered. Test system just executes them.

A team in my company has been using NI TestStand and LabView for years. It is cumbersome and an overkill for our project. We have looked at TestComplete. That fills the requirements, but it's still too complicated. So I'm looking for viable alternatives.

We have considered creating a custom test suite in Python or C++. It seems to be a good idea, but our development resources are quite limited. I would very much like to minimize the amount of work and not write it from scratch. Some elements - managing tests, running them and reporting results - are quite universal. Is there any effort to make a reusable library or framework for this purpose?

-- Kind regards, Tarmo Kuuse

Reply to
Tarmo Kuuse

re.

C++.

formatting link
formatting link

I have never used it but it is free. Keep us posted if you try it out.

Reply to
Marco

also:

Functional Testing Tools, the Open-Source Wave

formatting link

formatting link

Reply to
Marco

Thank you for the link. At first glance it seems just what is needed. A closer look will have to wait for the time being. I'll peek at the other links you provided also.

Winter holidays are a quiet, peaceful time - good for dreaming and researching. As soon as they are over, "normal" life returns.

--
Kind regards,
Tarmo Kuuse
Reply to
Tarmo Kuuse

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.