Test driven development?

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
I'm curious: what do people think of test driven development
when applied to embedded (read small microprocessors and
micro controller) systems?

Mike Harding


Re: Test driven development?


Quoted text here. Click to load it

You can read about how I used testing in a microcontroller-based
project here: http://www.guymacon.com/MATTEL/INDEX.HTM

--
Guy Macon <http://www.guymacon.com


Re: Test driven development?
Quoted text here. Click to load it

As the owner of a company that does validation and certification testing for
software used in safety critical applications, I'd say all software
development, embedded and otherwise should be test driven.

It never ceases to amaze me how low the quality of the "finished" software I
see.  And this is software headed for products where failure could directly
result in loss of life.

Far too many developers limit their testing to ideal cases at the end of
development and give no consideration at all to trying anything else.

I strongly encourage all developers to give testing the level of attention
it deserves.  Even in non-critical applications failures are costly, both to
your products and your reputation.

--
Scott
Validated Software Corp.



Re: Test driven development?

Quoted text here. Click to load it
for
I
directly

I think this is an issue of quality management and control, not coding
methodology. Requirements need to be made clear up front. I do agree that,
possibly, whoever is coding these devices hasn't really thought things
through; but is that their job? Given the nature of these systems, it sounds
like a quality systems audit would be in order.

I have yet to meet a coder who can test his own code as well, and as
objectively, as a good test technician/engineer. For the most part, asking
coders to qc/test their own code is a ticket to hell.

Quoted text here. Click to load it

Far too many specifications provide too little in the way of detail and
requirements, and it is not uncommon for people to expect too much for too
little -- especially those with no skill or experience in working with
developers. You must be familiar with the tendency of projects -- especially
poorly defined projects -- to balloon. This is caused by the vicious cycle
of, "What about this, and what about that...." Who gets screwed in the end
is the developer with the "you shoulda" statements. That's bullshit.

(IMHO)

Mike





Re: Test driven development?

[%X]

Quoted text here. Click to load it

I am sure that many of us have had, at some time, projects that were poorly
defined or became nightmares (only two in my case but I have seen them).
The real question becomes "what do we do to prevent projects falling into
such traps?"

Certainly getting the spec tied down properly first will stand you in good
stead. Ways around doing this are to:-

 * Take the spec apart and formulate questions about each and every
   apparent statement of requirements. Formally record all questions.

 * Obtain answers to each and every question formulated above and record
   these answers against each question complete with the source of the
   answer (not all the answers will be from the client).

 * Write a fully detailed technical specification based on the original
   requirements and the answers to questions. Present the document to the
   client, using any additional presentation aids that may be appropriate
   to aid understanding by the client.

 * Do not proceed any further unbtil the client has signed off on this
   technical specification meeting his requiremnents. Once he has signed
   acceptance any changes that are instigated by them can easily become
   chargeable.

 * Ensure you have a completely rigourous change management procedure in
   place to cope with any chanbges that are required of the specification,
   contract or design. This is a most impoirtant aspect of project
   management.

Of course, during development you will need to keep your client in the loop
and appraised of the impact on project timescale and cost every time they
go "cvould we have?". They can should also be permitted to observe that
your management of the project follows your published procedures in every
respect. That way the get a good feeling that you care about the quality of
the end result and may even assist in overcoming any difficult problems.
After all, system development should be a partnership.

--
********************************************************************
We've slightly trimmed the long signature. Click to see the full one.
Re: Test driven development?
Quoted text here. Click to load it

I mostly agree with your comments.  Especially that a good test
technician/engineer (or team) is a great addition to a project.  When a
project is designed with a critical application in mind, we generally see
good stuff.  This primarily includes requirements and design documents that
are well thought out and APPROVED before the real coding starts.

Where we see the worst nightmares is in code that was designed for a
non-critical app or an app not initially recognized as critical that must be
shoe-horned into a previously non-existent process.  It often seems to
restate the erroneous idea that "anybody can write code".   It's a fading
idea, but it surely isn't gone yet.

As you eluded to in your last paragraph, the "moving target" is alive and
well.

I disagree with your statement about coders qc/testing their own code,
unless they are in a company that has sufficient SQA that the tests are
designed and coded prior to the actual code.  Developers in a limited SQA
environment don't really have a choice.  It's surely not an ideal situation
but it is often unavoidable.

Scott



Re: Test driven development?
Comments below.

Doug

Quoted text here. Click to load it
software

That would be dead wrong. Coding methodology  is as much a part of
quality control as anything else. Incidently, "coders" is a job position
that has been extinct for a several decades.

Quoted text here. Click to load it

This never happens. Requirements change, thus the failure of the
Waterfall approach to project management. Project management
must be able to adapt to changing requirements.

Quoted text here. Click to load it

Absolutely it is their job!

Quoted text here. Click to load it

That too. But a audit is not a solution to the problem. Finding quality
problems after the fact is expensive. Quality must be built into the
methodology.

Quoted text here. Click to load it

That is because the test engineer is focused on comparing performance to
requirements and also as a second pass to make sure things are right.
This not an indictment on the work that "coders" do. Incidently, the
term "coder" pretty much died inthe 60's with the COBOL world. No
realtime embedded programmer would be referred to as a coder.

Quoted text here. Click to load it

That would be true if this were the only testing. Any responsible
organization would not be organized that way.

Quoted text here. Click to load it

That is a management issue. Developers do what they are told or allowed to
do. If you want better coverage in testing, then make it a part of your
process.

Quoted text here. Click to load it
especially

Developers are always the victims of poorly defined requirements. Poorly
defined requirements are often the norm since the desires of the customer
do change due to the fact that customer quite often cannot articulate
the nuances of the requirements, That is why iterative development
methodologies do better.


Quoted text here. Click to load it



Re: Test driven development?

Quoted text here. Click to load it

An embedded programmer who cannot at least test his/her code to reasonable
functionality isn't programming. More like "dumping".

--
Samiam is Scott A. Moore

Personal web site: http:/www.moorecad.com/scott
We've slightly trimmed the long signature. Click to see the full one.
Re: Test driven development?
Quoted text here. Click to load it

Many programmers test their code for reasonable functionality.
The real world is, however, often unreasonable.

Re: Test driven development?

Quoted text here. Click to load it

"Testing code well" is not the same as "Testing code to reasonable
functionality."  


Re: Test driven development?
Quoted text here. Click to load it

You misread.  He wrote "...who can test his own code AS well ..."  It's
a comparison, not an absolute.  Also, he's absolutely right.

Ed


Re: Test driven development?

Quoted text here. Click to load it
asking

I remember a DOS application I wrote maybe fifteen or eighteen years ago,
that was an interface to a mag strip reader (i.e. a credit card reader). I
was in the process of starting this business, and this was my first
customer. You can bet that I tested that code up-and-down, back-and-forth
and every which way to hell.

Anyway, I demo'd the s/w to the customer and everything worked great. Then
the guy takes the machine and a credit card and starts swiping the card back
and forth through the reader at a rate of three or four times a second. As
you can guess, error trapping be damned, the program crapped out.

I bet that just about everyone here can relate to this story at some level
or another. Do I write better, tighter code that is less prone to error than
I did fifteen years ago? You bet. Still, though, I make it a point to have
another person, who is knowledgeable in the area of the product, test my
code for me before I have the balls to say that its done.

I think one would be hard pressed to find a medium or large sized company
that develops and produces products, that does not have an independent QC
department responsible for both generating quality/reliability
specifications and performing tests to ensure conformance. This is the way
things are.

So, then, my opinion is this: developing and releasing a product that has
only been tested by the developer(s) isn't programming, and it sure isn't
good business.... its dumping.

Mike




Re: Test driven development?
Quoted text here. Click to load it
reasonable
back
than

So you dump it on the testers ;) Well, testing is boring, and it nice
to have testers. And to make the testers' job a bit interesting, of
course we will give them some bugs.

--
Thanks, Frank.
(remove 'x' and 'invalid' when replying by email)






Re: Test driven development?

Quoted text here. Click to load it

We are honor bound to do so. After all, these folks have families to
support.



Re: Test driven development?
On Fri, 20 Aug 2004 16:47:13 -0700, "Mike Turco"

Quoted text here. Click to load it

Isn't this just an example of deficient requirements capturing.
Basically it was a "typical"  use case that your software wasn't
designed for  -- and I'm not really talking about an obscure use case.
Having a good "tester" finding non-trivial validation flaws in the
software like the one you described, would certainly shatter my
confidence in whether the software is fit for use.

Also my observation is, writing better, tighter & less error prone
code would not have significantly "fixed" the problem in this case
(you were already doing most of this) as the fix to the problem would
have required architectural changes to your software.

If better analysis had been performed on the use cases, these
exceptional cases would have been revealled -- that's why experts are
generally employed.

Quoted text here. Click to load it


Ken.

+====================================+
I hate junk email. Please direct any
genuine email to: kenlee at hotpop.com

Re: Test driven development?


Quoted text here. Click to load it

Hi Ken,

I agree with what you're saying, but there's a dose of reality involved,
too. I'm not a big company, I'm self employed. Many of my customers are
small companies, too. The idea is to do the best you can with the money,
time and resources that you have. Certainly, not taking on jobs you can't
handle is a big part of that, and I think that's where a lot of consultants
screw up.

When I was consulting -- which I still do occasionally -- I was usually
hired because I brought something unique to the table in terms of skill and
aptitude. "Requirements", to my customer, usually meant functional
requirements. Product quality, in this case, means doing a good job. There
is no way in hell for me to bring in a quality engineer or a statistician
on-board for Joe Co's tape recorder on/off timer project.

Bottom line: business is business. We can talk all we want about creating an
ideal situation, at every level, to complete a project. The fact is we have
too little time, not enough money, hardly any memory space, funky compilers
and way too much pressure. I think I've done a damn good job on all of the
projects I've done over the years: my products keep on working and the
customers are happy.

Mike



Re: Test driven development?
On Tue, 31 Aug 2004 22:51:57 -0700, "Mike Turco"

Quoted text here. Click to load it

Hi Mike.
    I appreciate your comments and understand the constraints you
work to. My previous comments were intended to expand on this
discussion's thread, which included the unfavourable practice of "test
dumping".  The other abhor practice is using "test" personnel to
uncover requirements.

Ken.

+====================================+
I hate junk email. Please direct any
genuine email to: kenlee at hotpop.com

Re: Test driven development?
Quoted text here. Click to load it

It's worse than that IMHO. Most software developers don't design for the
failure paths. In no particular order, I have seen basic design problems in
code as follows:

1) Not checking return codes
2) Not exercising error conditions
3) Not root causing problems, only putting in "retry loops" to avoid getting
errors back
4) Not thinking through the design as potentially being used for more than
its current design in the future (expandability)
5) Not thinking about how unit testing can be accomplished manually or in an
automated fashion

It's a rare thing to run across an engineer who actually thinks in these
terms, so if you know one and are in the Portland, OR area, I'd like to hire
them. ;-)

-->Neil



Re: Test driven development?
what do you mean by test driven development ? Do you mean developing in
target and fixing all problems as you move forward ( bad ) or do you mean
desinging test features in to facilitate test at the end of development
integration phases ( good )


Quoted text here. Click to load it



Re: Test driven development?

Quoted text here. Click to load it

If you had gon to the link provided and read the introduction to the
subject matter on that web-site you would have known what the basis of TDD
is. See Mike's link (reproduced below)

   http://www.testdriven.com/modules/xoopsfaq/


--
********************************************************************
We've slightly trimmed the long signature. Click to see the full one.

Site Timeline