Re: Version Control in Embedded Systems

It does happen though. The first software version for the new Teac DVB-T STB recently released in Australia sucked so badly that users demanded an upgrade, and got it. You couldn't skip unused channels, and settings didn't save correctly.

Rob

Reply to
Rob Judd
Loading thread data ...

snipped-for-privacy@amleth.demon.co.uk ("Paul E. Bennett") wrote in news: snipped-for-privacy@amleth.demon.co.uk:

The extreme programming guys would take issue with this in that until you start delivering releases to the customers they tend to be very unclear as to what the sepcifications should be. Their approach is to have a development process that can cope with change, as that is a fact of life.

It can all depend on what you are doing with software. I have a worked on a number of systems where I have had to rely on third party components (ranging from middleware to Microsoft Office), and indeed the operating system and compilers. I have come across bugs in the compilers, bugs in Office, etc, etc. These are nasty and my ability to persuade Microsoft to fix things somewhat limited - I have to find the bug, find what caused it, and then find workarounds...

Software these days tends to depend on a heap of other stuff around it.

--
Robert Cowham
Reply to
Robert Cowham

An excellent point, highlighted many, many times in my own experience, as well. Few people really know what they want before they get to touch and feel what they don't want.

There are a few wonderful customers who have been down the road a few times and can provide both an excellent definition to start and an excellent background of experience to draw from. Some of those are even able and willing to pay for the job at hand, too. If anyone has a magic formula for causing that select set of customers to track you down and hire you, I'd love to hear it. I know one when I see one, but they are as rare as hen's teeth and I can't base a business on just that group.

But give someone something they don't want and they'll often find out some one fact quickly and let you know. Fix that and come back and they'll find yet another one that "slipped through" the last time.

Knowing this up front and still managing to get the right thing done in the end, on an acceptable schedule and cost, is some of what keeps the work interesting.

Jon

Reply to
Jonathan Kirwan

A prime example of the need to "fix the customer before fixing the system". The above siuation is wholly avoidable with the right approach. Remember System Design beigins by getting the spec right for yourself and the customer.

I have related in this and other groups my use of prototypes (on PC and in cardboard cut-outs) as aids in "spec-bashing sessions". It is quite extraordinary when you get a roomful of execs and operators pressing the imaginary buttons and making comments about what they expect to see happen as a result. The sessions need to be highly structured and a good moderator is absolutely vital. The result is usually a pretty good preliminary requirements specification that one can get ones teeth into and not bite much fresh air. Usually we do it all again a few weeks later but this time reviewing the spec very thoroughly and really stretching it hard. The second meeting everyone has to prepare for properly. As you develop the system you are going to continue to find the odd problem in the requirements specification. It is vital to raise a problem report on this with the client and ensure he responds to it promptly. Be on your guard to ensure that feature drift does not creep into the requirements without your noticing it (I know some clients try to do this). If your teams are on the ball you should be able to spot this specification drift easily enough. When you do quote the cost of adding the feature to the client and give him the option of changing the requiremenst and paying for it or to limit his needs to the original.

However good you are at getting the spec right first time there are many instances where a change must be implemented in a myriad of documents. You need to control the changes in a very orderly manner. Such changes may come from anywhere; testing; users of previous systems; the client maintenance etc. Encourage the filling in of problem reports (hint: only one problem per report as this eases closure later). Have a sound process for dealing with problem reports and ensure they are always under review (even the simply answered ones). Always record all problem reports submitted and track them until each problem is resolved (by a new version issue).

My simple system development process is partly described in the flowchart on the HIDECS page of my website and in papers I have presented to a couple of conferences. It is in fact the central change management core of the process. There is another diagram that deals with the range of documents required at each stage of development. You are invited to look and download the papers if you want to find out more.

--
********************************************************************
Paul E. Bennett ....................
 Click to see the full signature
Reply to
Paul E. Bennett

When I say "validation", I actually did mean validation and not verification. The process you describe is one of verification. In recent years the use of the term "validation" has become synonymous with that of "verification" -- even the FDA has done this. However by definition, validation is not verification.

Not at all -- I'm assuming that there is a Systems Engineering phase. Also I'll dispute that "in practice" that these requirements are NOT "done long before the system architecture is established". Why do I say that -- well Systems Engineering is not done in a lock-up in isolation. For a good System Spec to be produced it involves the collaboration of all stakeholders. The Systems Engineer acts as the mediator of the Domain Experts. He/she looks at the marketing requirements and with the inputs from the hardware, mechanical & software representatives (etc) produces a System Spec which would have a manageable development risk. What do the engineers do during this time? Well they're not sitting at their desks twiddling their thumbs or playing solitaire. They may be: o ringing up vendors for parts & availability o preparing & running prototypes o researching development tools o compiling a list of prospective micros to use o preparing pseudo Bills of Materials for costing

I do agree that a good System Spec should be produced before "real" development starts.

It should be noted that engineers are lazy sods (me included). To mitigate against development risk, they typically adopt a modus operandi or pattern of behaviour that has this tendency. This includes: o Adopting tried & proven methods o Utilising domain experts o Using mature architectures o Using mature components o Utilising a previous projects artefacts such as documents & code o Working within a reasonable comfort zone, that is, working with equipment that you know how to use, working with people you know, etc.

Now reading this you may think that engineers are pretty boring people (ever been to a party with one ;-) ), but obviously they do venture out and try new things. The point I'm getting to here, that is, it may be that the new project is similar to the previous one. This is the case for a company that is producing a limited product range. So the System Spec for the new product may be reasonably predictable.

Contractors are probably the laziest sods of all ;-) --- have you ever notice how they re-use aspects of their previous projects. However this usually isn't reflected in their invoice. (For the record, I have been a contractor :-) )

The above is great for "design" and I'm all for it, but the analysis of say, the Software system, still has to be done by an entity with a brain. Sure there are analysis tools which help greatly. I've used iLogix's Rhapsody and can appreciate the benefits of such technology.

Ken.

+====================================+ I hate junk email. Please direct any genuine email to: kenlee at hotpop.com
Reply to
Ken Lee

All good stuff. Where I work we use the term "train" rather than "fix". "Fix" has the implication that you want to kill someone --- which on the other hand with System Specs, may be the case ;-)

What's also useful from these sessions, are the use cases & scenarios as they could be used in System Validation and provide a guide to Software Verification.

Basically with new features (particularly when introduced in later design) you have to play hardball with the customer and state changes in terms of dollars ($$). Recording changes & problems provide good metrics that can give one insight as to the cost of change, for future projects.

It's often the case with projects that too much documentation is produced or not the right documents, particularly if the product is to be submitted to some regulatory body. Why too much? Typically for products that go to regulatory bodies, engineers will tend to produce a lot of "just-in-case" documents, may be because they haven't received guidance as to what documents to produce. If the product is to go for CE Mark or FDA (510K) submission, then the engineer should be made aware of what documents are mandatory at the onset.

I would recommend an online tool to track problems & issues and make it available to all team members (not just engineers). We use PVCS Tracker and the use of it is outlined in our work instruction so that everyone knows how it should be used.

+====================================+ I hate junk email. Please direct any genuine email to: kenlee at hotpop.com
Reply to
Ken Lee

Right: locking the conference room door.

John

Reply to
John Larkin

: > "John Larkin" wrote in : > message news: snipped-for-privacy@4ax.com... : > > On Mon, 14 Jul 2003 23:00:27 -0400, "Kevin Kramb" : > > wrote: : > > >

: > > >Is it your experience that hardware designs are normally right the first : > > >time? : > >

: > >

: > > It's interesting to compare. If I design a PC board full of parts, and : > > anything's wrong, I have to write an ECO to manufacturing to kluge the : > > board, and that's a very public admission of error. If it's too : > > serious or ugly to kluge, we have to roll the PCB rev - lots of work - : > > re-release the drawings, wait for new boards, build the next : > > iteration, etc. Again, very public and very expensive. So most : > > hardware works the first time, because the economic and social : > > pressures punish sloppy work. : > >

: > > Software, on the other hand, can be patched forever without traces : > > peeling off, and the entire process is pretty much done in private. So : > > software, even short routines, are seldom right the first, or even the : > > second, time. Interestingly, FPGA designs often tend to be like : > > software, sloppy and buggy on the first pass, at least by some people. : > > Anybody who designed a hard ASIC this way would be unemployed pronto. : >

: > Most ICs seem to take at least a couple of expensive tries to get right and : > many take more than that. It may vary between ASICs and full custom designs. : >

: > Ever considered how to apply a software patch to a few million TVs? Once it : > really goes out you are not going to patch it unless it is something very : > serious. : : It does happen though. The first software version for the new Teac DVB-T : STB recently released in Australia sucked so badly that users demanded : an upgrade, and got it. You couldn't skip unused channels, and settings : didn't save correctly. : : Rob

Hardware *bugs* are always more costly upfront because of the loss of material(say boards and other inputs) than software. Few years back Philips (one of my previous employer) had to move the DVD player out of the AsiaPacific market because it wouldnot play pirated VCDs, reason : developers coded only to the specification, which did not include the pirated format, but there were competetive products playing, so we needed to do the same to stay in the market. No one to blame but at the end Customer rules.

Reply to
Koushik Banerjee

If the tester happens to find a bug just after he starts testing, I think either the specs were not cleanly defined(or rather the test cases dont match the spec) Or the programmer is really a BAD one. In such cases it is better desirable to do away with that programmer and get someone else good to re-write the code (NOT revamp or fix the code), because it is easier writing something altogether new yourself, than to fix some shit of someone else. And anyways if the developer left behind some obvious bugs, his coding style would also be very bad and less readable to others, less maintainable and blah-blahs..

: Yes, we should use the most cost effective trade-offs available. By : some that will be rigorous code inspection, by others it will be a : solid test-harness. Others will find a combination of the two. : : There is one issue that I didn't see addressed. : : If a bug is found don't deny it. Sounds trite but I've seen it happen. : The engineer that I'm happiest with assumes the truth of a bug report, : tries to reconstruct it in his development environment, and then asks : the question "Is this isolated or is this found somewhere else in my : work (read Hardware and/or Software)?" IMHO, quick patches are : worthless, even if you have a zillion installations. You'll just have : to make another one. If your lucky the tester will catch your new bug : before it gets out the door. : : This question could and should also be applied to marketing decisions : (read GUI design and feature sets) : : Another point that ties in to Version Control (that's how we started : the thread :-} ). : There must be communication between the Development and Test Groups. : At the very least the Test Group (and actually the developer himself) : should ask "Did you touch shared modules?" The answer to this question : will define how extensive the regression test for introducing bugs to : other features will be. : : Yaakov

Reply to
Koushik Banerjee

It is right to get a customer "who knows what he wants" in software industry, but it is not the case for hardware. It is very expensive, and the customers _usually_ knows "what they want" in the hardware industry. Most companies into hardware do their own design and manufacturing, and others usually are already big and long-time players to know what would work with their design. The difference with a software and hardware can be tested on cost terms as: Software (say a particular version) once written can be used/sold "n" number of times with out any additional costs(barring packaging/shipping and other such costs), BUT Hardware except the designing , the actual hardware needs to be produced "n" number of times if you plan to sell it so many times (along with other costs of packagin/shipping ...)

So fixing a software bug when released into the market/into production even for embedded systems is easy , but for the hardware it is impossible since you need to replace the whole stuff or maybe if possible write something new in software to achieve the same functionailty , though not quite possible always, and any function thru hardware is performance efficient than done in software.

Koushik.

Reply to
Koushik Banerjee

It's rare to find them, though. So one cannot base their software business on such customers unless one has a magic formula here or a narrow enough market where this is often the case. I think some of the same issues hound hardware, as well. (In fact, I know some do, since I work in designing and building scientific and commercial instrumentation and I see the same issues come up in finding the _right_ system specifications.)

Ah, you mean the opposite of what I thought you mean. In other words, you mean that in hardware you *must* have a customer who knows what they want?

Plus, there is nature working for you. Nature/physics pretty much is the same where ever you go in the world and problems have a certain set of well known boundaries for their solutions. Software has to deal with physics, too, of course (the embedded work I do, for example), but facets can also often be more about addressing human likes and dislikes, which are vastly more fickle than nature. But then, so does hardware have to deal with this (ease of use.)

Indeed. Hardware is tangible.

I'm not sure exactly where you were going. I tend to understand and agree with you about the details you've mentioned. So perhaps all you wanted to do is highlight another aspect I'd failed to address.

The only thing I take issue with, if at all, is if you are saying that customers of hardware generally know what they want and that you *can* rely on having such customers when you do hardware design. If that is what you think or are arguing, then I have to disagree. Hardware customers are not always (or often, perhaps) all that knowledgeable about what they want. [Some are trained over time to _want_ only what can actually be made well!]

But the street of bankruptcy is littered with companies failing, in part, because of building what their perceived customers _told_ them they wanted. Failing to ask questions about what they are willing to pay for or failing to understand who the real customer (the one willing to actually pay) is and getting their information from the wrong base... and so on.

It's wonderful, as always, when one's customers are all well informed about their requirements, know what they need/want, know what they are willing to pay and for what, and have the where-with-all to actually pay a good price.

Still, I think many of the same problems exist in both domains, as far as customers are concerned. There are differences, of course. And you've painted some of those well.

Jon

Reply to
Jonathan Kirwan

I've only ever worked with one such developer - unfortunately it wasn't just his code that was problematic, it was his entire attitude to work. If he got challenging work he regarded it as being an attempt to stretch him past breaking point, if he got routine work he described it as an attempt to 'de-skill' him...

pete

--
pete@fenelon.com "there's no room for enigmas in built-up areas" HMHB
Reply to
Pete Fenelon

"Jonathan Kirwan" wrote in message news: snipped-for-privacy@4ax.com... :: >It is right to get a customer "who knows what he wants" in software : >industry, : : It's rare to find them, though. So one cannot base their : software business on such customers unless one has a magic : formula here or a narrow enough market where this is often the : case. I think some of the same issues hound hardware, as well. : (In fact, I know some do, since I work in designing and building : scientific and commercial instrumentation and I see the same : issues come up in finding the _right_ system specifications.) : : >but it is not the case for hardware. It is very expensive, and the : >customers _usually_ knows "what they want" in the hardware industry. : : Ah, you mean the opposite of what I thought you mean. In other : words, you mean that in hardware you *must* have a customer who : knows what they want? : : >Most : >companies into hardware do their own design and manufacturing, and others : >usually are already big and long-time players to know what would work with : >their design. : : Plus, there is nature working for you. Nature/physics pretty : much is the same where ever you go in the world and problems : have a certain set of well known boundaries for their solutions. : Software has to deal with physics, too, of course (the embedded : work I do, for example), but facets can also often be more about : addressing human likes and dislikes, which are vastly more : fickle than nature. But then, so does hardware have to deal : with this (ease of use.) : : >The difference with a software and hardware can be tested on : >cost terms as: : >Software (say a particular version) once written can be used/sold "n" number : >of times with out any additional costs(barring packaging/shipping and other : >such costs), : >BUT : >Hardware except the designing , the actual hardware needs to be produced "n" : >number of times if you plan to sell it so many times (along with other costs : >of packagin/shipping ...) : >

: >So fixing a software bug when released into the market/into production even : >for embedded systems is easy , but for the hardware it is impossible since : >you need to replace the whole stuff or maybe if possible write something new : >in software to achieve the same functionailty , though not quite possible : >always, and any function thru hardware is performance efficient than done in : >software. : : Indeed. Hardware is tangible. : : I'm not sure exactly where you were going. I tend to understand : and agree with you about the details you've mentioned. So : perhaps all you wanted to do is highlight another aspect I'd : failed to address. : : The only thing I take issue with, if at all, is if you are : saying that customers of hardware generally know what they want : and that you *can* rely on having such customers when you do : hardware design. If that is what you think or are arguing, then : I have to disagree. Hardware customers are not always (or : often, perhaps) all that knowledgeable about what they want. : [Some are trained over time to _want_ only what can actually be : made well!] : : But the street of bankruptcy is littered with companies failing, : in part, because of building what their perceived customers : _told_ them they wanted. Failing to ask questions about what : they are willing to pay for or failing to understand who the : real customer (the one willing to actually pay) is and getting : their information from the wrong base... and so on. : : It's wonderful, as always, when one's customers are all well : informed about their requirements, know what they need/want, : know what they are willing to pay and for what, and have the : where-with-all to actually pay a good price. : : Still, I think many of the same problems exist in both domains, : as far as customers are concerned. There are differences, of : course. And you've painted some of those well. : : Jon : You are right in saying that the problem exists in both domain, but I feel the problem exists more in software domain considering the knowledge base of the software companies, customers and the engineers (I am not saying the h/w engineers are better than s/w ones, but the number of new things done in h/w is less than in s/w)and this contributes to the artifacts that are needed to get a job done. My belief is that a company investing in h/w is dollar ($$) rich first to get all the right setup (say you need more than few computers and a garage to start with, you need a dust-free environment, some special dress, h/w and other stuff to get started), and after investing in all these, if they are not sure about their customers and their requirements, it is pretty much a ruin for the company. That is the company is likely to close down any day because of lack of business. But in sofware, you can rent a few machines, set up few tables in a garage, and get some hackers to start with. Once you feel that you are getting good business , you can expand, or simply stop paying the rent of the computers and garage, and you would be thrown out with very little to lose. The investment pattern is very different. But yes, hardware does get effected by software these days, say when the e-crash took place in 2000, all the switch manufacturing companies took a hit, since their customer was maily companies into e-commerce, and those either cancelled or postponed their orders. Lucent, Cisco took hits that way then.

And if you are talking of "usuability" case in hardware, I think the core functionality is what drives a product, other than the look & feel factor. And the core costs more than the UI factor, like how the front-panel looks like in a DVD player or the box looks like can be different in different regions, but most people would be more interested in what it can do (the core part), which is the expensive part.

Well otherwise both of them are same in most ways. You are right about that.

Koushik.

Reply to
Koushik Banerjee

Hardware design tends to have fewer (lots fewer!) bugs, but there may be another explanation: hardware is generally parallel and synchronous, while software is sequential within separate concurrent threads, and essentially asynchronous. So hardware logic designers have much better visibility and control over system states.

John

Reply to
John Larkin

... Which is exactly why I design my code to be synchronous. And why I have so few software problems.

I consider that designing hardware (esp. ASICs) taught me the most about writing good software ;).

Steve

formatting link
formatting link

Reply to
steve at fivetrees

Me too. I design embedded software the same way as I design logic: run nice clean little state machines at a regular rate, run each to completion every shot, and never assign resources dynamically. Fully decode states. "Inter-task" communications is via variables shared between state machines. No queues. No abstraction. No glamour. No bugs.

I've written a few RTOSs, but I find lately that I really don't need them. Luckily, I'm an engineer most of the time, so I view programming as a mildly tedious task to be finished, not as a lifelong career. So I just want to get it over the quickest, most reliable way, even if that involves simplistic grunt work that wouldn't look good on a resume.

John

Reply to
John Larkin

I think this is completely wrong, as regards consumer products. Color displays sell better than monochrome. Customers want the color of the box to match their other components, which is why audio/video equipment is either all silver or all black.

Selling consumer electronics is all about emotion, which is why advertisements are written and illustrated in particular ways.

MOST consumers don't know that there might be the identical decoder in two DVD players, one selling for $100 and one for $200. They will pay more for specific brands, particularly if they owned that brand before, or because someone they know has it. Other than that, price usually drives the decision.

And frankly, a good display, keypad, and case can cost several times what the core electronics does. Chip vendors are in a continuous race to the bottom, but that doesn't change the price of metal vs. plastic for the box.

--Gene

Reply to
Gene S. Berkowitz

C|N>K!

From what planet did you import this notion?

I take it you've never written firmware for mask-ROM, microcontrollers, which are soldered directly onto the product PCB, to control cost? Or code synthesized into a compacted ROM, within an ASIC?

Remind me not to hire you, for any projects with a half-million-dollar NRE or initial parts purchase. In fact, I think I'll put you on the as- sembly line, remanufacturing a few tens of thousands of product PCB's, to replace MCU's for a code change. Be careful not to peel any traces!

Once out of prototyping, the cost of changing embedded software is oft- en every bit as high as changing hardware--and sometimes worse. (After all, changing the odd SMT component, or adding a little flat-wiring is a bargain, compared to replacing an MCU).

--
egr
Reply to
Eric Roesinger

Koushik

Reply to
Koushik Banerjee

"Eric Roesinger" wrote in message news:3f2a92fb_4@newsfeed... : "Koushik Banerjee" wrote in message : news:bg8sac$lreg0$ snipped-for-privacy@ID-188659.news.uni-berlin.de... : >

: > : >

: > So fixing a software bug when released into the market/into production : > even for embedded systems is easy : : C|N>K! : : From what planet did you import this notion? : : I take it you've never written firmware for mask-ROM, microcontrollers, : which are soldered directly onto the product PCB, to control cost? Or : code synthesized into a compacted ROM, within an ASIC? : : Remind me not to hire you, for any projects with a half-million-dollar : NRE or initial parts purchase. In fact, I think I'll put you on the as- : sembly line, remanufacturing a few tens of thousands of product PCB's, : to replace MCU's for a code change. Be careful not to peel any traces! : : Once out of prototyping, the cost of changing embedded software is oft- : en every bit as high as changing hardware--and sometimes worse. (After : all, changing the odd SMT component, or adding a little flat-wiring is : a bargain, compared to replacing an MCU). : : -- : egr : : My statement was a relative one(though I had not mentioned that, but can be judged from the statement which follows). So what I meant that was (read relatively) easier to fix a bug in the software for embedded system than it is to do it with hardware. You are right that I have not written any firmware myself. My main job is configuration management, but I have written code also. I have worked for the european leader in CE, and that is how my idea on embedded system comes from. It might be possible that they have better process and technology working for them to get this done easily(they are in the market for more than 100yrs). At least I am aware that if they needed to fix a application bug in the DVD player, all that was needed was to create a CD with the flash, which would then be distributed to the dealers who would then place it in the tray of the DVD player, which would self-program itself, (this process took less than 5minutes). If that sound difficult, I know of no easier way even in software industry itself. A high range DELL server takes more than

5minutes to boot with NT/2K OS, leave aside Unix style OS.
Reply to
Koushik Banerjee

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.