On OS reliability

I believe Windows Vista does this as well, to some extent?

For the average user, it's a good solution -- the idea being that, most of the time, the driver's OK, so if it crashes once in awhile, it's better to re-start it and let the user keep working than to cripple their system. Of course, the fact that it *did* crash and had to be restarted should be prominently noted somewhere; I sure hope this doesn't make device driver authors feel they have license to debug their code less thoroughly. :-(

I dunno about the drivers, but Internet Exploder 7 will restart if it crashes for some reason.

Vista does have the restart manager, which is primarily to save the system state during a software install, so that Vista can gracefully restore Humpty Dumpty to his previous condition:

As a side benefit, it's also what puts Vista back into a usable state after a crash. In theory, it was also designed to minimize the number of reboots required to do installs and updates, but that hasn't really changed much.

XP and Vista both have a setting that allows some control over what happens after a crash under "Startup and Recovery" settings. It also controls crash log file generation and restarts. It's quite handy for remote systems that need to stay up. The choice is usually a long drive to the mountain top, a problematic remote reboot device, or the "reboot on crash" feature. The latter seems the least evil.

Agreed. According to my theory "features and functions get added faster than bugs get fixed" many of the system bugs will tend to become chronic and permanent. Attempts to fix the problem at the source have been heroic, but generally ineffective because of the rather "fluid" nature of modern operating systems. Therefore band-aids and kludges are a suitable expedient.

The industrial control sector needs reliability far beyond what is found in commodity hardware and operating systems. It would not do to regularly reboot a power plant or automated factory. So, watchdog timers and heartbeat monitors are epidemic. They keep an eye on things, and if something goes awry, they reboot, generate an alarm, and alert the operators.

It's also becoming popular in Linux devices. For example, DD-WRT firmware for wireless routers has a built in watchdog timer, which either requires an external device that belches regular timer ticks, or it can monitor an IO port:

During development of v24, there were some ummm... unstable beta releases which I non-cleverly deployed at paying customer locations. In retrospect, it was not a great idea. The timed daily reboots and watchdog timer (keyed by internet traffic) kept the memory leaks and table overflows from becoming a problem.

In my never humble opinion, ANYTHING that improves reliability (especially at little or no cost), is justified and perhaps required. Just because it's not elegant doesn't also make it not useful.

On a sunny day (Mon, 25 Aug 2008 11:46:00 -0700) it happened Jeff Liebermann wrote in :

I guess it all depends. The danger is in the Minix-3 case, that application writes no longer care, and the system then restarts at a high frequency. In fact all the applications I wrote, that once crashed, would crash again at exactly the same point (user input / config), so his solution solves nothing. Sure, firefox crashes sometimes even in Linux, and then, when I restart it, asks me if I want to carry on at the same point, and it usually works then. It is a complex piece of software with a lot of internet access all at the same time, like RSS feeds, so it does not _really_ restart at the exact same point.

Crashes happen (at least in the soft I write) when null pointers, or perhaps out of range pointers happen, hardly in any other case. Those need to be, and are, fixed. There are many issues, I cannot speak for MS OSses, but I often think other things, like memory errors, hardware problems, must be at the source of many peoples complaints.

As to watchdog, yes I have done industrial watchdogs, but even a watchdog in that case had a max restart count, after which it alerted some staff remotely. If something just keeps restarting and does not work right that is no good either,

the system

I agree to some extent. That does create a temptation for application writers to use the operating system to compensate for bad code. It's like that in any industry that emphasizes system safety over component reliability. For example, the safety interlock switches on many machines may be causing more accidents than they prevent. The workers feel protected by the safety interlocks, and therefore become careless. It would probably be better warning them of the dangers and making them responsible for the consequences. However, that's not fashionable these days.

The basic philosophy is to maximize successes instead of minimizing failures. If a product can stay up longer (max success) by using a watchdog timer, it makes no sense to have it crash, halt, and die (min failure). That works, but only to a point. I don't think the nuclear, medical, or space sectors would have any interest in a watchdog fix. They tolerate no level of failure and are willing to apply any means, including redundant processing, error correcting everything, remappable memory, and heavy duty testing, to insure that there are no failures. If there are any hints of failure, the problem gets fixed at the source, not covered up with a band aid. I can see a limited future for Minix-3 in such applications.

As a confirmed pessimist, I suspect we'll never achieve a utopian bug free code, whether OS, driver, or application. I'm a lousy programmer and have personally contributed to the problem. Sometimes, I step back, look at the entire industry as a whole, and marvel that things are working so well. Sure, there are bugs, but taken as a percentage of lines of code written and used in a typical home computah, the defect rate is far better than I would expect from such a complex conglomeration of often divergent efforts.

the system

We disable the watchdog on early production products, to make sure it's not needed. Then we enable it, just in case some outrageous emi glitch or something comes along.

I've shipped a few products that don't use the watchdog, ever, and it doesn't seem to matter.

It's not errors per line of code that matter, it's errors per product. Everybody's target and expectation should be zero.

John

We'll see how long that lasts. When the FDA (Food and Drug Abomination) tests for bugs (literally) in food products, the accept the fact that there are going to be fly eggs, rodent hairs, chemical residues, and insect parts in our food. Sure, everyone's target and expectation is zero. A sane person would ask why any of these bugs are tolerated. The answer is simple... it's too expensive to hit zero.

The same applies to software and bug fixing. Debugging is far more expensive than writing the original code. It's often easier to isolate the broken code module, define the interface specs, and re-write it from scratch. Sure, you can have bug free code, but nobody wants to pay for it. In some sectors (medical, space, nuclear, etc), there's a concerted effort in that direction, but there are still bugs.

If my guess(tm) is correct, there will come a time when the balance between consumer acceptance and code generation capabilities, define a balance point in what can be tolerated. If that happens, it won't be zero. For example, your automobile might be considered "safe" if it starts without a black box crash in 999 out of 1,000 attempts.

Personally, I see more and more bugs appearing for no better reason than the code size has grown. Cheap RAM has made it easy to quickly write sloppy and bloated code. Never mind efficiency when one's sandbox has lots of CPU cycles and scratch space to play with. What I find interesting is that although the number of bugs appear to be increasing, the number of visible bugs, that affect the average user, is decreasing. That's due to better simulation and testing tools as well as the general relaxation of the requirement to write tight code in limited memory space. The other bugs are still there, but with so much un-used code buried in today's devices, they don't appear except under exceptional circumstances (which only seem to happen to me).

I'm also amused at watching development produce updated products with resurrected old bugs. When time is limited, the very first thing to get tossed is regression testing. Without it, old bugs tend to reappear. It's usually because of sloppy version control. Sometimes, the bugs were never really fixed in the first place. When I received something to tear apart, one of my first tasks was to retest it for old bugs, fixed or otherwise. I don't think there was a "mature" product that I've worked over, that didn't have at least one old bug reappear out of nowhere.

Anyway, enough philosophy for one day. I'm not a programmer and am somewhat out of my depth. I have to deal with programmers, so I try to understand how their world works. It's not for me judge, just observe.

Each bug is more expensive than the effort it would have taken to prevent it in the first place. Ergo, the cheapest code has no bugs.

Yes. So don't make bugs in the first place.

It's often easier to

If it's an undocumented hairball, agreed. Keep rewriting it from scratch until you think there are no more bugs. There probably will be some.

But how do you "define the interface"? By reading the old code?

That's criminal levels of slovenly controls. And it's mostly normal.

John

I think Vista does it a lot more than "to some extent". It does a fairly good job of restarting and covering its tracks. The exception seems to be when some MFC applications cause a crash in the kernel. It then pops up the "send an error report" dialog etc. Normally all you see is some delay and something not working as expected. The blue screen of death is nearly gone.

Yep. I'm assuming modular code (instead of the monolithic spaghetti code that I scribble). According to the programming cookbook I subscribe to, each module has one and only one entry point, one and only one return point, and the usual mess of arguments and parameters to pass back and forth. As long as someone doesn't enter or exit in the middle of a module, sub-routine, or library, the interface specs for each is fairly easy to reverse engineer. At least that's the theory.

Fortunately, debugging code is NOT my job. It's a little like some of the foreign languages I "half" know. I can understand a few fairly well, but don't speak them. I never had to speak them, so I didn't bother learning. Same with programming. I can sorta read my way through most languages (except APL) because I had to, but can't write anything that works.

If the code is monolithic, non-modular, or a collection of stolen routines of undetermined origin, the interface specs will tend to be obscure and difficult to reverse engineer. I can't do anything with that kind of code.

Reminder: I are not a programmist.

This rules out thunks and throw-catches unless you loosen it up a bit.

If a routine passes references to a bunch of variables, it can be very hard to work out what is going on. Assignments can be as bad as GOTOs and passing by reference can be as bad as self modifying code.

For proof of these claim consider that the item to be passed is in fact machine code for some other machine that is being passed into a simulation of that machine.

I made a simple little box to test some hydraulic actuators a while back. The Navy QC department kept coming back with the same question; why no watchdog timer when the requirements clearly specified one? It took weeks to finally get someone there to understand that the negative-going zero crossing of every AC cycle caused a full reset, then the hardware spent four milliseconds doing some work, and then it went into an infinite loop (instruction jumps to itself) hard crash which ended with the next reset. Simple task, simple way to accomplish it.

"What happens if the microcontroller crashes?"

"It's crashing sixty time a second as we speak. Do you hve a problem with how it's working?"

Strange, but I do multiple entries all the time, and the occasional multiple exit. There's nothing wrong with any of that if it's correct. A subroutine often benefits from calling its own tail as a subroutine, or jumping into all or part of another subroutine.

Did somebody mention GOTO? Conditional GOTO is my preferred control element. It makes nice, clearly visible state machines.

Me neither. I hate to program, so I do it right the first time, to get it over with as soon as possible. A real programmer isn't motivated to do that, since he likes to program, and all he has to look forward to is more of it.

John

I like to program. But I like to program well. Does that mean I am not a "real" programmer?

Tim

For all your rants about the dangers of those "real" programmers, it seems to me the motivation is the same as you with electronics -- that's what's fun for them, and they're the kind guys you want writing code... it's just that far too many programmers have been allowed to assume that bug-free code really isn't all that bad and should be accepted as inevitable.

Your logic about bug-free code being the cheapest code to produce relies on the assumption that discovering bugs before they end up in a shipped product is trivially easy, but this isn't always the case. Indeed, I'm willing to bet that somewhere in your products there is at least one bug remaining -- it just hasn't been found by you, and either hasn't been found by your customers or your customers didn't consider it significant enough to harass you about it if there was a trivial workaround.

Still, the all-too-common programmer culture of expecting there are going to be lots of bugs and that it's better to write a large amount of code, do minimal testing (or let someone else test it), simply log the bugs into a database and then move on unless there are "critical" bugs -- figuring you'll eventually get around to a "bug fixing" stage to iron out all the less-critical bugs -- strikes me as far less productive than your software development philosophy.

I'm sometimes baffled by how programmers work... I worked on a certain handheld widget that displays a list of messages received (think cell phone text messaging), but if none had been received yet and you selected "display" while highlighting the "There are no messages" line, the widget crashed with a fatal error (no doubt a null pointer as Jan alludes to) -- requiring the widget to be disassembled and the battery removed/re-installed, since of course they hadn't gotten around to enabling a watchdog timer or written some "Fatal error, press here to reboot manually" code either. This was all nicely documented and scheduled to be fixed, but personally a bug that bad and easy to reproduce would goad me to immediately sit down and fix the problem... it almsot seems as though the programmer spent more time writing up a description of this bug than he would have fixing it.

I do know some very good programmers (and small companies full of them), though, where this sort of nonsense would never fly... yet in many places (especially those where the managers don't have technical backgrounds) this is par for the course.

Hey John -- has fast can you press the buttons on your boxes and still have them register? I've been arguing with some programmers lately that their requirement for a 100ms hold for a button press to be recognized is nuts... I easily can (and do) push up and down arrow keys at a rate exceeding 5Hz! (Assuming equally press/release times -- 100ms + 100ms = 5Hz.)

---Joel

If you indeed program well, you are a freak programmer.

John

On a sunny day (Tue, 26 Aug 2008 09:45:12 -0700) it happened John Larkin wrote in :

So, which one will it be? :

the system

exactly the same

time,

things,

complaints.

that case had a max

either,

Very true. Too many safeguards won't help either. 15 years ago I worked in a company that serviced mainframe computers. One of these machines kept failing for no good reason at all. In the end it turned out the power monitor board had a bad A/D converter on it which caused the machine to halt.

It's not trivial, but it's bot hard either. It does require discipline, and parts of the process, like carefully reviewing your own code, are boring.

Possibly, but such "bugs" aren't bugs if nobody triggers them. I have several complex products that are out in the firld in the thousands with no reported bugs. This ain't rocket science, this is just dumb being careful. The dumber the better, in fact.

Absolutely. It's especially expensive if the entire project is a failure, which something like half of big programming projects are.

I know two superb programmers who do everything fast and right. Neither has ever studied programming.

Some of the worst (in terms of most painful) code I've written is accelerating typematic pushbutton service/display refresh stuff.

John

On a sunny day (Tue, 26 Aug 2008 10:15:35 -0700) it happened John Larkin wrote in :

I'd rather the eject button was connected to a NMI, then to some Larkin polling, that happened to be at the same speed as the chair was vibrating.

'Honestly' (last words pilot) 'I press eject, nothing happens.'

Polled right in sync when the contact was off, for almost a second.