Predictive failures

Current and voltages outside of normal operation?

Cheers

Checking temperatures is good. An overload or a fan failure can be bad news.

We put temp sensors on most products. Some parts, like ADCs and FPGAs, have free built-in temp sensors.

I have tried various ideas to put an air flow sensor on boards, but so far none have worked very well. We do check fan tachs to be sure they are still spinning.

Blocking air flow generally makes fan speed *increase*.

There is a standard approach that may work: Measure the level and trend of very low frequency (around a tenth of a Hertz) flicker noise. When connections (perhaps within a package) start to fail, the flicker level rises. The actual frequency monitored isn't all that critical.

Joe Gwinn

Do connections "start to fail" ?

I don't think I've ever owned a piece of electronic equipment that warned me of an impending failure.

Cars do, for some failure modes, like low oil level.

Don, what does the thing do?

My conclusion would be no. Some of my reasons are given below.

It always puzzled me how HAL could know that the AE-35 would fail in the near future, but maybe HAL had a motive for lying.

Back in that era I was doing a lot of repair work when I should have been doing my homework. So I knew that there were many unrelated kinds of hardware failure.

A component could fail suddenly, such as a short circuit diode, and everything would work fine after replacing it. The cause could perhaps have been a manufacturing defect, such as insufficient cooling due to poor quality assembly, but the exact real cause would never be known.

A component could fail suddenly as a side effect of another failure. One short circuit output transistor and several other components could also burn up.

A component could fail slowly and only become apparent when it got to the stage of causing an audible or visible effect. It would often be easy to locate the dried up electrolytic due to it having already let go of some of its contents.

So I concluded that if I wanted to be sure that I could always watch my favourite TV show, we would have to have at least two TVs in the house.

If it's not possible to have the equivalent of two TVs then you will want to be in a position to get the existing TV repaired or replaced as quicky as possible.

My home wireless Internet system doesn't care if one access point fails, and I would not expect to be able to do anything to predict a time of failure. Experience says a dead unit has power supply issues. Usually external but could be internal.

I don't think it would be possible to "watch" everything because it's rare that you can properly test a component while it's part of a working system.

These days I would expect to have fun with management asking for software to be able to diagnose and report any hardware failure. Not very easy if the power supply has died.

Yes, they do, in things like vias. I went through a big drama where a critical bit of radar logic circuitry would slowly go nuts.

It turned out that the copper plating on the walls of the vias was suffering from low-cycle fatigue during temperature cycling and slowly breaking, one little crack at a time, until it went open. If you measured the resistance to parts per million (6.5 digit DMM), sampling at 1 Hz, you could see the 1/f noise at 0.1 Hz rising. It's useful to also measure a copper line, and divide the via-chain resistance by the no-via resistance, to correct for temperature changes.

The solution was to redesign the vias, mainly to increase the critical volume of copper. And modern SMD designs have less and less copper volume.

I bet precision resistors can also be measured this way.

Onset of smoke emission is a common sign.

The industrial method for big stuff is accelerometers attached near the bearings, and listen for excessive rotation-correlated (not necessarily harmonic) noise.

Good question.

Joe Gwinn

But nobody is going to monitor every via on a PCB, even if it were possible.

One could instrument a PCB fab test board, I guess. But DC tests would be fine.

We have one board with over 4000 vias, but they are mostly in parallel.

Big ships that I've worked on have a long propeller shaft in the shaft alley, a long tunnel where nobody often goes. They have magnetic shaft runout sensors and shaft bearing temperature monitors.

They measure shaft torque and SHP too, from the shaft twist.

I liked hiding out in the shaft alley. It was private and cool, that giant shaft slowly rotating.

There are a number of instruments available that look for metal particles in the lubricating oil.

Cheers

Phil Hobbs

And water. Some of our capacitor simulators include a parallel resistance component.

One customer used to glue bits of metal onto a string and pull it through the magnetic sensor. We did a simulator for that too.

Jet engines have magnetic eddy-current blade-tip sensors. For efficiency, they want a tiny clearance between fan blades and the casing, but not too tiny.

I think "outside" is (often) likely indicative of "something is (already) broken".

But, perhaps TRENDS in either/both can be predictive.

E.g., if a (sub)circuit has always been consuming X (which is nominal for the design) and, over time, starts to consume

Note that the goal is not to troubleshoot the particular design or its components but, rather, act as an early warning that maintenance may be required (or, that performance may not be what you are expecting/have become accustomed to).

You can include mechanisms to verify outputs are what you

You can, also, do sanity checks that ensure values are never what they SHOULDN'T be (this is commonly done within software products -- if something "can't happen" then noticing that it IS happening is a sure-fire indication that something is broken!)

And, where possible, notice second-hand effects of your actions (e.g., if you switched on a load, you should see an increase in supplied current).

But, again, these are more helpful in detecting FAILED items.

Why does your PC retry failed disk operations? If I ask the drive to give me LBA 1234, shouldn't it ALWAYS give me LBA1234? Without any data corruption (CRC error) AND within the normal access time limits defined by the location of those magnetic domains on the rotating medium?

Why should it attempt to retry this MORE than once?

Now, if you knew your disk drive was repeatedly retrying operations, would your confidence in it be unchanged from times when it did not exhibit such behavior?

Assuming you have properly configured a EIA232 interface, why would you ever get a parity error? (OVERRUN errors can be the result of an i/f that is running too fast for the system on the receiving end) How would you even KNOW this was happening?

I suspect everyone who has owned a DVD/CD drive has encountered a "slow tray" as the mechanism aged. Or, a tray that wouldn't open (of its own accord) as soon/quickly as it used to.

The controller COULD be watching this (cuz it knows when it initiated the operation and there is an "end-of-stroke" sensor available) and KNOW that the drive belt was stretching to the point where it was impacting operation.

The goal isn't to predict ALL failures but, rather, to anticipate LIKELY failures and treat them before they become an inconvenience (or worse).

One morning, the (gas) furnace repeatedly tried to light as the thermostat called for heat. Then, a few moments later, the safeties would kick in and shut down the gas flow. This attracted my attention as the LIT furnace should STAY LIT!

The furnace was too stupid to notice its behavior so would repeat this cycle, endlessly.

I stepped in and overrode the thermostat to eliminate the call for heat as this behavior couldn't be productive (if something truly IS wrong, then why let it continue? and, if there is nothing wrong with the controls/mechanism, then clearly it is unable to meet my needs so why let it persist in trying?)

You don't care about the real cause. Or, even the failure mode. You (as user) just don't want to be inconvenienced by the sudden loss of the functionality/convenience that the the device provided.

So, if you could predict the OTHER failure... Or, that such a failure might occur and lead to the followup failure...

But, likely, there was something observable *in* the circuit that just hadn't made it to the level of human perception.

Two TVs are affordable. Consider two controllers for a wire-EDM machine.

Or, the cost of having that wire-EDM machine *idle* (because you didn't have a spare controller!)

Again, the goal isn't to predict "time of failure". But, rather, to be able to know that "this isn't going to end well" -- with some advance notice that allows for preemptive action to be taken (and not TOO much advance notice that the user ends up replacing items prematurely).

You don't have to -- as long as you can observe its effects on other parts of the system. E.g., there's no easy/inexpensive way to check to see how much the belt on that CD/DVD player has stretched. But, you can notice that it HAS stretched (or, some less likely change has occurred that similarly interferes with the tray's actions) by noting how the activity that it is used for has changed.

What if the power supply HASN'T died? What if you are diagnosing the likely upcoming failure *of* the power supply?

You have ECC memory in most (larger) machines. Do you silently expect it to just fix all the errors? Does it have a way of telling you how many such errors it HAS corrected? Can you infer the number of errors that it *hasn't*?

There are (and have been) many efforts to *predict* lifetimes of components (and, systems). And, some work to examine the state of systems /in situ/ with an eye towards anticipating their likelihood of future failure.

The question is one of a practical nature; how much does it cost you to add this capability to a device and how accurately can it make those predictions (thus avoiding some future cost/inconvenience).

For small manufacturers, the research required is likely not cost-effective; just take your best stab at it and let the customer "buy a replacement" when the time comes (hopefully, outside of your warranty window).

But, anything you can do to minimize this TCO issue gives your product an edge over competitors. Given that most devices are smart, nowadays, it seems obvious that they should undertake as much of this task as they can (conveniently) afford.

That depends on many other unknown factors. Temperature sensors are common in electronics. So is current sensing. Voltage sensing too.

If the system is electronic then you can detect whether currents and/or votages are within expected ranges. If they are a just a little out of expected range then you might turn on a warning LED. If they are way out of range then you might tell the power supply to turn off quick. By all means tell the software what has happened, but don't put software between the current sensor and the emergency turn off. Be aware that components in monitoring circuits can fail too.

What system would you like to have early warnings for? Are the warnings needed to indicate operation out of expected limits or to indicate that maintenance is required, or both? Without detailed knowledge of the specific sytem, only speculative answers can be given.

Because the software designer didn't understand hardware. The correct approach is to mark that part of the disk as unusable and, if possible, move any data from it elsewhere quick.

I'd have put an SSD in by now, along with an off site backup of the same data :)

If it hasn't been used for some time then I'm ready with a tiny screwdriver blade to help it open. But I forget when I last used an optical drive.

That's why the furnace designers couldn't have anticipated it. They did not know that such a contition might occur so never tested for it.

There will always be sudden unexpected loss of functionality for reasons which could not easily be predicted. People who service lawn mowers in the area where I live are very busy right now.

Yes a power supply ripple detection circuit could have turned on a warning LED but that never happened for at least two reasons.

1. The detection circuit would have increased the cost of the equipment and thus diminished the profit of the manufacturer.
2. The user would not have understood and would have ignored the warning anyway.

Get feedback from the people who use your equpment.

Sure but you have to be the operator for that. So you can be ready to help the tray open when needed.

Then I probably can't, because the power supply may be just a bought in power supply which was never designed with upcoming failure detection in mind.

Things are sometimes done the way they've always been done. I used to notice a missing chip in the 9th position but now you mention it the RAM I just looked at has 9 chips each side.

I'm sure that's true.

Sensors cost money. And, HAVING data but not knowing how to USE it is a wasted activity (and cost).

Why not monitor every node in the schematic and compare them (with dedicated hardware -- that is ALSO monitored??) with expected operational limits?

Then, design some network to weight the individual observations to make the prediction?

Again, the goal is to be an EARLY warning, not an "Oh, S*it! Kill the power!!"

As such, software is invaluable as designing PREDICTIVE hardware is harder than designing predictive software (algorithms).

You don't want to tell the user "The battery in your smoke detector is NOW dead (leaving you vulnerable)" but, rather, "The battery in your smoke detector WILL cease to be able to provide the power necessary for the smoke detector to provide the level of protection that you desire."

And, the WAY that you inform the user has to be "productive/useful". A smoke detector beeping every minute is likely to find itself unplugged, leading to exactly the situation that the alert was trying to avoid!

A smoke detector that beeps once a day risks not being heard (what if the occupant "works nights"?). A smoke detector that beeps a month in advance of the anticipated failure (and requires acknowledgement) risks being forgotten -- until it is forced to beep more persistently (see above).

Which is why hardware interlocks are physical switches -- yet can only be used to protect against certain types of faults (those that are most costly -- injury or loss of life)

I'm not looking for speculation. I'm looking for folks who have DONE such things (designing to speculation is more expensive than just letting the devices fail when they need to fail!)

E.g., when making tablets, it is possible that a bit of air will get trapped in the granulation during compression. This is dependant on a lot of factors -- tablet dimensions, location in the die where the compression event is happening, characteristics of the granulation, geometry/condition of the tooling, etc.

But, if this happens, some tens of milliseconds later, the top will "pop" off the tablet. It now is cosmetically damaged as well as likely out of specification (amount of "active" present in the dose). You want to either be able to detect this (100% of the time on 100% of the tablets) and dynamically discard those units (and only those units!). *OR*, identify the characteristics of the process that most affect this condition and *monitor* for them to AVOID the problem.

If that means replacing your tooling more frequently (expensive!), it can save money in the long run (imagine having to "sort" through a million tablets each hour to determine if any have popped like this?) Or, throttling down the press so the compression events are "slower" (more gradual). Or, moving the event up in the die to provide a better egress for the trapped air. Or...

TELLING the user that this is happening (or likely to happen, soon) has real $$$ value. Even better if your device can LEARN which tablets and conditions will likely lead to this -- and when!

Actually, he DID understand the hardware which is why he retried it instead of ASSUMING every operation would proceed correctly.

That only makes sense if the error is *persistent*. "S*it happens" and you can get an occasional failed operation when nothing is truly "broken".

(how do you know the HBA isn't the culprit?)

So, any problems you have with your SSD, today, should be solved by using the technology that will be invented 10 years hence! Ah, that's a sound strategy!

Why don't they ship such drives with tiny screwdrivers to make it easier for EVERY customer to address this problem?

When the firmware in your SSD corrupts your data, what remedy will you use?

You're missing the forest for the trees.

Really? You can't anticipate the "gas shutoff" not being in the ON position? (which would yield the same endless retry cycle)

If they planned on ENDLESSLY retrying, then they must have imagined some condition COULD occur that would lead to such an outcome. Else, why not just retry *once* and then give up? Or, not retry at all?

And if they CAN'T be predicted, then they aren't germane to this discussion, eh?

My concern is for the set of failure modes that can realistically be anticipated.

I *know* the inverters in my monitors are going to fail. It would be nice if I knew before I was actively using one when it went dark!

That would depend on the market, right? Most of my computers have redundant "smart" (i.e., internal monitoring and reporting) power supplies. Because they were marketed to folks who wanted that sort of reliability. Because a manufacturer who didn't provide that level of AVAILABILITY would quickly lose market share. The cost of the added components and "handling" is small compared to the cost of lost opportunity (sales).

That makes assumptions about the market AND the user.

If one of my machines signals a fault, I look to see what it is complaining about: is it a power supply failure (in which case, I'm now reliant on a single power supply)? is it a memory failure (in which case, a bank of memory may have been disabled which means the machine will thrash more and throughput will drop)? is it a link aggregation error (and network traffic will suffer)?

If I can't understand these errors, then I either don't buy a product with that level of reliability *or* have someone on hand who CAN understand the errors and provide remedies/advice.

Consumers will replace a PC because of malware, trashed registry, creeping cruft, etc. That's a problem with the consumer buying the "wrong" sort of computing equipment for his likely method of use. (buy a Mac?)

Users often don't understand when a device is malfunctioning. Or, how to report the conditions and symptoms in a meaningful way.

I recall a woman I worked with ~45 years ago sitting, patiently, waiting for her computer to boot. As I walked past, she asked me how long it takes for that to happen (floppy based systems). Alarmed (I had designed the workstations), I asked "How long have you been waiting?"

Turns out, she had inserted the (8") floppy rotated 90 degrees from it's proper orientation.

How much longer would she have waited had I not walked past?

One wouldn't bother with a CD/DVD player -- they are too disposable and reporting errors won't help the user (even though you have a big ATTACHED display at your disposal!)

"For your continued video enjoyment, replace me, now!"

OTOH, if a CNC machine tries to "home" a mechanism and doesn't get (electronic) confirmation of that event having been completed, would you expect *it* to just sit there endlessly waiting? Possibly causing damage to itself in the process?

Would you expect it to "notice" if the drive motor APPEARED to be connected and was drawing the EXPECTED amount of current?

Or, would you expect an electrician to come along and start troubleshooting (taking the machine out of production in the process)?

You wouldn't pick such a power supply if that was an important failure mode to guard against! (that's why smart power supplies are so common -- and redundant!)

Then, we should all be using machines with MEGAbytes of memory...

Much consumer kit has non-ECC RAM. I'd wager many of the devices designed by folks *here* use non-ECC RAM (because support for ECC in embedded products is less common).

Is this ignorance? Or, willful naivite?

You have to be very careful that the additional complexity doesn't itself introduce new annoying failure modes. My previous car had filament bulb failure sensors (new one is LED) of which the one for the parking light had itself failed - the parking light still worked. However, the car would great me with "parking light failure" every time I started the engine and the main dealer refused to cancel it.

Repair of parking light sensor failure required swapping out the

The parking light is not even a required feature.

Monitoring temperature, voltage supply and current consumption isn't a bad idea. If they get unexpectedly out of line something is wrong. Likewise with power on self tests you can catch some latent failures before they actually affect normal operation.

A singular speculative spitball - the capacitive marker:

In-situ Prognostic Method of Power MOSFET Based on Miller Effect

... This paper presents a new in-situ prognosis method for MOSFET based on miller effect. According to the theory analysis, simulation and experiment results, the miller platform voltage is identified as a new degradation precursor ...

(10.1109/PHM.2017.8079139)

Danke,

Yes.

The old-school version was a magnetic drain plug, which one inspected for clinging iron chips or dust, also serving to trap those chips. The newer-school version was to send a sample of the dirty oil to the lab for microscope and chemical analysis. There are companies that will take your old lubrication oil and reprocess it, yielding new oil.

If there was an oil filter, inspect the filter surface.

And when one was replacing the oil in the gear case, wipe the bottom with a white rag, and look at the rag.

Nobody did electronic testing until very recently, because even expensive electronics were far too unreliable and fragile.

Joe Gwinn