Predictive failures

Current and voltages outside of normal operation?

Cheers

Checking temperatures is good. An overload or a fan failure can be bad news.

We put temp sensors on most products. Some parts, like ADCs and FPGAs, have free built-in temp sensors.

I have tried various ideas to put an air flow sensor on boards, but so far none have worked very well. We do check fan tachs to be sure they are still spinning.

Blocking air flow generally makes fan speed *increase*.

There is a standard approach that may work: Measure the level and trend of very low frequency (around a tenth of a Hertz) flicker noise. When connections (perhaps within a package) start to fail, the flicker level rises. The actual frequency monitored isn't all that critical.

Joe Gwinn

Do connections "start to fail" ?

I don't think I've ever owned a piece of electronic equipment that warned me of an impending failure.

Cars do, for some failure modes, like low oil level.

Don, what does the thing do?

Yes, they do, in things like vias. I went through a big drama where a critical bit of radar logic circuitry would slowly go nuts.

It turned out that the copper plating on the walls of the vias was suffering from low-cycle fatigue during temperature cycling and slowly breaking, one little crack at a time, until it went open. If you measured the resistance to parts per million (6.5 digit DMM), sampling at 1 Hz, you could see the 1/f noise at 0.1 Hz rising. It's useful to also measure a copper line, and divide the via-chain resistance by the no-via resistance, to correct for temperature changes.

The solution was to redesign the vias, mainly to increase the critical volume of copper. And modern SMD designs have less and less copper volume.

I bet precision resistors can also be measured this way.

Onset of smoke emission is a common sign.

The industrial method for big stuff is accelerometers attached near the bearings, and listen for excessive rotation-correlated (not necessarily harmonic) noise.

Good question.

Joe Gwinn

But nobody is going to monitor every via on a PCB, even if it were possible.

One could instrument a PCB fab test board, I guess. But DC tests would be fine.

We have one board with over 4000 vias, but they are mostly in parallel.

Big ships that I've worked on have a long propeller shaft in the shaft alley, a long tunnel where nobody often goes. They have magnetic shaft runout sensors and shaft bearing temperature monitors.

They measure shaft torque and SHP too, from the shaft twist.

I liked hiding out in the shaft alley. It was private and cool, that giant shaft slowly rotating.

You have to be very careful that the additional complexity doesn't itself introduce new annoying failure modes. My previous car had filament bulb failure sensors (new one is LED) of which the one for the parking light had itself failed - the parking light still worked. However, the car would great me with "parking light failure" every time I started the engine and the main dealer refused to cancel it.

Repair of parking light sensor failure required swapping out the

The parking light is not even a required feature.

Monitoring temperature, voltage supply and current consumption isn't a bad idea. If they get unexpectedly out of line something is wrong. Likewise with power on self tests you can catch some latent failures before they actually affect normal operation.

It was not possible to test the vias on the failing logic board, but we knew from metallurgical cut, polish, and inspect studies of failed boards that it was the vias that were failing.

What was being tested was a fab test board that had both the series via chain path and the no-via path of roughly the same DC resistance, set up so we could do 4-wire Kelvin resistance measurements of each path independent of the other path.

This can also be tested , but using a 6.5-digit DMM intended for measuring very low resistance values. A change of one part in 4,000 is huge to a 6.5-digit instrument. The conductivity will decline linearly as vias fail one by one.

Yep. And these kinds of things fail slowly. At first.

Probably had a calming flowing water sound as well.

Joe Gwinn

Yes, but the question was whether one could predict the failure of an operating electronic gadget. The answer is mostly NO.

We had a visit from the quality team from a giant company that you have heard of. They wanted us to trend analyze all the power supplies on our boards and apply a complex algotithm to predict failures. It was total nonsense, basically predicting the future by zooming in on random noise with a big 1/f component, just like climate prediction.

Millikelvin temperature changes would make more signal than a failing via.

They could repair a bearing at sea, given a heads-up about violent failure. A serious bearing failure on a single-screw machine means getting a seagoing tug.

The main engine gearbox had padlocks on the covers.

There was also a chem lab to analyze oil and water and such, looking for contaminamts that might suggest something going on.

Yes, cool and beautiful and serene after the heat and noise and vibration of the engine room. A quiet 32,000 horsepower.

It was fun being an electronic guru on sea trials of a ship full of big hairy Popeye types. I, skinny gawky kid, got my own stateroom when other tech reps slept in cots in the hold.

Have you noticed how many lumberjack types are afraid of electricity? That can be funny.

The real way to reduce failure rates is by designing carefully.

Sometimes BIST can help ensure that small failures won't become board-burning failures, but an RMA will happen anyhow.

I just added a soft-start feature to a couple of boards. Apply a current-limited 48 volts to the power stages before the real thing is switched on hard.

Built-in self test is mostly auto-calibration. You can use temperature sensitive components for precise measurements if you calibrate out the temperature shift and re-calibrate if the measured temperature shifts appreciably (or every few minutes).

It might also take out the effects of dopant drift in a hot device, but it wouldn't take it out forever.

Soft-start has been around forever. If you don't pay attention to what happens to your circuit at start-up and turn-off you can have some real disasters.

At Cambridge Instruments I once replaced all the tail resistors in a bunch of class-B long-tailed-pair-based scan amplifiers with constant current diodes. With the resistors tails, the scan amps drew a lot of current when the 24V rail was being ramped up and that threw the 24V supply into current limit, so it didn't ramp up. The constant current diodes stopped this (not that I can remember how).

This was a follow-up after I'd brought in to stop the 24V power supply from blowing up (because it hadn't had a properly designed current limit).

The problem had shown up in production - where it was known as the three back problem because when things did go wrong the excursions on the 24V rail destroyed three bags of components.

Agree.

Hmm. My first instinct was that they were using MIL-HNBK-317 (?) or the like, but that does not measure noise. Do you recall any more of what they were doing? I might know what they were up to. The military were big on prognostics for a while, and still talk of this, but it never worked all that well in the field compared to what it was supposed to improve on.

Not at the currents in that logic card. Too much ambient thermal noise.

Oh yes. And EEs frightened by a 9-v battery.

Joe Gwinn

Yes I've seen that a lot. The power rails in the production product came up in a different order to those in the development lab. This caused all kinds of previously unseen behaviour including an expensive flash a/d chip burning up.

I'd have it in the test spec that any missing power rail does not cause issues. And any power rail can be turned on and off any time. The equipment may not work properly with a missing power rail but it should not be damaged.

I had an intern, an EE senior, who was afraid of 3.3 volts.

I told him to touch an FPGA to see how warm it was getting, and he refused.

Some FPGAs require supply sequencing, as may as four.

LM3880 is a dedicated powerup sequencer, most cool.

Ok that doesn't surprise me. I'd want to be sure that the requirement is always met even when the 12V connector is in a position where it isn't sure whether it's connected or not. Or rapid and repeated connect/disconnect of 12V doesn't cause any issue.

We considered the brownout case. The MAX809 handles that.

This supply will also tolerate +24v input, in case someone grabs the wrong wart. Or connects the power backwards.

That's fine if the failure is clearly in the equipment itself, but what if it is in the way it interacts with something outside it, some unpredictable or unrecognised input codition? It works perfectly on the bench, only to fail when put into service ...again and again.

On a vaguely related rant, shamelessly hijacking your thread:

Why do recent mechanical hard drives have a "Annualised Workload Rate" limit saying that you are only supposed to write say 55TB/year?

What is the wearout mechanism, or is it just bullshit to discourage enterprise customers from buying the cheapest drives?

It seems odd to me that they would all do it, if it really is just made up bullshit. It also seems odd to express it in terms of TB read+written. I can't see why that would be more likely to wear it out than some number of hours of spindle rotation, or seek operations, or spindle starts, or head load/unload cycles. I could imagine they might want to use a very high current density in the windings of the write head that might place an electromigration limit on the time spent writing, but they apply the limit to reads as well. Is there something that wears out when the servo loop is keeping the head on a track?