Predictive failures

Replace drive and restore backup. It's happened a few times, and a friend had one of those 16 GB but looks like 1 TB to the OS SSDs from Amazon.

I don't mind how you pronounce it.

...

Reminds me of a tenant who just removed the battery to stop the annoying beeping.

...

Yes, but the question was whether one could predict the failure of an operating electronic gadget. The answer is mostly NO.

We had a visit from the quality team from a giant company that you have heard of. They wanted us to trend analyze all the power supplies on our boards and apply a complex algotithm to predict failures. It was total nonsense, basically predicting the future by zooming in on random noise with a big 1/f component, just like climate prediction.

Millikelvin temperature changes would make more signal than a failing via.

They could repair a bearing at sea, given a heads-up about violent failure. A serious bearing failure on a single-screw machine means getting a seagoing tug.

The main engine gearbox had padlocks on the covers.

There was also a chem lab to analyze oil and water and such, looking for contaminamts that might suggest something going on.

Yes, cool and beautiful and serene after the heat and noise and vibration of the engine room. A quiet 32,000 horsepower.

It was fun being an electronic guru on sea trials of a ship full of big hairy Popeye types. I, skinny gawky kid, got my own stateroom when other tech reps slept in cots in the hold.

Have you noticed how many lumberjack types are afraid of electricity? That can be funny.

My experience has been the smoke detectors too close (as the smoke travels) to the kitchen tend to suffer mysterious disablement. Relocation of the smoke detector usually solves the problem.

Joe Gwinn

The real way to reduce failure rates is by designing carefully.

Sometimes BIST can help ensure that small failures won't become board-burning failures, but an RMA will happen anyhow.

I just added a soft-start feature to a couple of boards. Apply a current-limited 48 volts to the power stages before the real thing is switched on hard.

Very interesting but are there any products out there which make use of this or other prognostic methods to provide information on remaining useful life?

I don't mind how you pronounce it.

Two comparators can make a window detector which will tell you whether some parameter is in a specified range. And it doesn't need monthly updates because software is never finished.

Reminds me of a tenant who just removed the battery to stop the annoying beeping. Better to inform the individual who can get the replacement done when the tenant isn't even home.

Well I don't recall putting anything much into a design which could predict remaining life. The only exceptions, also drawing from other replies in this thread, might be be temperature sensing, voltage sensing, current sensing, air flow sensing, noise sensing, iron in oil sensing, and any other kind of sensing which might provide information on parameters outside or getting close to outside expected range. Give that to some software which also knows how long the equipment has been in use, how often it has been used, what the temperature and humidity was, how long it's been since the oil was changed, and you might be able to give the operator useful information about when to schedule specific maintenance. But don't give the software too much control. I don't want to be told that I can't use the equipment because an oil change was required 5 minutes ago and it hasn't been done yet.

...

Oh yes I've had that too. I call them burnt toast detectors.

Yes, but you are limited in the relationships that you can encode in hardware -- because they are "hard-wired".

Software is finished when the design is finalized. When management fails to discipline itself to stop the list of "Sales would like..." requests, then it's hard for software to even CLAIM to be finished.

"Dinner will be served at the sound of the beep".

Most places have specific rules regarding the placement of smoke detectors to 1) ensure safety and 2) avoid nuisance alarms. (I was amused to disover that our local fire department couldn't cite the local requirements when I went asking!)

Add CO and heat detectors to the mix and they get *really* confused!

So, a WiFi/BT link to <whatever>? Now the simple smoke detector starts suffering from feeping creaturism. "Download the app..."

Remind the occupants once a week (requiring acknowledgement) starting a month prior to ANTICIPATED battery depletion. When the battery is on it's last leg, you can be a nuisance.

Folks will learn to remember at the first (or second or third) reminder in order to avoid the annoying nuisance behavior that is typical of most detectors. (there is not a lot of $aving$ to replacing the battery at the second warning instead of at the "last minute")

Most people don't. Most people don't design for high availability

When you have hundreds/thousands of devices in a single installation, then you REALLY don't want to have to be playing wack-a-mole with whichever devices have crapped out, TODAY. If these *10* are likely to fail in the next month, then replace all ten of them NOW, when you can fit that maintenance activity into the production schedule instead of HAVING to replace them when they DISRUPT the production schedule.

I have all of those things -- with the exception of knowing which sensory data is most pertinent to failure prediction. I can watch to see when one device fails and use it to anticipate the next failure. After many years (of 24/7 operation) and many sites, I can learn from actual experience.

Just like I can learn that you want your coffee pot started 15 minutes after you arise -- regardless of time of day. Or, that bar stock will be routed directly to the Gridley's. Because that's what I've *observed* you doing!

Advisories are just that; advisories. They are there to help the user avoid the "rudeness" of a piece of equipment "suddenly" (as far as the user is concerned) failing. They add value by increasing availability.

If you choose to ignore the advisory (e.g., not purchasing a spare to have on hand for that "imminent" failure), then that's your perogative. If you can afford to have "down time" and only react to ACTUAL failures, then that's a policy decision that YOU make.

OTOH, if there is no oil in the gearbox, the equipment isn't going to start; if the oil sensor is defective, then *it* needs to be replaced. But, if the gearbox is truly empty, then it needs to be refilled. In either case, the equipment *needs* service -- now. Operating in this FAILED state presumably poses some risk, hence the prohibition.

Built-in self test is mostly auto-calibration. You can use temperature sensitive components for precise measurements if you calibrate out the temperature shift and re-calibrate if the measured temperature shifts appreciably (or every few minutes).

It might also take out the effects of dopant drift in a hot device, but it wouldn't take it out forever.

Soft-start has been around forever. If you don't pay attention to what happens to your circuit at start-up and turn-off you can have some real disasters.

At Cambridge Instruments I once replaced all the tail resistors in a bunch of class-B long-tailed-pair-based scan amplifiers with constant current diodes. With the resistors tails, the scan amps drew a lot of current when the 24V rail was being ramped up and that threw the 24V supply into current limit, so it didn't ramp up. The constant current diodes stopped this (not that I can remember how).

This was a follow-up after I'd brought in to stop the 24V power supply from blowing up (because it hadn't had a properly designed current limit).

The problem had shown up in production - where it was known as the three back problem because when things did go wrong the excursions on the 24V rail destroyed three bags of components.

Perhaps this popular application "rings a bell"?

Battery and System Health Monitoring of Battery-Powered Smart Flow Meters Reference Design

Danke,

Agree.

Hmm. My first instinct was that they were using MIL-HNBK-317 (?) or the like, but that does not measure noise. Do you recall any more of what they were doing? I might know what they were up to. The military were big on prognostics for a while, and still talk of this, but it never worked all that well in the field compared to what it was supposed to improve on.

Not at the currents in that logic card. Too much ambient thermal noise.

Oh yes. And EEs frightened by a 9-v battery.

Joe Gwinn

...

No thanks. I have the same view of cameras. They won't be connecting outbound to a server anywhere in the world. But the average user does not know that and just wants the pictures on their phone.

That's one reason why you want feedback from people who use your equipment.

Preferably by me purchasing a new sensor and being able to replace it myself.

Yes I've seen that a lot. The power rails in the production product came up in a different order to those in the development lab. This caused all kinds of previously unseen behaviour including an expensive flash a/d chip burning up.

I'd have it in the test spec that any missing power rail does not cause issues. And any power rail can be turned on and off any time. The equipment may not work properly with a missing power rail but it should not be damaged.

There is no need for a manufacturer to interpose themselves in such "remote access". Having the device register with a DDNS service cuts out the need for the manufacturer to essentially provide THAT service.

OTOH, the manufacturer wants to "keep selling toilet paper" and has used that business model to underwrite the cost of the "device".

Everything, here, is wired. And, my designs have the same approach: partly because I can distribute power over the fabric; partly because it removes an attqck surface (RF jamming); partly for reliability (no reliance on services that the user doesn't "own"); partly for privacy (no information leaking -- even by side channel inference). My wireless connections are all short range and of necessity (e.g., the car connects via wifi so the views from the external cameras can be viewed on it's LCD screen as it pulls out/in).

All they know is when something BREAKS. But, my device also knows that (unless EVERY "thing" breaks at the same time). The devices can capture pertinent data to adjust it's model of when those other devices are likely to suffer similar failures because the failed device shared its observations with them (via a common knowledge base).

If it makes sense to do so. Replacing a temperature sensor inside an MCU is likely not cost effective, something folks are capable of doing nor supported as an FRU.

Not for most users here. They tried to put me on lsn/cgnat not long ago. After complaining I was given a free static IPv4. Most users wouldn't know DDNS from a banana, and will expect it to work out of the box after installing the app on their phone.

Most folks, here, effectively have static IPs -- even if not guaranteed as such. But, most also have AUPs that prohibit running their own servers (speaking about consumers, not businesses).

There's no reason the app can't rely on DDNS. Infineon used to make a series of "power control modules" (think BSR/X10) for consumers. You could talk to the "controller" -- placed on YOUR network -- directly. No need to go THROUGH a third party (e.g., Infineon).

If you wanted to access those devices (through the controller) from a remote location, the controller -- if you provided internet access to it -- would register with a DDNS and you could access it through that URL.

It is only recently that vendors have been trying to bake themselves into their products. Effectively turning their products into "rentals". You can buy a smart IP camera that will recognize *people*! For $30. Plus $6/month -- forever! (and, if you stop paying, you have a nice high-tech-looking PAPERWEIGHT).

I rescued another (APC) UPS, recently. I was excited that they FINALLY had included the NIC in the basic model (instead of as an add-in card as it has historically been supplied - at additional cost).

Ah, but you can't *talk* to that NIC! It exists so the UPS can talk to the vendor! Who will let you talk to them to get information about YOUR UPS.

So, you pay for a NIC that you can't use -- unless you agree to their terms (I have no idea if there is a fee involved or if they just want to spy on your usage and sell you batteries!)

In addition to the sleeze factor, it's also a risk. Do you know what the software in the device does? Are you sure it is benevolent? And, not snooping your INTERNAL network (it's INSIDE your firewall)? Maybe just trying to sort out what sorts of hardware you have (for which they could pitch additional products/services)? Are you sure the product (if benign) can't be hacked and act as a beachhead for some other infestation?

And, what, exactly, am I *getting* for this risk that I couldn't get with the "old style" NIC?

Sounds like they are really measuring gate threshold, or gate transfer curve, drift with time. That happens and is usually no big deal, in moderation. Ions and charges drift around. We don't build opamp front-ends from power mosfets.

This doesn't sound very useful for "in-situ" diagnostics.

GaN fets can have a lot of gate threshold and leakage change over time too. Drive them hard and it doesn't matter.

But vendors know that most people want it easy so the push towards subscription services and products which phone home isn't going to change.

Most people don't know or care what their products are sending to the vendor.

I like to see what is connecting to what with

I can also remote desktop from anywhere of my choice, with the rest of the world unable to connect.

Pretty much all of my online services are either restricted to specific IPs (cameras, remote desktop and similar). Or they have one or more countries and other problem IPs blocked. (web sites and email services).

None of that is possible when the vendor is in control because users will want their camera pictures available anywhere.