Approach to Finding the Root Cause of Failures

That's the Sherlock Holmes technique. It doesn't work very well. The list of NOTs to test is too big, and you are unlikely to include in the list the things you missed when you did the design.

--
John Larkin         Highland Technology, Inc 
picosecond timing   precision measurement  
 Click to see the full signature
Reply to
John Larkin
Loading thread data ...

Definitely can be fooled by the test itself. In my case a lot of times it has to do with common mode noise getting into the mearurements.

Yes, I would call it a step past troubleshooting but troubleshooting is a big part of it.

The phrase I like is that you have to "understand the problem"

That I normally do in the engineering phase but it's usually some error in the original engineering. Something designed on the edge and not worse case or even the documentation lacking enough information.

boB

Reply to
boB

One thing that helps to find intermittents is temperature testing. If you temp test new designs, you'll have a lot fewer bugs later.

--
John Larkin         Highland Technology, Inc 
picosecond timing   precision measurement  
 Click to see the full signature
Reply to
John Larkin

hat is rare, intermittent or obscure?

ure than I was when I was doing more design work. In many ways I think it is more challenging than design work. It takes a mindset that is differen t than design.

not being fooled by the results of your test

eighting factor of 1 to anything until you know you have the problem solved

an opposite conclusion when you repeat a test than what you concluded after the first test.

had they care and are smart, on the other hand if you go about chasing othe r peoples ideas (often conceived of to just demonstrate they are concerned in a meeting) you will never get an a clear path to troubleshoot the proble m in your own way.

matic.

esign phase, I no longer look at that as a curse, but as a blessing. It is going to come back and get you later.

d. As a designer you can show a days work for a days pay. In root cause y ou feel like you have accomplished nothing for a long time. Frequently, th ough , these problems are the most visible problems in an organization and can make a difference between losing a customer and keeping one.

ou find contradictions in your thinking.

I do not think I have ever designed something (not counting testing items a nd a few R&D jobs) that has not required temperature operation from -55C to +70C. I never really thought of that until now.

Reply to
blocher

-55 is severe. Did you test beyond the required range? A lot of things are first-order compensated for temperature.

A lot of timing/race conditions are temperature dependant. One guy we worked with used the wrong clock edge to strobe ADC data into an FPGA at 250 MHz. THAT was temperature dependant!

--
John Larkin         Highland Technology, Inc 
picosecond timing   precision measurement  
 Click to see the full signature
Reply to
John Larkin

Ah, the Sherlock Holmes technique.

Fails dismally because people's imagination is finite and the number of "Nots" is infinite.

Reply to
Tom Gardner

Are you quoting that WRT the input or the output? PSRR and CMRR are normally quoted input-referred, i.e. to find out the effect you have to multiply by the overall gain.

There are lots of parts that can have negative-dB PSRR as referred to the output.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs 
Principal Consultant 
 Click to see the full signature
Reply to
Phil Hobbs

e that is rare, intermittent or obscure?

ailure than I was when I was doing more design work. In many ways I think it is more challenging than design work. It takes a mindset that is diffe rent than design.

for not being fooled by the results of your test

a weighting factor of 1 to anything until you know you have the problem sol ved

aw an opposite conclusion when you repeat a test than what you concluded af ter the first test.

ne had they care and are smart, on the other hand if you go about chasing o ther peoples ideas (often conceived of to just demonstrate they are concern ed in a meeting) you will never get an a clear path to troubleshoot the pro blem in your own way.

blematic.

e design phase, I no longer look at that as a curse, but as a blessing. It is going to come back and get you later.

bad. As a designer you can show a days work for a days pay. In root caus e you feel like you have accomplished nothing for a long time. Frequently, though , these problems are the most visible problems in an organization a nd can make a difference between losing a customer and keeping one.

p you find contradictions in your thinking.

s and a few R&D jobs) that has not required temperature operation from -55C to +70C. I never really thought of that until now.

We typically do not test below -55. Things are usually pretty good to -40 then the last 15 degrees can be squirrely. Most of the parts are specified to -40 (no surprise). They still work at -55 just unknown parameters can fall out of spec. On the hot side we frequently run the units to 90 degree s or hotter to find find failures to see what breaks first. 85 is not usu ally too bad.

At the moment we have a design where the SPI bus is not reading correctly a t +70C. It was a copy of another design that we completely validated at +7

0 degrees. that is how corner cases work.
Reply to
blocher

that is rare, intermittent or obscure?

ilure than I was when I was doing more design work. In many ways I think i t is more challenging than design work. It takes a mindset that is differ ent than design.

or not being fooled by the results of your test

weighting factor of 1 to anything until you know you have the problem solv ed

w an opposite conclusion when you repeat a test than what you concluded aft er the first test.

e had they care and are smart, on the other hand if you go about chasing ot her peoples ideas (often conceived of to just demonstrate they are concerne d in a meeting) you will never get an a clear path to troubleshoot the prob lem in your own way.

lematic.

design phase, I no longer look at that as a curse, but as a blessing. It is going to come back and get you later.

bad. As a designer you can show a days work for a days pay. In root cause you feel like you have accomplished nothing for a long time. Frequently, though , these problems are the most visible problems in an organization an d can make a difference between losing a customer and keeping one.

you find contradictions in your thinking.

The NOT technique is a last resort. (it's how I found a leaky toggle switch... we had a bag of leaky switches, most circuits didn't care if there is a few meg ohm of resistance.) Before you pull all your hair out, you pull all the components out and replace 'em. But how do you know the replacement component is good! Quickly a knotty nightmare.

I find it best to get as much data as possible, and then sleep on it*. When you think about it 'actively' you tend to get stuck in your first assumption rut. (And if your first assumption had been right, it'd be fixed/found already. :^)

George H.

*or go explain the problem to someone else... not that they will be able to help (well they might) but because having to explain it makes you go over the whole circuit and may remind you of the part you haven't been thinking about.
Reply to
George Herold

I used to call it failure mode analysis.

Reply to
jurb6006

We had an Analog Devices SPI ADC that was flakey with temperature. They were no help. We spun the board and used a TI part.

--
John Larkin         Highland Technology, Inc 
picosecond timing   precision measurement  
 Click to see the full signature
Reply to
John Larkin

re that is rare, intermittent or obscure?

failure than I was when I was doing more design work. In many ways I think it is more challenging than design work. It takes a mindset that is diff erent than design.

for not being fooled by the results of your test

a weighting factor of 1 to anything until you know you have the problem so lved

raw an opposite conclusion when you repeat a test than what you concluded a fter the first test.

one had they care and are smart, on the other hand if you go about chasing other peoples ideas (often conceived of to just demonstrate they are concer ned in a meeting) you will never get an a clear path to troubleshoot the pr oblem in your own way.

oblematic.

he design phase, I no longer look at that as a curse, but as a blessing. I t is going to come back and get you later.

s bad. As a designer you can show a days work for a days pay. In root cau se you feel like you have accomplished nothing for a long time. Frequently , though , these problems are the most visible problems in an organization and can make a difference between losing a customer and keeping one.

lp you find contradictions in your thinking.

when it is laid out under their nose. It is never their fault :-)

had missed. (Who expects 0 dB?) On the customer's work bench they were ge tting noise in the audio that turned out to be from the DSP power consumpti on. They were using clip leads to provide power to the UUT and the on boar d capacitance wasn't enough to mitigate it. We told them to use better pow er connections and also used a larger cap.

piece of work they are. The other CP Clare part had a problem that virtual ly made it unusable, but they didn't point it out in the data sheet. I won der if they actually use engineers or if they just let high school kids des ign their ICs?

At higher frequencies aren't there many opamps that cross

0 dB PSRR. At least for one of the rails. (That's why God* invented the cap. multiplier.)

George H.

*or one of his offspring.... who did do the cap mult. first?

Reply to
George Herold

-------------------------------

** IOW another mindless troll.

** Analyse the actual failure first.

Something good service techs do every day, but few designers have a clue about.

Your dopey rules are all context free generalizations, so totally meaningless.

...... Phil

Reply to
Phil Allison

re that is rare, intermittent or obscure?

failure than I was when I was doing more design work. In many ways I think it is more challenging than design work. It takes a mindset that is diff erent than design.

for not being fooled by the results of your test

a weighting factor of 1 to anything until you know you have the problem so lved

raw an opposite conclusion when you repeat a test than what you concluded a fter the first test.

one had they care and are smart, on the other hand if you go about chasing other peoples ideas (often conceived of to just demonstrate they are concer ned in a meeting) you will never get an a clear path to troubleshoot the pr oblem in your own way.

oblematic.

he design phase, I no longer look at that as a curse, but as a blessing. I t is going to come back and get you later.

s bad. As a designer you can show a days work for a days pay. In root cau se you feel like you have accomplished nothing for a long time. Frequently , though , these problems are the most visible problems in an organization and can make a difference between losing a customer and keeping one.

lp you find contradictions in your thinking.

when it is laid out under their nose. It is never their fault :-)

had missed. (Who expects 0 dB?) On the customer's work bench they were ge tting noise in the audio that turned out to be from the DSP power consumpti on. They were using clip leads to provide power to the UUT and the on boar d capacitance wasn't enough to mitigate it. We told them to use better pow er connections and also used a larger cap.

piece of work they are. The other CP Clare part had a problem that virtual ly made it unusable, but they didn't point it out in the data sheet. I won der if they actually use engineers or if they just let high school kids des ign their ICs?

This was a telephone line isolation interface. One end was connected to th e phone line, an isolation capacitor (high frequency chopper) crossed the i solation barrier and the other side of the chip connected to the low voltag e CODEC circuit.

Not sure it matters if the spec was input or output referred since the circ uit has no gain, just isolation.

We had some low level audio frequency noise on the power rail (10 mV comes to mind) which showed up in the data as an audible tone which corresponded to the processing loop of the DSP. 10 mV seems like an acceptable amount o f noise in a power supply line, but I suppose normally PS noise is outside the audible range. The noise wasn't loud, but present. The fact that it c ame and went was what make it noticeable.

Compare to op amps where I typically see a large amount of PSRR in the audi o range, some 50 dB and up. The impact of 10 mV audio noise would not be m easurable in most op amp circuits.

--
  Rick C. 

  -+ Get 1,000 miles of free Supercharging 
 Click to see the full signature
Reply to
Rick C

I say to myself, "This was designed by an idiot. What stupid mistake did he make?"

There is a tendency to blame parts, when the problem is usually design.

--
John Larkin         Highland Technology, Inc 

Science teaches us to doubt. 
 Click to see the full signature
Reply to
jlarkin

The Cambridge Instruments example of that problem was an electron beam micr ofabricator which wrote patterns fine, but sometimes bits of the pattern we re half a micron away from where they ought to be.

We had to fly our chief engineer to America to sort it out - he was a brill iant diagnostician, but what turned out to be crucial was that he was an en gineering history buff. When he got into the lift to go up to see the machi ne he said "This is a hydraulic lift" which it was.

The large lump of wrought iron that served as the piston pushed up by water pressure was magnetic, and when the lift was up the ambient magnetic field at the electron beam microfabricator changed enough to move the electron b eam half a micron.

It became a legendary "super suss". The lab had to stop the lift from being used while a pattern was being written, which wasn't much of an inconvenie nce, and a lot cheaper than any other possible cure.

--
Bill Sloman, Sydney
Reply to
Bill Sloman

Whew, pot calling the kettle black!

--
  Rick C. 

  -+ Get 1,000 miles of free Supercharging 
 Click to see the full signature
Reply to
Rick C

This isn't actually useful. There are plenty of idiots around, so it can be productive, but there are a lot of circuits where you have to look quite h ard to see what the designer was worried about and work out what some osten sibly strange feature was intended to deal with.

Writing something off to idiocy prematurely could leave you with egg all ov er your face.

Or the lack of it.

in one circuit I was working on there was an op amp that clearly should hav e been oscillating, but it's output was loaded with a 100nF ceramic capacit or.

When I looked hard, it was actually oscillating, but at a very low amplitud e.

When I put in a more classical solution the oscillation went away and the D C offset at the amplifier went down.

The machine involved made about 95% of the single crystal GaAs manufactured in the free world at that time, and it may have had a little less thermal stress if made in machines that had been retrofitted with my version of th at circuit.

--
Bill Sloman, Sydney
Reply to
Bill Sloman

Identify the earliest necessary or possible precursor to the failure symptom, and search for that instead.

CH

Reply to
Clifford Heath

I guess i succeeded in my post then?

Reply to
blocher

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.