complexity, perversity, general evil

Nov 21, 2012 68 Replies

OK, in general, if you have a complex system with hundreds or thousands of causalities, there will be zillions of causal paths, many of which will cause system failures. Most of them will be not-obvious, because we've aleady thought about the easy ones and made sure they aren't going to make trouble. So there are, in effect, evil genies inside our systems creating intermittent problems just to torment us.



Less general: we have four products that have the same general control core: an NXP LPC3250 ARM processor, serial flash chip, FPGA, STM1818T powerup reset chip. At powerup time, the ARM reads our small boot loader program from flash, into CPU ram, and runs it. The boot in turn reads a small descriptor and then a big block of data to configure the FPGA. It then reads another small descriptor and then loads the application code, also into CPU RAM, and runs that.



Two of the products work fine. Two others have the following behavior: They fail to run when first turned on. If you power them down and up again, they are fine thereafter. If you store them for a few months, they do the first-time-crash thing again. My innovation is to bake them overnight at 120C, which has the same effect as longterm storage, namely it won't work at first powerup. That at least lets us try a few experiments per day.



Just for fun, we soaked one in water (to discharge every exposed electrical node on the board) and dried it at moderate temperature. This did not induce the failure mode. So if the time delay is an electrical node discharge, it's inside a chip.



All four products use parts out of the same bins. Two always work, the other two almost always have the startup problem. OK, we could have all sorts of hardware or software bugs, but what's with the months-long time constant?



John Larkin Highland Technology, Inc jlarkin at highlandtechnology dot com http://www.highlandtechnology.com Precision electronic instrumentation Picosecond-resolution Digital Delay and Pulse generators Custom laser drivers and controllers Photonics and fiberoptic TTL data links VME thermocouple, LVDT, synchro acquisition and simulation

CMOS memory, or perhaps DRAM that is not being initialized? Heat soaking could discharge the memory cells faster. An un-initialized memory location is one of those classic computer errors from the earliest days. Can you make your code zero all memory at startup? Does that change behavior? (This, of course, could make it NEVER start correctly, but then you'd have a solid bug to chase.)

Jon

does it try to load code from flash or is it just dead?

I remember many years ago having problems with a serial flash if the data pin or clock pin was at the wrong state the first time you asserted CE it just stopped working

-Lasse

Gut-level hunch: you've got some code (or some logic) which is dependent on the state of one or more bits of SRAM or DRAM which are not being explicitly initialized or reset. You're getting whatever state happens to be present in the memory as it powers up... this is often rather random, but I can well suspect that it may be highly dependent on the chipset process, small variations in the size or leakage of some transistors, etc. A single flip-flop might come up "true" or "false" depending on which transistor in a pair leaks more or less.

I suspect that both the long time constant, and the ability to induce failure by baking, may reflect slight changes in the leakage of in-chip components and insulators.

A race condition at power-up (e.g. reset timing, or the rise time and sequencing of multiple power supplies) might have a similar effect.

Dave Platt AE6EO Friends of Jade Warrior home page: http://www.radagast.org/jade-warrior I do _not_ wish to receive unsolicited commercial email, and I will boycott any company which has the gall to send me such ads!

Marginal rise time on the CR reset circuit, use a reset chip.

Cheers

Syd

Trouble with internal nodes is, DRAM time constants are on the order of hours, even when cool. (Which is a security attack method: while the target computer is running with whatever data in memory you're after, unplug it, quickly remove the memory module and plug it into your favorite DIMM reader board. Especially if cool, most of the cells likely contain the data you were after.)

Tim

-- Deep Friar: a very philosophical monk. Website:

formatting link

It seems to load and run our first chunk of code, our boot loader, OK. We seem to read the little headers OK, too. But the FPGA config fails, and the application code fails. So maybe we can read small chunks of serial flash but not big ones.

The data-out pin on the serial flash has to be pulled down to tell the ARM what address format the flash uses; we do that.

John Larkin Highland Technology, Inc jlarkin at highlandtechnology dot com http://www.highlandtechnology.com Precision electronic instrumentation Picosecond-resolution Digital Delay and Pulse generators Custom laser drivers and controllers Photonics and fiberoptic TTL data links VME thermocouple, LVDT, synchro acquisition and simulation

Could well be. There's no DRAM, but a classic 4-transistor SRAM cell can remember its state for a long, long time. The flop could be in CPU ram, or in some control register. So we could have a case of a variable or a register bit being used before it's properly initialized. My software guys are looking at that but can't see anything obvious. The C compiler is supposed to warn us about using uninitialized variables, but it won't know much about hardware registers.

Yeah, that too, but the long time constant is perplexing. We tried hard discharging all the supplies, to simulate the months of storage, and it didn't cause the hangup.

John Larkin Highland Technology, Inc jlarkin at highlandtechnology dot com http://www.highlandtechnology.com Precision electronic instrumentation Picosecond-resolution Digital Delay and Pulse generators Custom laser drivers and controllers Photonics and fiberoptic TTL data links VME thermocouple, LVDT, synchro acquisition and simulation

Floating FET gate?

Failure does not prove something is impossible, failure simply indicates you are not using the right tools... nico@nctdevpuntnl (punt=.) --------------------------------------------------------------

We use the STM1818T, and the reset waveform looks fine.

We do have a pushbutton reset, wired into the ST chip. If a new board powers up hung, pushing the button fixes it; it doesn't need a full power cycle, just a hard reset. Then it's happy for weeks or months.

John Larkin Highland Technology, Inc jlarkin at highlandtechnology dot com http://www.highlandtechnology.com Precision electronic instrumentation Picosecond-resolution Digital Delay and Pulse generators Custom laser drivers and controllers Photonics and fiberoptic TTL data links VME thermocouple, LVDT, synchro acquisition and simulation

e
n

but it is unlikely that there is any dynamic memory on chip it is all flipflops and sram

and all that is really needed is that the program counter is set so it starts executing code at the right place in ROM where the code that does the booting is

but there is of course alway the risk that the ROM has a bug and forgets to set a register before it checks a boot select pin or something but for that to do something random that register would have to not have been reset

-Lasse

I thought all of Larkin's designs always worked >:-} ...Jim Thompson

| James E.Thompson, CTO | mens | | Analog Innovations, Inc. | et | | Analog/Mixed-Signal ASIC's and Discrete Systems | manus | | Phoenix, Arizona 85048 Skype: Contacts Only | | | Voice:(480)460-2350 Fax: Available upon request | Brass Rat | | E-mail Icon at http://www.analog-innovations.com | 1962 | I love to cook with wine. Sometimes I even put it in the food.

Probably, but there are millions of them.

John Larkin Highland Technology, Inc jlarkin at highlandtechnology dot com http://www.highlandtechnology.com Precision electronic instrumentation Picosecond-resolution Digital Delay and Pulse generators Custom laser drivers and controllers Photonics and fiberoptic TTL data links VME thermocouple, LVDT, synchro acquisition and simulation

could you swap out the flash on a board to isolate if the problem is the flash or the CPU?

-Lasse

That which is most obviously correct, beyond any need of checking, is usually the problem.

Any supercaps onboard? It could be slowly self discharging over the long time period. I think (not sure) that they self-discharge faster when hot. If the supercap is in some way connected to the STM1818T, a longer charge time for the supercap might explain why it always starts on the 2nd try.

Any unconnected inputs? Never mind the net list. It might be a PCB fabrication error. Tediously walk a scope probe through the multitude of IC pins and see if anything is flopping in the breeze.

Check the PCB's for low resistivity from contamination. I always put test pads on my (now ancient) boards to allow testing for board leakage. While probably not really important for test equipment, it's vital for anything that has been in a marine environment[1]. If the PCB is loaded with some kind of ionic contaminant, any moisture will cause some conduction. However, because your heat soak also evaporated any moisture, humidity induced low resistivity is unlikely.

[1] My background is in marine radios.
Jeff Liebermann jeffl@cruzio.com 150 Felker St #D http://www.LearnByDestroying.com Santa Cruz CA 95060 http://802.11junk.com Skype: JeffLiebermann AE6KS 831-336-2558

OK, I'm sure you've done this, but my first thought is to watch how all the various power supplies turn on. A v(-) here before a v(+) there latches some thing over on the other board.....

George H.

Not easy in the surface mount world we live in, but if you could isolate the power pins to some critical chips, you could then isolate which chip is the culprit.

Baking chips is the classic way to discharge nodes insides chips. This should only be needed if some "beam" device was diddling the chip, like a FIB repair. That is, if the design is good.

This can be a chip design bug. You may recall my comment on POR design where you had to insure there were diodes to insure that the supply powering down also discharged internal nodes, else the POR timing could be off. In fact, you should verify the POR chip's first pulse (after the device has been sitting long enough) is clean.

For static CMOS logic, every signal node goes had diodes to both rails (maybe not for SOI). So they always behave well. But get a little clever in the logic design (domino or whatever you want to call it) or use a little analog circuitry and all bets are off. Discharge paths have to be eyeballed in design.

While Thompson would like to blame the board designer, this sounds like a chip problem. Well unless there is some register or memory location not being set.

Those kinds of problems are very difficult to find. You have to rethink and re-test everything step by step; and after all you find something very stupid. Like, uninitialized variable somewhere. Power sequencing, racing condition, intermittent shoot through or bus conflict or something is accessed before it gets out of reset state.

BTW. If an SMPS is synchronized to external clock, there is small glitch at the moment when the clock is turned on or off. That could cause a sequence of events... it took me a month to find the cause.

Vladimir Vassilevsky DSP and Mixed Signal Consultant

formatting link

That was crossing my mind, also. But it's likely that John is running very similar (perhaps identical) code on all four for operating the serial flash. Which would mean something electrical is different, though given the simplicity of the interface, not likely much... John did the obvious and made certain is wasn't some external cap holding charge.

I'm still wondering about the flash, though. One thing for sure -- what is stored on the flash is different. Similar routines and code and data may be located in different places in the flash. There may be problems with the flash, generally, that are in places which do not cause a problem for two... but do cause a problem for the other two. I'm thinking about the floating gates and perhaps some weaknesses in writing some areas vs other areas in the chip, which show up during readback... and, because the physical positioning of the data in the flash is different from product to product, it may show up in this way. Might have "month" delays for something like this, though I'm not sure exactly how repowering a unit would fix the issue for a while and then it would come back.

I wonder if playing around with varying, arbitrary delays at the start of the software might have an impact. It's very cheap to try, at least. Even if there is no good reason to believe it will have an effect. So I'd try it, merely on the chance that something might come of it.

Intriguing problem, though.

Jon

[snip]

Where did you get that hare-brained idea? Maybe I complained of a PCB fault, but I certainly don't recall doing so.

I have had foundry problems on CHIP's. But that's easily shown at test pattern probe.

My bet is a marginal POR... right on the hairy edge of failure for all items.

Larkin is just lucky that some pass. Wait till he gets a field failure >:-} ...Jim Thompson

| James E.Thompson, CTO | mens | | Analog Innovations, Inc. | et | | Analog/Mixed-Signal ASIC's and Discrete Systems | manus | | Phoenix, Arizona 85048 Skype: Contacts Only | | | Voice:(480)460-2350 Fax: Available upon request | Brass Rat | | E-mail Icon at http://www.analog-innovations.com | 1962 | I love to cook with wine. Sometimes I even put it in the food.

Join the Discussion

Have something to add? Share your thoughts — no account required.

Didn't find your answer?

Ask the community — no account required