weird reboots of embedded pc

Yikes! This sounds like something from a Steven King novel... :)

Hmm, short of calling in a witch doctor, have you tried shielding with a faraday cage?

And another idea, you could try putting a clamping zener (or MOV, neon lamp?) across the lockswitch to prevent any static discharges into the serial port. Even better, clamp the switch and buffer it with a schmitt buffer, powered from a heavily bypassed 78L05 or something. Pulling any fast communication wiring outside of the box sounds like trouble.

I'd seen very unusual things happen in RS-485 networks because of ground loop inductance. If you're using shielded cable, make sure only one end of it is grounded.

As others have said, the solution has got to be a process of elimination. At least with nothing connected, you can still duplicate the problem... perhaps touching the switch is adding capacitance to the system, allowing enough ambient noise to influence operation?

As for ground noise, could a high-power LC tank in the DC ground supply suppress the HF noise? Never tried it, just speculating.

Lastly... find the reboot trace on the PC, and add a 100k pullup resistor? :)

Good luck.

-M

Hi,

This is a verry strange one :)

The system; A large cabinet (casino slot machine) - made of wood.

-1 x standard pc motherboard (p4 1.5ghz)

-1 x standard ide disk

-1 x 17" flat panel with resistive touchscreen/usb controlled

-1 x key operated switch, connected to the CDS line of the serial port

The problem; If someone with a slight static charge touches the key to the keyswitch - it sometimes hard boots. As if the power was cycled.

We tried changing power supplies. We tried grounding everything, including the keyswitch. We tried with everything except power disconnected from the motherboard, still happens. (thats right, everything disconnected, including the keyswitch!)

It still happens, even when the keyswitch is completely disconnected - the keyswitch is also mounted in wood so makes absolutely no connection anywhere - but for some reason, touching it can reboot the machine.. wow.

Also, there are 10 machines, all worked fine for 3 months - this problem just 'started' one day - on all 10 machines. So we assumed an external influence caused it.

So, we replace all motherboards with a different brand - the problem disappeared. We added a regulator to feed a UPS for _each_ machine.

This ran for a few weeks - and now the problem has returned. Wow - how weird!.

any ideas ? this is killing me.

Alex.

I had something similar happen several years ago. I replaced the core memory on an old mainframe with SRAM. It would run fine for days and then all of a sudden glitch for a few seconds, then return to normal ops. This was in a DC8 flight simulator, so whenever the memory got corrupted things got very dramatic in the cockpit. We tried everything to no avail. Finally on one of the few times we actually observed it flaking, we had the memory unit pulled out of the rack and for some test or other we removed the DC supply from the chassis and by chance set it on top of a manila folder. We left it like that over the weekend, and on Monday morning the pilots log showed no problems in 48+ hours. By trial and error we were finally able to duplicate the problem. When a certain combination of switches, relays and rotary solenoids were engaged in the cockpit, lots of high frequency noise was getting interjected onto the chassis ground of the entire system. This was getting capacitively coupled through the mains input of the DC supply onto the DC rails. We ended up solving it by a combination of input power conditioning and insulating the DC supply's chassis from the rack.

If these things are in a casino, I'd assume that the AC mains is pretty noisy so you might be seeing something similar.

Just a thought...

Bob

I read in sci.electronics.design that Quack wrote (in ) about 'weird reboots of embedded pc', on Fri, 10 Dec 2004:

Did you try swapping used motherboards between machines? Could the BIOS be corrupted?

Your UPS trial won't have worked if the problem is noise in the grounding.

In these cases, brainstorming can sometimes work. Almost no test is 'too crazy to bother with'.

--
Regards, John Woodgate, OOO - Own Opinions Only. 
The good news is that nothing is compulsory.
The bad news is that everything is prohibited.
http://www.jmwa.demon.co.uk Also see http://www.isce.org.uk

Hi Bob,

Thankyou very much for sharing that experience. Atleast i know im not alone or imagining all this :)

Yes definantly. But i installed a voltage regulator to feed a ups for _each_ machine along with the new motherboards. Surely this must clean up the power ? :(

Its also not just displaying this problem at random times - once it has started, it wont go away until the motherboard is replaced with a new one. So something is cooking :(. slowly. argh.

Alex.

I read in sci.electronics.design that Mark Jones wrote (in ) about 'weird reboots of embedded pc', on Fri, 10 Dec 2004:

Or pushing volts of r.f. from the local radio stations on to the wire that was still connected to the switch. Been there!

'The switch doesn't work with only one wire connected, so why do I have to disconnect both of them?'

--
Regards, John Woodgate, OOO - Own Opinions Only. 
The good news is that nothing is compulsory.
The bad news is that everything is prohibited.
http://www.jmwa.demon.co.uk Also see http://www.isce.org.uk

Had similar as a long, ongoing, problem 15 years ago. Operator touched a keyboard in the control room and the computers 2 floors further down, crashed. Purpose built computers in a positive pressure, air conditioned room, with exquisitely clean power supplies and magnificent screening and earthing. Realtime control of a natural gas network, so people stressed out a little, when both the running and hot standby computers fell over.

Problem cured completely by spraying the control room carpeting with antistatic solution. Rein's solution would no doubt have the same effect.

Earthing the keyswitch should have been the first target but this didn't help . Possibly the series inductance was too high?. I'd think that touching the machine wooden casing and its laminate covering, impulsively transfers charge to it (the keyswitch act as a solidly connected charge path), as if to one plate of a capacitor. The other plate is *all* of the electronic components, casings, PCB tracking and earthed metalwork. A spike of capacitor charging current thus passes through the air dielectric via the myriad electronic circuit paths back to true earth. Some particularly sensitive point is noticing the transient current.

Try lining the wooden casing with aluminium cooking foil and connect to earth at a number of points (ie make a large area, good quality, 2nd capacitor plate) Such that any transient current has an easy path with little distance to travel, before it hits earth. regards john

Sounds pretty common. These problems are -great- revenue generators for consultants....

Advice:

1) You need to be able to cause the problem at will with 100% reliability. For this you need a static test 'gun', a standard rental item, though every ee development organization should have one readily available.

2) High-frequency/high-voltage electricity travels over surfaces, in this context wood is a pretty good conductor, as your experience demonstrates.

3) Your grounding effort wasn't done well - by definition - as it did not fix the problem.

4) Try a wide copper tape from the keyswitch mounting nut (or whatever hardware is on the inside of the box) to the AC entry panel ground. Do not connect this to anything else. Route it far away from anything electronic or conductive.

5) Spray conductive paint all over the inside of the box. The static discharge will flow on the outside surface and should (in theory) not effect anything on the inside.

5) Keep all AC power wiring far away from anything else.

6) Consider using a 1/2" wide copper braid from the machine ground to the building structural steel. Think lightning conductors ...

7) Isolate the keyswitch: use a mechanical flag to interrupt an optical sensor; a long plastic rod to activate a micro-switch; a relay; etc. etc. ...

Question: do you have problems with static discharge to the touch panel/lcd.

A 1.5GHz P4 to run a slot machine? -- back in my day, we did it with a Fairchild F8. Still used real reels and a chrome arm sticking out the side, though: nobody would trust the thing if they knew it was electronic. Kinda silly, though, not trusting something that is designed to rob you.

--
Nicholas O. Lindan, Cleveland, Ohio
Consulting Engineer:  Electronics; Informatics; Photonics.
Remove spaces etc. to reply: n o lindan at net com dot com
psst.. want to buy an f-stop timer? nolindan.com/da/fstop/

General advice on this topic:

Never use a standard motherboard in an embedded system. The quality of the layout and the noise margins are too much of an unknown. I would use a board from one of the well-known embedded vendors.

Never use any version of Microsoft Windows in an embedded system. There are too many thousands of bugs that MS refuses to address. Linux is good, BSD is better, QNX is the most stable, crashproof and bug-freeOS on earth. (A few other RTOSs come close).

Never run a program in an embedded system that was written by someone who knows general-purpose programming but is not an expert in real-time embedded systems programming.

"Guy Macon" wrote

Advice worth repeating. So I have.

BTW: QNX is the only system to use with embedded PC architecture, don't bother with anything else, sorry Linus.

--
Nicholas O. Lindan, Cleveland, Ohio
Consulting Engineer:  Electronics; Informatics; Photonics.
Remove spaces etc. to reply: n o lindan at net com dot com
psst.. want to buy an f-stop timer? nolindan.com/da/fstop/

QNX ... I read a couple dozen pages but it seemed all marketing. My best vision of an embedded PC would be to buy a PC and load say QNX instead of microsoft stuff. Yet still run DOS programmes that would run on a PC Is it this the case? or does everything have to be specially written for QNX?. regards john

Maybe you got a customer that comes around every few weeks and tries to get a payoff by tazering the beasts?

Rocky

Thanks for the advice :)

I totally agree, i love _real_ embedded boards and have never had such a problem with them.

The above problem is not my system, i am helping someone out. (really :) )

But i do like linux, i have never used QNX... i suppose i should try that out then.

Thanks.

Alex.

Nicholas,

Thankyou very much for that! I will go through that list and try _everything_ :).

Someone else mentioned 'faraday cage', i thaught of that too - and have attacked it with cooking foil, but on the old 'fried' motherboards.. I have not re-attempted that with the new boards. (perhaps the old ones were already too far gone).

Interesting that wood conducts static.. i had no idea... okay, off to build an 'easy path' for the static :).

thanks again

Alex.

(whoops)

Yes, the lcd/touch panel is surounded by an aluminium frame, and that can also (but less often) cause a reboot.

Alex.

< snip >

Actually there's nothing stange about it at all. You simply need to design for EMC.

Chances are that the humidity was high when you didn't have a problem. I guess the air has dried and now there's static about, you're seeing the weaknesses of the design.

In Europe ( and elsewhere ) we have to design to withstand electrostatic discharges or we can't legally sell our kit.

Graham

Go tell that to Linksys - I'm sure they care (not)!

....or here fwiw...

QNX et. al. are for people who want to write Infrastructure - i.e. Engineers - and that's not The Game anymore.

The real money are made by Integrating what is already there and making it usable or applicable for "normal people". That vein is only beginning to be tapped; there would be products for the next 3-5 years even if Linux development was frozen today, which it is not.

Curious, out of the different CPU's that are compatible with QNX, is there a "best" match?

I am with Frithiof on this one. QNX is a fine product, but it is no longer the only game in town, and the various flavors of Embedded Linux are really outstandingly good.

On the other hand, if I was designing a control system for a nuclear reactor, I would quit if they didn't let me specify QNX.