complexity, perversity, general evil

Since the boot loader works, how about adding a zero (one) memory routine right after the boot loader? If that forced a fail, or got rid of it, it would indicate an unitialized memory problem. If not, not so much (but still possible).

Maybe it's the FPGA and not the UC at all? Can you replace the FPGA firmware with just blinkin' light firmware (I always hook an LED to FPGAs for such debug) and see if that still fails? IOW, completely configure it with different and very simple logic.

Makes sense. Comment out this part until the FPGA problems are solved.

until it gets it right.

Uninitialized SRAM cell?

It sounds like some part of your power-on sequence is getting derailed by a race condition (there's a hidden timing that has to happen, not just a sequencing). So, adding some delays will fix it. The working units have sufficient delays due to their different code sizes...

Three possible ways a component can change over many days: electrolytic capacitors can get leaky (but that's usually not critical); magnetic elements (especially those with DC current) can demagnetize; impurities can migrate (usually electrons and gates, but also ions in quartz crystals).

It's possible that a crystal clock isn't stable enough, and that (when driven) the crystal gets better Q with time, as the drive causes ionic migration in the rock. Yeah, it's a long shot...

But, if the problem is related to temperature, you might be able to localize it to a few components, and apply freeze-mist selectively to find the sensitive one.

how about trying with an independent 50 MHz oscillator , instead of the clock source produced by the FPGA ?

You may have in the FPGA a PLL clock generator delaying the production of the 50 MHz signal.

Thanks, Jure Z.

Yep - At least for the 1768/1769 it's in the docs. Says you cannot access the Ethernet block unless the refclk is running. On a 1768/1769 it will just hang the processor and, if memory serves, the WDT wont go off either.

Also be careful because, at least on the 17xx series, some of the registers are not read/write. Read the docs carefully. Something like reg |= value will not work as you expect. Be careful to mask unused bits.

I also found POR to be a suggestion to some of the register blocks on some chips. I do not rely on POR values for the pin function and pin connect blocks. May or may not be in GPIO mode.

How are you programming the FPGA? SPI/I2C or ??

One more vote to uninitialized static RAM cell. A very small imbalance is enough for the cell to remember the last state at startup. That the error comes faster when warming the unit points also to this direction.

Years ago, I built a computer with Z80 and some of the first 64k dynamic RAMs. CP/M booted well up and run as long as the computer was used. After a short idle time (some tens of seconds), the system crashed completely. The culprit was a totally inoperative RAM refresh, so they did not forget in 2 ms, but rather in 20 s.

The FPGA doesn't configure. We always have a green LED on the config-done pin to indicate whet it's been successfully config'd, and it doesn't light up on our first powerup. A logic analysis of the first-time (unsuccessful) and subsequent (successful) config bitstreams look the same. So it may be the FPGA itself. The ARM crashing was, as the Brit mystery writers like to say, a red herring.

I'm thinking there's something we can do with the config code, like slow things down or something, or maybe bang nconfig a couple of times to wake it up.

The NXP chips are nice but quirky and not always well documented. We use the 1758 and 1768 in other products, but we like the horsepower of the 3250 in some places.

We use SPI1 to talk to the boot serial flash, and SPI2 to configure the Cyclone FPGA. The FPGA bitstream is in the serial flash too, run-length compressed to save time and space. The ARM reads the stuff in flash, decompresses it, and loads the FPGA.

Maybe my guys are rushing the nconfig pin timing.

ny

s,

But the changes it is causing - while they might be good for the planet (from some arbitrary planet-centred point of view) they are most unlikely to be good for us. See level rise, changing rainfall and more hurricanes, while they might conceivably represent long term benfits to some, are going to be very inconvenient for most.

It's actually well established, and has been for a decade or more.

It certainly would, but that - from the most recent studies - wasn't going to happen for a few tens of thousands of years anyway - apparently this would have been a long interglacial, even before we began to fiddle with the thermostat knob.

They aren't all that precise, bu they don't have to be all that precise to predict more warming than we want to have to cope with. There's been a paper publish recently that looked at humidity prediction and seems to be suggesting that some climate models are more reliable than others - sadly, these the models that predict more rather than less warming.

John's newspapers don't seem to have picked this. New Scientist commented on it on the editorial page.

Since it's rather difficult to unwind global warming once you've managed to achieve it, we'd be wise to act vigorously now, but we aren't wise, and John Larkin is representative of a large swathe of complacent and ill-informed foolishness.

Something is fishy.

How about the time between all the FPGA supplies stabilizing and configuration? Is nSTATUS going high after nconfig and before the data clock goes high for the first time? I think it actually has to have a pullup on it.. no LED + resistor ;-) Is the clock waveform and timing textbook? nRS pulled high for PPA?

Is it gobbling up the data and not showing init_done after the final data clock?

Yes-yes-yes. A friend of mine made PC XT on the breadboard. Without DRAM refresh, it booted MS DOS from floppy. You could launch Multi Edit and work in it. Everything was fine as long as you were pressing some keys. If you stop for 5 seconds or so, system crashed.

VLV

He wasn't smart enough to program the timer? The code was even published in the BIOS tech ref.

He was smart enough to build Turbo XT 8086 from scratch on CAMAC breadboard; without any schematics. It had CGA/HGS video onboard. All discrete components. At one day DMA-0 failed. Weird operation without refresh was discovered.

VLV

Can the FPGA config loader check if the config was accepted? Some Xilinx FPGAs are just a little flaky and will occasionally not accept a config load. Possibly the loader has some timing that is close to the edge. If the loader can determine the FPGA failed to load correctly, it can try again, hopefully everything has stabilized by that time.

Jon

Does this system use ALL static RAM, or is there some dynamic ram there, too? Some dynamic RAMs can hold at least partial data for amazing periods of time. I've read that the computer forensic guys can read data out of a PC's DRAM days after the system was last turned on, mostly by defeating the memory test.

Jon

The C compiler is supposed to warn us about using

Yikes! Many C compilers have some gaps in coverage of this. if the system uses interrupts, then all bets are off. The compiler can't possibly know the flow of control when interrupts are present.

Jon

yeah the Z80's built-in DRAM refresh counter only goes to 127, so 16K is the biggest DRAM chip you can use all of. if you use a 64K only 32K gets refreshed.

method until it gets it right.

That seems to be in the configuration data for the FPGA somewhere. Since the 50 MHz clock seems to be one of the things not config'ing properly it may give focus or be a collateral issue. I seems to be a sequence of initialization "undocumented feature". Double config'ing the FPGA may make it go away.

?-)

method until it gets it right.

Looks like you are on the trail now. But the other source of general power-up flakiness I have found is on-chip PLLs. I think they are analog circuits bodged out of digital ones by designers who don't know what they are doing. All the early versions of the micros I use seem to have errata there, sometimes never fixed.

Here's one:

Still does not work reliably with that circuit by the way.