Re: DRAM accommodations

Back in 1985 I was specifying 72-bit words to accomodate 64-bit data words and eight bits of error-detection and correction data/checksum.

That was built into every memory access - it didn't slow down the memory much, but it made it a lot more reliable.

Typical software reaction. You design error detection and error correction into the hardware, and the extra hardware can both correct most errors (when they can be corrected) and report all of the them - both those corrected and those that couldn't be corrected.

Data transmission systems can re-transmit damaged packets of data. and tend to go for checksums that merely detected errors in much longer words/packets, and reject the affected packets.

Running many day long computations for e.g. the euler project, involving giga byte memories, and require precise (not one off results), I have not encountered any wrong results caused by RAM failures.

Groetjes Albert

ECC is there so they can increase dram density. Increasing density and using multi levels per bit cell makes dram and flash cheaper and increases the raw error rate, and adding ecc bits makes it tolerably reliable, and that tradeoff is cost optimized.

It seems to work pretty well.

My PC here has 32Gbytes of dram and a terabyte of SSD, and seem fine. The combination runs Spice fast and the results work.

Are you really writing about MCU-s? My impression was that MCU-s which allow external memory (most do not) typically have WAIT signal so that external memory can insert extra wait states. OTOH access to external memory is much slower than to internal memory.

People frequently use chips designed for gadgets like smartphones of TV-s, those tend to have integrated DRAM controller and no support for ECC.

IIUC, if you need small amount of memory you should use SRAM...

I think that this estimate is somewhat pessimistic. The last case I remember that could be explanied by memory error was about 25 years ago: I unpacked source of newer gcc version and tried to compile it. Compilation failed, I tracked trouble to a flipped bit in one source file. Unpacking sources again gave correct value of affected bit (and on other bit chaged). And compiling the second copy worked OK.

Earlier I saw segfaults that could be cured by exchanging DRAM modules or downclocking the machine.

But it seems that machines got more reliable and I did not remember any recent problem like this. And I regularly do large compiles, here error in sources is very unlikely to go unnoticed. I did large computations were any error had nontrivial chance to propagate to final result. Some computations that I do are naturally error tolerant, but error tolerant part used tiny amount of data, while most data was "fragile" (error there was likely to be detected).

Concerning doing something about memory errors: on hardwares side devices with DRAM that I use are COTS devices. So the only thing I have is to look at reputation of the vendor, and general reputation says nothing about DRAM errors. In other words, there is nothing I can realistically do. On software side I could try to add some extra error tolerance. But I have various consistency checks and they indicate various troubles. I deal with detected troubles, DRAM errors are not in this category.

Do we know whether DRAM chips implement ECC internally? It seems an obvious thing for them to do. Of course it wouldn't help with bad solder joints on the DIMM, but it would help with many kinds of faults on the chip.

It seems unlikely. The only ECC coding that I got close enough to plan to implement used a 72-bit word to protect 64-bits of the word content.

I suppose there might be serial access DRAM chips that would spit out

I do not think that Cortex A is a MCU. More precisely, Cortex A core is not intended to be used in MCU-s and I think that most chips using it are not MCU-s. Distingushing feature here is if chip is intended to be a complete system (MCU) or are you expected to _always_ use it with external chips (typeical Cortex A based SOC).

PC-s, yes (small SBC-s too, but those were not used for really heavy computations). Concerning ECC, most PC-s that I used came without ECC. IME ECC used to roughly double price of the PC compared to not ECC one. So, important servers got ECC, other PC-s were non ECC.

I remeber memory corruption study several years ago that said that software was significant issue. In particular bugs in BIOS and Linux kernel led to random looking memory corruption. Hopefully, issues that they noticed are fixed now. Exact precentages probably do not matter much, because both hardware and software is changing. The point is that there are both hardware errors and software errors which without deep investigation are hard to distinguish from hardware ones.

Exactly.

Well, most of my computations were on machines without ECC. And I also had some machines sitting idle for long time. They had cached data in RAM and would use it when given some work to do.

Yes. In numerics using Newton style iteration small number of bit flips normally means that it needs to iterate longer, but will still converge to the correct result.

As I wrote, most computations were on non-ECC machines.

Well, I you are concerned you can implement low priority process that will read RAM possibly doing some extra work (like detecting unexpected changes).

When working with MCU-s I have internal SRAM instead of DRAM. And in several cases manufacturers claim parity or ECC protection for SRAM. But in case of PC-s hardware and OS are commodity. Due to price reasons I mostly deal with non-ECC hardware.

On PC-s that is BIOS and OS that I did not wrote. And AFAIK BIOS POST is detecting memory size and doing a little sanity checks to detect gross troubles. But I am not sure if I would call them "memory tests", better memory tests tend to run for days.

As I wrote I am doing computations and criteria are problem specific. For example I have two multidigit numbers and one is supposed to exactly divide the other. If not, software signals an error.

I read several papers about DRAM errors and I take seriously possiblity that they can happen. But simply at my scale they do not seem to matter.

BTW: It seems that currently large fraction of errors (both software and hardware ones) appear semi-randomly. So, to estimate reliabilty one should use statistic methods. But if you aim at high reliablity, then needed sample size may be impractically large. You may be able to add mitigations for rare problems that you can predict/guess, but you will be left with unpredictable ones. In other words, it make sense to concentrate on problems that you see (you including your customers). AFAIK some big companies have wordwide automatic error reporting systems. If you set up such a thing that you may get useful info.

You can try "error injection", that is run tests with extra component that simulates memory errors. Then you will have some info about effects:

- do memory errors cause incorrect operation?

- is incorrect operation detected?