We've all known about the importance of good grounding and bypass caps but this was a strange one.
I have a design based on the Analog Devices ADuC841, which has worked very well for some time. In a recent production lot, we started seeing failures at room temperature. Digging further, we had many failures at temperatures of 5C. These were soft failures where you'd get erratic operation and then have to reset the board after it had warmed up for it to work normally. Device was rated to -40C and an external clock was applied so weren't even close to design limit. Was all set to write it off as bad chips but a later spin of the board assembled by another vendor had a similar problem. Turned out to be entirely due to the decoupling caps being too far from the chip. In one case, there was even a via which further increased the impedance. Put an extra cap right at the main processor pins and problem gone. Just didn't expect that the problem would manifest itself when cold.