In message , Adrian writes
It has now run for a week, and I've yet to see a rsync operation fail, so (as predicted by some), it appears to be a hardware problem.
There have however been some other oddities (none of which appear to be problems), which may be of interest to some.
I gather a series of stats from each Pi every minute, so over time, performance patterns can be seen. These stats include CPU and GPU temperature, ping time (ping -c 1 $ROUTER) and the number of bytes in and out.
The target Pi, which has had it's OS upgrade to Buster has gone from a typical minimum ping time for the day of ~.450mS to ~.650mS. Other Pis have remained at their usual sort of times.
The source Pi (new SD card and new Pi) has changed from having the CPU and GPU temperature within 0.1 degrees of each other to now being ~5 degrees apart (CPU always lower than GPU). All other Pis are usually consistent in the 0.1 degree variation.
Other PIs on my network which have a basically static workload (e.g. the weather station), are showing interesting results around their daily rsync operations. Comparing (e.g.) Monday against Monday, I'm seeing the number of bytes being transferred dropping significantly during the operation, Incoming bytes by 17% and outgoing by 8%. Looking at the logs, the number of files being rsync'd is consistent, so unless those files have shrunk significantly (and I see no reason as to why they might have), then it looks as though the operation of the rsync process has improved significantly. The percentages vary from Pi to Pi, but in all cases, there is a noticeable downward trend in the network traffic at rsync time.
Adrian