Odd network behaviour

Apr 30, 2020 10 Replies

I have 7 PIs which rsync to one of their number, which then rsyncs to another one (this giving me a complete backup).



This has been working happily for years, until the last couple of days. For some time, I've been measuring bytes in/out, and this is reasonably consistent (it changes day to day, but usually in a predictable way). The last two days, the byte count on the Pi running the second backup has gone through the roof, and the logs are showing that this is when rsync is running. To put it into perspective, my Mbytes in / out for March were 4,171 and 478, so far this month (which doesn't include the latest rsync) have been 168,715 and 16,094.



The Rsync log summary for today shows :



sent 977,174 bytes received 106,060,387 bytes 62,613.37 bytes/sec total size is 65,569,967,535 speedup is 612.59



For the same day last week :



sent 968,084 bytes received 106,054,169 bytes 62,204.16 bytes/sec total size is 65,521,951,322 speedup is 612.23



Pretty much the same. So why is my byte counting showing so much difference ?



Byte counting is done by reading /sys/class/net/eth0/statistics/tx_bytes (or rx_bytes as appropriate), and taking the difference from the last reading, when the counter wraps, I take that into account.



Adrian


To Reply : replace "bulleid" with "adrian" - all mail to bulleid is rejected Sorry for the rigmarole, If I want spam, I'll go to the shops Every time someone says "I don't believe in trolls", another one dies.

There's not really a lot to go on here. I presume there has been no changes to any of the machines to account for this? How does this compare to the total amount backed up?

Assuming that really nothing has changed does a netstat -i show a sizable number of dropped packets? Are all the machines on ethernet or are some wireless?

Andrew Smallshaw andrews@sdf.org

All Pis are on Ethernet (I've never had much success with Pi WiFi). No obvious changes recently.

netstat -i shows no dropped packets, what time period does that cover ?

This morning's rsync seems to have worked OK (sensible byte count).

Thanks

Adrian

To Reply : replace "bulleid" with "adrian" - all mail to bulleid is rejected Sorry for the rigmarole, If I want spam, I'll go to the shops Every time someone says "I don't believe in trolls", another one dies.

Does fsck show anything interesting or report any errors?

You may find it easier to fsck it using a USB reader attached to another RPi.

Martin | martin at Gregorie | gregorie dot org

In message , Martin Gregorie writes

I've fsck'd the hard drive (where the backups go), and that comes up clean. For the SD card, that will have to wait until tomorrow, then I can get it onto a Ubuntu machine which has a card reader.

Thanks

Adrian

To Reply : replace "bulleid" with "adrian" - all mail to bulleid is rejected Sorry for the rigmarole, If I want spam, I'll go to the shops Every time someone says "I don't believe in trolls", another one dies.

for

es/sec

es/sec

Turn on progress with -P to see what files are being transferred.

Is the rsync doing an increment to backup of the Pi's storage? Check the

size of log files on the machine, as logs are some of the few things that change every day (log rotations included).

My dozen Pi's do a nightly incremental backup, and I noticed (from the temperature graphs initially, but then from the backup process log) that

on one of them, the transfers to the NAS were up. Turns out someone had left the Pi's camera display on an open browser tab, and coincidentally an update to raspimjpeg had turned nginx's access logging back on, creating a log entry every second or so - bah!

---druck

In message , druck writes

I am running it with -P (I snipped for brevity), and what is coming across is as expected.

They are consistent from week to week.

No login sessions on any of the Pis, and no browser sessions running either. Any such sessions are closed each night before I log off for the day. The rsync runs overnight, so I'm not around to watch it, but I could alter the timing.

Adrian

To Reply : replace "bulleid" with "adrian" - all mail to bulleid is rejected Sorry for the rigmarole, If I want spam, I'll go to the shops Every time someone says "I don't believe in trolls", another one dies.

Um.

I didn't spot the cause was large log files when I ran the backup first thing in the morning. The reason being the browser window had been closed the day before so the current log file wasn't growing, and the normal amount was transferred. However, the next night the transfers were huge again. The reason being the logs were rotated, so yesterdays large log file called access.log.1 was being renamed to access.log.2 at midnight, and being transferred during the 4am backup.

---druck

In message , Adrian writes

SD card was fsck'd on the Ubuntu box, and it also came up with a clean bill of health. I've not seen this problem now for several days, so it seems to have been a 2 day wonder.

Thanks for the input. Adrian

To Reply : replace "bulleid" with "adrian" - all mail to bulleid is rejected Sorry for the rigmarole, If I want spam, I'll go to the shops Every time someone says "I don't believe in trolls", another one dies.

weekly logfile rotation?

?I know that most men, including those at ease with problems of the greatest complexity, can seldom accept even the simplest and most obvious truth if it be such as would oblige them to admit the falsity of conclusions which they have delighted in explaining to colleagues, which they have proudly taught to others, and which they have woven, thread by thread, into the fabric of their lives.? ? Leo Tolstoy

I don't think so. I compared the stats with the same days the previous week, and they bore no resemblance at all. I usually see an increase in traffic when the logs rotate, (the rsync logs confirm this), but not a

30 - 40 fold increase as I was seeing here (see figure in my OP).

To be clear, the rsync logs are consistent on a week to week basis (e.g. comparing Tuesdays) both in terms of files copied and the number of bytes (minor variations in each, but a small percentage), but the number of bytes counted from /sys/class/net/eth0/statistics/*_bytes was showing a massive increase, and always at the tail end of the rsync session.

Adrian.

To Reply : replace "bulleid" with "adrian" - all mail to bulleid is rejected Sorry for the rigmarole, If I want spam, I'll go to the shops Every time someone says "I don't believe in trolls", another one dies.

Join the Discussion

Have something to add? Share your thoughts — no account required.

Didn't find your answer?

Ask the community — no account required