Odd network behaviour

I have 7 PIs which rsync to one of their number, which then rsyncs to another one (this giving me a complete backup).

This has been working happily for years, until the last couple of days. For some time, I've been measuring bytes in/out, and this is reasonably consistent (it changes day to day, but usually in a predictable way). The last two days, the byte count on the Pi running the second backup has gone through the roof, and the logs are showing that this is when rsync is running. To put it into perspective, my Mbytes in / out for March were 4,171 and 478, so far this month (which doesn't include the latest rsync) have been 168,715 and 16,094.

The Rsync log summary for today shows :

sent 977,174 bytes received 106,060,387 bytes 62,613.37 bytes/sec total size is 65,569,967,535 speedup is 612.59

For the same day last week :

sent 968,084 bytes received 106,054,169 bytes 62,204.16 bytes/sec total size is 65,521,951,322 speedup is 612.23

Pretty much the same. So why is my byte counting showing so much difference ?

Byte counting is done by reading /sys/class/net/eth0/statistics/tx_bytes (or rx_bytes as appropriate), and taking the difference from the last reading, when the counter wraps, I take that into account.

Adrian

--
To Reply : 
replace "bulleid" with "adrian" - all mail to bulleid is rejected 
Sorry for the rigmarole, If I want spam, I'll go to the shops 
Every time someone says "I don't believe in trolls", another one dies.
Reply to
Adrian
Loading thread data ...

There's not really a lot to go on here. I presume there has been no changes to any of the machines to account for this? How does this compare to the total amount backed up?

Assuming that really nothing has changed does a netstat -i show a sizable number of dropped packets? Are all the machines on ethernet or are some wireless?

--
Andrew Smallshaw 
andrews@sdf.org
Reply to
Andrew Smallshaw

All Pis are on Ethernet (I've never had much success with Pi WiFi). No obvious changes recently.

netstat -i shows no dropped packets, what time period does that cover ?

This morning's rsync seems to have worked OK (sensible byte count).

Thanks

Adrian

--
To Reply : 
replace "bulleid" with "adrian" - all mail to bulleid is rejected 
Sorry for the rigmarole, If I want spam, I'll go to the shops 
Every time someone says "I don't believe in trolls", another one dies.
Reply to
Adrian

Does fsck show anything interesting or report any errors?

You may find it easier to fsck it using a USB reader attached to another RPi.

--
Martin    | martin at 
Gregorie  | gregorie dot org
Reply to
Martin Gregorie

In message , Martin Gregorie writes

I've fsck'd the hard drive (where the backups go), and that comes up clean. For the SD card, that will have to wait until tomorrow, then I can get it onto a Ubuntu machine which has a card reader.

Thanks

Adrian

--
To Reply : 
replace "bulleid" with "adrian" - all mail to bulleid is rejected 
Sorry for the rigmarole, If I want spam, I'll go to the shops 
Every time someone says "I don't believe in trolls", another one dies.
Reply to
Adrian

for

es/sec

es/sec

Turn on progress with -P to see what files are being transferred.

Is the rsync doing an increment to backup of the Pi's storage? Check the

size of log files on the machine, as logs are some of the few things that change every day (log rotations included).

My dozen Pi's do a nightly incremental backup, and I noticed (from the temperature graphs initially, but then from the backup process log) that

on one of them, the transfers to the NAS were up. Turns out someone had left the Pi's camera display on an open browser tab, and coincidentally an update to raspimjpeg had turned nginx's access logging back on, creating a log entry every second or so - bah!

---druck

Reply to
druck

In message , druck writes

I am running it with -P (I snipped for brevity), and what is coming across is as expected.

They are consistent from week to week.

No login sessions on any of the Pis, and no browser sessions running either. Any such sessions are closed each night before I log off for the day. The rsync runs overnight, so I'm not around to watch it, but I could alter the timing.

Adrian

--
To Reply : 
replace "bulleid" with "adrian" - all mail to bulleid is rejected 
Sorry for the rigmarole, If I want spam, I'll go to the shops 
Every time someone says "I don't believe in trolls", another one dies.
Reply to
Adrian

Um.

I didn't spot the cause was large log files when I ran the backup first thing in the morning. The reason being the browser window had been closed the day before so the current log file wasn't growing, and the normal amount was transferred. However, the next night the transfers were huge again. The reason being the logs were rotated, so yesterdays large log file called access.log.1 was being renamed to access.log.2 at midnight, and being transferred during the 4am backup.

---druck

Reply to
druck

In message , Adrian writes

SD card was fsck'd on the Ubuntu box, and it also came up with a clean bill of health. I've not seen this problem now for several days, so it seems to have been a 2 day wonder.

Thanks for the input. Adrian

--
To Reply : 
replace "bulleid" with "adrian" - all mail to bulleid is rejected 
Sorry for the rigmarole, If I want spam, I'll go to the shops 
Every time someone says "I don't believe in trolls", another one dies.
Reply to
Adrian

weekly logfile rotation?

--
     ?I know that most men, including those at ease with problems of the  
greatest complexity, can seldom accept even the simplest and most  
obvious truth if it be such as would oblige them to admit the falsity of  
conclusions which they have delighted in explaining to colleagues, which  
they have proudly taught to others, and which they have woven, thread by  
thread, into the fabric of their lives.? 

     ? Leo Tolstoy
Reply to
The Natural Philosopher

I don't think so. I compared the stats with the same days the previous week, and they bore no resemblance at all. I usually see an increase in traffic when the logs rotate, (the rsync logs confirm this), but not a

30 - 40 fold increase as I was seeing here (see figure in my OP).

To be clear, the rsync logs are consistent on a week to week basis (e.g. comparing Tuesdays) both in terms of files copied and the number of bytes (minor variations in each, but a small percentage), but the number of bytes counted from /sys/class/net/eth0/statistics/*_bytes was showing a massive increase, and always at the tail end of the rsync session.

Adrian.

--
To Reply : 
replace "bulleid" with "adrian" - all mail to bulleid is rejected 
Sorry for the rigmarole, If I want spam, I'll go to the shops 
Every time someone says "I don't believe in trolls", another one dies.
Reply to
Adrian

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.