I rescue lots of kit from the tip. Presently, I am using 8 spindle servers to replace (reimplement) my SAN appliances (the appliances are closed source; file servers can be OPEN source!).
I was in the process of duplicating a spindle: # cd /0; tar cpf - * | (cd /4; tar xpf - ) and belatedly decided to make some changes to the structure of the source filesystem. So, killed off that job and "rm -r /4/*".
This always mucks with the free block count which is annoying; it's nice to df(1) and see identical block counts after a copy so: # umount /4; newfs -U -j /dev/da4p1; mount /4 makes the disk look pristine.
OK, restart the copy: # cd /0; tar cpf - * | (cd /4; tar xpf - ) And, lets see what sort of throughput we're getting: # iostat -c5 da0 WTF? *nothing*??? Literally *0* MB/s! # iostat -c5 da4 Hmmm... 150MB/s. That explains why the disk activity indicator for drive 4 is blinking like crazy! And, disk 0 is solid.
Open another session and I can see space being consumed on drive 4. And, the actual contents from drive 0 appearing as time passes!
After some 3 minutes, the activity indicator for disk 0 starts blinking...
Ah! Cached files from the prior "copy". Some 28GB of cached files! Well, it's good to see that the machine is making use of that memory (96GB). I wonder how much is set aside for the disk cache vs. network and other kernel buffers, etc.
Quite a disturbing experience (did I type the command incorrectly?)