[SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.

[SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives.
My life in code and photos recently approached half a Terabyte, the limit of my
USB external drive. Now with an HD dashcam the brink was in sight, so I
purchased a 3TB drive on sale at Fry's for $80. I wanted a bit for bit duplicate
of the old drive on the new one.
NO FEAR.
My first concern was the new drive is USB 3.0 and the old drive was 2.0. Not to
worry, that just means the 2.0 will be the slowness in the bottleneck factoring.
They are perfectly compatible in the eyes of the Pi and most other controller
devices. USB 3.0 is an investment in the future. This 3TB drive is my first USB
3.0 device. It powers solely from the Pi3+ and the Pi Zeros will be able to
connect with OTB to OTB USB cable (and my Android phone and dashcam dump)
according to plan, just not as fast as the drive is able.
An other concern was transfer software, what would make an identical copy?
Several answers were researched on the Google query "bit for bit disk copy"
later refined to "bit for bit disk copy in Linux command line". The simplest and
most powerful choice was the linux console command:
dd if=/dev/sdb of=/dev/sda &
See $> dd --help for details
dd, disk duplicate
if, input file, or other source in this case the unmounted b drive
of, output file, or other destination in this case the unmounted a drive
the drives were determined by order of attachment and the names were determined
by the command line 'fdisk ?l'
Which suited me fine as my Win PC is heavily burdened with other tasks and my
two Puppy Linux laptops are busy loaded with hundreds of SeaMonkey browser
pages, which tend to spike the processor load, locking up keyboard and mouse and
occasionally requiring manual restart - too flaky for the immediate task at
hand. However, there are two unburdened Raspberry Pi Zeros waiting for an
assignment and a Faspberry Pi 3+ running background CHRON scripts and finishing
up an online class 'Teaching Physical Computing with Raspberry Pi and Python'
from Raspberry Pi Foundation on FutureLearn.com and SSH in from the Puppy to the
Pi3 was the best choice, its quad core was speedier than the Pi0's and the 'dd'
command task runs in the background undisturbed at 20% CPU and 0.1% memory. It's
merrily zooming along dumping the old drive into the new as we speak.
References:
Clone a Hard Drive Using an Ubuntu Live CD
formatting link

11.2 dd: Convert and copy a file
formatting link

How To Clone An Internal Linux Partition To An External USB Disk Using DD
formatting link

The Best Disk Cloning App for Linux
formatting link

g4u - Harddisk Image Cloning for PCs
formatting link

What I learned: dd can be both a powerful and potentially dangerous command when
ill considered.
Mistakes I made: The good news was that I made this minor mistake which was
quite unrelated to the disk duplication via SSH on a Puppy laptop console
window. The same window in which I launched the DD command I wanted to see what
so far had been written to the unmounted, unformatted three terabyte external
USB spinning hunk of iron. I should have launched another command console window
that I could kill without affecting the dd transfer. I typed 'cat /dev/sda' and
my window filled with ascii interpretations of machine code. That answered my
question - yes, something has now been transfered onto the unmounted,
unformatted USB external drive, but it also answered quite loudly, as each '08'
byte rang the terminal bell and there were plenty of them. I succeeded in
annoying myself. Humorous enough but I couldn't stop the display with control c,
z or x and I couuln't figure out how to interrupt it, so I killed the command
console process window on the puppy, which also halted the dd transfer. Ah well,
live and learn.
--
   All ladders in the Temple of the Forbidden Eye have thirteen steps. 
There are thirteen steps to the gallows, firing squad or any execution. 
 Click to see the full signature
Reply to
DisneyWizard the Fantasmic!
Loading thread data ...
It will go much faster with bs=64k added to the arguments.
--
Steve O'Hara-Smith                          |   Directable Mirror Arrays 
C:>WIN                                      | A better way to focus the sun 
 Click to see the full signature
Reply to
Ahem A Rivet's Shot
On Sat, 08 Apr 2017 19:46:53 -0700, "DisneyWizard the Fantasmic!" declaimed the following:
Since the RPi3 is still only USB 2.0 port-wise, data transfer speeds will be about the same -- barring the old drive being very inefficient.
--
	Wulfraed                 Dennis Lee Bieber         AF6VN 
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/
Reply to
Dennis Lee Bieber
Probably even faster with (a little higher) blocks... and using raw devices (/dev/rsd{a,b}) may also help.
Reply to
Raymond Wiker
Not much faster with bigger blocks they tend to get split up to 64k blocks between the driver and the hardware IME.
--
Steve O'Hara-Smith                          |   Directable Mirror Arrays 
C:>WIN                                      | A better way to focus the sun 
 Click to see the full signature
Reply to
Ahem A Rivet's Shot
ls -l /dev/rs* ls: cannot access /dev/rs*: No such file or directory
In fact e raw devices are IIRC /dev/sd[abcd] etc. The partitions are /dev/sd[abcd][12345...]
--
Future generations will wonder in bemused amazement that the early  
twenty-first century?s developed world went into hysterical panic over a  
 Click to see the full signature
Reply to
The Natural Philosopher
For different devices on different interfaces I have found the opposite to be true. Large blocks need to be buffered and written to and retrieved from memory. Small ones may fit into faster cache.
--




/ \  Mail | -- No unannounced, large, binary attachments, please! --
Reply to
Axel Berger
In Linux raw devices means the cache is bypassed; see ?man raw?. For this application I wouldn?t bother.
--
http://www.greenend.org.uk/rjk/
Reply to
Richard Kettlewell
Ah. wasn't aware of that semantic. I've always used 'raw' to refer to 'unpartitioned' on block devices.
--
If you tell a lie big enough and keep repeating it, people will  
eventually come to believe it. The lie can be maintained only for such  
 Click to see the full signature
Reply to
The Natural Philosopher
Il giorno domenica 9 aprile 2017 04:47:48 UTC+2, DisneyWizard the Fantasmic! ha scritto:
also the most stupid choice in this case. dd will copy EVERYTHING, even empty space. And also partition table etc. so the result will be a 3TB disk with a 500GB partition, that you then have to expand using parted or similar.
Much better (and faster) in this case to mount both HDs and do a cp -a.
Bye Jack
Reply to
jack4747
The drive is nearly full, so there won't be much empty space. If the drive is reading one file at a time it will likely have to perform seeks in a number of fragmented files (the likelihood of this is increased by the fullness of the drive) while dd will read each track consecutively.
But copying the whole disk rather than just individual files you don't need to format the target drive before copying.
I doubt there's much in it - it's possible dd will be faster because of fragmentation.
Reply to
Rob Morley
to:
yes, but you end up with a target that has a partition the same size of the source, even if the target is bigger...
dd-ing will preserve fragmentation, copying will remove fragmentation.
other cons of dd is that if the copy is interrupted for some reason, it nee d to be restarted from the beginning because the target will not be usable until the end. With cp at least the copied files are usable. Better solution would be rsync, so if the copy is interrupted, it will resu me without problem.
Bye Jack
Reply to
jack4747
That's easily fixed. Format before, or re-size after - looks like swings and roundabouts to me.
If fragmentation is a problem in use (not normally with Linux, but sometimes with Windows IME) you can just copy the fragmented files once you have all that lovely extra disk space.
They're copies - you already have the originals on the source drive, why would you need them on the target before the transfer is complete?
You can stop and resume dd quite easily - send SIGUSR1 to show progress, then kill dd. You resume by giving appropriate values to skip and seek using the record count produced by SIGUSR1. Or just send the signal occasionally and redirect output to a file so you have a record of a fallback point if the dd encounters some insurmountable problem like a host crashing.
Reply to
Rob Morley
On Wed, 19 Apr 2017 14:32:42 +0100, Rob Morley declaimed the following:
OTOH: future use of the destination drive may be faster because the file copy command may serve to defragment it
--
	Wulfraed                 Dennis Lee Bieber         AF6VN 
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/
Reply to
Dennis Lee Bieber
Linux doesnt suffer significant degradation from fragmentation if you avoid MSDOS style partition types.
--
The New Left are the people they warned you about.
Reply to
The Natural Philosopher
Innocent question:
Then why is this in the man for mount?
Mount options for btrfs
autodefrag Disable/enable auto defragmentation. Auto defragmentation detects small random writes into files and queues them up for the defrag process. Works best for small files; not well-suited for large database workloads.
Been looking to set up a RAID 1 to keep a pair of 4TB drives sync'd up.
Reply to
Sidney_Kotic
I've seen this in action, I like to dd an uninstalled Raspbian...just in case. dd a 8GB microSD into a iso image on the desktop computer. Which ends up being 8GB in size, although compression usually works well on them. dd that 8GB image to a 32GB and end up with 24GB of unallocated space.
So basically in order to simply use dd the drives should be the same, otherwise there's more involved.
Reply to
Sidney_Kotic
Maybe BTRFS is a clone of Microsoft?
I bet it doesn't say that that with ext 2,3,4
--
You can get much farther with a kind word and a gun than you can with a  
kind word alone. 
 Click to see the full signature
Reply to
The Natural Philosopher
Fragmentation of the files isn't the issue. If the smaller drive is nearly full and/or contains lots of small files, using cp -a will be far slower. For each file copied to the disc, the directory, allocation structures and journal logs must be updated so it maintains a valid filing system at all times. This will cause far more writing to the disc than the simple one block written for each block read that dd performs.
---druck
--
This email has been checked for viruses by Avast antivirus software. 
https://www.avast.com/antivirus
Reply to
druck
I forgot to add...I pretty much use rsync. Not the fastest thing but it does work and cares not one whit about drive/microSD size. I run openSUSE on a couple of computers and backup my home directory, plus a couple more, with a nightly cron job onto a 16GB USB thumb drive. I actually use 2 16's and rotate them daily, been known to save my bacon.
Reply to
Sidney_Kotic

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.