[SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
[SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives.

My life in code and photos recently approached half a Terabyte, the limit of my  
USB external drive. Now with an HD dashcam the brink was in sight, so I  
purchased a 3TB drive on sale at Fry's for $80. I wanted a bit for bit duplicate  
of the old drive on the new one.

NO FEAR.
My first concern was the new drive is USB 3.0 and the old drive was 2.0. Not to  
worry, that just means the 2.0 will be the slowness in the bottleneck factoring.  
They are perfectly compatible in the eyes of the Pi and most other controller  
devices. USB 3.0 is an investment in the future. This 3TB drive is my first USB  
3.0 device. It powers solely from the Pi3+ and the Pi Zeros will be able to  
connect with OTB to OTB USB cable (and my Android phone and dashcam dump)  
according to plan, just not as fast as the drive is able.
An other concern was transfer software, what would make an identical copy?
Several answers were researched on the Google query "bit for bit disk copy"  
later refined to "bit for bit disk copy in Linux command line". The simplest and  
most powerful choice was the linux console command:

dd if=/dev/sdb of=/dev/sda &

See $> dd --help for details
dd, disk duplicate
if, input file, or other source in this case the unmounted b drive
of, output file, or other destination in this case the unmounted a drive
the drives were determined by order of attachment and the names were determined  
by the command line 'fdisk ?l'

Which suited me fine as my Win PC is heavily burdened with other tasks and my  
two Puppy Linux laptops are busy loaded with hundreds of SeaMonkey browser  
pages, which tend to spike the processor load, locking up keyboard and mouse and  
occasionally requiring manual restart - too flaky for the immediate task at  
hand. However, there are two unburdened Raspberry Pi Zeros waiting for an  
assignment and a Faspberry Pi 3+ running background CHRON scripts and finishing  
up an online class 'Teaching Physical Computing with Raspberry Pi and Python'
from Raspberry Pi Foundation on FutureLearn.com and SSH in from the Puppy to the  
Pi3 was the best choice, its quad core was speedier than the Pi0's and the 'dd'  
command task runs in the background undisturbed at 20% CPU and 0.1% memory. It's  
merrily zooming along dumping the old drive into the new as we speak.

References:
Clone a Hard Drive Using an Ubuntu Live CD
https://www.howtogeek.com/howto/19141/clone-a-hard-drive-using-an-ubuntu-live-cd/

11.2 dd: Convert and copy a file
http://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation

How To Clone An Internal Linux Partition To An External USB Disk Using DD
http://linuxpitstop.com/how-to-clone-an-internal-linux-partition-to-an-external-usb-disk-using-dd/

The Best Disk Cloning App for Linux
http://lifehacker.com/5891933/the-best-disk-cloning-app-for-linux

g4u - Harddisk Image Cloning for PCs
http://www.feyrer.de/g4u/

What I learned: dd can be both a powerful and potentially dangerous command when  
ill considered.

Mistakes I made: The good news was that I made this minor mistake which was  
quite unrelated to the disk duplication via SSH on a Puppy laptop console  
window. The same window in which I launched the DD command I wanted to see what  
so far had been written to the unmounted, unformatted three terabyte external  
USB spinning hunk of iron. I should have launched another command console window  
that I could kill without affecting the dd transfer. I typed 'cat /dev/sda' and  
my window filled with ascii interpretations of machine code. That answered my  
question - yes, something has now been transfered onto the unmounted,  
unformatted USB external drive, but it also answered quite loudly, as each '08'  
byte rang the terminal bell and there were plenty of them. I succeeded in  
annoying myself. Humorous enough but I couldn't stop the display with control c,  
z or x and I couuln't figure out how to interrupt it, so I killed the command  
console process window on the puppy, which also halted the dd transfer. Ah well,  
live and learn.



--  
   All ladders in the Temple of the Forbidden Eye have thirteen steps.
There are thirteen steps to the gallows, firing squad or any execution.
We've slightly trimmed the long signature. Click to see the full one.
Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On Sat, 08 Apr 2017 19:46:53 -0700

Quoted text here. Click to load it

    It will go much faster with bs64%k added to the arguments.

--  
Steve O'Hara-Smith                          |   Directable Mirror Arrays
C:>WIN                                      | A better way to focus the sun
We've slightly trimmed the long signature. Click to see the full one.
Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.

Quoted text here. Click to load it

Probably even faster with (a little higher) blocks... and using raw
devices (/dev/rsd) may also help.

Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On Sun, 09 Apr 2017 18:16:09 +0200

Quoted text here. Click to load it

    Not much faster with bigger blocks they tend to get split up to 64k
blocks between the driver and the hardware IME.

--  
Steve O'Hara-Smith                          |   Directable Mirror Arrays
C:>WIN                                      | A better way to focus the sun
We've slightly trimmed the long signature. Click to see the full one.
Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On 09/04/17 17:16, Raymond Wiker wrote:
Quoted text here. Click to load it
ls -l /dev/rs*
ls: cannot access /dev/rs*: No such file or directory

In fact e raw devices are IIRC /dev/sd[abcd] etc.
The partitions are /dev/sd[abcd][12345...]

--  
Future generations will wonder in bemused amazement that the early  
twenty-first century?s developed world went into hysterical panic over a  
We've slightly trimmed the long signature. Click to see the full one.
Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
Quoted text here. Click to load it

In Linux raw devices means the cache is bypassed; see ?man raw?. For
this application I wouldn?t bother.

--  
http://www.greenend.org.uk/rjk/

Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On 09/04/17 18:06, Richard Kettlewell wrote:
Quoted text here. Click to load it
Ah. wasn't aware of that semantic. I've always used 'raw' to refer to  
'unpartitioned' on block devices.

--  
If you tell a lie big enough and keep repeating it, people will  
eventually come to believe it. The lie can be maintained only for such  
We've slightly trimmed the long signature. Click to see the full one.
Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
Raymond Wiker wrote:
Quoted text here. Click to load it

For different devices on different interfaces I have found the opposite
to be true. Large blocks need to be buffered and written to and
retrieved from memory. Small ones may fit into faster cache.

--  




/ \  Mail | -- No unannounced, large, binary attachments, please! --

Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On Sat, 08 Apr 2017 19:46:53 -0700, "DisneyWizard the Fantasmic!"

Quoted text here. Click to load it

    Since the RPi3 is still only USB 2.0 port-wise, data transfer speeds
will be about the same -- barring the old drive being very inefficient.

--  
    Wulfraed                 Dennis Lee Bieber         AF6VN
     snipped-for-privacy@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
Il giorno domenica 9 aprile 2017 04:47:48 UTC+2, DisneyWizard the Fantasmic! ha scritto:
Quoted text here. Click to load it

also the most stupid choice in this case.
dd will copy EVERYTHING, even empty space.
And also partition table etc. so the result will be a 3TB disk with a 500GB partition, that you then have to expand using parted or similar.

Much better (and faster) in this case to mount both HDs and do a cp -a.

Bye Jack

Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On Wed, 19 Apr 2017 05:41:53 -0700 (PDT)
snipped-for-privacy@gmail.com wrote:

Quoted text here. Click to load it


The drive is nearly full, so there won't be much empty space.  If the
drive is reading one file at a time it will likely have to perform seeks
in a number of fragmented files (the likelihood of this is increased by
the fullness of the drive) while dd will read each track consecutively.

Quoted text here. Click to load it

But copying the whole disk rather than just individual files you don't
need to format the target drive before copying.
Quoted text here. Click to load it

I doubt there's much in it - it's possible dd will be faster because of
fragmentation.


Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.

to:
Quoted text here. Click to load it

yes, but you end up with a target that has a partition the same size of the
 source, even if the target is bigger...

Quoted text here. Click to load it

dd-ing will preserve fragmentation, copying will remove fragmentation.

other cons of dd is that if the copy is interrupted for some reason, it nee
d to be restarted from the beginning because the target will not be usable  
until the end.
With cp at least the copied files are usable.
Better solution would be rsync, so if the copy is interrupted, it will resu
me without problem.

Bye Jack

Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On Wed, 19 Apr 2017 06:56:18 -0700 (PDT)
snipped-for-privacy@gmail.com wrote:


Quoted text here. Click to load it


That's easily fixed.  Format before, or re-size after - looks like
swings and roundabouts to me.
Quoted text here. Click to load it

If fragmentation is a problem in use (not normally with Linux, but
sometimes with Windows IME) you can just copy the fragmented files
once you have all that lovely extra disk space.
Quoted text here. Click to load it

They're copies - you already have the originals on the source drive,
why would you need them on the target before the transfer is complete?

Quoted text here. Click to load it
You can stop and resume dd quite easily - send SIGUSR1 to show
progress, then kill dd.  You resume by giving appropriate values to  
skip and seek using the record count produced by SIGUSR1.
Or just send the signal occasionally and redirect output to a file so
you have a record of a fallback point if the dd encounters some
insurmountable problem like a host crashing.


Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
declaimed the following:

Quoted text here. Click to load it

    OTOH: future use of the destination drive may be faster because the
file copy command may serve to defragment it <G>
--  
    Wulfraed                 Dennis Lee Bieber         AF6VN
     snipped-for-privacy@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On 19/04/17 16:57, Dennis Lee Bieber wrote:
Quoted text here. Click to load it
Linux doesnt suffer significant degradation from fragmentation if you  
avoid MSDOS style partition types.

--  
The New Left are the people they warned you about.

Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.

Quoted text here. Click to load it

Innocent question:

Then why is this in the man for mount?

    Mount options for btrfs

        autodefrag
               Disable/enable auto defragmentation.  Auto defragmentation
               detects small random  writes into files and queues them up
               for the defrag process.  Works best for small files; not
               well-suited for large database workloads.

Been looking to set up a RAID 1 to keep a pair of 4TB drives sync'd up.

Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On 19/04/17 18:55, Sidney_Kotic wrote:
Quoted text here. Click to load it

Maybe BTRFS is a clone of Microsoft?

I bet it doesn't say that that with ext 2,3,4



--  
You can get much farther with a kind word and a gun than you can with a  
kind word alone.

We've slightly trimmed the long signature. Click to see the full one.
Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On 19/04/2017 14:32, Rob Morley wrote:
Quoted text here. Click to load it

Fragmentation of the files isn't the issue. If the smaller drive is  
nearly full and/or contains lots of small files, using cp -a will be far  
slower. For each file copied to the disc, the directory, allocation  
structures and journal logs must be updated so it maintains a valid  
filing system at all times. This will cause far more writing to the disc  
than the simple one block written for each block read that dd performs.

---druck


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On Wed, 19 Apr 2017 19:59:06 +0100

Quoted text here. Click to load it
That too. Having thought about the whole process of copying
file-by-file, ignore the file  fragmentation thing as insignificant,
unless all the file system overhead is cached in RAM and only
occasionally written to disk.   :-)
(And directory fragmentation is probably a bigger factor anyway.)


Re: [SOLVED] {dd if=/dev/sdb of=/dev/sda &} duplicates terabyte drives with ease.
On 04/19/2017 04:41 AM, snipped-for-privacy@gmail.com wrote:

Quoted text here. Click to load it

I've seen this in action, I like to dd an uninstalled Raspbian...just in case.
dd a 8GB microSD into a iso image on the desktop computer.  Which ends up being  
8GB in size, although compression usually works well on them.
dd that 8GB image to a 32GB and end up with 24GB of unallocated space.

So basically in order to simply use dd the drives should be the same, otherwise  
there's more involved.

Site Timeline