Failure to clone SD card with compression

Why not use gzip on its own rather than as part of a pipeline? I.e. do the dd to a file as you have been, then separately gzip that file. (Or better bzip2 or xz it, these days.) To use it, gunzip it then dd the uncompressed image.

It still should work though, but that's a good first test.

Why are you using count and conv with dd?

I'd suggest 'partimage' instead.

/dev/sdc and / are on different filesystems, yes? Just checking. Use gzip with piped input without -c. Maybe try without noerror to see if an error occurs. I'm not exactly sure which buffers get padded with the sync option, but that sounds like maybe the output is not identical to the input, which would definitely be wrong.

...

I'm not entirely sure why it's necessary, but I've seen a lot of instructions for this sort of thing that have "conv=sync,noerror" in them.

See A Dumas reply which echoes my own misgivings.

Also, Chinese whispers.. The number of times I research a topic and find the same instruction on ten different websites that is actually obsoleted by the version of code I am running...in reality probably someone once, got it to work using that, and everyone else has just copied it blindly. You have stumbled on a case where it doesn't seem to work, and you will write the next definitive guide, to be plagiarised a hundred times..

The main advantage of using sync,noerror is to replace bad blocks and keep going.

Is that likely to cause problems with slower storage media, when a slow read may be interpreted as a failure but a different application might be less fussy and wait for it to complete? I'm guessing it would be the same bits of file-system/kernel code running in both cases so no different, but maybe not.

I should have tried to run without suppressing errors. I think The Natural Philosopher is right about Chinese whispers, I used these options without looking into dd options carefully. Actually it's odd I used conv=noerror as I usually like to check for errors.

The conv=sync was also a bad idea as it fills out short inputs with zeroes. When I ran the restore without conv=noerror it gave me this warning: dd: warning: partial read (32768 bytes); suggest iflag=fullblock

So I think what it was doing was filling the rest of the block with zeroes on each partial read then writing the output. I also think the operation completed more quickly without this argument which would have been because it would have been reading only 32k then writing a whole

The number of whispers saying the same thing misled me into assuming it was gold information. Damn Chinese! (joke - nothing against the Chinese). I was obviously a bit lazy.

So I tried the following lines which worked: # dd bs=1M count=3752 if=/dev/sdc | gzip -c >/backfile.img.gz # gunzip -c /backfile.img.gz | dd bs=1M count=3752 iflag=fullblock of=/dev/sdc

Actually it now strikes me that I should also use iflag=fullblock on the intial SD card read. Thinking about it, whenever you use a count it would seem that iflag=fullblock should always be used, otherwise the count is a count of read operations.

The reason for the block count is that the SD card I used was 16GB but the partitions were much smaller - about 3.8GB - so there was no point writing beyond this. Also if you read the whole SD card you'd get a much larger file unless you first filled the empty space with zeroes. I used a spreadsheet to check the figures reported by fdisk. There were 2 partitions but the first started 4MB into the card.

So problem solved, thanks for all the replies.

Final solution: # dd bs=1M count=3752 iflag=fullblock if=/dev/sdc | gzip -c >/backfile.img.gz # gunzip -c /backfile.img.gz | dd bs=1M count=3752 iflag=fullblock of=/dev/sdc

Also, you don't need the count on the last dd that is doing the writing. It will just write the contents of the file, which is the correct length.

If at all possible post this up on a website where the next poor sucker will find the better 'magic spell'

Also, you don't need -c on gzip because if there's no input file, output will go to stdout automatically. So you do need it on gunzip, yeah.

Ok my writeup of the command lines is:

To create a compressed image file: (make sure the count is correct or omit it for safety count can be found from fdisk use End block address of last partition +1, x 1024 for bytes, divide by your bs (here=1M))

# dd bs=1M count=3752 iflag=fullblock if=/dev/sdc | gzip >/backfile.img.gz

To restore the SD card: # gunzip -c /backfile.img.gz | dd bs=1M iflag=fullblock of=/dev/sdc

I would, but don't have my own. Please suggest best place - a forum?

You can even use # zcat /backfile.img.gz | dd bs=1M of=/dev/sdc

"zcat filename" is the equivalent of "gunzip -c filename", but slightly easier to type.

I don't know exactly what the iflag=fullblock does. Does it improve performance?

iflag=fullblock does absolutely nothing unless you specify a value for ibs. If ibs is specified (not bs=, but ibs=), then dd will do multiple reads up to the value of ibs before continuing. This (usually) means that dd will spawn multiple processes, which can cause other issues when used in a pipe...

It is vital in this case. See this from "info dd": Note if the input may return short reads as could be the case when reading from a pipe for example, `iflag=fullblock' will ensure that `count=' corresponds to complete input blocks rather than the traditional POSIX specified behavior of counting input read operations. Also see my previous post in this thread about "conv=sync".

No. bs also affected: "bs=" overrides any ibs= and obs= settings

When in doubt, check the source code as I did. The internet is not a substitute for knowledge.