Failure to clone SD card with compression

I am having difficulty using gzip for compressing images of SD cards. If
I use command lines such as
# dd bs=1M count=3752 if=/dev/sdc conv=sync,noerror of=/backfile.img
# dd bs=1M count=3752 if=/backfile.img of=/dev/sdc conv=sync,noerror
for making the image and restoring it to another card, it works ok.
However if I use compression like this:
# dd bs=1M count=3752 if=/dev/sdc conv=sync,noerror | gzip -c
/backfile.img.gz
# gunzip -c /backfile.img.gz | dd bs=1M count=3752 of=/dev/sdc
conv=sync,noerror
it fails to restore the image correctly. The MBR and the partitions
cloned are not the same as the original.
What error(s) am I making?
Reply to
Russell Gadd
Loading thread data ...
Why not use gzip on its own rather than as part of a pipeline? I.e. do the dd to a file as you have been, then separately gzip that file. (Or better bzip2 or xz it, these days.) To use it, gunzip it then dd the uncompressed image.
Reply to
Roger Bell_West
It still should work though, but that's a good first test.
Why are you using count and conv with dd?
--
Karl Marx said religion is the opium of the people. 
But Marxism is the crack cocaine.
Reply to
The Natural Philosopher
I'd suggest 'partimage' instead.
Reply to
ray carter
/dev/sdc and / are on different filesystems, yes? Just checking. Use gzip with piped input without -c. Maybe try without noerror to see if an error occurs. I'm not exactly sure which buffers get padded with the sync option, but that sounds like maybe the output is not identical to the input, which would definitely be wrong.
Reply to
A. Dumas
...
I'm not entirely sure why it's necessary, but I've seen a lot of instructions for this sort of thing that have "conv=sync,noerror" in them.
--
Apparently I lack some particular perversion which today's 
employer is seeking.                 --- Ignatius J Reilly
Reply to
Adam Funk
See A Dumas reply which echoes my own misgivings.
Also, Chinese whispers.. The number of times I research a topic and find the same instruction on ten different websites that is actually obsoleted by the version of code I am running...in reality probably someone once, got it to work using that, and everyone else has just copied it blindly. You have stumbled on a case where it doesn't seem to work, and you will write the next definitive guide, to be plagiarised a hundred times..
--
The biggest threat to humanity comes from socialism, which has utterly  
diverted our attention away from what really matters to our existential  
 Click to see the full signature
Reply to
The Natural Philosopher
The main advantage of using sync,noerror is to replace bad blocks and keep going.
Reply to
ray carter
Is that likely to cause problems with slower storage media, when a slow read may be interpreted as a failure but a different application might be less fussy and wait for it to complete? I'm guessing it would be the same bits of file-system/kernel code running in both cases so no different, but maybe not.
Reply to
Rob Morley
I should have tried to run without suppressing errors. I think The Natural Philosopher is right about Chinese whispers, I used these options without looking into dd options carefully. Actually it's odd I used conv=noerror as I usually like to check for errors.
The conv=sync was also a bad idea as it fills out short inputs with zeroes. When I ran the restore without conv=noerror it gave me this warning: dd: warning: partial read (32768 bytes); suggest iflag=fullblock
So I think what it was doing was filling the rest of the block with zeroes on each partial read then writing the output. I also think the operation completed more quickly without this argument which would have been because it would have been reading only 32k then writing a whole 1M. However this is just an impression, I didn't time it.
The number of whispers saying the same thing misled me into assuming it was gold information. Damn Chinese! (joke - nothing against the Chinese). I was obviously a bit lazy.
So I tried the following lines which worked: # dd bs=1M count=3752 if=/dev/sdc | gzip -c >/backfile.img.gz # gunzip -c /backfile.img.gz | dd bs=1M count=3752 iflag=fullblock of=/dev/sdc
Actually it now strikes me that I should also use iflag=fullblock on the intial SD card read. Thinking about it, whenever you use a count it would seem that iflag=fullblock should always be used, otherwise the count is a count of read operations.
The reason for the block count is that the SD card I used was 16GB but the partitions were much smaller - about 3.8GB - so there was no point writing beyond this. Also if you read the whole SD card you'd get a much larger file unless you first filled the empty space with zeroes. I used a spreadsheet to check the figures reported by fdisk. There were 2 partitions but the first started 4MB into the card.
So problem solved, thanks for all the replies.
Final solution: # dd bs=1M count=3752 iflag=fullblock if=/dev/sdc | gzip -c
# gunzip -c /backfile.img.gz | dd bs=1M count=3752 iflag=fullblock of=/dev/sdc
Reply to
Russell Gadd
Also, you don't need the count on the last dd that is doing the writing. It will just write the contents of the file, which is the correct length.
Reply to
Dom
If at all possible post this up on a website where the next poor sucker will find the better 'magic spell'
--
The biggest threat to humanity comes from socialism, which has utterly  
diverted our attention away from what really matters to our existential  
 Click to see the full signature
Reply to
The Natural Philosopher
Also, you don't need -c on gzip because if there's no input file, output will go to stdout automatically. So you do need it on gunzip, yeah.
Reply to
A. Dumas
Ok my writeup of the command lines is:
To create a compressed image file: (make sure the count is correct or omit it for safety count can be found from fdisk use End block address of last partition +1, x 1024 for bytes, divide by your bs (here=1M))
# dd bs=1M count=3752 iflag=fullblock if=/dev/sdc | gzip >/backfile.img.gz
To restore the SD card: # gunzip -c /backfile.img.gz | dd bs=1M iflag=fullblock of=/dev/sdc
Reply to
Russell Gadd
I would, but don't have my own. Please suggest best place - a forum?
Reply to
Russell Gadd
You can even use # zcat /backfile.img.gz | dd bs=1M of=/dev/sdc
"zcat filename" is the equivalent of "gunzip -c filename", but slightly easier to type.
I don't know exactly what the iflag=fullblock does. Does it improve performance?
Reply to
Dom
iflag=fullblock does absolutely nothing unless you specify a value for ibs. If ibs is specified (not bs=, but ibs=), then dd will do multiple reads up to the value of ibs before continuing. This (usually) means that dd will spawn multiple processes, which can cause other issues when used in a pipe...
Reply to
CPMDude
It is vital in this case. See this from "info dd": Note if the input may return short reads as could be the case when reading from a pipe for example, `iflag=fullblock' will ensure that `count=' corresponds to complete input blocks rather than the traditional POSIX specified behavior of counting input read operations. Also see my previous post in this thread about "conv=sync".
Reply to
Russell Gadd
No. bs also affected: "bs=" overrides any ibs= and obs= settings
Reply to
Russell Gadd
When in doubt, check the source code as I did. The internet is not a substitute for knowledge.
Reply to
CPMDude

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.