Create NDIF disk image from RasPi SD card

On Wed, 18 Jul 2018 15:36:41 +0200, snipped-for-privacy@apple-juice.co.uk (D.M. Procida) declaimed the following:

Crawling around the official site (I normally just grab the NOOBS full image)

Both are available as direct ZIP files, or as Torrent (which I've never used). If you fetched via Torrent, the format may be influenced by the client you used -- Firefox on Win10 is showing ".zip.torrent" if I click on the torrent link.

A Google search for NDIF seems to imply that it is a deprecated Apple-specific format (replaced by UDIF).

On Windows, if one has /identical spec/batch/ SD cards, I'd probably use Win32DiskImager to create an .IMG file from a configured SD card, then change cards and use Win32DiskImager to write the IMG file to the new cards. Note that minor differences between cards that claim to be the same size/speed can cause the imager program to fail (if one card reserves a few MB more for wear-leveling space the usable size becomes smaller than the IMG file one started with)

--
	Wulfraed                 Dennis Lee Bieber         AF6VN 
	wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

To be clear, I don't care much about the format of the disk image, just about having *a* disk image or file, created from an SD card, that I can then re-use.

Daniele

If you're on a Mac as I think you might be, insert the SD card.

$ diskutil list

look at the list of devices shown, find the one that's the right size, probably at the bottom of the list. For instance it might be shown as disk3. If you get this wrong you might nuke your system disc, so be careful.

$ diskutil unmountDisk disk3

if it's still showing in Finder (if you click the eject button in Finder it might disappear from the list).

$ sudo dd if=/dev/rdisk3 of=your-image.img bs=1m

Will make an image of the filesystem, the same size as your SD card (16GB,

32GB, whatever). If you want compression, try something like:

$ sudo dd if=/dev/rdisk3 bs=1m | xz > your-image.img.xz

Theo

On a sunny day (Wed, 18 Jul 2018 19:42:40 +0200) it happened snipped-for-privacy@apple-juice.co.uk (D.M. Procida) wrote in :

For an image from a SDcard to say a hardisk it makes no difference, as long as the size is OK, and preferable back to the same card for 'restore'.

I have no idea how to do it in MS windows ..

In Linux take the sdcard, put it in your 'puter, type dmesg see what device is just found then type dd if=dev/sd? of=cardimage.img where ? is the card device detected, probably sdb, but always check!

and to restore dd if=cardimage.img of=/dev/sd?

You need enough disk space.

You cannot, I repeat CANNOT backup from a raspi live system to some external disk or vice versa that way. The OS should not be running.

And PLS do not type the wrong device names, that may kill your PC OS.

What OS or whatever is on the card does not matter, it is just copying data. I do this on a regular basis, and it has rescued some old cards with a lot of stuff on it too. For that I have a big 1 TB external USB harddisk on the PC as backup, normally not even connected. Magnetic media last a long long time.

To my mind a disk *image* includes the things which nmake up its format, so if you really want a disk image you have its format whether you like it or not! :-)

--
Chris Green

Thanks, excellent instructions.

I didn't know it wasn't necessary to create the .img file before starting (which is what I was trying to do).

And, ejecting in the Finder does something more than just unmounting - if you eject, then the volume won't be available for disk operations. So it has to be an unmount, either in the Terminal or in Disk Utility.

Daniele

On Wed, 18 Jul 2018 19:49:02 +0100, Chris Green declaimed the following:

I suspect the OP meant the format (type) of the generated image file itself, and not that of the file system within the image. NDIF vs IMG vs ISO, etc.

--
	Wulfraed                 Dennis Lee Bieber         AF6VN 
	wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

Possibly, but given that some of the answers just did:- dd if= of= this doesn't apply really does it? You'll get an exact copy of the Pi's whole file sytem onto the other device.

--
Chris Green

Hello Theo,

T> $ sudo dd if=/dev/rdisk3 of=your-image.img bs=1m

At the Pi 3B+ this gives the error: dd: invalid number '1m'

Use 1M instead of 1m ;-). I am now writing Raspbian Stretch 2018-06-27 img from a DownLoad dir on a 32 GB usb card which is /dev/sda to another 32 GB card as /dev/sdb. The 16 GB SDcard booted from had too little free space as temporary storage. About 10 minutes I know the result ;-).

sudo dd if=/media/pi/rootfs/home/pi/Downloads/Raspbian/2018-06-27-raspbian- stretch.img of=/dev/sdb bs=1M

4600+0 records in 4600+0 records out 4823449600 bytes (4.8 GB) copied, 567.882 s, 8.5 MB He's ready and it works well. Ofcourse many things need configuration.

Henri.

Daniele is on a Mac, where it's 1m. Similarly for *BSD (from where Macs get their dd). On Linux it's 1M.

It's really rather annoying they have this difference.

(you can do without the 'bs' blocksize setting, but the copy will go a lot more slowly as it'll copy one byte at a time)

Theo

Well, if you have a linux box handy with an SD card reader:

Let's call the sd card reader /dev/sdz, so to make an image of the card:

dd if=/dev/sdz of=mypiimage.img

and to copy one to a new card

dd if=mypiimage.img of=/dev/sdz

The account you use will have to be a memeber of the group disk, or prepend sudo before the dd.

--
Consulting Minister for Consultants, DNRC 
I can please only one person per day. Today is not your day. Tomorrow 
isn't looking good, either. 
I am BOFH. Resistance is futile. Your network will be assimilated.

On a sunny day (21 Jul 2018 13:11:58 +0100 (BST)) it happened Theo wrote in :

Actually 512 bytes at the time, and that is often the exact sector size of SDcards, so normally no need to specify a bigger blocksize. Try it, makes no difference whatsoever, except for reporting perhaps. What does make a difference is fast cards, and a fast card reader.

From man dd ibs=BYTES read up to BYTES bytes at a time (default: 512) obs=BYTES write BYTES bytes at a time (default: 512)

Hello Jan,

RM>> (you can do without the 'bs' blocksize setting, TM>> but the copy will go a lot more slowly as it'll copy one byte at a time)

JP> Actually 512 bytes at the time, JP> and that is often the exact sector size of SDcards, JP> so normally no need to specify a bigger blocksize.

Sorry but of course there is a difference in speed versus blocksize.

JP> Try it, makes no difference whatsoever, except for reporting perhaps.

See my results below.

JP> What does make a difference is fast cards, and a fast card reader.

Yes, that too.

JP> From JP> man dd JP> ibs=BYTES JP> read up to BYTES bytes at a time (default: 512) JP> obs=BYTES JP> write BYTES bytes at a time (default: 512)

Remark that bs ibs and obs. I donot know the different effects of it. May be next round when I make new versions for the skippers.

See here my results of several days and hours of SDcard filling with the latest Raspbian Stretch Linux, OpenCPN and inland ECDIS navigation charts.

pi@raspberrypi:~ $ sudo dd if=/dev/sda of=/dev/sdb bs=1M

14784+0 records in 14784+0 records out 15502147584 bytes (16 GB) copied, 1707.88 s, 9.1 MB/s

1707.88 seconds is about 48 minutes.

Other tries with the same card reader/writers for source and targets with all 16 GB Kingston microSDcards Class 10 rated max. 80 MB/sec.:

pi@raspberrypi:~ $ sudo dd if=/dev/sda of=/dev/sdb bs=4M

3696+0 records in 3696+0 records out 15502147584 bytes (16 GB) copied, 1709.89 s, 9.1 MB/s

pi@raspberrypi:~ $ sudo dd if=/dev/sda of=/dev/sdb bs=4M

3696+0 records in 3696+0 records out 15502147584 bytes (16 GB) copied, 1940.11 s, 8.0 MB/s

pi@raspberrypi:~ $ sudo dd if=/dev/sda of=/dev/sdb bs=1M

14784+0 records in 14784+0 records out 15502147584 bytes (16 GB) copied, 1682.93 s, 9.2 MB/s

pi@raspberrypi:~ $ sudo dd if=/dev/sda of=/dev/sdb

30277632+0 records in 30277632+0 records out 15502147584 bytes (16 GB) copied, 6219.24 s, 2.5 MB/s pi@raspberrypi:~ $

103 minutes, that's much more than the first 48.

For the last Kingston SDcard, I had to use no blocksize, i.e. the default one, because it was a slower 16 GB Class 10 microSDcard rated max. 45 MB/sec. The source SDcard was a newer Kingston 16 GB Class 10 microSDcard rated max. 80 MB/sec. Using a bigger blocksize resulted in a not good filled slower card, especially the EXT4 partion was not copied right and had to much free space, en many directories and files missing ;-(. Even the Raspbian SDcard Copier could not handle this different source and target speeds, only DD with default blocksize did the right copy trick. So beware. And always dismount the source and target media before starting a backup with the Raspbian SDcard Copier or the DD-command. Otherwise you will damage the card's contents when dismounting the original blank fat image on the just copied one, i.e. small fat and large ext4 parts.

This night my 32 GB version is being copied as a backup. Unluckily I forgot the bs parameter this time, so it will take a lot more time I think. But that does not matter when I sleep. Tomorow I can report the exact result ;-). May be I should store SDcard images at an external HDD.

Another difference I remarked was that the Raspbian SDcard Copier left less free space on the destination card after copying the same sized source card. The 16 GB source card had 2.0 GB free, and the identical make and type target version only 1.9 GB, strange. When making the backup with the DD-command, they were exactly the same. But be very carefull when using DD, as it can be very dangerous when making mistakes in source and destination parameters, or inserting the cardreaders in a different order. That insertion order influences the device name, i.e. which one is /dev/sda and wich one /dev/sdb? So good luck in copying.

Geetings van Henri.

Block size does matter, sometimes rather a lot.

There are two components to this: the syscall overhead and write amplification. Syscall overhead is the number of times it has to shuttle back and forth between userland and kernel. On this Mac:

$ dd if=/dev/zero of=/dev/null bs=1m count=1024

1024+0 records in 1024+0 records out 1073741824 bytes transferred in 0.041791 secs (25693158687 bytes/sec)

$ dd if=/dev/zero of=/dev/null bs=1 count=1g

1073741824+0 records in 1073741824+0 records out 1073741824 bytes transferred in 1781.858768 secs (602596 bytes/sec) (the fans also went up to turbo for that time)

$ dd if=/dev/zero of=/dev/null count=2m

2097152+0 records in 2097152+0 records out 1073741824 bytes transferred in 3.446420 secs (311552811 bytes/sec)

- so if you're copying to an SSD it would make a difference. (but less so on a Pi, due to its woeful I/O)

However, there's another issue which is write amplification. If you're using an unbuffered write, which is what /dev/rdisk on Mac is, you'll be transmitting those 512 byte writes through to the SD card. The SD card erase block size is probably somewhere 128KB-1MB. If you write a 512 byte block, what's actually happening is it might read 1MB into a buffer, overwrite 512 bytes, then write 1MB back to a new block in the flash. So every 512 byte write amplifies to 1MB. Good SD cards hold some buffering so they can coalesce such writes but such buffering is costly for the microcontrollers in the card, so nasty SD cards just write the block back, doing vastly more work than they needed.

I have here such a nasty 2GB SD card, and I get:

$ sudo dd if=/dev/zero of=/dev/rdisk2 bs=1m dd: /dev/rdisk2: Input/output error

1929+0 records in 1928+0 records out 2021654528 bytes transferred in 268.585332 secs (7527047 bytes/sec)

$ sudo dd if=/dev/zero of=/dev/rdisk2 dd: /dev/rdisk2: Input/output error

3948545+0 records in 3948544+0 records out 2021654528 bytes transferred in 14503.783763 secs (139388 bytes/sec)

- a factor of 54 slower. By this I would assume that each write is amplified (roughly) 54 times, and deduce this (small) SD card has a flash block size of 32KiB.

In case you think the fault is using unbuffered mode, we can run the same test in buffered mode. It's better than 512 byte blocks unbuffered, but much worse than 1M unbuffered.

$ sudo dd if=/dev/zero of=/dev/disk2 bs=1m dd: /dev/disk2: end of device

1929+0 records in 1928+1 records out 2021658624 bytes transferred in 2222.524882 secs (909622 bytes/sec)

$ sudo dd if=/dev/zero of=/dev/disk2 dd: /dev/disk2: end of device

3948545+0 records in 3948544+0 records out 2021654528 bytes transferred in 2711.264947 secs (745650 bytes/sec)

Theo

On a sunny day (22 Jul 2018 18:59:16 +0100 (BST)) it happened Theo wrote in :

Interesting, I do not have a MAC, just Linux PCs. I find on my cards hardly any difference, reason why I started leaving out the bs= option.

909622 / 745650 is a factor 1.2199 So probably did not notice that.

The card itself, especially in write more, is the limiting factor.

Do not have a free card ATM, but here the read timings for a 8 GB raspi card on my Core5 laptop, card in /dev/sdc

~ # hdparm -t /dev/sdc /dev/sdc: Timing buffered disk reads: 50 MB in 3.03 seconds = 16.48 MB/sec

Without bs= ~ # dd if=/dev/sdc of=/dev/zero

16003072+0 records in 16003072+0 records out 8193572864 bytes (8.2 GB) copied, 472.769 s, 17.3 MB/s dd if=/dev/sdc of=/dev/zero 4.94s user 24.64s system 6% cpu 7:52.81 total

With bs= ~ # dd if=/dev/sdc of=/dev/zero bs=1M

7814+0 records in 7814+0 records out 8193572864 bytes (8.2 GB) copied, 474.15 s, 17.3 MB/s dd if=/dev/sdc of=/dev/zero bs=1M 0.07s user 16.86s system 3% cpu 7:54.45 total

So it is actualy slower with bs=1M specified ;-) Of course that is pure coincidence, system processes affect things. It does show the card read speed sets the speed, and matches almost exactly what hdparm showed.

Writing to SDcards is usually much slower. Then you get the same uncertainty, factor 1.2 does not mean so much, could be other way around man hdparm says to run it a couple of times too for a better idea.

On Sun, 22 Jul 2018 18:57:30 GMT, Jan Panteltje declaimed the following:

I wouldn't consider read timings to be of significance. As has been mentioned, SD cards -- to write to a block -- have to have an erased (all

1s) block available. However, erasure takes place in allocation units which are larger than file-system blocks. So before a "random" write can occur, the card has to erase a complete allocation unit, copy over data "before" the block to be written, write the new block, and if the I/O now shifts to another random block, copy the rest of the old allocation unit into the new one -- then put the old allocation unit into the free list.

Class-10 cards are rated for streaming a single (video) file to a freshly formatted card, and may be totally trashed by small random (photo creation/deletion) files. Oh, and that is FAT file system -- just the open file and the open FAT area, for two allocation units at a time.

The better cards (and most Class-2/4/6) are capable of maintaining (buffering) 4-8 open allocation units, which much better supports random I/O and usage of non-FAT (eg; journal ling) file systems. Things like alternating writes to two files -- both files can be in separate open allocation units without triggering unit closure, erasure, rewrite...

--
	Wulfraed                 Dennis Lee Bieber         AF6VN 
	wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

On a sunny day (Sun, 22 Jul 2018 17:43:33 -0400) it happened Dennis Lee Bieber wrote in :

Do not move the goal posts.

First, to MAKE a backup you need to read the card. ~ # dd if=/dev/sdc of=/dev/zero

16003072+0 records in 16003072+0 records out 8193572864 bytes (8.2 GB) copied, 472.769 s, 17.3 MB/s dd if=/dev/sdc of=/dev/zero 4.94s user 24.64s system 6% cpu 7:52.81 total

~ # dd if=/dev/sdc of=/dev/zero bs=1M

7814+0 records in 7814+0 records out 8193572864 bytes (8.2 GB) copied, 474.15 s, 17.3 MB/s dd if=/dev/sdc of=/dev/zero bs=1M 0.07s user 16.86s system 3% cpu 7:54.45 total

No difference whatsoever, set by the card's specified read speed if on a good system like this one, with nothing else running: ~ # hdparm -t /dev/sdc /dev/sdc: Timing buffered disk reads: 50 MB in 3.03 seconds = 16.48 MB/sec

That read happens (if things are done right) much more often than a 'restore' where you write the backup back to the card.. The restore may even never happen if things are done right.

On the write timing issue, it will very likely just be limited by he specified card write speed, if it is a half way decent card it will take care of the allocation.

Straw man Note that writing an image back to a card is totally unrelated to what sort of filesystem is in use.

Well, I recently bought let me see, MB-MC32DA/EU Samsung Evo+ 32 GB Micro SD class 10 met adapter That was in March, is used in my drone, and in my video camera. read speed up to 80 MB/sec. write speed up top 20 MB/sec.. UHS-I-speed class 1 (U1), class 10

Best ones I could find.

I have some older 8 GB and 16 GB cards in the raspis, but I am sure as I just went for the best in those days that those have a decent control chip.

Also have a bunch (from ebay) 2 GB cards for things that do not support > 2 GB, or do not need that and the speed, such as auto-pilot drone:

old WiFi webserver, etc. And of course I wrote the software to read / write to some of those cards myself in asm actually All sector based... :-)

You can use hdparm to get some info on the cards you have. Not sure you can still get older cards, the old 2 GB ones I got from ebay ware almost more expensive than the new fast ones.. getting rare for people who do not want to dump old equipment, sellers can ask what they want, like for antiques :-)

OK,

But next time I need to write back a raspi backup image (hope it never happens) I will show the write timings with - and without bs=1M.

:-)

I'm pretty sure it was you who moved them from talking about writes to reads!

With flash the reading is very different to writing, and the performance of one doesn't tell you anything about the other.

[Snip pretty much the same read results as before]

No, you misunderstand, it's not the filing system in the image that matters, but what filing system the controller in the SD card is optimised for. On some cards if you do not write to a card in the way it would be written to by a FAT32 filing system (use a different filing system or even raw writes), you get very poor performance.

We look forward to it.

---druck

Hello Jan,

HD> This night my 32 GB version is being copied as a backup. HD> Unluckily I forgot the bs parameter this time, HD> so it will take a lot more time I think. HD> But that does not matter when I sleep. HD> Tomorow I can report the exact result ;-). HD> May be I should store SDcard images at an external HDD.

So here are the results as promised:

pi@raspberrypi:~ $ sudo dd if=/dev/sda of=/dev/sdb

60456960+0 records in 60456960+0 records out 30953963520 bytes (31 GB) copied, 10383.6 s, 3.0 MB/s pi@raspberrypi:~ $

10384 seconds is 173 minutes or 2 hours and 53 minutes. pi@raspberrypi:~ $ sudo dd if=/dev/sda of=/dev/sdb bs=1M

29520+0 records in 29520+0 records out 30953963520 bytes (31 GB) copied, 3811.36 s, 8.1 MB/s

64 minutes for a 32 GB Class 10 card. A little bit slower as the 16 GB did took about 28 minutes. May be I should use 2M buffers for a 32 GB card the next time? You see here that higher up the buffer from default 512 KB block size up to 1 MB has a big effect in speed!

173 minutes versus 64 for the same amount is almost a 3 times increase! That's the proof that setting a right blocksize does matters really.

HD> Another difference I remarked was that the Raspbian SDcard Copier left HD> less free space on the destination card after copying the same sized HD> source card. HD> The 16 GB source card had 2.0 GB free, HD> and the identical make and type target version only 1.9 GB, strange. HD> When making the backup with the DD-command, they were exactly the same. HD> But be very carefull when using DD, as it can be very dangerous when HD> making mistakes in source and destination parameters, or inserting the HD> cardreaders in a different order. That insertion order influences the HD> device name, HD> i.e. which one is /dev/sda and wich one /dev/sdb?

You unluckily did not answer this point.

A new quirk from Rasbian Stretch (necessarry for a Pi 3B+) is that you cannot read a 100 % backup of a SDCard of the same size. At older versions of Raspbian (Jessie/Wheezy) this was no problem. First I thought my SDcards or reader/writer were defective. Then I remarked I could read a 16 GB backup SDcard when booted from a 32 GB Raspbian Stretch OS. The other way round also worked. But I can not read a 100 % 16 GB backup when booted at a 16 GB card. The same happens to a 100 % 32 GB backup when booted from a 32 GB card.

In the SDcard copier util I saw a new option to tick: New Partition UUIDs. I think that will make them different from the original booted from. Yes, I found the answer today in the help text from the SDcard Copier util:

Under Raspbian Stretch and later versions, you cannot mount two partitions with the same UUID, so you will not be able to mount a cloned SD card when booted from the disk from which it was cloned. If you need to do this, check the "New Partition UUIDs" box before copying.

Ok, that's a change we have to live with since 14 March 2018 at the Pi 3B+. I luckily now know a solution around this new quirk. I helps a lot if you have at least two USB2 (micro-)SDcard readers/writers.

Even if you donot have a Pi 3B+ but older ones, it is still a good idea to start using the last raspbian Linux, in this case Stretch over Jessie or Wheezy.

But I still do not understand why the resulting copy has a lower free space? That's why I am using now the dd Linux command as showed above, which results of course in the same UUIDs, but with the same free space. So good luck in copying.

Greetings van Henri.