Create NDIF disk image from RasPi SD card

Similarly, I use two USB 2.5" hard drives (currently a pair of 375GB WD Blue hard drives) to take weekly backups. This is mild paranoia: I use two, rotated in sequence, so one is always offline in a firesafe and so fairly safe from damage.

However, IMO the key trick is using rsync to do the backups. Its fast because it always does the minimum work needed to make a full backup: it only copies files and directories that have changed to the backup disk and removes files from the backup that have been deleted from the disk being backed up. The firdst backup to a hew disk is always slow because everything is backed up: subsequent backups can be 20-30 times faster depending on how much has changed/

There is a related program, rsnapshot, that preserves deleted files, but I haven't used it. I think it works like this: like rsync, it initially makes a full backup as the base snapshot and then preserves a set of snapshots, with the files and directories in each being either a copy of a file that changed or a symlink the most recent backed-up copy. Either of these is probably faster than using dd for backups and is certainly much faster than using an archiver such as tar (with or without compression), zip, lharc, etc.

--
Martin    | martin at 
Gregorie  | gregorie dot org

On a sunny day (Tue, 7 Aug 2018 10:04:06 +0000 (UTC)) it happened Martin Gregorie wrote in :

I use a backup script per program code I work on. You can use cp -urp if you want recursive. Only use mkdir the first time to create the directories on the target device after checking it was really mounted, else it can leave you in the illusion you are copying to some device, while it actually copied to local directories created on /mnt/...

mount /dev/sdg1 /mnt/sdg1 ##mkdir -p /mnt/sdg1/compile/pantel/weather_sensor > /dev/zero cp -up * /mnt/sdg1/compile/pantel/weather_sensor/ df sync umount /dev/sdg1 echo df

Takes only a second to run, say with an formatted USB stick in /dev/sdg That is level 1 backup. I do that every time I add a new feature to the code, or write a lot of code. Very useful as you can get old working code back if you get some strange errors with the new code. sdg is in this case the USB stick I always carry with me.

The dd thing for the whole raspi image I only do every now and then, sometimes with month in between, sometimes days if a lot was changed or a lot was added with apt-get.

I have similar scripts for email backup in /root/mail/ (I use pine). The result is that my emails are still all nicely organized and go back to 1998 on this system (been using pine since then), same for Usenet as I wrote the reader... And easy to search for anything in /root/mail with grep... So much faster.... Simplicity.

The best way to avoid losing source code, scripts, web pages, etc is to use a version control package in addition to conventional backups.

I generally use CVS but am also using git for a couple of projects.

CVS is quite old now, but has two advantages:

- its source repository is always separate from the source code. It may be in a different user on the same host, but it can equally be on a separate server on your LAN

- everybody commits to the same repository

- as the repository is a mirror of your source tree, it can sometimes be convenient to go in as the CVS user and modify it with cp, mv and friends or to edit source files (these contain the original source with sets of comment+changes appended for each commit

git seems pretty good, but:

- its repository is a large, opaque file structure in the root directory of the source tree, so if it does wrong you're stuffed.

- you can, however, push the local repository to update a remote one (on github or your own server) and pull a fresh update from the remote repository followed by an update to bring your source tree into line with the local repository. Both have the advantage that, if you completely screw up an edit, you just delete the wrecked file and run an update, which pulls the latest version out of the repository.

In both version control systems, if its a shared project you need to run an update to pull down the latest commits before editing anything ('cvs update' for CVS and 'git pull; git update' for git.

I've been using version control systems for over 3 decades and would not dream of writing code, scripts or web pages without using version control.

My local CVS repository is on my house server and is used for all projects regardless of whether they are developed on the server, one of my laptops or my Raspberrypi.

--
Martin    | martin at 
Gregorie  | gregorie dot org

I remember a guy I used to work with at ICL had a sign above his desk saying "Never underestimate the information transfer rate of a truck-load of mag tapes" which is the same principle.

That was in the days of reel-to-reel mag tapes that were about 18 inches diameter - not even the QIC cartridges or DAT tapes. The joys of going down to the firesafe in the bowels of BRA01, to get the next backup tape to load into the tape drive and to take last night's tape to the safe.

Are tapes still used much for computer backups, or has everyone gone over to remote off-site backup onto hard disc? I used to backup my own things to CD-RW, back in the days when I could fit all my documents, emails and photos onto a single CD. It was a real PITA having to wait while the drive added a new session to store the new or changed files, and then incur an even longer delay while the disc was wiped clean to recover all the space that was lost from obsolete versions of a file.

Backing up to memory stick or external HDD has speeded things up dramatically.

On a sunny day (Tue, 7 Aug 2018 12:08:55 +0000 (UTC)) it happened Martin Gregorie wrote in :

Yes, I am aware of those systems, sometimes I look at code, sometimes I download things. Maybe my ego or way of coding is not suited for multi-contributing coders.. Last time somebody emailed me with some ideas causing me to re-write some code his way, after some back and forward emails finally got to the point where he said: 'Why do you not do it that way', to which I replied 'That was the way the original was done'. I do not always say 'why' (almost never actually) I write code in some way, most is based on small tests and incremental code. Gets complicated and hard to follow it seems for others... So what I do is use scripts, when I think I have tested enough and it MAY be of use to others, I just run a script called to-web version_number from the coding directory and that puts it on my website. After that I update the VERSION define, and all version references in the documentation. The program-xx.tgz is the release and also my backup. It is then on a US server, my own server, and later on my backups. Here a screen shot of the current working directory of a very simple program I released as 'weather_sensor-0.2.tgz' a while ago: http://217.120.43.67/nuclear/ws.gif

Not even on the main web site yet, only on the raspi server as http://217.120.43.67/nuclear/weather_sensor-0.2.tgz bks is the backup script, to-net is the script that makes the ..tgz and sends it to the website and test1 is how I tested the code (command line option etc).

I have some programs on some other sites, like this one that reads data from seven segment displays

But almost all public releases are at this link:

Some programs are thousand of lines of code, I would not want anyone else changing one character anywhere in it :-) It could make it better, or worse, but would break my view, oversight. Several times I just gave the advice: Split it off under your own program name. And so it happened. Saves me answering a hundred email questions about things.

That is an other aspect, cute, sometimes I get a bug report about code I wrote 20 years ago. If it is justified I fix it make a new release. Happened some month ago.

On a sunny day (Tue, 7 Aug 2018 13:41:10 +0100) it happened "NY" wrote in :

Not so long ago I think it was Sony that came out with a yet bigger tape just for that

IBM and Sony cram up to 330 terabytes into tiny tape cartridge:

\

330 *tera*bytes? Blimey. I bet it takes ages to locate the file that you want to retrieve in the event of file corruption or accidental deletion. That's where hard discs are good for backups because they are random access. I use Microsoft SyncToy for doing incremental backups, which has the advantage that the backup is stored as an exact file-and-folder copy of the original, so you don't need proprietary software to decode a huge backup file to extract the file you want. Mind you, it doesn't store multiple versions of the same file...

:-)

Nearer 10.5 inches IIRC, holding 2400ft of 0.5 inch tape.

I think you'll find LTO cartridges are still quite widely used for archival storage.

tape. The latest spec (LTO-8) stores 12TB in each cartridge.

Bit of a change from the old ICL drives which, IIRC held about 8 MB per reel.

I went directly from floppy disk backups to DAT because I didn't trust CD- RW and anyway it was both slow and lacking in capacity. When I eventually ditched Windows for RedHat Linux (around 2003), I used the Amanda package to manage DAT backups. I was quite happy with that, but a few years later, after Redhat introduced Fedora distro, the DAT drives I was using (capacity around 4GB) became too small for what I needed, so I switched to using removable USB 2.5" hard disk drives with backups managed by custom scripts wrapped round rsync. That's where I've stayed: the only difference is that the disks have got a bit bigger since then.

--
Martin    | martin at 
Gregorie  | gregorie dot org

Yep tapes are for disaster recovery, online or nearline backups are much more practical for day-to-day mistake correction.

--
Steve O'Hara-Smith                          |   Directable Mirror Arrays 
C:\>WIN                                     | A better way to focus the sun 
The computer obeys and wins.                |    licences available see 
You lose and Bill collects.                 |    http://www.sohara.org/

Yup.

A 4-dimensional cartridge? No wonder they hold so much data. :-)

Depends on block size. Short blocks mean that the 0.6-inch inter-record gaps are longer than the data blocks themselves. But if you used a decent block size (e.g. 4K) you could get about 20 MB on a 2400-foot reel at 800 bpi. Newer, higher-density drives could do even better.

--
/~\  cgibbs@kltpzyxm.invalid (Charlie Gibbs) 
\ /  I'm really at ac.dekanfrus if you read it the right way. 
 X   Top-posted messages will probably be ignored.  See RFC1855. 
/ \  Fight low-contrast text in web pages!  http://contrastrebellion.com

I wish. Periodically the Pan/XFCE/Fedora combo screws up ^C paste operations by pasting somewhere I didn't intend. This was one of them...

Yeah, that was a guess and I don't remember the actual capacities except that real tapes held a lot more than George 3 emulated tapes (245 Kword IIRC). Not that that prevented one idiot G3 site I knew from using one of the old ICL tape sorts with emulated worktapes to sort quite large production workloads when there was a good, fast disk sort available.

--
Martin    | martin at 
Gregorie  | gregorie dot org

I disscovered that te cost of a hard disk wasa lot less than a DAT drive so went straight from CDs to multiple nachines with disks in them

Nowadays USB drives are not much ,more expenmsive than a tape cartridge

--
"Socialist governments traditionally do make a financial mess. They  
always run out of other people's money. It's quite a characteristic of them" 

Margaret Thatcher

On a sunny day (Tue, 07 Aug 2018 11:47:00 +1200) it happened snipped-for-privacy@f1208.n80.z2.binkp.net (Henri Derksen) wrote in :

Interesting question. Took it, removed the dust from it, and plugged it in the raspi running the webserver. Expected it to crash (How much power does it draw?), but: root@raspberrypi:~# dmesg [516954.521704] usb 1-1.3: new full-speed USB device number 5 using dwc_otg [516954.674780] usb 1-1.3: New USB device found, idVendor=03ee, idProduct=6901 [516954.674817] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [516954.674833] usb 1-1.3: Product: MITSUMI USB FDD 061M [516954.674847] usb 1-1.3: Manufacturer: MITSUMI [516954.678831] usb-storage 1-1.3:1.0: USB Mass Storage device detected [516954.691847] scsi0 : usb-storage 1-1.3:1.0 [516955.701095] scsi 0:0:0:0: Direct-Access MITSUMI USB FDD 061M 2.00 PQ: 0 ANSI: 0 CCS [516955.814079] sd 0:0:0:0: Attached scsi generic sg0 type 0 [516959.925614] sd 0:0:0:0: [sda] Attached SCSI removable disk

So far so good. Now to find a floppy..

Found some floppy with Linux filesystem root@raspberrypi:~# fdisk /dev/sda You will not be able to write the partition table. fdisk: unable to read /dev/sda: Invalid argument no go lots of noises from the drive trying, disk is write protected it seems..

How about reading raw data? root@raspberrypi:~# dd if=/dev/sda bs=512 count=1000 > q1 dd: reading /dev/sda': Input/output error

8+0 records in 8+0 records out 4096 bytes (4.1 kB) copied, 6.42305 s, 0.6 kB/s

Let's see what we got from the boot sector etc: root@raspberrypi:~# hexedit q1

YES!!! lots of zeros and some data!

Other disk, old write protected backup ext2 format root@raspberrypi:~# fdisk /dev/sda You will not be able to write the partition table. Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel with disk identifier 0x72867184. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/sda: 1 MB, 1474560 bytes

1 heads, 3 sectors/track, 960 cylinders, total 2880 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x72867184

Device Boot Start End Blocks Id System

Still no sign of sda1, lemme see if I can mount it root@raspberrypi:~# mkdir /mnt/sda1 root@raspberrypi:~# mount /dev/sda1 /mnt/sda1 mount: special device /dev/sda1 does not exist

Nope

root@raspberrypi:~# mount /dev/sda /mnt/sda1 mount: block device /dev/sda is write-protected, mounting read-only AHA!!!

....

-rw-r--r-- 1 root root 28816 Jan 3 2002 new-timed.txt

-rw-r--r-- 1 root root 28794 Jan 3 2002 new-timed-strange-chars-test.txt

-rw------- 1 root root 0 Jan 3 2002 new-timed.ssa

-rw-r--r-- 1 root root 29238 Jan 3 2002 moving-new-timed.txt

-rw-r--r-- 1 root root 29251 Jan 3 2002 moving-new-timed-1.txt ....

-rw-r--r-- 1 root root 3727 Jan 3 2002 demo.ppml

-rw-r--r-- 1 root root 138 Jan 3 2002 cd-id.ppml ...

't works

I am not going to write to any of these disks, and have no free 3.5 inch disk.

root@raspberrypi:~# mount /dev/root on / type ext4 (rw,noatime,data=ordered) devtmpfs on /dev type devtmpfs (rw,relatime,size=219744k,nr_inodes=54936,mode=755) tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=44784k,mode=755) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=89560k) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) /dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) /dev/sda on /mnt/sda1 type ext2 (ro,relatime)

@PID: FM 2.02 Hello Jan,

HD>> Do you know if it is possible to get that USB Floppy disk drive working HD>> at a Raspberry Pi?,

JP> Interesting question. JP> Took it, removed the dust from it, and plugged it in the raspi running JP> the webserver. JP> Expected it to crash (How much power does it draw?), but:

That was/is also my point of interest. May be you can measure what it draws in Amps.?

I also digged out my Iomega 3,5" 1,44 MB USB2-Floppy diskdrive. This afternoon I got it working, and could even write a new Test directory and a large textfile in it. This evening/night It dont want to work again, at the first attempts. I unplugged everything, including RJ45 EtherNet Cable, and only left the USB dongle for wireless keyboard and mouse and the USB-floppy drive, and now it works when plugged in into the bottum ports as close to the Pi3B print. I think a powered USB2 hub shall be necessarry for the power hunger of this floppydrive.

JP> root@raspberrypi:~# dmesg

What a long list does that command "dmesg" on Raspbian Stretch gives to me.

JP> Now to find a floppy..

JP> Found some floppy with Linux filesystem JP> root@raspberrypi:~# fdisk /dev/sda JP> You will not be able to write the partition table. JP> fdisk: unable to read /dev/sda: Invalid argument JP> no go JP> lots of noises from the drive trying, disk is write protected it seems..

Hm. Yould better try a new emty floppy disk. I even have some new unused box of floppy disks.

JP> Let's see what we got from the boot sector etc: JP> root@raspberrypi:~# hexedit q1

He, were did you get that "hexedit" from? It did not work at mij Raspbian Stretch command line prompt. I only installed "Jeex", but am not familiair with it.

JP> YES!!! lots of zeros and some data!

Ofcourse ;-).

JP> Other disk, old write protected backup ext2 format JP> root@raspberrypi:~# fdisk /dev/sda JP> You will not be able to write the partition table. JP> Device contains neither a valid DOS partition table, nor Sun, SGI or OSF JP> disklabel JP> Building a new DOS disklabel with disk identifier 0x72867184. JP> Changes will remain in memory only, until you decide to write them. JP> After that, of course, the previous content won't be recoverable. JP> Warning: invalid flag 0x0000 of partition table 4 will be corrected by JP> w(rite)

Yopu really did not have a floppy disk with some free space to write on?

JP> Command (m for help): p

What does P do?

JP> Still no sign of sda1, lemme see if I can mount it JP> root@raspberrypi:~# mkdir /mnt/sda1 JP> root@raspberrypi:~# mount /dev/sda1 /mnt/sda1 JP> mount: special device /dev/sda1 does not exist

Strange.

JP> Nope

JP> root@raspberrypi:~# mount /dev/sda /mnt/sda1 JP> mount: block device /dev/sda is write-protected, mounting read-only JP> AHA!!!

So, use a new floppy disk.

JP> 't works

He, he ;-)

JP> I am not going to write to any of these disks, and have no free 3.5 inch JP> disk.

Strange. It is a good idea to always have some media with free space. You never know what comes on your way you want to store rapidly. So always have some free floppy's, USB-sticks and microSDcards etc.. Of course you could make an image of that floppy and write that to another medium with enough free space to store that image, i.e. harddisk. Then you can testwrite that floppy and after that write back that backupped image you earlier made.

JP> So it works, write should work too on a not write protected disk. JP> Same for other filesystems.

Yes.

JP> What surprises me, is that the rapi wallwart (voeding in Dutch) does not JP> go through its knees. JP> Must take some current that drive motor seeking back and forward.

The supply power is not the problem, what the Pi lets through does. It is NOT the 4 x 0.5 Amps total for each USB2.0 ports, you would expect.

Even if you have the original 2.5 Amps power supply, and the Pi takes 0.5 A from it, there are NOT 2.0 Amps available for USB devices! ;-(. So a powered USB hub is needed for this USB2 floppy device.

Henri.

On a sunny day (Sat, 11 Aug 2018 01:38:00 +1200) it happened snipped-for-privacy@f1208.n80.z2.binkp.net (Henri Derksen) wrote in :

That would require cutting the lead and putting an amp meter in, I'd like to keep it in one piece.

Good!

Yes, P3 uses even more power than my old raspi.

I have a big box full of old floppies, but all are write protected, backups of projects I did long time ago.

Looks like I did apt-get hexedit It is the hexadecimal editor, one of the most useful tools I have.

Perhaps, but did not want to take a change with my backups.

Once you started fdisk /dev/sda you can type more commands, typing 'm' shows those, command 'p' prints the partition table. As it shows no partition table (sda1 for example) that means the whole floppy is used. So that is why I cannot mount /dev/sda1, but must mount /dev/sda. Be careful with fdisk, it can delete and create partitions too.

I got these raspis and power supplies from

(no connection with them) so far the stuff works OK.

Yes, well I had a wireless keyboard adaptor in one USB slot, and the floppy in the other, no need to reboot even, this is an old raspi with only 2 USB slots I use for webserver. Usually my rapis crash if you plug _anything_ in the USB.

Free floppies? The whole thing is only 1.44 MB, why still use floppies except to get back old code from backups? What will you use it for?

Greetings Jan

PS you often (?) see me using commands like l

Those are aliases to reduce typing.

You can make those aliases by editing ~/.bashrc and adding those, like I did here:

# some more ls aliases alias ll='ls -alF' alias la='ls -A' alias l='ls -rtl' alias d='ls' alias lb='ls -rtl --color=none'

l (ls -rtl) is useful as it shows what you did last after scrolling of the screen, including modification times and length.

One thing is bash, I do not use bash on my Linux PCs, but use zsh as shell. zsh saves a lot of typing if you are working on a terminal all the time,

Have not put zsh on my raspis yet, would improve things.

Greetings Jan

On latest Raspbian: $ apt-cache search hexadecimal bless - A full featured hexadecimal editor hexedit - view and edit files or devices in hexadecimal or ASCII jeex - visual editor to view and edit files in hexadecimal okteta - hexadecimal editor for binary files radare - free advanced command line hexadecimal editor radare-gtk - free advanced command line hexadecimal editor with graph functionality radare2 - free and advanced command line hexadecimal editor wxhexeditor - hexadecimal editor for massive files

Quick evaluation: bless depends on Mono, so strike that. hexedit seems good but Raspbian (Debian Stretch) repo version 1.3-1 out of date, see

jeex looks like abandonware, homepage gone. Also depends on gtk+. okteta is part of KDE. radare2 (and all their other tools) looks amazing wxhexeditor, like the name suggests, is graphical and cross-platform. Doesn't look very neat or organised, though. Version (no) control all over the place. No recent edits.

Or keep them all in one file ~/.bash_aliases which, if it exists, is loaded from the standard .bashrc (in Raspbian/Debian Stretch).

Hello Martin,

MG> Similarly, I use two USB 2.5" hard drives (currently a pair of 375GB WD MG> Blue hard drives) to take weekly backups. This is mild paranoia: I use MG> two, rotated in sequence, so one is always offline in a firesafe and so MG> fairly safe from damage.

I never want WD drives ;-(. I have to many friends who got problems wit that make and types.

MG> However, IMO the key trick is using rsync to do the backups. Its fast MG> because it always does the minimum work needed to make a full backup: it MG> only copies files and directories that have changed to the backup disk MG> and removes files from the backup that have been deleted from the disk MG> being backed up. The firdst backup to a hew disk is always slow because MG> everything is backed up: subsequent backups can be 20-30 times faster MG> depending on how much has changed/

Yes that's ideal, it is called mirrorring. The destination drive is after the selective copy a complete imaga of the source media.

But what happens when at the destination there is a file called ProjectX, and on the source media it became a directory called ProjectX, with several files more in them. Is then de original file ProjectX deleted before making a directory ProjectX at the destination, or does rsync give an error that the directory ProjectX cannot be created, as there is already a file with that name? Then te backup copy process is interrupted for an answer to gon on.

MG> There is a related program, rsnapshot, that preserves deleted files, but MG> I haven't used it. I think it works like this: like rsync, it initially MG> makes a full backup as the base snapshot and then preserves a set of MG> snapshots, with the files and directories in each being either a copy of MG> a file that changed or a symlink the most recent backed-up copy.

Did you test this?

MG> Either of these is probably faster than using dd for backups and is MG> certainly much faster than using an archiver such as tar (with or MG> without compression), zip, lharc, etc.

Yes of course, but is it working secure?, and has that been extremely tested? Because everyone wants to be sure to have good, complete and accurate backups.

I once bought and paid for a commercial program called !SaveStore for RISC OS, but it could NOT do secure mirror backups. I.e. to delete the files at the destination that were removed from the source media, and copy only the missing objects. It did the copy partially, and not always the deleting. I tested that many times and it was very unreliable ;-(. I tried wit with a simple set of subdirs at a RAMdisc.

!DirSync is a much more reliable directory and file copying program for this kind of work. But there something goes wrong either. The problem lies in files that are the same, but treated as different, or just the other way roud. I.e. the name, date and length are the same, but in that case you have to do a binary test for equality. I.e. they still can be different. Same name, date and length are the same, but the contenst could nog be 100 % identical. Then you have to make a decission what to do?

And when files are 100 % identical, the oldest version should be kept, as that is likely to be the original, and the other one "stamped". Many programmers always stamps even unchanged files, arg... ;-(. Please donot do that. So now I am mostly copying manually, but that takes more time ;-(. I am using a RISC OS tool very much to stamp the original file to its original date ansd time stanp, if there are more incarnations of the 100 % idential contents and name of that file(s).

Now I am busy to backup all my USB sticks to an external USB2 HDD with an old Win2K laptop I once did in 2017, and I have to think what could be faster, delete a complete subdirectory, or copy all over it and delete the removed items. If you do not do the latter, the backup is always increasing bigger and bigger. So I like the mirror trick, but there is not much backup software that does that good on many platforms, i.e. Windows, RISC OS, and Linux. So, if you (or someone else) have suggestions, I am all ears to receive your comment with the only half one ear I have ;-).

Henri.