Shrinking the SD card occupancy

Wise man. A backup *isn't* a backup until it has been demonstrably restored. There are plenty of tales of large companies buying and using expensive backup systems which failed to restore when the inevitable happened..

--
Rob
Reply to
Rob
Loading thread data ...

Yes, it's something I should do, but I have more faith in a full disk (SD) image backup than in the file-by-file methods mentioned by others. It's very simple. I know the disk writing works OK - that's how the cards got their OS in the first place. Surely /reading/ the SD card should be just as reliable .... ahem!

--
Cheers, 
David 
 Click to see the full signature
Reply to
David Taylor

FWIW, at least one reason for using a rsync/rsnapshot/tarball method to back up files over ethernet or USB is the avoid yanking and replacing the SD card for each backup. Judging by the comments I've seen about the fragility of the onboard SD card reader, using a backup method that doesn't involve physically removing the card may be a good idea.

--
martin@   | Martin Gregorie 
gregorie. | Essex, UK 
 Click to see the full signature
Reply to
Martin Gregorie

What about a compromiss: Using dd running on the Pi via ssh connection over ethernet to create an image of the complete SD card for a backup, and using Winimage/dd/whatever on the desktop PC to write the image to an SD card for restoring the system.

This way, the SD card has only to be removed for restoring the system, and not every day for the backup.

I fell the same way as David: A full card image is easy to understand and seems to be robust against software changes.

Does this aproach work? dd would run in the middle of operation and not when the system is properly shut down.

Can the output of dd be transported safely to the terminal for storing the data? I do _not_ see at this moment how I would safe that data using putty on my Windows desktop. However, I have the opportunity to store the image directly on the Pi on an attached USB hard disk drive; it also works as file server anyway.

Best regards, Kurt.

Reply to
Kurt Stege

My main reasons why not are that:

- dd images aren't compressed AFAIK, and so will be a lot slower to copy over Ethernet. Besides, I'm not sure that you can do that. I've only ever used it to move stuff between locally mounted disks.

- the other problem is that your smallest backup unit is the partition which, given that the default Raspbian disk layout puts everything in a single ext4 partition, means that you're shutting yourself out of the possibility of combining recovery with a Raspbian upgrade to the latest image - and when Wheezy eventually moves on you'll want to reflash the SD card from the new Wheezy equivalent and drop your own stuff back into its /home structure.

This is very easy if you've been using tar and/or rsync to back up /home, and even easier if you've done the stuff I mentioned such as symlinking /usr/local/bin into home/ and keeping copies of stuff in /etc that you've manually edited. Symlinking /usr/local/* is particularly useful if you've followed the convention of only putting packages you downloaded and installed as extras in your distro in /bin and /usr/bin while putting everything else (downloads from elsewhere and stuff you wrote) into the /usr/local/* hierarchy. If you replace /usr/local with a symlink to, say, home/local this stuff gets backed up with /home and, after a reinstall, all you do to get access to it back is to flop your /home backup onto the SD card, delete /usr/local and put the symlink back in place, the work of a minute or two and the fastest you'll ever restore stuff you need.

There's a bit more detail here:

formatting link
and look under Linux upgrades in the How-tos section.

As I said, keeping a separate on /home is actually far more robust because its useful during a distro upgrade or when swapping distros: a full dd card image isn't at all helpful in these circumstances.

All of these schemes can run when you like. If the RPI and the place the backup is saved are on 24/7 you can use a cron job to run them overnight. Otherwise, do them during a coffee or lunch break.

You use the scp utility (part of PuTTY), ftp or rsync to do the copy. rsync has Windows ports and it very fast because it only copies changes and deletes files you've removed from the master. Once its on the Windows box you'd back it up as part of your next Windows backup run.

Also a possibility. My current backup system uses rsync to make backups on a pair of USB disks. I keep them offline in a fire safe and alternate them, so I have two offline backups at all times (except when a backup is actually being made). HTH

Martin

--
martin@   | Martin Gregorie 
gregorie. | Essex, UK 
 Click to see the full signature
Reply to
Martin Gregorie

This means you make a copy while the system is running and the card is changing. The resulting copy is not consistent: it may be that a file is created whose directory entry is on your copy but the data is not. When you restore your backup you will have a file with garbage.

Copying a partition is only reliable when the partition is not mounted, or there is some sort of snapshot mechanism. (like the Linux volume manager can offer)

Reply to
Rob

You could put gzip in the pipeline, or use ssh?s built-in compression.

The natural way to upgrade to jessie would be a dist-upgrade, not reflashing. That won?t touch /home.

If you do want to upgrade by reflashing, though, it?s possible to mount the partitions from inside a whole-card backup (use kpartx to create the device nodes) in order to retrieve /home and so on.

All that said, I fully agree that whole-card backups are a cumbersome approach at best.

I have a similar strategy.

--
http://www.greenend.org.uk/rjk/
Reply to
Richard Kettlewell

Of course there can be higher-level inconsistencies if you backup a working system by copying files (rsync or whatever).

Arranging for the fs to be read-only (either from boot, or remounting it) should do the trick, though this can be rather disruptive...

--
http://www.greenend.org.uk/rjk/
Reply to
Richard Kettlewell

However, if you're not using the system for anything else[*] while you do the backup and restrict it to the /home structure, the chances of any files being changed during the backup run is vanishingly small.

Why do you think I suggested doing overnight backups or at least doing them during a coffee or lunch break?

There's an advantage, too, in restricting the backup to just the /home structure plus any symlinked stuff you've moved into it, e.g. /usr/local/ because backup becomes much faster. There's really little point in backing up anything else because the rest of the filing system is easily replaceable after a disaster by reflashing the (replacement) SD card with a freshly downloaded copy of the base system.

[*] yes, I do mean shutting down mail clients, web browsers, etc during a backup and Postgres isn't being used for anything. I do that as a matter of course during my weekly rsync backup. Postgres is shut down during my daily overnight backup run even though that's probably not necessary.
--
martin@   | Martin Gregorie 
gregorie. | Essex, UK 
 Click to see the full signature
Reply to
Martin Gregorie

You will still be copying a mounted filesystem which means that when you restore it, it will be treated as dirty and needing to be cleaned.

Also, I think there is no separate /home fs in the default install, so it can only be done in special cases. When there is a separate /home, it is better to umount it before copying.

Usually, people want to backup the fs because they don't want to lose all those small things they never thought about, like configuration files in /etc, the collection of installed software, their logfiles, etc.

When you only care about /home it is easier to just tar or cpio it.

Reply to
Rob

I'm not talking about backing up a complete fs in this case, merely the /home directory structure. The backup copy would also be just a directory structure, regardless of whether it is physically a tarball or a directory structure duplicated by rsync. Either can be copied back into partition without any 'dirty copy' issues: I've done it several times with no significant side issues.

A compressed tarball will take longer to make and copy to the backup medium but OTOH it won't need any chown action to restore file ownership, which might be needed if you restore the structure saved by rsync.

Indeed, but its irrelevant: both rsync and tar can backup and restore a single directory tree within a partition.

Covered previously. After a clean install its much easier to diff/replace/ edit the relatively few config files you've kept copies of than it is to ferret through a tarball to find them. Besides, just dumping the entire contents of /etc back onto a new install is a good way of fubaring it.

I'd probably use a compressed tarball, but rsync is a lot faster and rsnapshot may be even better - keep multiple versions of changing files, about as fast as rsync and apparently minimal disk space overheads. I'm thinking of replacing my overnight compressed tarball backup with rsnapshot. In this case there will be no file ownership issues because the backup medium is a directly attached USB disk drive.

--
martin@   | Martin Gregorie 
gregorie. | Essex, UK 
 Click to see the full signature
Reply to
Martin Gregorie

This is not at all what Kurt was suggesting! He suggested copying the SD card as an image using dd, presumably by copying the device node to an external file. That is copying a live mounted fs, with danger for inconsistencies as a result.

Copying by file is entirely different. That usually works with Linux, especially when the system is mostly idle. Restoring is tricky but can be done. Better is to restore on another system with the card mounted somewhere.

rsync with the proper flags will also restore file ownership and even hardlinks (also an issue often forgotten by naive backup programs)

Again, you are discussing a different method than was suggested in the article I replied to.

That is why it is so convenient to just mirror the entire card. Only don't do it on a live system when you don't have LVM to snapshot it, that is my point.

Reply to
Rob

dd by default writes to stdout or reads from stdin, so it can be used with ssh or netcat to send bytes over the network.

--
?? 100% natural
Reply to
Jasen Betts

You keep confusing the issues here. There are differences between archives, user directory backups and system backups, and you need to choose the tools to fit the task at hand.

*n*x have three tools for these, cpio, tar and backup/restore. cpio and tar have overlapping uses, as they have outgrown their more narrow tasks. These copy files to backup media. Tar has gotten rather good at keeping the file metadata intact too.

But both of these are for user mode backups.

For system mode there is backup/restore. They take a snapshot at the i-node level, and handle file system changes below it while the backup is going on. They backup whole file systems, and preserve hard and soft links, device nodes etc. just as they were. They also handle incremental backups very well.

These are file system tools, and you need different versions for different file systems.

I take a daily tar archive of my home directories; and an incremental dump every three days, plus a full dump every other week or so.

I shun "media copies", although these may be a lot more useful on journalling file systems as long as you take the whole snapshot between log rolls; i.e. on a reasonably idle file system. In that case a new mount will just do a log roll, instead of needing a full fsck.

ext3, ext4, xfs, zfs and jfs are such file systems. ext2, fat, ffs, efs are not.

-- mrr

Reply to
Morten Reistad

True, but when copying at the partition level things are much worse.

Yes, that is sufficient as well. However, the system normally will run only in very rudimentary form when the root fs is mounted readonly. This usually can be done only in "single user mode" where there is no network. The copy would have to go to a USB memory key or somesuch.

Reply to
Rob

Yes, but the thread had already moved on: I suggested that just copying the /home directory structure was a better way to do it for better speed and flexibility.

Even if the backup is to an fs on a different host? Under those conditions I'd think there's a possibility of rsync setting the file ownership to that of the owner of the rsync process on the backup copy end, especially if the user lists don't match between hosts.

I'll admit that I don't know the answer to this because so far I haven't needed to restore from such a backup: on the system I backup that way /home *IS* a separate partition and so doesn't get formatted when I do a clean reinstall.

Nope - you're apparently replying to stuff I wrote.

While my point is that you can do it fairly easily and without using LVM and, in fact, the discipline of saving copies of config files as you edit them is in fact a useful timesaver when you next do a clean install of the next version of your distro.

I may be a tad biased here because I've never managed a successful upgrade from one version of Fedora to the next, but the clean install route always works. Maybe others have this problem too because from F18 onward the upgrade option has vanished: a clean install is now the only option.

--
martin@   | Martin Gregorie 
gregorie. | Essex, UK 
 Click to see the full signature
Reply to
Martin Gregorie

Yes, I keep forgetting about netcat. Setting up the remote copy as an xinetd service would be a convenient way of managing things too.

--
martin@   | Martin Gregorie 
gregorie. | Essex, UK 
 Click to see the full signature
Reply to
Martin Gregorie

At that time I thought you meant a separate partition for /home and copying *that* as a whole partition (not by file). Having a separate /home partition is quite usual on disk installations on PCs, but I don't think it is often done on the Pi. Maybe I am wrong.

We regularly rsync between servers at work, but all servers are in a single NIS domain so all user lists are the same. It works OK, even with hardlinks, symlinks etc. We don't use ACLs though.

What I normally do is save copies of original config files before modifying them, taking diffs at reinstall time, and then patching the new system either manually or using patch.

And of course backup the whole thing to do disaster recovery restores.

Reply to
Rob

OK, understood. I don't really see the point of a separate /home partition on a Pi, partly because I simply can't see its content getting large and certainly because, at least at present I can't see whi I'd need more than one 'normal' user. Besides, everything I really care about preserving that is on a Pi will have been committed to my central CVS repository.

However, giving /home its own partition on a more general purpose box is extremely useful. I've do wonder why it isn't a standard option in Linux installers.

I periodically wonder about using NIS, but so far haven't seen any strong reason for doing so. No ACLs here either. Also no RAIDs so far, though I rather wish Linux had a port of ZFS available: I kike the sound of its self-healing abilities.

Basically we 're doing the same thing except that my diffs compare the latest patched version with the one in the new distro.

The only place this has really bitten me so far was the last time, when Apache 2.2.3, with its monolithic .conf file was replaced with Apache

2.2.4, which fragmented the configuration in the same way that the shells did with /etc/profile and /etc/profile.d/* - the change-over needed care but the end result will be much easier to maintain.

I do that on a weekly basis (with rsync) but even when it is a real disaster, i.e. a disk dies, AFAICR I've always recovered to a new disk with a fresh install followed by pulling /home out of the latest overnight backup. This is followed in due course by using CVS to refresh the stuff that was being worked on.

--
martin@   | Martin Gregorie 
gregorie. | Essex, UK 
 Click to see the full signature
Reply to
Martin Gregorie

If you are going to use dd in connection with ssh or netcat, why do you need dd? You can do things like "ssh < /dev/sda1" or "netcat ... < /dev/sda1" instead of piping the data through dd.

--
Robert Riches 
spamtrap42@jacob21819.net 
 Click to see the full signature
Reply to
Robert Riches

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.