Disaster recovery, how to automate as far as possible?

Re: Re: Disaster recovery, how to automate as far as possible? By: Chris Elvidge to The Natural Philosopher on Mon Aug 02 2021 11:56 am

Yeah, but he wants a bootable backup image I think.

Also rsyncing a live system may cause inconsistencies in the backup image. You have to be careful with those.

--
gopher://gopher.richardfalken.com/1/richardfalken

Re: Re: Disaster recovery, how to automate as far as possible? By: Chris Green to Richard Falken on Mon Aug 02 2021 05:51 pm

Then, your work is easy.

Get PiCore. It is a Tiny Core Linux port for the Raspberry Pi. YOu can configure it to boot from the SD card, copy the whole operating system to RAM, and then forget the SD card exists.

You just remaster the SD card with the configuration you want for your DNS and DHCP. The content of the SD card is never changed. There is no data persistence unless you manually -or scriptedly- command the OS to copy data to disk.

This allows you to have a BASE operating system which never changes stored in the SD. If there is data you want to save, you trigger a "store to disk" command, which saves the changes between the running system in RAM and the base system in the SD as a tar file.

And then you can send that tar file over a network.

Which is GREAT because you may have two PiCore SDs ready, use one for production, command it to save the tar overlay every 24 hours and send the tar to a backup server periodically. If you ever need to recover using the second SD, you just copy the tar file to the second SD and boot. Just copy. No imaging, no uncompressing, no anything.

Still I recommend the backup to be done with services stopped.

--
gopher://gopher.richardfalken.com/1/richardfalken

Recently (like last week-end) my Pi that runs local DNS died, I did an 'apt update;apt upgrade', the /boot partition went read only and there was no kernel to boot from (the "7 short flashes" code from the LEDs was actually quite handy). Presumably the SD card failed in some way.

I have good backups and was able to recover everything but it took quite a while (like Sunday morning) and, as a result I've been wondering if there's something I can do better.

My existing backups are incremental backups taken daily of /etc and /home thus restoring the system meant that I got a new clean image, wrote my backups of /etc and /home to it and was back running again. However there were a few gotchas on the way, a few symbolic links from /etc had to be mended as well as installing some extra packages (fortunately I list packages I've added when I add them - very important!).

So, what could I have done to make things quicker/easier?

My incremental backups are automatic and run every night, when they're more than a month old the system weeds some of them out but I have monthly backups for some years back now. I'm not aiming to change this.

However some sort of nightly 'image copy' would make what I had to do on Sunday much easier. It has to be automatic (i.e. run by anacron or cron) or it won't get done. This is a headless system so it has to be non-GUI too (with the advantage that as it's a "Lite" installation there's only about 1.5Gb of it to back up).

Is there some sort of "make an image copy of this system" I can run on the Pi to write an image to another system? I'd just keep a single image I think and overwrite it every night, it's a "restore and get it running quickly" backup I want, not the same as I have with the incremental backups. If the image is bad for some reason then I can always fall back to my incremental backups.

What's recommended (if anything)?

--
Chris Green

Chris Green wrote on 02-08-2021 at 11:29:

Apart from any backup strategy: boot from a USB disk (SSD would be good) which almost certainly more reliable and probably faster than SD card.

Well one way is to take the SD card out put it in a reader and take an image

I am not sure if you can do that over a network in a live system.

--
Karl Marx said religion is the opium of the people. 
But Marxism is the crack cocaine.

There are many possibilities.

Running from mirrored drives provides the fastest recovery from drive failure - not possible with SD cards on a Pi but should be possible booting over USB. You do need to check from time to time and replace bad drives promptly.

Running a failover server provides the fastest recovery from system failure - setting up can be tricky and you need something to tell you when a system is down otherwise you may not notice until the failover goes down.

Network booting everything from a central server is great and makes it easy to arrange snapshots - until the central server dies - so don't go that way unless you can afford to failover that server. It's also terrible for maintaining partial service on batteries in power outage.

A better option is probably to keep a master image (several copies) and make a regular backup that is the increment against the master - then restoring is a bit like assembling a docker image. Copy the master and apply the latest increment.

Whatever you do do not I implore you overwrite a single image every night. Murphy will ensure that the crash happens in the middle of that operation and then your system and the backup are hosed[1]. Keep at least two images.

[1] This is not imagination - where I saw it happen they hadn't changed the backup tape in the several months the system had been there and the disc failed in the middle of the backup run. You should have seen the faces when I said "Where's last night's tape, this one is corrupt ?". They were *really* good at changing tapes and rotating the weeklies after that!

--
Steve O'Hara-Smith                          |   Directable Mirror Arrays 
C:\>WIN                                     | A better way to focus the sun 
The computer obeys and wins.                |    licences available see 
You lose and Bill collects.                 |    http://www.sohara.org/

Yes, I'm considering that too! :-)

Can an old Pi 2 boot from SSD? I know a Pi 4 can.

--
Chris Green

That's a little difficult to automate! :-)

--
Chris Green

You don't want an _image_ copy of a running system - there's all sorts of things you don't need/want copied - /dev /proc /sys etc. Just do an ssh/rsync --one-file-system from the backup machine. E.g.

rsync --archive --one-file-system --exclude=lost+found --exclude=media

--exclude=save --exclude=var/cache --exclude=swap --exclude=*~ --delete

--delete-excluded --info=STATS1 "machine-to-backup:/" "directory-for-backup"

--
Chris Elvidge 
England

It needs far more than that for what I want to do.

Apart from anything else there are two file systems to back up:-

Filesystem Type 1M-blocks Used Avail Use% Mounted on /dev/root ext4 29270 1318 26731 5% / /dev/mmcblk0p1 vfat 253 49 205 20% /boot

I want to have available some sort of clone/image of the system that I can quickly create a Pi SD image from. So the nightly backup will create a clone/image/whatever (on another system preferably) and from that clone/image/whatever I will be able to (quickly) create a working SD card to plug into the Pi.

--
Chris Green

That's overkill! :-)

Similarly overkill.

Yes, but this takes me to where I am already, it's essentially what I did when my Pi died. The 'master image' was simply a newly created Pi SD card and I applied my incremental changes. It was simple enough to do but did take some time.

Maybe, but (as I said) if the overnight image is bad I simply fall back to what I did this time. I know it's sound and works, just takes a bit of time.

What I'm after is a way to have a "ready to plug in" SD card that doesn't need any extra work on it before using it. Or, nearly as good, at least an image on disk that I can write to an SD card that will "just work".

--
Chris Green

I don?t know of a good tool to make image backups as such, though it?s going to boil down to something equivalent to

ssh root@pi cat /dev/mmcblk0 > /path/to/backup.img

...done when the system is idle (if there?s lot of IO going on then the filesystem inside the image will be internally inconsistent).

I had to restore a Pi from backups recently after the SD card failed. My backups are whole-system backups made with rsync, so once I?d figured out what shape the partitions should be in order to be bootable, it was straightforward to restore the backup onto the new card. It worked first time.

IMO the requirement for an SD card is a weakness in the Pi design, due to their poor reliability. I don?t know what best alternative approach is, though, given the need to keep cost and physical size down while remaining flexible.

--
https://www.greenend.org.uk/rjk/

Chris Green wrote on 02-08-2021 at 12:46:

Only the 2B v1.2, apparently:

But because it's not USB3 the speed is probably not better than SD card, I think, and an SSD might be overkill. Maybe a "high endurance" SD card would be a better investment.

Which is sort of where I came in. :-)

How do I make (and maintain up to date) an image of the SD card?

--
Chris Green

So it looks like the way to go is simply to create a new SD card with the Lite OS on it and then rsync all the files from the running system to that card (less a few things as noted elswehere in this thread).

I guess that's not so difficult to automate.

The BeagleBone Black has an on-board eMMC memory with an SD slot as an alternative, I don't know if the eMMC is inherently any better than an external SD though.

--
Chris Green

I use timeshift on my main PC and laptops.

I make the backups on a different slab of spinning rust to the system disk, obviously. I believe it can be configured to use a network drive for the backups and I see it is available in the PI repository.

On the few occasions I have needed to recover a system it has worked very well.

--
Nev 
It causes me a great deal of regret and remorse 
that so many people are unable to understand what I write.

Try to separate things into read-only (preferably all things needed to boot and run the system) and read-write (home, ...). Then put the read-write stuff on storage other than a sd-card, e.g. hdd. Make a image of the read-only sd-card, and daily backups of the other stuff.

HTH, Stefan

It doesn't do what I want though.

Its basic function in life is to restore your system to how it was some time ago, hence its name. (I.e. it does incremental backups, just like I do already, and very handy they are too)

It doesn't do remote backups/copies (which I'd prefer, though not essential).

It doesn't provide a restore for a non-bootable system, you have to create a bootable system, install timeshift on that and then do a restore. That's as much work as my restoring from my home-made incremental backups.

I repeat! :-) :-) :-)

I want a system that provides me with either a ready to boot SD card with my current system on it or, an 'image' that I can quickly copy to an SD card and can boot.

--
Chris Green

Set up your backup cards (use at least two and cycle them), using the same partition scheme as your prime card) and run rsync twice (once on each partition for each backup session) because that's fast - if there are no changes, rsync only scans the partition being backed up.

If you install your locally developed scripts, programs and man pages in the /usr/local/* file structure, make sure thats backed up too.

BTW, full marks for keeping a list of the extra packages you've installed, but have you thought of structuring that list as a script containing a list of "apt get package-name" statements? Its a nice, zero- cost way of speeding up disaster recovery.

Using rsync the way I've suggested will do that: cloning the backup disk then requires:

- creating the two partitions on the new disk

- running dd to copy their contents to the new disk.

--
Martin    | martin at 
Gregorie  | gregorie dot org