... and how do I quickly restore from that? You've effectively described what I already do with my incremental backups. I do daily incremental backups of everything that changes.
My issue was it took a while to:-
Create a clean new Pi system Add/move users as required do 'apt update;apt upgrade' restore /etc and /home from my incremental backups
That's pretty much what it would take to restore from the backups you're suggesting above.
-- Chris Green
Well, a high quality high endurance sd card would certainly be an additional benefit.
I don't think you can get much closer than you already are, except with that other suggestion: mirrored disks, which you rightly deemed to be overkill. You can however dramatically accelerate the restore from fresh image with something like Ansible playbooks (or your own bash script implementation).
Actually I'd say that's a large part of it. I'll sketch here how you could make an image of your Pi:
- Create an empty file the size of your SD card or at least large enough to hold the file systems.
- Put your partition table from Pi's SD card on there and create the root file system.
- On my Pi at least the /boot partition on a Pi can be unmounted during runtime so you can do that and then image it directly into your backup SD card image and remount. Offset calculation left as an exercise.
- For the / partition, mount your corresponding partition inside the SD card image and use the rsync stanza from your namesake above.
And that should do it. I've never done this exactly but I don't see an issue either. You could even use a real SD card as the target instead of an image but if you do it every night it might eat up the card...
For a more advanced solution you could go to the zfs filesystem and use the snapshot feature there. I don't know zfs nearly well enough to do that though.
Instead of automatic system backups you could look at automatic/simplified provisioning. i.e. taking a standard OS image and automatically installing/configuring everything you need.
You still need to backup your data, but backing up the OS is a bit old fashioned.
I'm assuming you want to get up and running ASAP after a disaster? So I would prepare for it by doing your normal incremental backups then copying them (at your leisure) to a (or many) spare SD Card at regular intervals. Then when the inevitable happens you should only need to swap the most recent backup SD card into the dead Pi.
Or have I still not got what you want right in my head?
-- Nev It causes me a great deal of regret and remorse that so many people are unable to understand what I write.
It is faster then the SD card. The SD card interface is not particularly fast.
I have 2 Pi servers, a Pi3B and an old Pi1b+.
Each boots from it's SD card, but the cmdline.txt specifies that the root filesystem is on the USB attached HD (spinning rust). The disks are both WD disks with built in USB2 interface.
I have it so the sd card /boot partition is mounted readonly, and there is a second SD partition (usually unmounted) where I keep a copy of the root partition on the USB disk. An rsync script(*) running at least daily, mounts the SDcard 2nd part. and updates it from the running root partition, then unmounts. That way the SD card is only "at risk" during the update. I get aprox. 1.3MB per day updates - log files mostly.
If the harddrive dies I will take out the SD card and on my desktop computer alter the cmdline.txt file to specify root on the SD card - alter the /etc/fstab file as needed - but back in Pi and reboot. Then go about replacing the failed HD.
The other data partitions on the Harddrive are backuped up elsewhere and would have to be restored onto any replacement USB disk, along with a copy of the root partition.
I use rsync for my backup purposes, and have recovered my Linux Intel desktop when it's boot disk died, using a similar scheme (it has 2 disks
- but grub is a Bl**dy can of worms and a grub rescue CD was needed).
Jim
(*) make sure you specify the -XAH options to rsync as well as the usual
-a and -x (don't cross file system boundaries), and whatever delete option you prefer.
aaah didn't follow closer this thread, but it is continuing for some time now If you want to keep the SD card ... then just replicate the SD card to the backup and prepare to write back whenever needed.
BTW borg backup does deduplication and is very space efficient. I already restored one CFLASH card when the old one suddenly died on the firewall. Needless to say you need a second machine anyway for doing the restore and a USB adapter etc.
The best way to deal with root system backups is lvm snapshots - you have to plan this before partitioning. It pays off at the end.
I solved the problem in my case by eliminating the SD card. I configured the RPis for diskless boot from the server. On the server changing things is a peanut butter. I guess each avg. NAS can do this, so it does not have to be expensive and power consuming machine - it is sufficient to have linux on it. A single SD card is and will remain the risk in the equation. But if you insist using SD card for whatever reason, better have a ready to use copy in your pocket. Not that they are so bad, but you don't know when exactly it will fail
On Mon, 2 Aug 2021 13:56:15 +0100, Chris Green declaimed the following:
Depends on the SD card, to some extant. I've seen one with quite atrocious I/O, even though it was "rated" faster than some slower SanDisk.
Class 10 cards are rated on streaming a single file to a freshly formatted card [video]; Class 2/4/6 are based on multiple short files, possibly including fragmentation from deletes [still image cameras]. As a result, higher end Class 4/6 cards can easily out-perform a cheaper Class
10 whose internal controller was optimized for FAT and streaming (only two "allocation unit" buffers; if jumping around for small files, the card is continuously updating allocation units).-- Wulfraed Dennis Lee Bieber AF6VN wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/
The Pi I want to back up runs only DNS/DHCP for my LAN, there are other services running but none are actually in use.
-- Chris Green
On Mon, 2 Aug 2021 15:57:49 +0100, Chris Green declaimed the following:
No... /etc and /home would be on the external hard-drive -- no restore for their contents, unless the hard-drive itself fails. If the SD card fails, you swap the card and reboot. You do have to keep the SD card image up-to-date -- so periodic swapping to run the apt update/upgrade cycle may be recommended.
-- Wulfraed Dennis Lee Bieber AF6VN wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/
You have it right, that's most of what I want to do.
The only thing is that I want to automate "... copying them (at your leisure) to a (or many) spare SD Card at regular intervals", I suppose this is do-able though. I *know* it won't get done otherwise! :-)
-- Chris Green
Does rpi-clone do what you want? Scripting for rsync with lots of options for cloning an SD card
You can't automate "periodic swapping", so (knowing me) it won't get done.
-- Chris Green
Yes, I think it's about the closest to what I want that I've come across. It's just a long bash script so I can some/all of it if I decide to roll my own.
-- Chris Green
I don't think you're going to get that with any kind of online backup. I image Raspberry Pi systems by popping out the SD card, shrinking the root filesystem as small as possible with gparted (running on another computer), and piping dd into xz (or gzip if I'm in a hurry) to make an image backup. Clonezilla would probably work too, but it's a bit trickier to use if you need to restore to a smaller SD card.
_/_ / v \ Scott Alfter (remove the obvious to send mail) (IIGS(
But, almost as fast as using rsync, I think (disclaimer: not tried) would be to write a bash script that:
- wipes and recreates the two partitions on the backup SD card with the same names as the corresponding partitions on the live SD card
- uses dd to copy both partitions from live to backup card.
Thats pretty much what I did last time I migrated a working RPi filing system to a larger SD card, except that it was done manually rather than run by a bash script. The new card was bootable without further hackery, so maybe that would offer a better way to make a bootable backup.
I don't recall that being a particularly slow process. However, even if it is slow, running it overnight as a cron job would make its speed irrelevant.
-- Martin | martin at Gregorie | gregorie dot org
Copy the image to a new sd-card (or have already one prepared) and re-attach the read-write storage to the pi. That's work for 10 minutes. It won't get much faster.
There's one thing I didn't mention: don't do frequent apt update/upgrade cycles. Otherwise separation into read-write and read-only doesn't make too much sense. If you want frequent upgrades use only a hdd as storage and boot from it.
These three steps are not necessary if you have an image of the read-only stuff. Just dd the image to an sd-card and put it in the pi.
That's not necessary. Just reattach the read-write storage.
Only the read-write storage. And that may even be a ramdisk in the pi's RAM. You can even write a script which automatically creates the ramdisk and fills it from the read-write backup at boot-time. If done so, recovery desaster is just putting a prepared sd-card into the pi and booting (10 seconds).
HTH, Stefan
Now that is quite a neat idea, thank you.
-- Chris Green
What I do is start by taking a copy of the SD card as an image file on to my NAS drive. Where the Pi's are using USB sticks, they are set up with with a boot and a root partition, just like an SD card, so the image can be updated with the boot partition from the SD card, and the root partition from the USB stick.
Every night I run a cron script to rsync the days changed files from the boot and root partitions of each Pi to the corresponding partition of the image file on the NAS. That way if any of the machine's storage fails, I can immediately write the image on to a new SD card (or USB stick) and be back up running in minutes.
I have other scripts which run monthly to archive the images on to off line storage. This runs zerofree on image files to blank any free space on them, so they compress better with pigz (parallel gzip, other algorithms compress better but take much longer). The archived files are named with date, so I can keep several copies and delete them when they are too old. I make sure I've got an up to date archive copy before I do any system upgrade, so I can go back to the previous version if problems only show up after the nightly backup has updated the previous image.
---druck
Yes, I think this sounds a good approach. One can mount the image file on the backup system so that it's partitions and files are visible so - as you say - you can rsync the changes across.
It'll take a little configuring and setting up but not a lot and once running smoothly I can go back to ignoring it all again! :-)
Thanks for all the input everyone.
-- Chris Green
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.