In getting ready to try the latest "full-upgrade" for RaspiOS I decided to save a copy of my home directory on another machine, just in case the upgrade goes wrong.
It turns out to be a fairly big job, mostly because of dotfiles.
Is there any simple rule of thumb for cleaning up before backing up? I'd like to avoid destroying useful things (saved login credentials) that notes and memory might not record accurately. The rest can go.
Well, in genral, anything in ~/.config and/or files that have a .conf or .cfg suffix are configuration files for programs you have installed so are worth keeping if you want to keep your configurations.
Files in .cache directories or with similar suffix are cache files and just preserve state to speed things up a little so can usually be discarded.
Then there are .login, .profile, .bashrc, .bash which are your login and shell environment and configuration. If you have changed them you probably want to preserve them.
However, having said all that, I'd recommened copying the whole of /home out to a backup and then copying it back in after the upgrade. Restoring cache files won't do any harm.
I never clean out .dotfiles. As others have said most of them are parameter sets and/or data stores for the more complex applications you use, such as news readers, mail readers and LibreOffice - you really, really want to keep these dotfiles!
That said, I don't bother backing up the caches some programs create: long ago I configured my web browsers (Firefox and Brave) to empty their caches when they're closed. Another candidate for this treatment is Google Earth, though you have to remember to empty its cache manually (go to Tools:Options:Cache to do that)because automatically emptying its cache in exit isn't a configurable option, and I exclude stuff like the .tracker cache and similar from all backups, which I do with rsync. Rsync is recommended because its hugely faster than making tar, gzipped tar or zip backups. Avoid the latter because zip doesn't fully understand Linux file permissions and ownership, unlike tar, which does.
If you're planning to do a full upgrade on the same SD card, either use rsync to make a logical backup or use Clonezilla
formatting link
to duplicate the SD card. If you use Clonezilla, mount the EXT partition on the newly copied SD card and empty the /boot directory in it - this should be empty since its only use on an RPi is as a mountpoint for the FAT32 boot partition.
To get clonezilla, download an iso image from their website and burn it to a CD, which makes the CD into a bootable Debian system. Booting that and using it to copy both partitions to a new SD card (expanding them as needed) should leave yo with a new, bootable SD card ready for doing in in-situ Raspi upgrade.
--
--
Martin | martin at
Gregorie | gregorie dot org
In the end, that is what I did. As it happened, the upgrade went without a hitch (so far!) and I haven't needed to restore anything at all.
I was somewhat stunned at the number and volume of files transferred,
2.8 GB, 2.0 of them .cache files.
Where does the chromium browser store login credentials? A Web search didn't turn up any ready answers for the case of the Pi. It appears that Mac and Windows put them in system directories.
Yep, they are bulky, and not worth backing up since in general they're temporary files and contain nothing that doesn't already exist elsewhere.
BTW, speaking of backups: my house server gets automatically backed up by a cron job that runs at 03:00 every dat. Originally this made a compressed .tgz (gzipped tar archive) file each day. About 3-4 years back this was taking 3.5 hours to run each night: not a problem since the system was never doing anything else at the time, but I was already planning to scan in my CD collection, which ended up at 145GB, so I switched over to using rsnapshot to make the same backup: this now takes
9 minutes plus a weekly run of about 10 minutes.
For those who don't know, rsnapshot is a backup subsystem that manages the number of daily and weekly backups that are kept. The daily run backs up changes made during the previous 24 hours, while the weekly run concatenates the last week's daily backups into a single weekly backup.
Pass - I don't and won't use the Chromium Browser or the Google Search engine for that matter, because I want to minimise the amount of my data that is stolen and monetised by Google as possible. AFAICT the *only* Google code I use is Google Pro, and thats only because there is not non- Google equivalent available - and don't forget that it was NOT developed by them - they bought it in during the first Internet boom.
--
--
Martin | martin at
Gregorie | gregorie dot org
On Sun, 6 Dec 2020 17:35:23 -0000 (UTC), bob prohaska declaimed the following:
Well, a quick scan of
ls -Ral .config/chromium
shows a likely candidate in
.config/chromium/Default: total 736 drwxr-xr-x 25 pi pi 4096 Dec 1 13:12 . drwxr-xr-x 17 pi pi 4096 Aug 22 15:06 ..
-rw------- 1 pi pi 0 Dec 19 2019 000003.log drwx------ 2 pi pi 4096 Aug 22 15:04 AutofillStrikeDatabase drwxr-xr-x 3 pi pi 4096 Aug 22 15:04 blob_storage drwxr-xr-x 2 pi pi 4096 Aug 22 15:04 BudgetDatabase
-rw-r--r-- 1 pi pi 28672 Aug 22 15:04 Cookies
-rw-r--r-- 1 pi pi 0 Aug 22 15:04 Cookies-journal
-rw------- 1 pi pi 16 Dec 19 2019 CURRENT
-rw------- 1 pi pi 1315 Aug 22 15:05 'Current Session'
-rw------- 1 pi pi 1094 Aug 22 15:06 'Current Tabs' drwx------ 2 pi pi 4096 Dec 19 2019 databases drwxr-xr-x 2 pi pi 4096 Aug 22 15:04 data_reduction_proxy_leveldb drwxr-xr-x 2 pi pi 4096 Jul 1 13:21 'Extension Rules' drwxr-xr-x 5 pi pi 4096 Aug 22 15:03 Extensions drwxr-xr-x 2 pi pi 4096 Aug 22 15:04 'Extension State'
-rw-r--r-- 1 pi pi 24576 Jul 1 13:29 Favicons
-rw-r--r-- 1 pi pi 0 Jul 1 13:29 Favicons-journal drwxr-xr-x 4 pi pi 4096 Nov 18 2019 'Feature Engagement Tracker' drwx------ 3 pi pi 4096 Apr 27 2020 'GCM Store' drwxr-xr-x 2 pi pi 4096 Nov 18 2019 GPUCache
-rw------- 1 pi pi 16384 Apr 27 2020 heavy_ad_intervention_opt_out.db
-rw------- 1 pi pi 0 Apr 27 2020 heavy_ad_intervention_opt_out.db-journal
-rw-r--r-- 1 pi pi 118784 Aug 22 15:06 History
-rw-r--r-- 1 pi pi 0 Aug 22 15:06 History-journal
-rw-r--r-- 1 pi pi 466 Aug 22 15:05 'History Provider Cache' drwx------ 5 pi pi 4096 Dec 19 2019 IndexedDB
-rw------- 1 pi pi 1267 Aug 22 15:03 'Last Session'
-rw------- 1 pi pi 1022 Aug 22 15:03 'Last Tabs' drwx------ 4 pi pi 4096 Dec 19 2019 'Local Extension Settings' drwx------ 3 pi pi 4096 Dec 19 2019 'Local Storage'
-rw------- 1 pi pi 0 Dec 19 2019 LOCK
-rw------- 1 pi pi 46 Dec 19 2019 LOG
-rw-r--r-- 1 pi pi 22528 Apr 27 2020 'Login Data'
-rw-r--r-- 1 pi pi 0 Apr 27 2020 'Login Data-journal' drwx------ 3 pi pi 4096 Dec 19 2019 'Managed Extension Settings'
-rw------- 1 pi pi 50 Dec 19 2019 MANIFEST-000002
-rw-r--r-- 1 pi pi 45056 Aug 22 15:02 'Network Action Predictor'
-rw-r--r-- 1 pi pi 0 Aug 22 15:02 'Network Action Predictor-journal'
-rw------- 1 pi pi 1633 Aug 22 15:06 'Network Persistent State'
-rw------- 1 pi pi 0 Jul 1 13:29 .org.chromium.Chromium.OAMYZt
-rw-r--r-- 1 pi pi 0 Nov 6 2019 page_load_capping_opt_out.db-journal drwxr-xr-x 2 pi pi 4096 Aug 22 15:04 'Platform Notifications'
-rw------- 1 pi pi 30119 Dec 1 13:12 Preferences
-rw-r--r-- 1 pi pi 16384 Nov 6 2019 previews_opt_out.db
-rw-r--r-- 1 pi pi 0 Nov 6 2019 previews_opt_out.db-journal
-rw------- 1 pi pi 53248 Apr 27 2020 QuotaManager
-rw------- 1 pi pi 0 Apr 27 2020 QuotaManager-journal
-rw-r--r-- 1 pi pi 95 Nov 6 2019 'Secure Preferences' drwx------ 5 pi pi 4096 Dec 19 2019 'Service Worker' drwxr-xr-x 2 pi pi 4096 Aug 22 15:04 'Session Storage' drwxr-xr-x 3 pi pi 4096 Aug 22 15:04 shared_proto_db
-rw-r--r-- 1 pi pi 20480 Apr 27 2020 Shortcuts
-rw-r--r-- 1 pi pi 0 Apr 27 2020 Shortcuts-journal drwxr-xr-x 2 pi pi 4096 Aug 22 15:04 'Site Characteristics Database' drwxr-xr-x 3 pi pi 4096 Nov 18 2019 'Sync Data' drwx------ 3 pi pi 4096 Dec 19 2019 'Sync Extension Settings'
-rw-r--r-- 1 pi pi 20480 Apr 27 2020 'Top Sites'
-rw-r--r-- 1 pi pi 0 Apr 27 2020 'Top Sites-journal'
-rw-r--r-- 1 pi pi 2537 Nov 6 2019 'Translate Ranker Model'
-rw------- 1 pi pi 2019 Aug 22 15:04 TransportSecurity drwx------ 2 pi pi 4096 Dec 19 2019 VideoDecodeStats
-rw-r--r-- 1 pi pi 131072 Aug 22 15:04 'Visited Links'
-rw-r--r-- 1 pi pi 69632 Apr 27 2020 'Web Data'
-rw-r--r-- 1 pi pi 0 Apr 27 2020 'Web Data-journal'
"Login Data" appears to be an SQLite3 database. (Having sftp to my Windows machine and opening it in both SQLiteStudio and "DB Browser for SQLite" reveals I have no saved login data, so the hexdump below is safe to include).
pi@rpi3bplus-1:~$ hexdump -C .config/chromium/Default/Login\ Data
On Sun, 6 Dec 2020 19:35:18 -0000 (UTC), bob prohaska declaimed the following:
You'd have to ask Google... Though before graphical file managers embedded spaces were rarely found, since they have to be escaped on the command line. Graphical systems that use text boxes can take almost anything and once you have that "anything" the OS functions for creating/opening files don't care. The normal escape for Linux appears to be a \ before the space. Windows, instead requires the entire path to be inside " marks if any component has a space.
Tab-completion on the command line does help. One just types enough of the path to identify the start of the file name, then hits and the OS fills in the name with any escapes (or shows a list of names with similar starts -- so those could require including the escape to isolate).
FYI: The Xerox CP/V OS allowed for other non-printable characters in file names -- like BEL, which would cause the terminal to beep when the name is displayed. Trick was, on anything faster than a 300baud terminal, one could not tell /where/ the BEL was while the name was spewed to the screen. One could have two files that "looked" like the same name on screen with non-printing characters at various locations...
example example
I haven't used Chromium on Windows but seem to have a
C:\Users\Wulfraed\AppData\Local\Chromium\User Data
directory, so that would be in the Windows equivalent of "home directory"
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/
In most unix shells you can use \ to escape a single character or use single or double quotes (in double quotes variable expansion happens in single quotes it doesn't) or if those seem too much typing then ? matches any single character including space. These conventions predate Linux by quite some time.
--
Steve O'Hara-Smith | Directable Mirror Arrays
C:\>WIN | A better way to focus the sun
The computer obeys and wins. | licences available see
You lose and Bill collects. | http://www.sohara.org/
In fact, it seems that the boot partition is mounted on /boot as part of the boot process, so on thinking about it, the only effect of it having content *should* be that the files in it occupy space in the filesystem partition but are inacessable while Raspbian is running. Consequently they would be better removed so that other, more useful files can occupy that space.
Note that, if the entire filesystem is backed up using tar, rsync or zip pointed at '/' while Raspbian is running from that SD card, then the backed up archive *will* correctly show that /boot contains the entire content of the boot partition. IOW, cleaning out /boot after restoring an offline SD card from the archive will at worst do no harm and at best will release some tens of MB of filespace.
I've always assumed that is why you said that /boot should be emptied when moving the two partitions to a bigger card. Doing so makes sense, is fast and easy to do, so why not do it?
--
--
Martin | martin at
Gregorie | gregorie dot org
You shouldn't be backing up files under /boot, the boot partition needs to be backed up separately.
Most backup software will not cross filing system boundaries, so you don't backup several TB of mounted NAS drives rather than just your SD card. If rolling your own backups, make sure you specify option not to follow mounts or links to other filing systems.
If you've backed up with the correct options /boot on the root filing system will be empty, along with all the other places that should not be backed up (/dev /proc /media /mnt /tmp /run ecetera)
However, a simplistic backup run on an RPi that writes '/*' to a .tgz archive or uses rsync to make a full filesystem backup to a different SD card or USB disk will do exactly that.
Its very easily done and, at first glance, looks like a neat trick.
A somewhat more knowledgeable person may make a TGZ archive of '/' plus a zip backup of the boot partition. This also looks like a neat trick, but also hides a similar problem.
All I'm trying to do is to explain the pitfalls in both backups and how to fix them should these be the perp's only backups.
Bottom line: if you want to copy a working RPi system to a new SD card, specially if you want a bigger filing system, don't mess with archivers, go straight for the jugular and use Clonezilla.
Indeed, but this us a special case: the RPi boot process mounts the boot partition on /boot and any backup of '/' made on a running RPi with rsync
*will* include the contents of the boot partition unless you've configured the rsync run to exclude /boot/* . Don't ask me how I know!
If this is something you do all the time you'll know it already: a friend is like this he's constantly setting up RPis for people who like music and has the parameters engraved on his brain. I, like most of us only do this stuff as long intervals and so, for us, that's why I'm saying that using CloneZilla is pretty much a one stop shop if you only have to transfer an RPi filesystem to an new SD card every few years.
NOTE: I didn't run out of space on mt EXT file system: I was only kicked into making the SD card switch because RPiHQ decided it would be a good idea to put copies of all bootable kernels in everybody's boot partition, which had the unintended consequence of overflowing my boot partition when the update tried to install an new, bigger kernel set.
--
--
Martin | martin at
Gregorie | gregorie dot org
Which is why all rsync based archive scripts should use the -x flag to rsync which prevents crossing filesystem boundaries, it's not just /boot you need to skip but *all* mounted filesystems.
--
Steve O'Hara-Smith | Directable Mirror Arrays
C:\>WIN | A better way to focus the sun
The computer obeys and wins. | licences available see
You lose and Bill collects. | http://www.sohara.org/
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.