Adding a hardware swap partition

There was this conventional separation into two levels:

Absolutely core, indispensable stuff:

/bin /lib /sbin

Somewhat less important stuff:

/usr/bin /usr/lib /usr/sbin

The core point being that things in the first group had no dependencies on things in the second group. So, for example, there would be no executable in /bin or /sbin that depended on a shared library in /usr/lib. But everything could safely depend on things in the first group, since pretty much by definition, you wouldn’t have a functional system without that.

That distinction has gone out the window nowadays. It was too much trouble to maintain, since nobody really cares about it any more. Hence the merging of the two groups.

Since you ask...8-)

On rare occasions my 8GB Pi5 slowed to a crawl. Eventually I realized it was happening when I had two browsers (firefox and chromium) both running with too many tabs open. The system was short of memory. It needed swap to cope this those occasions.

Hardware swap is _supposed_ to be faster than a swap partition, which made it, at least in principle, more attractive. Also, in principle, a physical swap partition can be placed _between_ (in the sense of seek stroke) /uar and /, minimizing the amount of head movevent. This was true in the days of st506 disks, I'm not sure how true it is with SATA. So, I inquired about how this might be done under RasPiOS. It's not hurgely hard under FreeBSD and I thought there might be some nifty tools for the purpose in RasPiOS.

The answers received have now led me to question the orignal premises. No nifty tools have come to light. The relative speed of file-vs-hardware swap haven't been quantified. The need for occasional swap use remains, but it's sporadic. Maybe a swapfile is good enough.... it's certainly easier.

It was a surprise to learn that a single partition is somehow required for RasPiOS to function correctly. If true, it's a good thing to know.

Given that RasPiOS has an /sbin directory, I'm pretty sure the machine will come up at worst in single-user if /usr can't be mounted normally. Whether /home/, /var/ and /tmp/ can be links to /usr is less clear to me at this point.

The original plan would have left the machine with a /root of around

Busy trying to drink from a firehose 8-) And, thanks to those at the pumps!

bob prohaska

It may still be true with moving-head disks, but all that geometrical information is hidden from the OS these days, behind linear sector addressing done within the drive itself.

I think the basic thinking nowadays is that, if you’ve having to do disk I/O, you’ve already lost several orders of magnitude in transfer speed, and no amount of trying to optimize head movement is going to make much of a difference to that.

In other words, if you have to swap, your system is in trouble, anyway.

Separate /usr probably still works, but I doubt it gets much testing.

It’s really not going to make swapping significantly faster. Nor is trying to find an optimal location for swap - you have quite possibly already spent more time on the matter in this thread than you could ever possibly save.

$ ls -ld /sbin lrwxrwxrwx 1 root root 8 Nov 17 2023 /sbin -> usr/sbin

However I believe this is neither here nor there since initramfs is responsible for mounting /usr if it’s separate. If it fails to mount you will most likely end up in an initramfs shell (which is a pretty reasonable recovery environment).

You might run into trouble with AppArmor. Not particularly hard to fix.

If all this is about adding some swap space then you are massively overcomplicating matters. If there’s no unpartitioned space left then just create a swapfile and get on with your life. If you do anything more complicated than that then make sure your backups are up to date first.

Can you guarantee that /usr will be mounted at boot time?

I remember having to use /bin/sh in init.d scripts because /usr/bin/ksh didn't exist yet. That may have changed.

A long time ago, and not on an rpi. AIX or HP/UX or something.

Anyway, I'd keep /usr under / just to be safe.

/home on its own partition.

It depends what you mean by ‘boot time’. On a mainstream Linux system /usr needs to be mounted by the time initramfs hands over to the real root and init. So in practice unit files/init scripts/etc can safely assume it’s mounted.

Historically making sure /usr was mounted (if it was separate) was init’s job (via an init.d scripts), so very early in boot, /usr might not have been there. But mounting filesystems was, necessarily, a pretty early step in the boot process, so by the time it was running (say) the init script for an X display manager, /usr would have been mounted.

For all I know HPUX and AIX still work that way, I’m glad to say I don’t have to care any more.

Agreed

It was also that you could then make /usr an NFS share, to be shared by all the computers on the network. /bin /lib etc would be local to the machine (perhaps in ROM, on a small HDD or even on floppy) which was just enough to connect to the network, mount the NFS and load all the real stuff from /usr over the network. If the NFS mount failed for some reason, you still had some local tools to fix problems.

Of course network boot has moved on a lot since then.

Theo

With PXE boot and the Linux pivot_root() system call, you could have the entire root file system mounted via NFS.

Thanks. :-)

So far, I follow you.

Assuming you are using a mechanical disk (I don't think you said anywhere, did you?), there is some (a smallish?) gain to be had for minimizing head movement. If you are using an SSD through some SATA-USB dongle, then you can completely forget about worrying where on the disk to put swap... the end is fine.

Carrying on with the assumption that you are using a mechanical disk...

Someone else already pointed out that /sbin is (on Raspberrry Pi OS and many others) just yet another sym link. On current systems where /bin is just a link to /usr/bin, except for occasional updates for log files, I doubt that your system would be accessing / (the "without /usr" version of /) very often. (Unless your /tmp is not a tmpfs and you are using /tmp a lot, for whatever reason.)

You also didn't tell us where /home is going to be. Were you thinking of having it on its own partition? That decision may be more significant than any possible benefit from separating /usr from /.

That is true.

You mean "a single partition for / and /usr" ? I'm not sure that it is required, but I wouldn't be surprised if having /usr on its own partition caused problems. But I'm happy to be educated about that!

See above.

/home certainly can, and if /usr gets mounted "early enough" during boot, maybe /var and /tmp can be too. But if you do that, you now have a minuscule amount of stuff sitting there by itself on /, and I can't imagine the benefit of having swap between "almost nothing" and "almost everything".

OK. I thought it your original message you said something about making / as small as possible.

Tricky to not drown. Although I've seen dogs trying to drink from garden hoses, it might just be a matter of scale.

Cheers. Jim

If I was going to set up swap it seemed prudent to make it as fast as I reasonably could. I didn't (and so far still don't) have a clear idea how much difference a swap file versus a swap partition makes.

I'm a little embarassed to admit I didn't know /sbin was merely a link. /sbin is part of / on FreeBSD and I never thought to check RasPiOS.

Typically I put home on /usr/home when I'm setting up a machine by hand. In this case I planned to leave it in /, supposing it wasn't important to the question at hand.

Point taken 8-)

The plan was to resize / to what's occupied now, add a 16GB swap partitiion, add another partition after that and copy the contents of usr to the new partition. The next step would be to clean out the old /usr tree and mount the new usr partition on the old mountpoint. That would free space in /.

Since /sbin isn't a freestanding directory, this seems impossible without booting from a separate root device, likely a microSD. I didn't see this hurdle coming, at all. It isn't a fatal blunder, but it's a blunder.

In the meantime it has developed the machine has a swapfile, but it was limited to 2GB, which on an 8GB machine makes little sense. The limit is now removed and the machine has about 16GB swap.

It'll take some time to see how the machine behaves. At the moment YouTube seems quite sluggish with only 80MB swap in use. Perhaps my notion that lack of memory was the main culprint is mistaken.

I've definitely gotten a snootful, but have learned at lot in the bargain.

Thanks for writing,

bob prohaska

The only way to be find out is to measure it under a representative workload, but the likely outcome is that the difference will be negligible or undetectable.

You seem to have a misapprehension about /sbin. It’s not some kind of “stuff you need to boot” directory. It’s for utilities used for system administration and other root-only commands.

The way to make a separate-/usr system bootable is to ensure that the initramfs mounts /usr, as already discussed. You don’t need a separate device for that.

But those utilities are needed during the partition resize, creastion, copying and mounting processes if I want to "clean out" the duplicated /usr files.

This touches on something I wasnt aware of. I don't see an initramfs manpage. There apropos references to it, but nothing explaining its purpose. What's it for?

Thanks for writing!

bob prohaska

I am sure Richard can do much better, BUT AIUI initramfs is a RAM based filesystem that is responsible for installing the final file system, as such it loads complete as a ram image off the boot disk and then can mount disks, (load the final OS ?) and start to use it.

I am unclear as to whether the kernel it uses is a stripped down or a final version however

This may or may not be correct

"The kernel uses the initramfs as a temporary rootfs, and the initramfs will likewise include an init process that runs in the initramfs stage. The job of this pre-init is to mount the real rootfs and then switch over, handing control to the standard init process. "

Yes. You’d also need things conventionally found in /bin (today, /usr/bin) for that.

It’s there to bridge the gap between the boot loader and the root filesystem.

A typical Linux boot process looks like this:

2) The computer’s built-in firmware runs. This would be UEFI on a Mac or PC, or the EEPROM on a Pi.

4) The boot loader loads a Linux kernel.

What we want to do next is run the kernel, which should then mount the root filesystem and run init. But to do that the kernel may need a number of things:

- device drivers for the physical storage device containing the root file system

- if root filesystem is across a network, then network drivers and network configuration

- if the storage is encrypted, a driver for the encrypted storage, and the passphrase or key that unlocks the storage

- if the root filesystem is on software RAID or LVM, it needs the kernel components for those and may also need to run user-space components to properly configure access to them

- the driver for the filesystem used by the root filesystem (ext4, zfs, etc)

In principle many of these things could be hard-coded into the kernel, and many years ago they were - we all rebuilt our kernels manually with just the drivers we thought we needed, and put configuration (e.g. root device name) on a kernel command line in the boot loader configuration.

However, a Linux distribution generally wants to support as wide a range of possible configurations as possible and adding all possible drivers that anyone might need would make the kernel unreasonably large. So the drivers are kept in separate files and loaded when needed.

This creates a chicken-and-egg problem: to mount the root filesystem we need device drivers (and maybe a lot more than that), but if all of those things are on the root filesystem, the kernel can’t load them yet.

The solution is an initramfs, which is essentially a filesystem archive which contains the kernel modules and user-space executables necessary to mount the root filesystem. So booting continues:

6) The boot loader executes the Linux kernel.

8) The kernel executes the ‘init’ program found within the ramdisk. This is usually a shell script (though it doesn’t have to be).

10) When the real root filesystem has been mounted, the initramfs contents are deleted (to save memory) and the real filesystem is moved to its proper location, i.e. the / directory.

There are other ways to achieve this, but the above is more or less how it is typically done in Debian-based systems (which includes the Raspberry Pi OS).

If you want to see what’s inside the initramfs, have a look in /boot. You’ll see one or more files with initrd in the name (for historical reasons). You can unpack them with unmkinitramfs, e.g.:

$ mkdir initrd $ unmkinitramfs /boot/initrd.img-6.12.74+deb13+1-amd64 initrd/ $ ls -l initrd/main/ total 28 lrwxrwxrwx 1 richard richard 7 Mar 15 06:16 bin -> usr/bin drwxr-xr-x 3 richard richard 4096 Mar 19 21:50 conf drwxr-xr-x 7 richard richard 4096 Mar 19 21:50 etc -rwxr-xr-x 1 richard richard 6787 May 13 2025 init lrwxrwxrwx 1 richard richard 7 Mar 15 06:16 lib -> usr/lib lrwxrwxrwx 1 richard richard 9 Mar 15 06:16 lib32 -> usr/lib32 lrwxrwxrwx 1 richard richard 9 Mar 15 06:16 lib64 -> usr/lib64 drwxr-xr-x 2 richard richard 4096 Mar 15 06:16 run lrwxrwxrwx 1 richard richard 8 Mar 15 06:16 sbin -> usr/sbin drwxr-xr-x 8 richard richard 4096 Mar 19 21:50 scripts drwxr-xr-x 7 richard richard 4096 Mar 19 21:50 usr

The initramfs here has its own private /usr l-)

The relevance of all this to the separate /usr configuration that I’m trying to dissuade you from is that “the root filesystem” has to include /usr (the real one, not the initramfs’s private /usr).

Typically that is done by having it all be in one big root filesystem but as long as /usr is there by step 11, then in at least in theory, you should be golden. If it isn’t then quite early in the process you’ll find that something assumes it is present and falls over in its absence; on a modern Debian-derivce system the specific issue will be that init (i.e. systemd) is in /usr, so failure will be immediate.

That’s basically right - there’s just one kernel, it starts out quite stripped down and the initramfs is responsible for, er, clothing it up.

No words but "thank you!", that's an eloquent tutorial. Clearly my goal was mistaken.

bob prohaska

Richard Kettlewell snipped-for-privacy@>> Richard Kettlewell snipped-for-privacy@>>> You seem to have a misapprehension about /sbin. It’s not some kind of