problem moving root to hard drive (library files not found, kernel panic)

Hi all, I'm stumped with this one.

I've got a running raspbian system: the root partition is on the SD card, while everything else is on various partitions on a connected hard drive.

The hard drive has a spare partition, sda3, and I've rsync'ed root from the SD card onto that.

I've changed cmdline.txt to reflect the moved root partition:

original: console=...... root=PARTUUID=6c586e13-02 ..... new: console=s.... root=PARTUUID=b9a242b4-1890-4c02-808b-36961834b2f6 ...

and fstab (in both the new and old /etc directories) original: PARTUUID=6c586e13-02 / ext4 defaults,noatime 0 1 new: PARTUUID=b9a242b4-1890-4c02-808b-36961834b2f6 / ext4 defaults,noatime 0 1

The system tries to start up off sda3, and gets as far as running /sbin/init. But then it complains about various library files not loading, starting with:

ERROR: ld.so object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored

(I had to video the console to get that :-)

followed by a kernel panic.

I know the libarmmem*.so files are there, as it still boots off the SD card when I restore the original cmdline/fstab files.

AFAICS what I've done reflects various how-to's off the web. But something's amiss, clearly. Any help would be appreciated, thanks.

--
Mike Scott 
Harlow, England
Reply to
Mike Scott
Loading thread data ...

Have you also added this to cmdline.txt

rootwait rootdelay=15

If you haven't it can fault loading the kernel as you've described.

---druck

Reply to
druck

Thanks for that. No, only rootwait was there. However, I've now tried your suggestion; same kernel panic though.

cmdline.txt contains console=serial0,115200 console=tty1 root=PARTUUID=b9a242b4-1890-4c02-808b-36961834b2f6 rootfstype=ext4 rootwait rootdelay=15 elevator=deadline fsck.repair=yes rootwait

same problem if I try root=/dev/sda3.

However, if I use root=UUID=..... (getting the UUID from blkid), the behaviour changes... it just hangs, the last console messages being along the lines of

7.5 sda: sda1 sda2 ...... 7.5 sda 0:0:0:0 [sda] 21.5 waiting for root device UUID=...... 24.8 random: crng init done

Characters typed on the keyboard echo, but nothing else happens.

Rider: should I use the UUID or the PARTUUID in cmdline.txt, or doesn't it matter?

--
Mike Scott 
Harlow, England
Reply to
Mike Scott

Try redoing your rsync from /dev/mmcblk0p2 (root partition) to /dev/sda3 (new root partition) with -avx (x = don't cross filesystem boundaries; this will ignore things like /boot, /proc, /dev) I.e mount /dev/sda3 on (e.g.) /mnt Then "rsync -avx / /mnt/" (as root)

My cmdline.txt now looks like (note this is all one line and I use btrfs on the new root partition): # cat /boot/cmdline.txt dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=5cf6b9c2-9894-cd47-933e-ad3840ae4e5c rootfstype=btrfs elevator=deadline fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles net.ifnames=0 biosdevname=0 usbhid.mousepoll=0

Don't change the original /etc/fstab file (the one on /dev/mmcblk0p2).

This is on a RPi3+, boot on /dev/mmcblk0p1, root on /dev/sda2

--

Chris Elvidge, England
Reply to
Chris Elvidge
[rootwait and rootdelay booting from USB]

Um. Its probably worth copying all partitions back to the SD card, altering cmdline.txt and fstab and seeing if it still boot from there. If it doesn't it' something other than booting from USB.

ASFAIK: It has to be PARTUUID

---druck

Reply to
druck

On 23/10/2019 11:44, Chris Elvidge wrote: ....

Thanks for the comments (and to druck).

I've used rsync -n to cross-check the two root partitions - apart from inconsequential stuff (/tmp, cups) they're the same.

Keeping the original (card) fstab file the same makes no difference.

I don't know if it makes a difference, but I /think/ this is a plain pi3B. Other partitions are fine on the HD, but the only way I can make this go is by having root on the SD card.

I'm beginning to wonder if there might be something odd about the USB connection: I'm using a spare sata drive in a usb adapter. It works ok on ubuntu and freebsd desktop machines though. I don't have another spare to try.

--
Mike Scott 
Harlow, England
Reply to
Mike Scott

That will check the file contents are the same, but there could be differences in permissions if it wasn't copied preserving attributes with as Chris mentioned. I suggest following his advice rather than mine

- redo the SD card to USB copy with rsync -vaxHAXP which should make an identical copy.

Although they way I've always made USB drives for the Pi, is a copy a working SD card to it using dd, so the contents are intially 100% identical. I then expand the partition on the USB device to the desired size, edit /etc/fstab on the USB, and modify cmdline.txt on the SD card.

There's no reason why this wont work on the 3B. I've used a USB to SATA and a SSD with a 3B, 3B+, 4B with no changes to the setup when migrating.

I have several different USB3 to SATA2 and USB3 to SATA3 and they all work with the Pis. There isn't any noticable difference in performance between SATA2 and SATA3 on the USB2 3B and 3B+, but is a bit quicker on the USB3 4B.

---druck

Reply to
druck

....

Hi; thanks for your patience.

I've redone the rsync (nothing interesting changed), changed cmdline.txt on the card (with rootwait and rootdelay=15), and edited fstab (on the new HD partition). Still the same kernel panic. It looks increasingly as though although the HD is detected correctly, there's a problem reading it at that point.

Looks like I'll have to sacrifice another system's backup disk to get a spare usb drive to try in place of this one. Tedious :-{

--
Mike Scott 
Harlow, England
Reply to
Mike Scott

On mine (Raspbian Buster 9.11) /etc/ld.so.preload only contains the line:

# cat /etc/ld.so.preload /usr/lib/arm-linux-gnueabihf/libarmmem.so

and:

# ls -l /usr/lib/arm-linux-gnueabihf/libarmmem*

-rw-r--r-- 1 root root 22K Mar 14 2019 /usr/lib/arm-linux-gnueabihf/libarmmem.so

---------------------------------------------

Now looked at another system (Raspbian 10.1) and I see several libarmmem* files in /usr/lib/arm-linux-gnueabihf/ with the same line as yours in /etc/ld.so.preload

$ cat /etc/ld.so.preload /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so

$ ls -l /usr/lib/arm-linux-gnueabihf/libarmmem* lrwxrwxrwx 1 root root 16 Apr 30 10:22 /usr/lib/arm-linux-gnueabihf/libarmmem-aarch64.so -> libarmmem-v7l.so

-rw-r--r-- 1 root root 9512 Apr 30 10:22 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so

-rw-r--r-- 1 root root 17708 Apr 30 10:22 /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so lrwxrwxrwx 1 root root 16 Apr 30 10:22 /usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so -> libarmmem-v7l.so

Are all the softlink files available in your install? What do you get if you run the 'ls -l' command above?

--

Chris Elvidge, England
Reply to
Chris Elvidge

Further to my last, these are on my 10.1 install (512Mb SD card and 16Gb USB stick):

root@redraspberry:~# df -lh | grep ^/dev/ /dev/root 15G 1.6G 13G 12% / /dev/mmcblk0p1 490M 41M 449M 9% /boot

root@redraspberry:~# cat /etc/fstab | grep ^PART PARTUUID=4a80fd15-01 /boot vfat defaults 0 2 PARTUUID=f69d5e19-01 / ext4 defaults,noatime 0 1

root@redraspberry:~# blkid | grep [ap]1 /dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="KINGMAX512" UUID="8215-480C" TYPE="vfat" PARTUUID="4a80fd15-01" /dev/sda1: UUID="7336da1d-b3dc-4639-978f-d7d1376946ab" TYPE="ext4" PARTUUID="f69d5e19-01"

root@redraspberry:~# cat /boot/cmdline.txt dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=f69d5e19-01 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Does your system match?

--

Chris Elvidge, England
Reply to
Chris Elvidge

You could try to copy the root partition using dd.

$ sudo dd if=/dev/mmcblk0p2 of=/dev/sda2 bs=1M

I had good results with that. The destination partition has to be at least as big as the source partition. When the SD card is bigger than the root partition on the HD or SSD, you have to avoid trouble by removing init=/usr/lib/raspi-config/init_resize.sh from cmdline.txt prior to the first boot, so the root partitiion on the SD card is not expanded.

After the dd, the new root filesystem can be expanded with resize2fs.

Before (destination already partitioned) : $ lsblk NAME SIZE TYPE MOUNTPOINT sda 232.9G disk

+-sda1 256M part /media/pi/SSDBOOT +-sda2 14.8G part /media/pi/ssdroot +-sda3 218G part /media/pi/{some_GUID} mmcblk0 29.7G disk +-mmcblk0p1 256M part /boot +-mmcblk0p2 5.6G part /

After: pi@raspberrypi:~ $ lsblk NAME SIZE TYPE MOUNTPOINT sda 232.9G disk

+-sda1 256M part /media/pi/SSDBOOT +-sda2 14.8G part / +-sda3 218G part /extra mmcblk0 29.7G disk +-mmcblk0p1 256M part /boot +-mmcblk0p2 5.6G part /media/pi/sdroot

Notes:

-sda1 is meant to keep a copy of /boot

-sda2 is root and swap

-sda2 is an extra partition for home and data, so a new Raspbian install can leave those alone.

--
Regards, 
Kees Nuyt
Reply to
Kees Nuyt

.....

I think I may be getting closer to the issue. Sorry if this is longish; work in progress.

The spare disk I mentioned is actually half-free, partitioned into two

500Gb chunks, one free. So I stuck it in in my desktop machine, converted the spare partition to ext4, and used rsync -ax on it, and tweaked the fstab and cmdline.txt to match. Stuck it in the PI, and bingo, it boots happily.

But I don't like an all-in-one partition, so, I started over (on the dekstop box), and repartitioned for a fresh root, var, usr, and so on. I mounted the new root, and created mount points, and mounted those partitions. Then again used rsync -ax , edited the fstab in the new etc (using device names), and updated the SD card's cmdline.txt.

I stuck the SD card and disk onto the PI -- kernel panic.

Hmmm. Should have worked.

OK, back to the desktop box. This time, I rsync into place the entire SD card into the new root partition (which was large enough). Restore my fstab to mount the other partitions on top of the (unwanted) directories. This really ought to behave the same as the last failed effort, but no: this time it boots.

Conjecture (for now) - ld.so.preload moaned about missing files from /usr/lib. But these aren't available until /usr is mounted, which I presume must happen /after/ the preload stuff. Maybe I'll go back to the original setup and put these files into the root partition, so they're present before /usr is mounted. Bet it works then :-}

(Oh, and thanks again to all who've commented. It's useful to see another POV and keeps the thoughts flowing.)

--
Mike Scott 
Harlow, England
Reply to
Mike Scott

On 24/10/2019 15:43, Mike Scott wrote: ....

Yes, that does seem to be the issue.

I put /usr/lib/arm-linux-gnueabihf onto the root partition so it's available at boot time. It gets mounted over later.

For reference, cmdline.txt contains

console=serial0,115200 console=tty1 root=PARTUUID=b9a242b4-1890-4c02-808b-36961834b2f6 rootfstype=ext4 rootwait rootdelay=15 elevator=deadline fsck.repair=yes rootwait

fstab on the HD root partition is proc /proc proc defaults 0 0 PARTUUID=6c586e13-01 /boot vfat defaults 0 2

# keep root on sd card for now ##PARTUUID=6c586e13-02 / ext4 defaults,noatime 0 1 # don't forget to update /boot/cmdline.txt to match PARTLABEL=rootPI0 / ext4 defaults 0 1

PARTLABEL=usrPI0 /usr ext4 defaults 0 1 PARTLABEL=varPI0 /var ext4 defaults 0 1 PARTLABEL=homePI0 /home ext4 defaults 0 1 PARTLABEL=mmediaPI0 /mmedia ext4 defaults 0 1

PARTLABEL=swapPI0 none swap defaults 0 0

(Although the fstab entry for / isn't used.)

Thanks all.

--
Mike Scott 
Harlow, England
Reply to
Mike Scott

I wish you'd explained that bit at the beginning.

--

Chris Elvidge, England
Reply to
Chris Elvidge

Yes. But it's only after the fact that I see the issue. Which, in all the "how to"s I've looked at, isn't mentioned. I'd call it a bug - anything that's needed to boot should be available in the root file system. The arm-specific libraries are needed, but aren't in the right place.

Surely I can't be the only one to try moving /usr to a separate partition, when it's standard practice anywhere else?

Anyway, problem resolved, and thanks again to all for their patience.

--
Mike Scott 
Harlow, England
Reply to
Mike Scott

See:

formatting link

Quote:/usr on its own filesystem is useful in some custom setups. But instead of expecting the traditional Unix way to (sometimes mindlessly) distributing tools between /usr and /, and require more and more tools to move to /, we now just expect /usr to be pre-mounted from inside the initramfs, to be available before 'init' starts. The duty of the minimal boot system that consisted of /bin, /sbin and /lib on traditional Unix, has been taken over by the initramfs of modern Linux. An initramfs that supports mounting /usr on top of / before it starts 'init', makes all existing setups work properly.

There is no way to reliably bring up a modern system with an empty /usr. There are two alternatives to fix it: move /usr back to the rootfs or use an initramfs which can hide the split-off from the system.

--

Chris Elvidge, England
Reply to
Chris Elvidge

It is possible to have a minimal /usr in the boot partition and mouunt the proper one over it post boot.

I.e. you can mount drives on a populated mount point.

--
?There are two ways to be fooled. One is to believe what isn?t true; the  
other is to refuse to believe what is true.? 
 Click to see the full signature
Reply to
The Natural Philosopher

It's times like this that I wish usenet had the head slapping emoji.

---druck

Reply to
druck

Interesting discussion. I'm somewhat old fashioned, I guess. That little discussion is totally new to me - and I would note that in freebsd a /usr partition is standard, while mint is very happy to use a separate /usr.

And in my days (long gone!) as system manager of SunOS systems, /usr (in particular) was network mounted and shared, because some workstations had no [expensive] disk at all. It made network-wide updates easy, and the small root partition had all the "personality" for each machine.

But things have moved on while I've not been looking. I'll certainly take your thoughts on board for the future. Thanks.

--
Mike Scott 
Harlow, England
Reply to
Mike Scott

See my comments to Chris. When practice varies across time & systems, there are bound to be problems that pop up. The odd thing is, that in all the web searches I did (before posting here), there wasn't a hint of this issue. Hopefully that's now changed.

--
Mike Scott 
Harlow, England
Reply to
Mike Scott

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.