problem moving root to hard drive (library files not found, kernel panic)

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
Hi all, I'm stumped with this one.

I've got a running raspbian system: the root partition is on the SD  
card, while everything else is on various partitions on a connected hard  
drive.

The hard drive has a spare partition, sda3, and I've rsync'ed root from  
the SD card onto that.

I've changed cmdline.txt to reflect the moved root partition:

original:
console=...... root=PARTUUID=6c586e13-02 .....
new:
console=s.... root=PARTUUID=b9a242b4-1890-4c02-808b-36961834b2f6 ...

and fstab (in both the new and old /etc directories)
original:
PARTUUID=6c586e13-02  /               ext4    defaults,noatime  0       1
new:
PARTUUID=b9a242b4-1890-4c02-808b-36961834b2f6  /               ext4  
defaults,noatime  0       1

The system tries to start up off sda3, and gets as far as running  
/sbin/init. But then it complains about various library files not  
loading, starting with:

ERROR: ld.so object  
'/usr/lib/arm-linux-gnueabihf/libarmmem-$.so' from  
/etc/ld.so.preload cannot be preloaded (cannot open shared object file):  
ignored

(I had to video the console to get that :-)

followed by a kernel panic.

I know the libarmmem*.so files are there, as it still boots off the SD  
card when I restore the original cmdline/fstab files.

AFAICS what I've done reflects various how-to's off the web. But  
something's amiss, clearly. Any help would be appreciated, thanks.



--  
Mike Scott
Harlow, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 22/10/2019 16:12, Mike Scott wrote:
Quoted text here. Click to load it

Have you also added this to cmdline.txt

    rootwait rootdelay15%

If you haven't it can fault loading the kernel as you've described.

---druck


Re: problem moving root to hard drive (library files not found, kernel panic)
On 22/10/2019 20:10, druck wrote:
Quoted text here. Click to load it

Quoted text here. Click to load it

Thanks for that. No, only rootwait was there. However, I've now tried  
your suggestion; same kernel panic though.

cmdline.txt contains
console=serial0,115200 console=tty1  
root=PARTUUID=b9a242b4-1890-4c02-808b-36961834b2f6 rootfstype=ext4  
rootwait rootdelay15% elevator=deadline fsck.repair=yes rootwait

same problem if I try root=/dev/sda3.

However, if I use root=UUID=..... (getting the UUID from blkid), the  
behaviour changes... it just hangs, the last console messages being  
along the lines of
7.5 sda: sda1 sda2 ......
7.5 sda 0:0:0:0 [sda]
21.5 waiting for root device UUID=......
24.8 random: crng init done

Characters typed on the keyboard echo, but nothing else happens.

Rider: should I use the UUID or the PARTUUID in cmdline.txt, or doesn't  
it matter?


--  
Mike Scott
Harlow, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 23/10/2019 11:00, Mike Scott wrote:
Quoted text here. Click to load it

[rootwait and rootdelay booting from USB]
Quoted text here. Click to load it

Um. Its probably worth copying all partitions back to the SD card,  
altering cmdline.txt and fstab and seeing if it still boot from there.  
If it doesn't it' something other than booting from USB.

Quoted text here. Click to load it

ASFAIK: It has to be PARTUUID

---druck

Re: problem moving root to hard drive (library files not found, kernel panic)
On 22/10/2019 16:12, Mike Scott wrote:
Quoted text here. Click to load it

Try redoing your rsync from /dev/mmcblk0p2 (root partition) to /dev/sda3  
(new root partition) with -avx (x = don't cross filesystem boundaries;  
this will ignore things like /boot, /proc, /dev)
I.e mount /dev/sda3 on (e.g.) /mnt
Then "rsync -avx / /mnt/" (as root)

My cmdline.txt now looks like (note this is all one line and I use btrfs  
on the new root partition):
# cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1  
root=PARTUUID=5cf6b9c2-9894-cd47-933e-ad3840ae4e5c rootfstype=btrfs  
elevator=deadline fsck.repair=yes rootwait quiet splash  
plymouth.ignore-serial-consoles net.ifnames=0 biosdevname=0  
usbhid.mousepoll=0

Don't change the original /etc/fstab file (the one on /dev/mmcblk0p2).

This is on a RPi3+, boot on /dev/mmcblk0p1, root on /dev/sda2

--  

Chris Elvidge, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 23/10/2019 11:44, Chris Elvidge wrote:
....
Quoted text here. Click to load it

Thanks for the comments (and to druck).

I've used rsync -n to cross-check the two root partitions - apart from  
inconsequential stuff (/tmp, cups) they're the same.

Keeping the original (card) fstab file the same makes no difference.

I don't know if it makes a difference, but I /think/ this is a plain  
pi3B. Other partitions are fine on the HD, but the only way I can make  
this go is by having root on the SD card.

I'm beginning to wonder if there might be something odd about the USB  
connection: I'm using a spare sata drive in a usb adapter. It works ok  
on ubuntu and freebsd desktop machines though. I don't have another  
spare to try.




--  
Mike Scott
Harlow, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 23/10/2019 16:52, Mike Scott wrote:
Quoted text here. Click to load it

That will check the file contents are the same, but there could be  
differences in permissions if it wasn't copied preserving attributes  
with as Chris mentioned. I suggest following his advice rather than mine  
- redo the SD card to USB copy with rsync -vaxHAXP which should make an  
identical copy.

Although they way I've always made USB drives for the Pi, is a copy a  
working SD card to it using dd, so the contents are intially 100%  
identical. I then expand the partition on the USB device to the desired  
size, edit /etc/fstab on the USB, and modify cmdline.txt on the SD card.

Quoted text here. Click to load it

There's no reason why this wont work on the 3B. I've used a USB to SATA  
and a SSD with a 3B, 3B+, 4B with no changes to the setup when migrating.

Quoted text here. Click to load it

I have several different USB3 to SATA2 and USB3 to SATA3 and they all  
work with the Pis. There isn't any noticable difference in performance  
between SATA2 and SATA3 on the USB2 3B and 3B+, but is a bit quicker on  
the USB3 4B.

---druck

Re: problem moving root to hard drive (library files not found, kernel panic)
On 23/10/2019 11:44, Chris Elvidge wrote:
Quoted text here. Click to load it
....

Hi; thanks for your patience.

I've redone the rsync (nothing interesting changed), changed cmdline.txt  
on the card (with rootwait and rootdelay15%), and edited fstab (on the  
new HD partition). Still the same kernel panic. It looks increasingly as  
though although the HD is detected correctly, there's a problem reading  
it at that point.

Looks like I'll have to sacrifice another system's backup disk to get a  
spare usb drive to try in place of this one. Tedious :-{



--  
Mike Scott
Harlow, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On Thu, 24 Oct 2019 10:35:44 +0100, Mike Scott

Quoted text here. Click to load it

You could try to copy the root partition using dd.

$ sudo dd if=/dev/mmcblk0p2 of=/dev/sda2 bs=1M

I had good results with that.
The destination partition has to be at least as big as the
source partition. When the SD card is bigger than the root
partition on the HD or SSD, you have to avoid trouble by
removing
   init=/usr/lib/raspi-config/init_resize.sh
from cmdline.txt prior to the first boot, so the root partitiion
on the SD card is not expanded.

After the dd, the new root filesystem can be expanded with
resize2fs.

Before (destination already partitioned) :
$ lsblk
NAME         SIZE TYPE MOUNTPOINT
sda        232.9G disk
+-sda1       256M part /media/pi/SSDBOOT
+-sda2      14.8G part /media/pi/ssdroot
+-sda3       218G part /media/pi/
mmcblk0     29.7G disk
+-mmcblk0p1  256M part /boot
+-mmcblk0p2  5.6G part /

After:
pi@raspberrypi:~ $ lsblk
NAME         SIZE TYPE MOUNTPOINT
sda        232.9G disk
+-sda1       256M part /media/pi/SSDBOOT
+-sda2      14.8G part /
+-sda3       218G part /extra
mmcblk0     29.7G disk
+-mmcblk0p1  256M part /boot
+-mmcblk0p2  5.6G part /media/pi/sdroot

Notes:
-sda1 is meant to keep a copy of /boot
-sda2 is root and swap
-sda2 is an extra partition for home and data, so a new Raspbian
install can leave those alone.

--  
Regards,
Kees Nuyt

Re: problem moving root to hard drive (library files not found, kernel panic)
On 22/10/2019 16:12, Mike Scott wrote:
Quoted text here. Click to load it

On mine (Raspbian Buster 9.11) /etc/ld.so.preload only contains the line:

# cat /etc/ld.so.preload
/usr/lib/arm-linux-gnueabihf/libarmmem.so

and:

# ls -l /usr/lib/arm-linux-gnueabihf/libarmmem*
-rw-r--r-- 1 root root 22K Mar 14  2019  
/usr/lib/arm-linux-gnueabihf/libarmmem.so

---------------------------------------------

Now looked at another system (Raspbian 10.1) and I see several  
libarmmem* files in /usr/lib/arm-linux-gnueabihf/ with the same line as  
yours in /etc/ld.so.preload

$ cat /etc/ld.so.preload
/usr/lib/arm-linux-gnueabihf/libarmmem-$.so

$ ls -l /usr/lib/arm-linux-gnueabihf/libarmmem*
lrwxrwxrwx 1 root root    16 Apr 30 10:22  
/usr/lib/arm-linux-gnueabihf/libarmmem-aarch64.so -> libarmmem-v7l.so
-rw-r--r-- 1 root root  9512 Apr 30 10:22  
/usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
-rw-r--r-- 1 root root 17708 Apr 30 10:22  
/usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so
lrwxrwxrwx 1 root root    16 Apr 30 10:22  
/usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so -> libarmmem-v7l.so

Are all the softlink files available in your install? What do you get if  
you run the 'ls -l' command above?


--  

Chris Elvidge, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 24/10/2019 12:43, Chris Elvidge wrote:
Quoted text here. Click to load it
.....

I think I may be getting closer to the issue. Sorry if this is longish;  
work in progress.

The spare disk I mentioned is actually half-free, partitioned into two  
500Gb chunks, one free. So I stuck it in in my desktop machine,  
converted the spare partition to ext4, and used rsync -ax on it, and  
tweaked the fstab and cmdline.txt to match. Stuck it in the PI, and  
bingo, it boots happily.

But I don't like an all-in-one partition, so, I started over (on the  
dekstop box), and repartitioned for a fresh root, var, usr, and so on. I  
mounted the new root, and created mount points, and mounted those  
partitions. Then again used rsync -ax <sdcard root> <new root>, edited  
the fstab in the new etc (using device names), and updated the SD card's  
cmdline.txt.

I stuck the SD card and disk onto the PI -- kernel panic.

Hmmm. Should have worked.

OK, back to the desktop box. This time, I rsync into place the entire SD  
card into the new root partition (which was large enough). Restore my  
fstab to mount the other partitions on top of the (unwanted)  
directories. This really ought to behave the same as the last failed  
effort, but no: this time it boots.

Conjecture (for now) - ld.so.preload moaned about missing files from  
/usr/lib. But these aren't available until /usr is mounted, which I  
presume must happen /after/ the preload stuff. Maybe I'll go back to the  
original setup and put these files into the root partition, so they're  
present before /usr is mounted. Bet it works then :-}

(Oh, and thanks again to all who've commented. It's useful to see  
another POV and keeps the thoughts flowing.)

--  
Mike Scott
Harlow, England

resolved: Re: problem moving root to hard drive (library files not found, kernel panic)
On 24/10/2019 15:43, Mike Scott wrote:
....
Quoted text here. Click to load it

Yes, that does seem to be the issue.

I put /usr/lib/arm-linux-gnueabihf onto the root partition so it's  
available at boot time. It gets mounted over later.

For reference, cmdline.txt contains

console=serial0,115200 console=tty1  
root=PARTUUID=b9a242b4-1890-4c02-808b-36961834b2f6 rootfstype=ext4  
rootwait rootdelay15% elevator=deadline fsck.repair=yes rootwait

fstab on the HD root partition is
proc            /proc           proc    defaults          0       0
PARTUUID=6c586e13-01  /boot           vfat    defaults          0       2

# keep root on sd card for now
##PARTUUID=6c586e13-02  /               ext4    defaults,noatime  0       1
# don't forget to update /boot/cmdline.txt to match
PARTLABEL=rootPI0       /               ext4    defaults        0       1

PARTLABEL=usrPI0        /usr            ext4    defaults        0       1
PARTLABEL=varPI0        /var            ext4    defaults        0       1
PARTLABEL=homePI0       /home           ext4    defaults        0       1
PARTLABEL=mmediaPI0     /mmedia         ext4    defaults        0       1

PARTLABEL=swapPI0       none            swap    defaults        0       0


(Although the fstab entry for / isn't used.)



Thanks all.



--  
Mike Scott
Harlow, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 24/10/2019 15:43, Mike Scott wrote:
Quoted text here. Click to load it

I wish you'd explained that bit at the beginning.


--  

Chris Elvidge, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 24/10/2019 16:28, Chris Elvidge wrote:
Quoted text here. Click to load it
Yes. But it's only after the fact that I see the issue. Which, in all  
the "how to"s I've looked at, isn't mentioned. I'd call it a bug -  
anything that's needed to boot should be available in the root file  
system. The arm-specific libraries are needed, but aren't in the right  
place.

Surely I can't be the only one to try moving /usr to a separate  
partition, when it's standard practice anywhere else?

Anyway, problem resolved, and thanks again to all for their patience.

--  
Mike Scott
Harlow, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 24/10/2019 16:37, Mike Scott wrote:
Quoted text here. Click to load it

See:
https://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken/

Quote:/usr on its own filesystem is useful in some custom setups. But  
instead of expecting the traditional Unix way to (sometimes mindlessly)  
distributing tools between /usr and /, and require more and more tools  
to move to /, we now just expect /usr to be pre-mounted from inside the  
initramfs, to be available before 'init' starts. The duty of the minimal  
boot system that consisted of /bin, /sbin and /lib on traditional Unix,  
has been taken over by the initramfs of modern Linux. An initramfs that  
supports mounting /usr on top of / before it starts 'init', makes all  
existing setups work properly.

There is no way to reliably bring up a modern system with an empty /usr.  
There are two alternatives to fix it: move /usr back to the rootfs or  
use an initramfs which can hide the split-off from the system.





--  

Chris Elvidge, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 24/10/2019 17:44, Chris Elvidge wrote:
Quoted text here. Click to load it

Interesting discussion. I'm somewhat old fashioned, I guess. That little  
discussion is totally new to me - and I would note that in freebsd a  
/usr partition is standard, while mint is very happy to use a separate  
/usr.

And in my days (long gone!) as system manager of SunOS systems, /usr (in  
particular) was network mounted and shared, because some workstations  
had no [expensive] disk at all. It made network-wide updates easy, and  
the small root partition had all the "personality" for each machine.

But things have moved on while I've not been looking. I'll certainly  
take your thoughts on board for the future.  Thanks.


--  
Mike Scott
Harlow, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 24/10/2019 16:28, Chris Elvidge wrote:
Quoted text here. Click to load it

It's times like this that I wish usenet had the head slapping emoji.

---druck


Re: problem moving root to hard drive (library files not found, kernel panic)
On 25/10/2019 08:16, druck wrote:
Quoted text here. Click to load it

See my comments to Chris. When practice varies across time & systems,  
there are bound to be problems that pop up. The odd thing is, that in  
all the web searches I did (before posting here), there wasn't a hint of  
this issue. Hopefully that's now changed.


--  
Mike Scott
Harlow, England

Re: problem moving root to hard drive (library files not found, kernel panic)
On 24/10/2019 15:43, Mike Scott wrote:
Quoted text here. Click to load it

It is possible to have a minimal /usr in the boot partition and mouunt  
the proper one over it post boot.

I.e. you can mount drives on a populated mount point.

--  
?There are two ways to be fooled. One is to believe what isn?t true; the  
other is to refuse to believe what is true.?

We've slightly trimmed the long signature. Click to see the full one.
Re: problem moving root to hard drive (library files not found, kernel panic)
On 22/10/2019 16:12, Mike Scott wrote:
Quoted text here. Click to load it

Further to my last, these are on my 10.1 install (512Mb SD card and 16Gb  
USB stick):

root@redraspberry:~# df -lh | grep ^/dev/
/dev/root        15G  1.6G   13G  12% /
/dev/mmcblk0p1  490M   41M  449M   9% /boot

root@redraspberry:~# cat /etc/fstab | grep ^PART
PARTUUID=4a80fd15-01  /boot     vfat    defaults          0       2
PARTUUID=f69d5e19-01  /         ext4    defaults,noatime  0       1

root@redraspberry:~# blkid | grep [ap]1
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="KINGMAX512" UUID="8215-480C"  
TYPE="vfat" PARTUUID="4a80fd15-01"
/dev/sda1: UUID="7336da1d-b3dc-4639-978f-d7d1376946ab" TYPE="ext4"  
PARTUUID="f69d5e19-01"

root@redraspberry:~# cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1  
root=PARTUUID=f69d5e19-01 rootfstype=ext4 elevator=deadline  
fsck.repair=yes rootwait

Does your system match?

--  

Chris Elvidge, England

Site Timeline