Pi 5 and NVMe SSD

I’m expecting to free up an NVMe SSD soon, and my Pi 5 is currently making do with an SD card, so I’ve been looking into options for putting the two together.

Has anyone used this case?

Another easily discoverable option is:

Are there any other good options out there?

Print your own Richard. That's what I will be doing :-)

Alternatively wait a couple of months till I have proven my design...it should fit an NVME equipped Pi 5, although it's designed for three SATA SSDs.

I don't know about cases, but the rPi5 £5 official active cooler is good. Good in that it cools passively, with the fan only spinning up momentarily at startup and under extreme load (in my case only under a stress test).

The cooler provides quite a lot of protection to the top of the board, I assume the nvme-base will protect the base.

So the need for additional case protection just comes down to the 40 pin gpio, and how much you want to press the on off button (tricky without a case).

I got rid of my desire for an nvme with my opi5. Due to being PCIe 1 lane, it wasn't that much faster, and there was significant additional effort installing the OS, compared to a micro SD. Obviously, you may have a different use-case.

That surprises me, could anyone with an NVMe base run this script which measures the maximum sequential and random access read/write speeds. You'll need to install the fio package and be in a directory on the disc under test.

#!/bin/bash TESTFILE=fio-tempfile.dat fio --name SeqRead --eta-newline=5s --filename=$TESTFILE --rw=read\ --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio\ --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=30\ --group_reporting | grep bw= fio --name SeqWrite --eta-newline=5s --filename=$TESTFILE --rw=write\ --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio\ --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=30\ --group_reporting | grep bw= fio --name RndRead --eta-newline=5s --filename=$TESTFILE --rw=randread\ --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1\ --iodepth=1 --direct=1 --numjobs=1 --runtime=30 --group_reporting\ | grep bw= fio --name RndWrite --eta-newline=5s --filename=$TESTFILE--rw=randwrite\ --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1\ --iodepth=1 --direct=1 --numjobs=1 --runtime=30 --group_reporting\ | grep bw= rm $TESTFILE

More than dd'ing the SD card image on to the NVMe, using gparted to expand the linux partition, and then changing the boot device in /boot/cmdline.txt and /etc/fstab ?

---druck

I posted some stats back in December ( hdparm -Ttv), on the opi5 the nvme was only giving 186 MB/s seq read, not that much more than a USB3 SSD at 137 MB/s.

I think, like the oPi5, the rPi5 also has the handicap that it only allocates PCIe 3.0 x 1 to nvme.

So yes, nvme is faster in benchmarks, but nowhere near the 3000MB/s you might see on a PC.

Perhaps more pertinently, the user experience improvement using the Pi as an HTPC was negligible. I guess the performance I care about wasn't constrained by disk read speed. So now I just run a SD card for the OS, and pretty much everything else is NAS anyway.

I'm actually really happy with the rPi5 as a HTPC. I'm sitting in front of the TV typing this.

On the oPi5 there was DD (which I hate), and editing files on the boot partition, on the onboard persistent ram. Not difficult, but still enough to make mistakes. Whereas the Raspberry Pi Imager makes writing a SD card really easy.

I've got at least one spare nvme sitting around, I'm not going to put it on the rPi5.

My oPi5 is also booting off micro SD even though it has an nvme drive. If Armbian ever rollout a stable version for the oPi5 I will stick it on the nvme again, but I did it so many times I got pissed off.

In stark contrast, Pi OS has been solid on the rPi5.

I think Jeff Geerling saw 900MB/s ?

Remember, I'm testing nvme on an Orange Pi 5 not a Raspberry Pi 5, but, I feel quite inadequate. However, it is what I get.

Here are the results from druck's script:

READ: bw=200MiB/s (209MB/s), 200MiB/s-200MiB/s (209MB/s-209MB/s), io=6000MiB (6291MB), run=30060-30060msec WRITE: bw=199MiB/s (209MB/s), 199MiB/s-199MiB/s (209MB/s-209MB/s), io=6000MiB (6291MB), run=30085-30085msec READ: bw=36.0MiB/s (37.8MB/s), 36.0MiB/s-36.0MiB/s (37.8MB/s-37.8MB/s), io=1081MiB (1133MB), run=30001-30001msec READ: bw=48.1MiB/s (50.4MB/s), 48.1MiB/s-48.1MiB/s (50.4MB/s-50.4MB/s), io=1443MiB (1513MB), run=30001-30001msec

Maybe my result is so bad because I'm only using a KingSpec nvme drive? It's supposed to do 3500MB/s, but I've never tested it in anything that could handle that.

I'm not going to use a top range drive on a Pi. The Orange Pi isn't doing anything else, isn't busy.

Shouldn't that file size be much larger to prevent it from being cached ? I get ridiculous numbers trying that script on a PC.

I was curious, so I did a bit of digging, it appears my nvme drive is running as PCIe gen 1. The oPi5 should be gen 3, The nvme drive should be gen 3.

$ lspci | grep -i nvme

$ sudo lspci -vv -s 0004:41:00.0 | grep LnkSta

LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded) LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete EqualizationPhase1-

Which, from my poor understanding, implies a theoretical top speed of

I don't have an rpi5, but they also default to gen1, a config.txt option can force gen3

dtparam=pciex1_gen=3

does an opi5 have similar?

I looked, I tried, I failed!

I tried adding the following line to /boot/armbianEnv.txt

''' dtoverlay=pcie-gen2 '''

I also tried editing the dtb overlay: /boot/dtb/rockchip/rk3588s-orangepi-5.dtb

which was already configured for gen 2

Seriously, the oPi5 looks like a brilliant device, it feels like a brilliant device, but nothing quite works properly, my advice is to buy the rPi5, every time.

I can't speak for Armbian, but config.txt is a RPi thing - it's config for the GPU (originally) or boot firmware (4+) to set things up, rather than Linux. I don't know what the oPis do, but unless they clone rPi settings I wouldn't expect it to work.

That may well make a difference, but don't know how they set up PCIe. Also, devices start up as gen1 and then mutually agree to upgrade. Perhaps something isn't right to do that. I don't know if it's firmware or Linux upgrading, but maybe look at dmesg for anything eith 'pci' in it for clues? Signal integrity can be a reason for the upgrade to fail.

That's my general feeling. I think Armbian are trying to increase their level of support for various board, but they are just volunteers with no help from vendors.

Theo

On 21/02/2024 10:22, Guido Giorgi wrote: [Snip non standard quoting]

The first 3 lines look good, the last one should be a random write, not a read, so I fear the script didn't make it out of the post intact.

---druck

Check the line breaks didn't get messed up, as the last line should be WRITE.

---druck

Yes, that script was on a Pi with an SD card, so didn't want to spend too much of it's write life. Try with 2g for an SSD.

---druck

=$TESTFILE--rw=randwrite

should be

=$TESTFILE --rw=randwrite

Yep, for clarity, I don't want to slag off Armbian. So far, Armbian is the best distro I've tried for the oPi5. My main gripe is the lack of hardware support, i.e. a good GPU driver, which is something outside Armbian's scope.

I don't really care about the NVME running gen 1. Maybe I will buy a NVME base for the rPi5, when the official case is adapted to work with it.

I’ve ordered one of these.

Not an rPi, not even an oPi, but an interesting looking device ...

A carrier board for a rockchip compute module, gives four M.2 slots (each only single lane of gen3 if using all four)

That sort of issue could be the kicker with the friendlyelec ...