It does work on the Pi, I've taken and restored backup using this method many times. As long as you use the correct flags on rsync (a bit of a black art) all the file attributes are preserved and Linux will work.
Yes I'd forgotten to mention that bit in my post, but don't forget to modify cmdline.txt on the FAT partition, as that will either contain a device name or a partition uuid of the original disc.
---druck
I saw it, but thougt that the "sda"-part was just as an example. Did not know at that time it is name of the whole disc. I simple have too little knowledge about handling/naming discs in linux. Last night when I wrote the question if "raspberrypi" in the dd command would represent the whole disc, it crossed my mind a few times that maybe sda represented the whole disk.
But now you all have helped me arrive at the goal: One command that fixes the whole copying. And also learned a bit. So a big thanks to all, and sorry about my ignorance :-)
An order for the NVME, case and fan will be sent today.
Best regards
Something like that. But..
In *nix the whole disk is normally given at connection time a device name like /dev/sda whereas its *partitions* would be /dev/sda1, /dev/sda2 etc etc.
when its mounted under /media it may have another name.
The mount command will show what devices you have and where they are mounted. So with the disks attached try
mount | grep '^/dev'
To identify the device name.
E.g. when I use the above command on as USB attached raspberry PI bootable drive on my desktop machine I get this:
/dev/sda5 on / type ext4 (rw,relatime,errors=remount-ro) /dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro) /dev/sdb1 on /media/leo/bootfs type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2) /dev/sdb2 on /media/leo/rootfs type ext4 (rw,nosuid,nodev,relatime,uhelper=udisks2)
This shows that the actual PC is running off /dev/sda. but the Pi disk has been given the name /dev/sdb
The fact that its been automounted on /media is irrelevant You can only mount *partitions* but you want the whole disk - in this case /dev/sdb2
Ta. I thought it probably was. My Pi server build could not handle 3 USB drives so I need to reconstruct it with just 2, which means cloning the OS onto a fraction of a larger disk.
Then creating another partition for all the data under /home
lsblk
I have not seen anybody mentioning sync Once the dd command is done, type 'sync'
In some settings the writing is not quite done yet, the sync forces the write of the caches to the device. I see this on my pc running ubuntu with 32 Gb RAM.
Never mind sync.
It's important to wait anyway on an SSD/nvm until it has finished its internal business.
For any copy process.
dd'ing 500 Gb to an ssd disk, I've seen sync taking 30 s or more and sync is of course started AFTER dd is done
So - how do you know it is done its internal business? Not all drives have blinking LEDs
sync makes it easy to know
Even sync may not be enough. SSDS/NVM have their own internal caching.
Then the question remains - how do you know when its done?
from man sync you get "sync - Synchronize cached writes to persistent storage"
In other words, doing sync at least clear the cache, which must be better than nothing? or?
Or are you saying that ripping the power to a newly sync()ed disk may mess it up?
as in
dd .... sync rip out usb cord for a 2.5" ssd sata in a usb case or something similar ?
True for all drives, unfortunately.
It’s bloody stupid, because the drive caching is on the wrong side of the drive interface. Better to leave it to the OS, which can use main RAM for its filesystem caching, on the fast side of that drive interface.
When a drive says to the OS driver “write is done”, it should mean “write has gone to actual persistent storage”, not “write is in my cache”.
The sync syscall (and command) will flush those too.
Unless you’ve paid extra for a drive with a huge cache I would expect the extra delay while the on-drive cache is flushed to be absolutely tiny in human terms, and certainly tiny compared to flushing the OS’s cache, which can be multiple gigabytes.
References:
Yes.
I am on the edge of my comfort patch here. If I were building an SSD I would have a diode and a large capacitor inside it to make sure all its caches were dumped to NVRAM before the voltage collapsed completely.
But on a big unit this could take a bit of time.
What happens between a SATA/USB plug and the actual NVRAM is a bit of a mystery.
We know its nothing like a 1:1 correlation between 'sector' and physical RAM location. We knows that physical RAM locations are regularly shuffled for 'wear levelling' When is all this done? What happens if, during it, there is power failure?
I honestly do not know, hence the warning to leave the SSD for a few seconds before yanking any power cords
It can do no harm
And that is my point. sync will do no harm either, but it might save you, especially when you are dd()ing an image smaller that the RAM of the computer. Like I have 32 Gb RAM, and dd an image of 4 Gb (like a headless) onto a SD-card to run an old pi. The interface is slow, yet dd reports done within a minute. sync takes a long time to flush it over.
Just waiting a few secs will end in disappointment
I think they cause grief in the postgres mail lists some 15-20 years ago. They were called 'lying IDE-disks' and were not popular in that crowd.
Well I will be at some stage dd-ing about 60GByte across to a 2TB disk, but given the hassle, sync and a cup of coffee wait before unplugging it, is likely.
RK claims the wait afterwards isn't necessary. Completing 'sync' is enough. He generally doesn't comment unless he knows.
I simply do not know how a command to something purporting to resemble a hard drive, that isnt can guarantee all data in it is flushed.
We might talk about different things. I'm thinking of what the OS is caching. That sync is flushing that part of the OS cache.
Not whatever clever memory that is inside a storage device.
looking at
""" The kernel keeps data in memory to avoid doing (relatively slow) disk reads and writes. This improves performance, but if the computer crashes, data may be lost or the file system corrupted as a result. sync ensures that everything in memory is written to disk. """
So, what I think is that dd writes to a slow device, and it is cached by the OS. sync forces the OS to actually write to slow device.
This may not be true with new and shiny fast NVM storage, but the principle holds.
do a sync after a dd operation.
‘Lying disks’ are those that either disregard flush operations, or lie about whether they have a write-back cache at all. That is certainly a stupid outcome, though as a response to operating systems or applications that are over-eager to flush, you can see why there’d be pressure from marketing to do it, and acceptance from market segments that either don’t value data integrity or alternatively assume that the storage is unreliable and address the issue some other way.
I think what Lawrence is complaining about is the fact that the default behavior, even for a non-lying disk, is when a SATA device returns a response to the host, this may indicate only that data has been transferred to an internal write-back cache rather than the underlying medium.
But that’s just the normal engineering response to high physical IO latency.
Recall that both traditional hard disks and SSDs do not have a 1:1 mapping between the logical block read/writes requested by the host. In a hard disk it takes time to reach the correct track, and the order of writes from the host may not match the track order. In an SSD multiple logical blocks are grouped into pages and pages must be written in a single operation.
The same logic turns up elsewhere. The write() syscall completing normally only indicates that data has been transferred to the operating system’s RAM cache, not to your SSD or hard disk (and certainly not to a remote disk, when using a network filesystem). A memory write instruction on a modern CPU only transfers a value to an internal write buffer; the data may only reach the external DRAM hundreds of cycles later, or not at all if the same location is written again soon.
The alternative is absurdly slow write IO. Usually there are some combination of flush operations, synchronous IO modes, barrier operations, etc, to allow data integrity requirements to be met without sacrificing performance globally (and this is true both at the Linux syscall layer and in the SATA protocol ... provided of course your disk does not lie).
I've used the "conv=fsync" option to dd so that sync is done automatically. Since the man page is rather vague about it, I did a web search to double check that I wasn't imagining things and found this page which describes the behaviour with some examples:
The tests at that link were done with a RAM disk, and the kernel was caching writes to that, so it doesn't look like the caching system is smart enough to know when a device is fast enough that the cache isn't required.
Caches cannot increase overall transfer bandwidth, they can only reduce latency. Which we don’t care about, because OS filesystem cache flushing is a batch-type operation anyway.
Which is why having these drives lie about when their writes have completed is such a dumb idea.
Have something to add? Share your thoughts — no account required.
Ask the community — no account required