Disk imaging strategy

On a sunny day (Sun, 02 Nov 2014 08:25:31 -0700) it happened Don Y wrote in :

Not quite sure whatyouwant, but I have done this a lot: start some recue disk, plug in some USB disk. mount the partition you want, then: tar -zcvf partition_sda1_image.tgz /dev/sda1

I've done this with my Raspberries SDcards by just putting the card in the PC and mounting it.

script: /usr/local/sbin/backup-raspi-sd-card

echo "Unmount and remove any USB memory sticks" echo "Insert and power seagate USB harddisk, type ENTER when ready" read user_reply

sleep 4 mount /dev/sdc8 /mnt/sdc8

cd /mnt/sdc8/backups/raspberry_pi/raspi_backups/

ls -rtl df

echo "check for enough diskspace,

It doesn't get interesting until you get to IBM CKD disks...

For older disks, that are usually relatively small, that is probably the best choice.

Be sure to fsck or chkdsk first.

I usually use an ATA or SATA to USB adaptor, and then copy from the USB side. I presume you are copying partitions, not the whole disk including the partition table.

-- glen

The problem comes with newer disks. E.g., I keep ~1T on each workstation and *only* drag "current projects" onto them counting on the file servers to maintain most of my stuff "semi-offline".

So, you can easily have just 5 or 10% of the disk "in use" but 90% of it "dirty". Dirt doesn't compress nearly as well as "virgin media" :>

Yes, of course. The point is that I am willing to expend a fair bit of effort -- including "unscripted" actions -- to get the initial "master" disk image "Correct". But, want most of that effort to be in the native OS instead of having to implement hooks for every conceivable file system type.

I am actually doing both.

I originally crafted a quick and dirty solution that allowed me to build "rescue partitions" on laptops that were being donated to disadvantaged children. The point was to remove the "maintenance" issue from that (pro bono) activity: "Your computer is 'broke'? Well, do this when the machine boots up and your disk will be restored to the EXACT condition it existed at the time you received it!" (which is typically much further developed than any "factory restore" would have created).

In those cases, *everything* but my partition was restored (some laptops have one or two additional "partitions" besides the "restore" partition). This allows me to really distance myself from whatever bad practices in which they may have engaged.

The fact that *they* have to assume responsibility for the "repair" ALONG WITH RECOVERING ANY PRECIOUS FILES BEFORE DOING SO tends to discourage them from putting much "at risk".

But, I also needed to be able to spool the image off onto offline media for my own archives (so, if I come across another "Model 123" laptop, I can just load the disk from my offline copy of the "restore partition").

As this seems to have worked pretty well, I want to extend it to address a wider variety of machines -- esp those that I run, here! As I keep most precious files on servers, it is relatively easy for me to adjust to bare metal restores (I already do this on this "Internet Machine" as a simple way of ensuring it's integrity).

But, more machines means more file system types, etc.

And regardless of interface? ATA? SCSI? USB? IEEE-1394? SAN? ??? Good luck with that.

Raw block access is OS agnostic, but it has a lot of drawbacks: it happily replicates damaged filesystems and a naive image can't be restored to a smaller volume even if the volume could hold the live data.

It's better to have a program that understands the volume's file table format and can copy only live files [as files]. Most systems use some variant of either inode or FAT, so there isn't too much to figure out. The only oddballs I'm aware of are record and log-structured systems which you aren't likely to encounter.

I'm not a fan of imaging backup ... too many ways to get burned.

The big file approach can take a lot of time and can cause failures if something suddenly needs file space when the disk is full.

Also, quite a few filesystems shortcut writes of all zeros and just mark the affected blocks empty in their headers. So to be safe you have to fill write some other value.

Yeah, until it blows up in your face.

George

If the machine can access the medium, then what do I care about the hardware interface?

For every (?) machine, there are certain assumptions that can be made at IPL: one of which is that it can retrieve some number of bytes from the boot volume. I.e., it need not be aware of the file system's structure or even how to *write* to that medium. But, it must have the ability to fetch the initial loader.

What you load is largely a matter of your discretion. If you don't need to understand what you are restoring (or imaging), then all you need to do is ensure you can extend that initial capability to one that allows you to read (write) arbitrary portions of the medium.

If you want to image/restore a medium that is not natively supported, tough!

Point isn't to restore to arbitrary media but, rather, to the medium from which it came.

As to "damaged filesystems", deal with that BEFORE you make the image!

I've not encountered a problem, yet -- and have been going the Clonezilla route for my home machines for quite some time (storing the images on removable SATA drives in a large shoebox).

Note, also, that it isn't a "backup" mechanism but, rather, a "restore" mechanism. I.e., once the image is created, it's only ever *restored* (if you want to update the image, it's an expensive operation)

Sure! You wouldn't, for example, use it on a "live server". But, that's the whole point -- you would image the media BEFORE the server ever went live!

All you need to do is ensure that whatever you write is very compressible. Much more so than "unconstrained DEADBEEF". E.g., you could tailor your compressor to recognize the 512 byte sequence: "123234u349tuepdfjg;skjdgpa9sufwrtd....sdklfsopriujh" and replace it with a one byte "sector is empty" code (where "empty" really means "contains the aforementioned 512 byte sequence")

The problem is creating "partition_sda1_image.tgz" *without* being concerned with the underlying filesystem. So, you have no knowledge (from the filesystem layer) of the "valid" contents of the volume (vs. blank/deleted content).

Then, restore it *without* the rescue disk.

You could use a program that already has this knowledge, like Partimage

or (I'm pretty sure) Acronis True Image. I know that Partimage knows tricks like splitting the image into files of a given size (so it fits on removable media), embedding comments into the image files, keeping checksums on the image files, etc, and I would assume that Acronis knows many of those same tricks.

OK, neither of these will help you back up that Amiga FFS partition while the disk is plugged into your System/360, but for 95% of the situations that people actually have in 2014, they work pretty well. :)

Sometimes a fast way to do this is to create (ahead of time) a file of a given size full of nulls, then compress it with gzip. On your target, unzip that file multiple times, renaming the resulting file, until the disk is as full as you want it. gzip is available for lots of platforms, and its file-writing code seems to be fairly well optimized.

In other words, you do something like this ahead of time

dd if=/dev/zero of=1g.bin bs=1048576 count=1024 gzip -9 1g.bin

and bring a copy of gzip and the 1g.bin.gz file with you to the target system. Then, on the target system, you do something like

gzip -d 1g.bin.gz; mv 1g.bin 1g.bin.01 gzip -d 1g.bin.gz; mv 1g.bin 1g.bin.02 gzip -d 1g.bin.gz; mv 1g.bin 1g.bin.03

until you fill up the disk. (You could say something like "zcat 1g.bin.gz > 1g.bin.01" instead, but then you're using the shell's file handling, not gzip's.)

To get the target disk all the way full, you could make a selection of compressed files of various sizes, and keep expanding the biggest one that will fit. You could have just one compressed file, and "manually" fill up the remaining part of the disk with dd or equal. Or you could have just one compressed file, and decide that filling up the last fraction of 1 GB or 10 GB or whatever isn't that important.

Some other thoughts...

I have found that sometimes NTFS doesn't really like to be full, or close to full (like, say, over 90 or 95%). It doesn't corrupt itself, but it gets realllly slooooow when doing anything. I have seen this on NT4, 2000, and XP; I haven't tried it on anything later.

If the disk you are trying to image is an SSD, you may wish to avoid writing more files to it than necessary. OK, one set of extra writes (for the null files) probably won't kill it, and it will probably be a couple of more years before SSDs start showing up in donated computers, but it might be worth thinking about.

Matt Roberds

On a sunny day (Sun, 02 Nov 2014 12:06:10 -0700) it happened Don Y wrote in :

Sure you can dd that partition, but now you really are in trouble. Its safer to tar a filesystem (that should NOT be currently running, else you are in trouble too), you can always untar it into an other filesystem (ext2, ext4, reiserfs, etc) that is compatible. The other thing I have noticed is that when copying partitions of disk images to bluray with my LG writer, some bytes at the start seem to get changed, could be an error, but this does not seem to happen when copying to a filesystem. The other thing is in case restore of an image to a similar device, I found that for example 8 GB card 1 (as source) has a different _real_ size than 8 GB card 2 (same make, same type, same specified size, bought at the same time) this could be related to bad sector managing of FLASH. If the new card (or disk) that you copy to, is smaller, then you are in trouble too. I _always_ run a full byte by byte compare after burning a bluray... wrote a program for that dvdimagecmp. I use it to check FLASH images too. All things of importance these days go to bluray kept in a dark metal case. It has close to a thousand disks in it.

Na, never work on a running filesystem, UNLESS its just a file or you know _exactly_ what is going on.

Yes, my Clonezilla approach to date builds on partimage/partclone. But, you shouldn't *need* to know anything about the filesystem as that just effects "efficiency" (compressibility?)

I figured a little piece of code that open|creat a file, pushes a particular byte stream to that file until an error is signaled, then closes it and tries for *another* file (in case the fwrite error was caused by max file size) until it dies. Then, unlinks all. Report the total number of bytes written (so the "user" can get a feel for whether or not the total "seems about right").

Don't rely on much of anything from the "OS"!

Goal isn't to ensure every unused sector is "cleared" but, rather, that

*most* are! E.g., you can't be sure the OS won't be dicking with the medium while you're trying to "fill" it. So, you could get unlucky and find that the OS unlinks a HUGE file (or even a subdirectory full of big files!) just after your code decided the disk was "full". Now, all that unlinked space has escaped your "filling". :-/

I would only have to do this once on each machine. If I am going to install that image on many machines, then the pain is spread over those multiple instances. E.g., if I spend a day "building" and imaging *a* machine but can replicate that to 10 or 20 others in

*another* day, then it's a huge win!

I have two goals. The first is to be able to create machines that users can "maintain" (amusing choice of words given that the only hook I am giving them is to "start over"! :> ) without requiring assistance from others (e.g., me, in particular). These will tend to be PC's almost exclusively (laptop/desktop). But, no idea what MS will throw at them in the future (hopefully, dealing with bare iron keeps MS's efforts out of my hair -- even if I have to resort to pulling drives!)

[I will probably PXE boot each machine and let them pull down the disk image from which to initialize their disk media. For machines that can't PXE boot, make a generic CD that does the same thing. This should allow me to process multiple machines concurrently (goal is to be able to do a dozen machines a week without increasing the pro-bono time that I'm spending... ~10hrs/wk)]

The other goal is to address the machines that I build here for my own use. Hysterically, that has been done by installing OS, imaging disk to secondary storage as "OS_Installed". Installing /opt and /usr/local stuff and then imaging as "Localization". Then, installing whatever apps and imaging as "Applications". etc.

But, this is tedious to do and tedious to restore: have to boot a separate utility, attach secondary store with the desired image(s), etc.

I got spoiled by watching how convenient SteadyState was! Then, my subsequent hack to allow *explicit* COMPLETE restores from a hacked "factory restore" partition. More recently, my latest "home-grown" restore mechanism (which favors PC's).

It's just *really* cool to be able to get VM style "reinitialization" of a system in a matter of minutes -- without the hassles of having a VM!

On Sun, 02 Nov 2014 10:42:55 -0700, Don Y Gave us:

I think you have the wrong "idear" about how they work.

Compression of ANY file has NOTHING to do with the state the disk file tables or previous written sectors are in.

Absolutely NOTHING.

Now, IF you are compressing a volume, you should use a method that does not fuss with empty sectors. It defines their location and stores THAT info, NOT "empty but dirty" sector data.

You lie, and you likely do not even know why.

By imaging the volume OFF-LINE (i.e., when the OS is NOT in control of the hardware), you ensure that the filesystem's state is self-consistent (assuming you shut the OS down cleanly, etc.). This neatly avoids the issues of "locked files" and privilege that might otherwise interfere with your imaging of the *entire* filesystem (e.g., even those parts not directly accessible to "applications" -- regardless of privilege)

But, this comes at the cost of not knowing which parts (sectors) of the medium actually have "content" that must be preserved! So, you have to include every sector in your image!

[we'll ignore any chicanery that tries to hide information in unused sectors]

If, just prior to creating the image, an application has constructed a large file full of completely random (i.e., noncompressible) data AND THEN DELETED THAT FILE, the sectors previously occupied by that file are now full of uncompressible data. EVEN THOUGH THE "SYSTEM" WILL NEVER *LOOK* AT THAT "data" AGAIN (because it has been "deleted").

[Of course, if the filesystem/hardware automatically scrubs deleted sectors, then the issue changes -- not necessarily better OR worse!]

So, your "image" will contain a chunk of data virtually identical in size (and content!) to the "deleted data" present on these dirty parts of the medium (because you MUST preserve it as you don't know if it is "live data" or not without knowledge of the filesystem's structure).

If, instead, you deliberately create a huge file (one that effectively consumes ALL of the unused space on the medium) and fill that file with VERY compressible data (YOU define what is maximally compressible by knowing how YOUR compressor will operate!) and *then* delete the file, you still have the problem of having to image "the sectors previously occupied by that file" BUT, now those sectors contain highly compressible data! (assuming the system doesn't get around to overwriting them with cruft between the time you delete them and the time the system comes to a halt).

Experience teaches you to always leave a bit of the medium "unused" (i.e., not present in ANY partition) to accommodate "small" variations between drives. Many megabytes on a 100G drive is "noise". Likewise, a GB on a TB drive is similarly "noise".

I've found this easier than worrying about resizing partitions (which typically requires the partition to have been "defragmented" beforehand to ensure there is nothing present at the tail end of the partition).

That is exactly the point: you're running on bare metal knowing exactly what that metal GUARANTEES to the bootstrap loader. No need to worry that some flakey driver is going to have to be accommodated, etc. No need for ANY "OS" at all! Every opcode that executes is one that you wrote (aside from the "well behaved" BSP/BIOS/OFW/etc.

You obviously know at least a little about Clonezilla - what do you find to be the problem with it? If you don't want to make a direct copy of the entire disk image, then you need to know the filesystem (or autodetect it) - that's the only way to be able to tell the important parts from the unimportant parts. And once you have that information, surely partimage/partclone will do the job?

With an unprotected system like MSDOS booted on a floppy or a flash disk, a disk editor can copy the sectors on one partition to another. Simtel\msdos has those editors, I believe, but searching for a simtel site is required. The simplest procedure is to format the first half of a disk and use the other half for the backup image.

Hul

Try detecting the filesystem (as a hack, do a mount from Linux without specifying the filesystem, and see what system was mounted - it will autodetect most types).

This has many disadvantages. On many systems, you'll create a sparse file, taking no space on the disk. On some SSD's (ones that don't have compression or other zero detection), you will cripple the disk's performance. And on disks where you actually end up writing zeros to the disk, it will take forever.

How about replacing the 1 TB harddisk with an 80 MB SSD? Then you enforce the rule that only a small amount is on the disk locally, you can use a simple dd with compression, and everything works faster.

To avoid thread drift (and the pissing and moaning that comes with it), this doesn't address the question I asked... (i.e., remove the FS-specific code from partclone and then tell me how to make it work!)

How does that give me 1T of storage?

What I currently do is similar -- except no DOS, etc. (just write a boot loader that effectively does the decompress & copy without the overhead of a "real OS").

Not using compression is highly wasteful of disk space (for the "restore image"). If the image is to co-reside on the medium with the live data, then it'd be nice not to have to "throw away" half the medium for this "feature"

E.g., the laptops that I build for students tend to have ~160G drives that I can cut into a "system" partition (which I want to be able to restore on-demand) as well as a "data" partition (which I will leave to the student to maintain... if their data gets clobbered, that's THEIR problem; at least the machine will still be runnable after recovery)

That's not the issue. The problem statement is:

- provide a means of imaging a volume WITHOUT KNOWLEDGE OF THE FILESYSTEM(S) THAT IT MIGHT (or might NOT!) contain

- do so with a goal of minimizing the image preserved

Only if you are writing /dev/zero to the medium. E.g., an OS can't decide that

*arbitrary* (yet highly repeatable/compressible!) data is "blank" without SOME way of recording what that data *was*. So, if you want to assume zeroes can be ignored, then you *must* record "fives" when they are presented to you (Or, "supercalifragilisticexpialidocious", etc.)

There are tradeoffs with every design decision. I can tar(1) to a PPT device. It wouldn't be *practical* to do this for a MB file! (but, that shouldn't preclude me doing it for a smaller file -- where *I* define my own idea of "smaller"... perhaps even hundreds of KB!)

The point is to leave the policy decision to the user, not the person who codes it ("Oh, this would be bad for XYZ so I won't allow it to work with XYZ")