Shutdown behavior and flash storage devices

If it's a USB stick, SCSI has an UNLOAD command which is what you do when you eject removable media like a CD. I would hope that a USB stick (USB mass storage = SCSI protocol) would use it as a hint to flush its write cache and get ready to be removed, but I don't actually know if it does.

Maybe they can also detect the USB signalling going away, which is what happens when you pull the stick, a few ms before the power disconnects, due to the layout of the USB A connector.

Enterprise SSDs have power loss protection so you can pull the power at any time and they'll tidy up nicely using stored energy in a bank of capacitors.

Theo

Two (transient) copies of each "chunk" of data. Update the second copy (on or before impending power fail). When complete, toggle a pointer to reference it as the most recent (assuming you don't have metadata that automatically provides that sort of information).

The "first" copy can now be removed and its space reallocated to some other "second copy"

Then, move on to the next "chunk". Design your algorithms so that you can tolerate some chunks being "stale" (not updated in time) wrt the others.

Don, this strategy is obviously good (standard?) for say disk writes etc., but I don't know if it will be OK for flash as it involves erase then write, AFAIK "erase" is the most killing part. Perhaps they do a lot of "refresh", i.e. read while they can then rewrite the same locations? I don't really know, just wondering how they do it.

Don, I don't think that works. The entire contents of the volume can be corrupted if power is removed during wear-leveling operations that are invisible to the host OS. And I guess you are assuming no file system is in use (because file system internals would still get corrupted)?

It is not necessary to do this. Here's a very simple way to recover with no data loss ever. It is simply "log-only" or "journal-only" storage.

The device keeps 10% capacity reserved (pre-erased) (or less, this is a performance number). The driver has enough RAM to track where every logical block is (this isn't much RAM). And then writing is always done as a logfile--write whatever the user data is starting at block 0, and move up. After writing user data, write a log entry (this can be just right after the user data). Writes never go to the block the user indicates, they always go only to the log pointer. Reading looks up where that block's latest copy is, and reads that.

When you're at less than 10% space left (near the end of the device for first-pass writes), you simply read from +10% from the current log write position, and write it to the log (compacting it to avoid re-writing any blocks which are now obsolete) until you free up space. Since many blocks are rewritten, when we compact them we free space. Once copied, then erase the blocks which were compacted. Then just keep growing the "log" through the newly erased blocks.

Upon power up, scan entire device for the log info, and rebuild the RAM index.

Power can be removed at any time, and fully recovered, although obviously the most recent data may be lost, but we can roll back to the last consistent state just like journaled file systems (the log entries need a checksum so we can validate them). Wear leveling is achieved by design.

There are lots of small ways to improve the above, so do that.

I have no idea what actual devices do, I suspect it's something much more complex and less robust.

Kent

If you have no knowledge of how the flash medium is written - which would normally be the case with USB sticks or SD cards etc. - the most efficient way to mitigate data loss probability is to do as much write cacheing as you can afford at system level. Of course again you will be prone to data loss on unexpected power loss but updating the write cache will be done at (much) longer intervals and in relatively brief bursts, thus the probability to be hit gets lower. The rest can only be got rid of by having large enough caps on the power supply and an early warning that power has been lost (usually at the input of the stepdowns and you are running on fumes.

Of course you can employ various strategies to mimick flash drive behaviour at a higher level, Don suggested one, Kent also did, but these mean having different drivers for different media and all the consequences of that. I prefer to talk to standard SCSI or ATA behaving things and leave it to them to handle their unique specifics. [I remember how I *had to* introduce write cacheing at system level, in the mid 90-s I started to do backups under DPS to magneto-optical devices; early days DPS would simply update its CAT (Cluster Allocation Table) every time it got modifier and these sectors got corrupted before the backup would finish.... :-) ].

Dimiter

====================================================== Dimiter Popoff, TGI

======================================================

You're missing the point/role of the internal controller in the mix.

Imagine 8 blocks of memory in the device. Blocks 1, 2 and 3 (no special significance) contain highly static code/data. They get written *once* when the device is manufactured. So, each has seen *1* program/erase/write cycle.

Meanwhile, blocks 4-8 are constantly hammered on. The code is constantly updating the "files" that occupy those blocks (the files don't TOUCH the first 3 blocks, by definition).

After 1,000 updates, blocks 4-8 have seen 1,000 program/erase/write cycles.

The underlying FLASH device will have a MAXCYCLES specified, based on the technology used (SLC, MLC, TLC, QLC, etc.), process geometry, etc. For consumer devices, this number tends to be lower -- because consumers tend to want to purchase "big" instead of "durable" (so, MLC/TLC/etc. technology).

Assume the MAXCYCLES is 1,002 (actual numbers are unimportant -- if Dave feels he will be taxing the medium over the life of his design).

So, after two more updates, the device is *broken*. Because blocks 4-8 will each have hit their 1,002 cycle limit and stopped working (yeah, I know it's not a brick wall; but there is *some* number at which ECC errors prove unmanageable and the controller marks those blocks (4-8) as bad.

Meanwhile, 1,2 and 3 each have 1,001 cycles of wear available -- that they aren't (and WON'T!) be using. Had those 3,003 cycles been "shared" among all of the blocks, then the files being stored in 4-8 would still be writable (even though the next write might be into blocks 1, 3, 5, 6 and 8).

So, the controller has to deliberately *move* a copy of the data in blocks

1, 2 and/or 3 into 4-8 -- consuming one cycle of 4-8. But, freeing up 1001 cycles in 1, 2, 3!

You, on the outside, can't tell how it is doing this, when it is doing this or even *if* it is doing this! So, you can't predict what parts of the flash will be corrupted as power goes away. Maybe the VTOC gets hosed (so you can't even FIND the files). Or, some bookkeeping metadata used by the controller.

The logging idea works for magnetic media where what's there, stays there, and all you have to worry about is whether or not the *new* stuff made it in "under the wire", or not.

As an analogy, next time your RAID array is rebuilding, cycle power. See how easily it sorts out WHERE it was, in the process (and think about the resources that it spent to make that possible -- all in a $5 thumb drive??)

One would assume you'd adopt such a strategy regardless -- as the durability of FLASH is limited. (write a piece of code that just keeps rewriting the contents of FLASH and see how long before you get write/verify errors)

The usual trick is to just have a mirror of the FLASH contents and flush that to the media when you're pretty sure the ship is sinking.

But, if this is *large*, then you risk only partial writes before power drops to an unusable/unreliable level.

I'm not sure that will work with consumer devices. You have no way of knowing how long it will take to reliably flush ALL of the data out to the physical store.

Remember, if you have a USB interface, then the interface limits the write rate as well as the media in the device.

Exactly. I use "bare" devices so I can see how they respond -- because the datasheet gives me hard limits.

So, I have to worry about a write *failing* (in which case, it's as if I lost power prematurely) but not *other* "unchanged" data being corrupted in the process (neglecting read/write disturb events).

I think much of that is inconsistent with Dave's idea of "consumer" (which I read as "inexpensive")

Well I think tens of gigabytes for small drives nowadays, what can you do. Cacheing it all just is not feasible. What DPS does is cache directories and CATs on a 4k piece basis (regardless of the disk block size, 512 2048, 4k) and maintains a bitmap for its dirty regions - so when it updates to the medium it writes at least 4k and no more than twice 4095 excessive bytes or something like that.

Oh with consumer devices you have not built yourself it will probably never work, of course. They are unlikely to give you an early warning in the first place. If you want to be safer you just use good rotating disks.... I think I read somewhere they do their soft shutdown using the energy in the rotor (and do that since times immemorial), but I have never verified this.

Dimiter

====================================================== Dimiter Popoff, TGI

======================================================

A number of upper tier memory device manufacturers sell flash drives, USB sticks, SS Sata drives and M2 drives with "UPS" capabilities. When they sense a loss of power they will have just enough onboard juice to power down in a controller manner.

I have used drives from InnoDisk (

) on Linux boxes inside machines with hard on/off switches. With regular drives we'd see boot problems after 20 to 30 power cycles, with the "UPS" protected ones we could do hundreds of hard cycles without ill effect.

Antoon

What's the (typical) price premium for that "feature"?