NAND flash misery

Vladimir, if flash were a viable and reliable replacement for HDDs this would have happened for years by now, the costs would have gone down. They are not, and given their limited number of write cycles they are bound to stay out of the way of normal disks (which have achieved an amazing level of performance). Thanks for posting the details you measured, I'll know now to not buy a 2G SD-card for my camera (not that I need any more than the 1G I have now, I have never used > 1/3 its capacity anyway :-) ).

Didi

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

Original message:

I thought the IDE interface was supposed to hide that from the host by mapping in spares?

Isn't it also supposed to do wear leveling behind your back?

Flash is at the moment becoming a viable replacement for many applications, both high end and low end - witness flash disks turning up in everything from servers, to high end ultraportable notebooks, to low end ones like the EeePC and even cheaper competitors.

This wasn't reasonable until the most recent generations of devices started beating the performance and price point of the 1.8" mechanical drives.

I don't necessarily think flash will be a replacement for large, cheap mechanical drives, but for applications that only need a few GB, or for applications where space and weight count and 32-64 GB is enough, it's already gaining market penetration.

Write cycles could be an issue, but many of the ultraportable gadgetry applications will see system replacement before that happens. And the replacement will probably be 2 - 4 x the GB/$ of the original.

It clearly does that, but only to a limited amount. Also, when a file gets broken because of the bad block, it is too late to replace it.

There is a whole lot of things that the internal controller could probably do; but we can only guess about what it actually does.

Vladimir Vassilevsky DSP and Mixed Signal Design Consultant

Just because the device is manufactured with bad blocks, does not

*automatically* mean that more will follow. (At least you have not demonstrated that).

It's not like a load of bad blocks appearing on a magnetic drive, where you know the whole drive is probably on the way out.

It's the same for conventional disks isn't it?

--

John Devereux

It also explains why the 8GB Photo Hard Drive I bought for my Nikon is rock-solid and I've had many intermittent and non- reproduceable problems with larger FLASH cards.

FWIW, for an 4GB flash device, 2% would be 80MB.

You'll also notice that most (all?) flash manufactures follow the "1GB=3D1,000,000,000 bytes" rule common for disk drives. Leaves them plenty of space (6.8%, in fact) for manufacturing defects and wear leveling.

When the platter densities of (mechanical) HDD's went up at a certain point manufactures of HDD's had to resort to error correction schemes to obtain reliable operation. A modern HDD would be unusable without ECC. It appears that high density flash is going the same direction.

NAND flash always has defects in manufacturing - the devices are designed to cope with a certain level of faults to make manufacturing cheaper (the same applies to many other types of chips, and hard disks). Each sector in NAND has extra space for error correction and detection (IIRC, 512 byte sectors are actually 528 bytes in size). Bad blocks can be detected and marked during manufacture and testing, and blocks that go bad (due to wearing out) are detected in use and the data moved to different blocks.

The same thing is done with hard disks - the controller detects bad blocks, and re-maps them. There are a few differences, however - on hard disks, you get bad blocks in manufacturing but it is rare that a good block goes bad in use. With flash, the controller can almost always spot a bad block and recover the data (since it's normally a single bit failure, the ECC will fix it), while on a hard disk you lose data. And on flash, a remapping makes no difference to performance - on a hard disk, it's equivalent to file fragmentation.

CF cards and other earlier flash devices are not that great at wear levelling and bad block handling (that's one of the reasons for flash-specific file systems like YAFS and JFFS2). Modern IDE, SATA and SAS flash drives are far better. Good manufacturers quote MTBF numbers that are orders of magnitude higher than for hard disks, and wear is no longer a practical issue for larger flash disks (I've seen flash disks spec'ed for *continuous* 20 MB/s writes for years). See also - a 4 GB Eee PC disk should be fine for a normal user for 25 years. And since wear is levelled across a disk, a 128 GB disk will survive 32 times as much use for the same time.

The utterly bad blocks are detected at manufacturing; however there is a bunch of the unreliable blocks which takes hours of testing to discover. If the bad block is detected when in use, it means that the data is lost already. It is too late to hide it by remaping.

The actual erase block size in NAND flash is something like 32/64/128/256KB, being bigger for the higher capacity devices. What it implies: any write operation through the IDE interface is actually read - copy - erase - modify - write at the controller level. The other surcumstance of that is the speed penalty for the writes misaligned to the erase block size. Since this kitchen is hidden behind IDE, there is no point in using YAFS or JFS or such. A disk cache with the blocks of 32k makes a lot of sense though.

Vladimir Vassilevsky DSP and Mixed Signal Consultant

The point of ECC - Error checking and *correcting* - is that slightly bad blocks do not lead to lost data. The most common problem in flash blocks (excluding any totally failed blocks found in manufacturing) are single-bit errors - bits that can't erase or program properly. These single-bit errors do not lead to data loss, and the flash controller can easily detect and correct them. It is even possible that the controller will continue using a block with bad bits, and will not disable the block until a certain number of bad bits have been found (I don't know what error rates are used here in practice).

YAFS and JFFS are designed for when you have direct access to the flash

- when you are using a controller that handles the wear levelling and block placement (such as for CF cards and IDE/SAS/SATA controllers), you should not use flash-specific file systems of that sort.

True, but HDDs don't wear out with writing - and flash does. This is a major advantage flash does not promise to catch up with - at least for the time being.

Didi

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

Le Fri, 27 Jun 2008 11:10:29 -0500, Vladimir Vassilevsky a écrit:

"... generally impossible to apply an error correction scheme ..." Are you really a serious guy or you comes here on subjects that you do not master?

I will try to make the kind of "Vlad response" : Is it a hobbyist project ? I think this is. mmmhh, Let see ... who cares about the safety of hobbyists projects ?

--
HBV

HDDs wear out through use. The lifetime is roughly dependant on the time the hard disk has been powered up, and how much the head is moved. It's thus fairly independent of the size. Flash wears out through erase-write cycles on blocks. So the more blocks you have to spread the wear, the longer the lifetime, and the more you read rather than write, the longer the lifetime. So as flash drives get bigger, they are surpassing HDDs for reliability and lifetime. For common desktop usage, a 32 GB flash disk will probably far outlast a typical hard disk - with

256 GB and bigger flash disks, even high quality hard disks are no longer competitive on reliability and lifetime for real applications.

The big issue is the cost per GB - hard disks are still much much cheaper. A second issue is speed - standard hard disks are significantly faster than standard flash disks. But that will change - flash speeds are easily scaled (just use several devices in parallel) once the price is right.

A HDD has one big problem: the lifetime tends to go to the end when it falls on the floor. Yes there are the special HDDs, accelerometers, suspensions and such; however it makes HDD much less attractive then a flash alternatives. Besides, a flash card is easy to swap.

But the small and light HDD is more likely to survive the contact with the floor :)

The flash write endurance (as stated by the manufacturers) is at the order of millions of cycles. This is more then enough for many applications; there is really no point to bother about the wear leveling. The problem I am facing is the infant mortality of the unreliable cells.

A common x300 8GB Lexar CF card from WallMart sustains the read/write speeds at the order of 25...30MB/s. Those are the numbers that I measured; the upper limit is because of the hardware, not the flash.

The falling on the floor, weight/size, power consumption, instant readyness and the heat dissipation are the problems of HDDs which flash doesn't have.

Vladimir Vassilevsky DSP and Mixed Signal Consultant

You blockheads should bow to the opportunity of receiving a lesson of wisdom.

Good comment, but where is meat? Would you suggest a solution which:

1) Small, low power, robust and swapable. 2) Sustain the read/write speeds of 5MB/s at the least 3) Compatible with standard flash card readers regardess of OS

Vladimir Vassilevsky DSP and Mixed Signal Design Consultant

Un bel giorno Vladimir Vassilevsky digitò:

If with "typical" you mean "the grade used for low-cost consumer electronics", you are right. But NAND normally come at least in two or three different grade options: no bad blocks, 2% initial bad blocks without dynamic bad blocks (until the minimum number of erase cycles), 2% bad block with dynamic bad blocks, etc.

I believed that solid-state drives would have taken over a lot easier, too. Instead the good old rotating junk continues to resist very well, except in very hot or very vibrational or very space-restrained applications.

--
emboliaschizoide.splinder.com

If you are planning on dropping your drive, flash certainly has the advantage!

Write endurance is not normally in the millions for NAND flash - perhaps you are thinking of NOR flash, which is inherently more reliable (but costs more per bit)? With wear levelling, write endurance is not a big issue for modern flash disks - without it, you would wear out parts of your disk (such as the FAT on FAT formatted drives).

Millions of cycles is what stated in the datasheets for the finished devices like flash cards or jump drives. We can only guess what methods are actually employed inside. Perhaps the wear leveling is taken care off. However the reliability appears to be surprisingly low, especially for the higher capacity devices.

Vladimir Vassilevsky DSP and Mixed Signal Design Consultant