Re: Have ext3 on SanDisk CF but can't disable write-back caching as kernel instructs

...

Is it important for the device not to self-destructed in the field?

...

CF are close to hard drives, but not identical.

...

You might not have a choice!

...

If the Sandisk controller does not handle the necessary command function, no other compact flash will.

It does not matter. You will destroy the CF after a few hundred thousand cycles, probably within days with a Journalized file system.

I'm not sure if *any* filesystem can help to make this reliable: AFAIK, the behavior of the disk device itself in case of a power loss (e.g. during a write operation) is undefined. It may well happen that the disk's sector formatting gets corrupted, rendering the disk unusable. IMHO, no filesystem software can compensate for that.

Rob

--
Robert Kaiser                     email: rkaiser AT sysgo DOT de
SYSGO AG                          http://www.elinos.com
Klein-Winternheim / Germany       http://www.sysgo.de

The sector (if it *really* is only a single one) gets *damaged* from the drive electronics' point of view. Surely you can find the garbled sector: the drive does this for you by returning an I/O error if you attempt to access it (usually after performing prolonged retries and recalibrates). The only way to cope with such a defect is to add the bad sector to the drive's defect list, so the drive will avoid it in the future and access a spare one instead. If the drive is capable of doing this transparently (I wouldn't bet on it), then a journalling filesystem may be able to recover, but only so many times (because sooner or later, the drive will run out of spare sectors).

The behavior of a disk drive in case of power loss during active write is completely unspecified in general. You may be able to find specific models designed to cope with such situations, but in general, there is no way to predict what will happen.

Btw, Compact Flashes appear to be particularly ill-suited for this situation: There have been several reports from IMO trustworthy people on this ML that a sudden power loss during write operation can render a CF *completely* unaccessible.

.. only if the *media* is 100% predictable ...

Assuming a reliable media for storage (e.g. Flash memory (*not* CompactFlash!) or DiskOnChip devices), JFFS2 is said to be up to the challenge.

Rob

--
Robert Kaiser                     email: rkaiser AT sysgo DOT de
SYSGO AG                          http://www.elinos.com
Klein-Winternheim / Germany       http://www.sysgo.de

You may find that the _whole_device_ does not respond any more to the normal command set. Bingo!

[Yes, I have seen such behaviour with my own eyes. Twice.]

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88   Web: www.denx.de
I am a computer. I am dumber than any human and smarter than any  ad-
ministrator.

It wasn't a duplicate, BTW. So apparently the "rejected post to moderated group doesn't make it to crossposted groups either" rule applies when posting via Google Groups too. In case anyone was curious...

No, it was auto-rejected because the linux.kernel newsgroup "is currently read only because of administrative reasons".

As to your comment about tech support, since my question is rather esoteric and involves the interaction of multiple areas of the kernel plus the SanDisk hardware, it's unlikely that contacting SuSE (the makers of the distribution we started with) would yield anything. We did try asking SanDisk some questions about the details of their wear-leveling system, and they were not helpful. Asking them Linux-specific questions would be even less likely to get a useful response.

I'm just hoping there's some collective wisdom out there I can tap in to...

Naturally we'll be doing that, but if we're heading down the wrong path entirely, I want to know now.

The person who put that warning in the kernel presumably knew what the risks were of not turning off write caching on a disk with a journalled filesystem -- I need to find out what those risks are.

I guess if no one here knows the answer I'll try either posting to the linux-kernel mailing list or directly emailing the guy who wrote the patch to output that "cache flushing failed. disable write back cacheing for journalled file systems" message.

-- Dan Harkless snipped-for-privacy@harkless.org

Yes, preventing the system from self-destructing is even more important than not corrupting our data. This is why our root filesystem is in an initrd ramdisk that lives in the /boot partition, which we don't have mounted when we're booted.

The hardware is pretty much fixed at this point (we've already gone through initial "qual-testing", for instance). We definitely won't be able to change the design to use a hard drive rather than a CF disk, and even if we were able to, I can't imagine a HD being more reliable in this application than a solid-state disk.

I wonder why they wouldn't allow disabling write caching. Naturally performance suffers with it turned off, but if it's required to turn it off to use a journalled filesystem, you'd think they'd support it...

Well, that's definitely not true. We've gone for much more than a matter of days and the CF has not been destroyed. We did some calculations (admittedly doing a lot of hand-waving, since, for instance, we can't get the details of SanDisk's wear-leveling) and it was our belief that the disks would far outlast the service lifetime of the unit...

-- Dan Harkless snipped-for-privacy@harkless.org

Yes, we have experienced this. Due to the inflexibility of SuSE's init.d startup/shutdown ordering scheme, they try to shut down PCMCIA prior to taking down PCMCIA network interfaces, which caused our boxes (until I kludged a fix) to hang at the "Shutting down PCMCIA" step.

One time I powered my system off during this hang (at which time I guess writes were still being done to Flash), and when I booted back up, the CF disk had I/O errors that were not fixed by re-making the partitions and filesystems.

Did some research and found a post saying that some CF disk reader peripherals have low-level format capability, but haven't looked further into that yet.

Luckily, our device has a large capacitor which will keep us running for up to a second if we lose AC power. We plan to have a device driver which will detect the power loss and lock out all other tasks besides the driver to prevent there from being any active writes to Flash as we lose capacitor power.

The hardware was already decided before I joined the company, so I'm not sure what the decision-making process was that settled on CompactFlash, but we do need to have read/write speeds not massively slower than hard disks, and relatively large capacity (250 MB). Would Flash memory or DiskOnChip fulfill those constraints?

-- Dan Harkless snipped-for-privacy@harkless.org

Several years ago i do some tests in with low level HD disk writes an NEVER got really damaged sectors, but some with checksum errors after power down. But if I overwrote them with new data, everything was ok. I can't beleive that sectors of a hard disk can

Some of this rumors I heard, too.

Regards,

--
Bernhard Roessmann
Don't Fear The Penguins!

Can you give me some information about the vendor and type of the CF you used?

Regards,

--
Bernhard Roessmann
Don't Fear The Penguins!

The first one was a SimpleTech 32 MB CF (don't remember exact type, this was 1 ... 1.5 years ago). When it died, I repeated the test with a no-name CF card which died within the first 100 cycles of write/power-off. I was not in the mood of ruining more cards at that time.

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88   Web: www.denx.de
In accord with UNIX philosophy, Perl gives you enough  rope  to  hang
yourself.              - L. Wall & R. L. Schwartz, _Programming Perl_

I will suggest you to do some research on CF to IDE. If you use 2.4 kernel, hack ide.c is must. Then, why not to put most of your system to cramfs and read-only and just keep a small piece of the CF to r/w. Even the power lost, your system will never lost after boot. I used to put my /etc, /var and /tmp on ram disk when system-init and s-link to cramfs system. Here is what I did:

1. keep a /etc somewhere cramfs and copy it to ramdisk at the boot time.
2. cramfs /etc s-link to ramdisk /etc mount point.
3. Then, copy your extra setting from r/w(if exist) to overwrite /etc setting. If not exist, well, you still have the default one. JFFS or JFFS2 can be done on CF but you will need mtd-cf module to simulate MTD device(never try).

pojen

AFAIK, hdparm only works with IDE drives. All the USB drives I've seen show up as SCSI devices.

--

-John (JohnThompson@new.rr.com)

A harddisk has no limitation on the count of rewrites of any sector. The CF cards has (though its complicated and well hidden). So this is not a case of reliability but of specification. If you want to use a CF card for some kind of application that permanently rewrites some data you will quickly destroy it. There is a file system for linux that is written exactly for the purpose of using Flash in the best possible way, but it only works correctly with naked flash chips and can't be updated for CF cards, as there is no specification how exactly the CF card handles the "spreading" of data, and supposedly it's different for different brands.

-Michael

The CF decides by itself how to sore the sector. Internally they use flash pages much bigger than sectors and do a complicated housekeeping involving a RAM at least as big as a flash page. So if you write a sector, other sectors already stored in the Flash will be moved. If power is turned off in the process, internal allocation tables can be damaged, destroying everything stored in the device. I heard that there are (or will be) CF cards that include a battery to prevent this, but I did not yet see any specs.

-Michael

I understand ;-)

I will try this with two Sandisk CF cards (one consumer, one industrial) in the next weeks. And if I don't forget it, I will post the results here :-)

Regards,

--
Bernhard Roessmann
Don't Fear The Penguins!

Our understanding is that the SanDisk CF's built-in wear-leveling will prevent the disk from wearing out for many, many years. Do you know this not to be the case?

Yes, that filesystem is JFFS2, and was discussed elsewhere in the thread.

-- Dan Harkless snipped-for-privacy@harkless.org

Research on what aspect of it?

Yes, we're using the 2.4.20 kernel that SuSE 8.2 comes with, tweaked to not build modules we don't need.

Hack it in what way?

As I mentioned previously, we're running the root filesystem out of an initrd ramdisk. It lives on the /boot partition, which we don't mount, so there's no danger of corrupting it.

The concern is corrupting our read-write data partition (the whole point of our device is to exchange a bunch of data).

-- Dan Harkless snipped-for-privacy@harkless.org

Our CompactFlash disk is attached to the IDE bus, not USB.

-- Dan Harkless snipped-for-privacy@harkless.org

Of course it tries to do so.

I don't know the Method used by Sandisk, but from another brand I learned that they do the built-in wear-leveling in a way that they measure the time used to program the words in a page and if that time gets too long they use one of the spare pages. (BTW.: JFFS (on a "naked" flash chip) uses all pages in a cyclic way.)

In any case changing a single byte forces a complete Flash sector (some

512 K Byte, depending on the internals of the device) to be rewritten. So you can roughly calculate the count of possible write requests when you know the page size the count of spare pages and the number of allowed (re-)write accesses to a page.

Using a journalling file system will make things much worse as each write will result in additional updates of the Journal.

-Michael