SD Card FAT support issues (dosfs, fatfs,fatlib)

There is a group that is contemplating adding LFNs to dosfs.

Observe that there are two known bugs in the current version of dosfs:

1. Error crossing cluster boundaries on FAT32 volumes.
2. Cannot open file in root directory. (All the applications where I use the code work inside a subdirectory to avoid root entry limitations on FAT12/FAT16 - that's how this bug escaped).

I have patches for both of these bugs but haven't finished testing them yet.

I don't know about their decisions, but this was an *explicitly* stated design decision in dosfs. A "working directory" is an operating system concept, because it belongs to a process. Observe that dosfs was designed to be friendly to a multitasking operating system; it uses no global variables, and a consequence of this is that the caller needs to keep track of a "working directory", if such a concept exists.

names.

code

Timescales? Links? I've done the proof of concept stage for the device and can talk SPI to the Card. So I'm looking to move on quickly.

I was using a FAT16 image 512 MB, 32 sectors/cluster. So I did think I was bit by this.

I don't think this was my issue either because the file was in a subdirectory (MYDIR1) but there may have been files in the root at the time. I will redo the tests with a clean file system.

OK, great.

Nor

Yes, I understand this. I think in a multitasking enviroment there is more to be done than this though. A certain amount of synchrization is required. What if two processes each create a file in the same directory? They need to avoid private copies of directory sectors, fat sectors etc. Not that this is a problem for me because I am single threaded. The reason I used a working directory was because it was easy to implement if the FS code but worse in the application. My (limited) understanding of FAT is that a current working directory would amount to a cluster number for the start of the directory data and so could be used to avoid scanning the directory tree every time. If thats so then a cluster number could be used as a directory handle passed in addition to the file path. In that case DFS_OpenFile(...) would take an addition DIRHANDLE parameter (say), plus a new function to obtain such a handle.

Thanks very much for your prompt help.

Peter

... snip ...

Nor will they. There are patent problems involved.

-- "I'm the commander--see, I don't have to explain -- I don't need to explain why I say things. That's the interesting thing about being the President. Maybe somebody needs to explain to me why they say something, but I don't feel like I owe anybody an explanation." - George W. Bush, 2002-11-19

I've no idea of timescales. The project is an open-source bootloader, the name eludes me at the moment, but I will look it up when I get home. I've had a few technical/bureaucratic email exchanges with two people involved with the project.

I will not be offering LFN support in dosfs - I made it clear to this other group that while they're welcome to do whatever they like with my code, they will be forking permanently away from my code and I won't help them maintain it, nor will I roll their changes into the dosfs I am offering to the public.

LFNs are not going to become part of dosfs for the following reasons:

1. Not part of the design goals. dosfs is designed for minimal embedded systems.
2. _AFAIK_ (and I could be wrong) dosfs is currently patent-pure. Adding LFN support definitely infringes on a MS patent related to storing multiple filenames in a single filesystem.
3. Not required for any application where I use it. dosfs is used as a drop-in black box in several paid projects I still maintain. I'm not making API changes that aren't necessary for those applications.

Hmm. There are no known bugs with FAT16 except the "can't open root file" bug. If you can zip up your image file and send it to me, I might be able to interpret what is happening.

Sure, there's additional work required in the OS layer. I'm just saying that dosfs can't enforce working directories because it doesn't know who owns it - anyone who calls dosfs is expected to provide, with every single API call, a pointer to user-supplied data structure(s) that define(s) _all_ information required to execute the API function.

Hmm. That's an interesting idea. Could also be implemented in the FS layer - or just above it - using a MRU list of directories vs. start cluster.

I understand that there are patent problems although it I thought that this was still an open question. If this is now resolved in MS's favour then I won't go down that route.

So, what is the current status?

Peter

and

:-( I'll take that as too late for my project.

Fair enough. No need to entangle your project with patent disputes. If this patent is confirmed then I won't use LFN either. I'll have to find some other way such as force long names to short but not include the long name records.

was

that should be didn't.

I will make sure that I can reproduct the problem from clean, and if there is still an issue I'll take you up on your offer to look.

required.

need to

OK. So the problem is you don't know other processes' working directories only this one's (if the working dir was in the data structure). I think that problem is at a similar level to the problem of multiple processes modifying the same sector. Global info is needed to co-ordinate modifications, such as a locked sector table (or cache).

working

directory

A directory start clucter cache that would help even without a working directory. I do something like this in my Flash file system. In my case, because I am single threaded I can do stuff like flush the cache on directory modifications

Peter

I'm not aware that there has ever been any litigation about this. I'm not even sure if MS actively enforce it. However it isn't a gray area: the way LFNs are implemented on FAT volumes is patented - and hence I'm not going to implement them (never mind that I have no use for them either).

Have you noticed that almost no embedded devices that support FAT support LFNs?

Support for FAT filesystems per se is supposedly covered by some MS patents also, but the area is much murkier here, and I certainly feel it's an acceptable risk to use FAT as a filesystem in a fielded appliance.

The patents are clearly not rigorously enforced, but it's simply a headache I don't want to deal with. If I had a compelling reason to use LFNs, I might reconsider that - until such a time, I won't get my hands dirty on the issue. (Supporting LFNs makes life very difficult in constrained systems. Filenames no longer have a fixed length, and paths can become obscenely long. Some extension-parsing code - not mine - may break because the filename can legitimately contain periods other than that separating the extension from the filename. The extension might not be three letters long, and tables for identifying extensions have to be enlarged. Etc, etc.).

of

R0.02a

like

data,

would

two

can

names.

discarded

add

Nor

You may also want to take a look at EFSL, which is an open source file system library that has a port for ARM7. It can be found at:

Mike

Back in February, I had sent you some patches; I don't think you may have received them ;) They fix a few bugs and add mixed-case filename support to dosfs. I have put the diffs (sorry, they aren't unified) and a text file at:

The patched version should work on any svr4 compliant *nix o/s.

Regards,

Michael Grigoni Cybertheque Museum

I should have noted in my previous post that my patches address the root directory problem and also add FAT16 < 16MB support.

Regards,

Michael Grigoni Cybertheque Museum

Nitpick: My code is compliant to the letter with Microsoft's instructions on how to select a filesystem (based on the total number of clusters in a volume). If you changed that piece of code, you just "fixed" something that was broken by design and entirely spec-compliant :)

The page on MSDN where I read the details said something like "people use all kinds of incorrect methods and slightly wrong numbers when identifying the filesystem on a volume... here's the gospel:".

If you just need a SD card FAT storage module, look at the ROGUE Robotics uMMC. As it has no source of RTC, the data/time tags on the files aren't filled in. It uses a 5V TTL/CMOS level serial interface. Up to 4 files may be open at a time.

Don

may

Thanks. In fact I already have working hardware so I'm don't want to go beyond that. Having said that, the expansion interface that I am using does have a serial port available (at 3.3V).

Peter

Thanks, I will look into this.

Peter

I have just recently got DOSFS working in a media player project I'm doing with an AT91SAM7S ARM chip under WinARM, GCC 4.1.0. I had looked at efsl and fatlib as well as some other libraries but DOSFS seemed the least complicated to me and I didn't need the couple of things it doesn't have (yet) like the ability to create subdirectories.

FYI: I had to add the GCC __attribute__((packed)) to all of the FAT structures *as well as* the union. Happily there are no alignment issues. The biggest challenge after that was correctly getting the FAT16 disk image onto my media (raw NAND flash via my own sector-to-flash management layer). After that it all worked fine.

I guess if you're using an SD card it will already be formatted, or easily can be.

GCC spits out a couple of warnings: one that isn't valid; the other simply an unused variable (de in DFS_UnlinkFile). (I also had warnings about different signedness of the strings (char vs. uint8_t) in places but today they are gone. Not sure why?)

Anyway, many thanks to Lewin for making this code available and under such a generous license.

-Dennis.

Oooh. __Good__ catch, I completely forgot to document this (I guess you are running with an ARM?). There's a historical reason why it isn't in there, but I won't bore you with it.

It's fun to work with... :)

With SD and MMC you don't need to perform the low-level NAND management, fortunately.

De nada.

Yep. Like I mentioned, my target is an AT91SAM7S ARM chip under WinARM, GCC 4.1.0. I think the OP is also targeting ARM which is why I thought I should share this. It would probably be good to include this caveat in your readme.txt file (which is otherwise great). On the other hand, most ARM developers will probably pick this up immediately. I'm sure I will will next time 8)

A good way to avoid this sort of problem is to add padding explicitly, so that all your data is correctly aligned, and make sure it always uses the natural alignment for the data. That way it should be correct for (almost) all architectures. I compile with the -Wpadded flag on, so that if the compiler has added alignment padding, I see it straight away.

Unfortunately creating subdirectories was the first thing I needed to do :-(

Indeed.

I second that.

For now I have gone with fatfs because it had the nearest set of features to my requirements and some API docs. It isn't perfect but I'm feeling confident. I have one signed/unsigned warning and I had to set a macro for misaligned access (using ARM7). It took about an hour to add working directory support. I have this running on my target now. I managed to copy files at about 70Kbyte/s using a ~2 Mbit/s SPI link using a 51 MHz ARM7 running out of slow flash over a 16 bit bus (thumb) and zero optimization effort. I'm pretty happy with that.

In terms of behaviour the only problem so far is that . and .. don't work. I made some changes that sort of fixed that for FAT16/12 but I need to look into FAT32. The sort-of is that . doesn't work for the root directory. The FAT32 problem is it doesn't work when .. is the root (to be honest, I expected 16&12 to be a problem there and 32 to just work).

What is a problem is that the code contains very little explanation except for the extern API.

Peter