Re: Flash File System for Dataflash

Hi Stefan

Before this get totally out of hand, let me just give you a quick rundown.

The basic Idea is not to tie a File Allocation table to a specific aria where it will wear out the flash. Rather to consider the file object as a piece of memory like you would in a linked list. This means you just have to find the first (indexed) peace and then you can access all the files header info?s sequentially. But let?s stick the name FAT for clarity.

The System starts by initializing the Flash, determining how big the Flash Page sizes are and then setting the appropriate offset addresses for wear level info, File info and data.

Once the flash type is determined and initialized it will allocate (malloc) ram for a ?FAT? page, A ?FATBAK? page and the amount of open files you want, let?s say 4. Then it scans the flash from start page (configurable) to end page, looking for a fat page and a fatback page. If not found, it?s clearly not formatted so it will be formatted with callbacks for the systems that has output and input. If found (the pages are indexed), the start page can be trace from there without the need to continue stepping through the flash.

Now that we have a fat start page and a backup start page we can load that page into RAM, read out file headers happily and find the file start addresses with an offset in the media.

To open a file with name ?FILE?, the fat entries will be stepped through until such a file is found, the Object will be loaded with start page, file number etc. the handle (pointer to object) will be returned.

So with a 2MB flash that generally uses 512 to 528 byte pages you roughly need 3K for the 2 fats and the 4 files you may want open at the same time. The SPI code for hardware, the wear leveling code, sector disturb/refresh code, the file systems and a multitude common file routines for text file manipulation all fit in 32K block on an 8051.

I have stored bitmaps and text files of up to 500K ? ¼ of the specific flash?s size without effort on this very small system. It is not intended for the larger Linux or ARM environment where as you mentioned, there are many available systems for.

Now I maintain my original point, this may be useful for some and it certainly works on the 8 different applications I rolled it out on of which there is about 3000 units in the field. If someone has a need to store and retrieve data on flash memory, this may work for them and for that reason I would like to put it out there, I am sure yours is much better and I?m really not trying to compete with you or anyone for that matter. So if you are not in favor of such, I take note, I apologize, hell ill even stand corrected, but it may just be suitable for one of the millions of people who were not going to buy you product anyway.

Please ? this intended to help, not for financial gain of any sort and I?m sure it will not affect your business in any way.

Still, I thank you for your input but I take exception to terms like ?Mickey mouse stuff? let part while things are still reasonably civil.

--------------------------------------- This message was sent using the comp.arch.embedded web interface on

s 2

hahaha, DO NOT WANT !

ess

e 2

0.1% chance of failure, is that suitable for IEC61508 SIL -3 ?

seem

str=

th=

Try to keep in mind that it could be hardware failure!

--------------------------------------- This message was sent using the comp.arch.embedded web interface on

Hi,

r

Ok

fopen() and his brothers belong to the C standard I/O library () and not to the filesystem.

Filesystems export only: open (), close (), read (), write (), lseek (), et cetera

Filesystems compile and run perfectly without the standard library code.

s

ss

2 s n

Years ago, I extracted the FAT16 filesystem from Pat Villani's FreeDOS project and made it architecture independent. The only function architecture dependent were:

media_init() media_read () /* read a sector */ media_write () /* write a sector */

I also extracted the MINIX standard I/O library and compiled it over the FAT16 filesystem.

The code was debugged on Visual C 6.0. As a media, I used a usb disk plugged in the PC.

The code is in production on hundreds of embedded devices (M16C+ dataflash). All works nice. Never had problems.

It was a very important C exercise... That I won't ever repeat again in my life.

FAT filesystems are risky on embedded systems for the following reasons:

1) Unless you purchase a commercial product, the second copy of the FAT is never used to back up if the first fails 2) FAT16 don't do WEAR LEVELLING, that means that after 10.000/50.000 of write sessions your Dataflash may fail 3) Speaking of embedded systems: in many many cases, there's absolutely no need to create new files at run time. 4) Power fail failure during write sessions may corrupt FAT and files

Back then I was very excited running the FreeDOS filesystem and the MINIX stdio library on a 16 bit M16C microcontroller... Production volumes now are too big and I don't wanna risk...I save my data at fixed flash addresses, use 32 CRC, and keep 2 copies of the data.

Suggestions for you :-)

Dataflash Write

------------------

1) A write must be followed by a read, to check if you wrote the sector correctly 2) Poll the status register of the dataflash, and wait for the write to be completed. Don't assume the 3) Use the extra 8 bytes of each sector to store the CRC (16 bit is fine) 4) You may want to use ECC code correction. I remember correctly, 8 bytes may correct up to 8 bit on a buffer of 256. 5) Use wear levelling if the application is supposed to write frequently onto the Dataflash. 6) Don't go 20 MHz... Slow down to 4 or 8 MHz

Dataflash Read

------------------

1) Each time you read a sector onto the Dataflash, do it two times on 2 different buffers and then compare their content.

The best filesystem for embedded sytems is a journaling fileystem. Each time you make a write, you add a record to the log. In case of Power Failure, you lose only the very last write and the filesystem is safe.

Hope this'll help Enrico Migliore

Thank you for the input

The code is now available at

This little system changed a lot from its inception. Speed was of course an issue, especially since it originally rolled out on a standard 8051 12 clock cycle CPU with a 36 Meg crystal. This only runs at 3 MIPS so the data flash speed was not reached as I did not have a spi integrated in the 8051, all IOs clocked from code making effective speed very slow. The above code was ported onto the DK3300 dev kit from ST so its slightly better but still not fast and I didn?t bother to use spi port again.

What you mentioned is mostly done with the exception of a CRC check it just became to taxing on the system. I used the extra bytes but just to get a few more bytes in a page, it bit my bad when one production run came with the D type flash in binary address mode. This is an irreversible setting so I had to quickly build in a check for that setting, luckily the ram buffers are allocated dynamically after the page size is determined to handle a variety of data flash sizes (I mostly use 16Mbit/2Mbyte with 512/528 bytes per page).

The application that first ran the system had about 40 files, hardly ever bigger than 20K and exclusively text, I mixed the use of fixed location memory with normal fat like structure. A few setup type files never changed, and the rest constantly changed. In the traditional fat like scheme every time I wanted to remove data from the front of the file, I would open the file read out the spent data, write the remaining data to a new file, remove original and rename the new to the old. Slow, risky and a huge source of wear. It irritated me so much that I designed this system with the ability to just shift the file start page/position in the fat so the file can shrink from the front without rewriting all the content. I also tagged the pages with file number, prev page, next page and an index. This allows an integrity check on the chain of pages which I do at startup or at user request to look for any lost/unlinked pages.

In the demo above I write 100 files of 10K bytes (1Meg total used), then rewrite them, then append another 5K (1.5Meg total used) then clip, as I call it, 1k from the front (1.4K total used) then rename them all and lastly delete them all.

I?m comfortable with the current level of performance for my needs. However, the very rapidly changing data like usage meters I store raw in the flash for both speed and wear purposes.

In the error I experienced, it looked liked some other process messed up my FAT buffer. I was quite impressed with the recovery routine that picked up all the loose pages and reconstructed the FAT. The fat backup becomes the fat in this case and any loose pages are removed. But to test that properly I will have to step into that section of code and disrupt the fat update process to simulate a failure. I just haven?t felt compelled to do that yet. Hopefully I get some energetic contributors that feel like digging into that part of the code or maybe simulate the flash with another small micro or even a PC, then we can drop chars and pages as we please to see how robust the code is.

thanks again for the input.

--------------------------------------- This message was sent using the comp.arch.embedded web interface on

ver

e

=92s

ress

he 2

be

rts

if

s

can

Yes, wear levelling & power failure scenarios are not well handled with FAT16 and it is well handled in the JFS. Further, there is no much necessity to create files in runtime in embedded. Agreed w.r.t. embedded systems .

Karthik Balaguru

Huh?

If there is no need to create files at runtime, then there is no need for the file system. You can include the data as the constant arrays.

BTW, there is a whole big area of the embedded systems for data acquisition, monitoring and logging. All they do is creating and writing files.

Vladimir Vassilevsky DSP and Mixed Signal Design Consultant

In most of the scenarios constant arrays are preferred in embedded. But Yes, if you go in for lot of logging then it is required.

Karthik Balaguru

Hi,

there might be an error in the code...

Compile your code with Visual C after activating the LEVEL 4 option which is the most severe (the Visual C Express edition is now free from Microsoft). Make sure the compiler issues absolutely no warnings.

Use function pointers to call all the functions that are architecture dependent. Don't be scared by function pointers: even 8 bit microcontrollers with

1k flash can handle them correctly. In this way, your C code will compile cleanly on ANY architecture.

See the following example: the client of your code, will have to make

2 function calls, before using the filesystem:

#include

void main (void) { fs_config_t *conf;

fs_init ();

conf->disk_sector_read = dataflash_read; conf->disk_sector_write = dataflash_write;

fs_config (&conf);

}

Nice feature.

Good luck pal :-)

ciao, Enrico Migliore

Hi,

p.

y

forgive pal, but this is the very first time I hear that.

Function pointers addresses are calculated at compile time and stored in flash.

I use them on HCS08 microcontrollers and never had a problem.

If you statically link the architecture dependent code, you force a potential user to modify your code.

k

Ok.

You might want to have PC-lint check your code for buffers overrun.

r

he

ar

Ok

Enrico Migliore

Calculation and storage are not the issue. _Calling_ functions via those pointers is.

The problem is that some architectures (most notably the 8051) just don't have enough stack space (nor pointer registers) to afford putting anything besides the return address in there. It forbids itself to put automatic variables or function arguments on these systems' stacks. So C compilers for such architectures allocate even automatic variables statically. To reduce the impact of that on RAM consumption, they analyze the call tree in order to find out which functions' automatic data are never needed simultaneously, so they can statically be allocated to the same location. That optimization is called data overlaying.

Function pointers make that analysis hard --- potentially impossible, even.

Consider yourself having been spared a worthwile, educating experience.

We're talking embedded systems here (and potentially small ones at that)

--- users don't get to decide whether they want to modify the code or not. More often than not, they won't even know that there is code running in that thing they just bought.