Interleaved Read/Write to Compact Flash Card

The title of my post is a bit misleading. Ideally, I would like to start reading out the CF card before the file has been completely written, which will result in interleaved read/writes, each at datarates at least twice my input data rate. Something like this:

- write 32 sectors to address A at > 4 MByte/s

- read 32 sectors from address B at > 4 MByte/s

- write 32 sectors to address A+32 at > 4 MByte/s

- read 32 sectors from address B+32 at > 4 MByte/s

and so on...

That's the ideal case. In reality, there is the system bus, OS and file system overhead, and random delays in the card itself.

You can be absolutely sure that it won't work unless you have buffering for at least 100 msec.

See above.

Vladimir Vassilevsky DSP and Mixed Signal Design Consultant

Vladimir,

I forgot an important piece of information: there is no file system. I'm writing the card in "raw" mode using an FPGA. I'm writing only the data bytes, starting at sector 0, until the 10 GB file is complete. There is nothing else on the card.

Cheers,

Guy.

You can expect a huge delay (~100ms) at the very beginning of the process, and random delays (~10..100ms) while in the operation. This is a typical behavior of CF card. Although the average sustained transfer rate can be 20-30MB/s or even higher.

Vladimir Vassilevsky DSP and Mixed Signal Design Consultant

On Fri, 16 Apr 2010 13:12:20 -0700, Guy Eschemann wibbled:

Look at Transcend Flash - they've got some more expensive cards that (on paper) write rather faster than your figures.

eg this fast bastard:

(87-92MB/s max)

They have slower ones as well ;->

That's why CF cards don't specify minimum write speeds, right? What about SDHC cards? Do they exhibit similar random delays? If not, is there any technology (be it a harddisk drive) that will allow me to write these 10 GB of data without unpredictable interruptions?

Many thanks, Guy.

Even a harddrive will recal the heads every so often.

If your using an FPGA, why not use a RAM stick ?

You said you are not using a file system, so Windows/Linux compatibility is not an issue.

( How did you intend to get that data out of the CF card ? )

RAM would be cheaper anyway.

Why add the overhead ?

hamilton

On a hard drive with some luck, yes. If you verify the areas you will write to in advance to be fast writable (i.e. continuous, the drive has not reassigned any sectors) you can be pretty sure things will work for some time. But you will not know when you will run out of luck, of course...

For 100% uninterrupted write I vaguely remember a discussion on the T13 reflector perhaps 10 years ago when some new "streaming" features were being introduced or were about to be introduced. I have never needed them so I don't know what happened and if they are any good for your purpose (I believe they were just error tolerant, drive would not bother retrying a write if it failed) since my memories on that are very vague, but it may be worth a glance, probably they are in the ATA standards.

But as Vladimir suggested, just buffering for 100+ mS and using normal command sets may be the simplest and sanest way to go.

Dimiter

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

Original message:

Hamilton,

the system is connected to a PC via a gigabit ethernet interface. Ideally, the data will be transmitted in real-time to the PC and logged to the card for backup. In case something goes wrong with the network or the PC during the data acquisition, the data (which can be worth tens of thousands of dollars) will still be readable from the card (via ethernet) after the acquisition has completed.

As you said, I could also use a few DIMM modules to store the whole file, but flash gives me additional immunity against power interruptions. Also, since this is for an embedded application that will be built for the next 10 years, I'm looking for solutions that will be available a bit longer than the few years these DIMM modules typically have.

Thanks,

Guy.

Are these random delays of up to 100 ms specified somewhere? To design a large-enough buffer, I need to know at least the maximum duration of these pauses, and the minimum interval between two consecutive pauses.

Cheers,

Guy

Hi Guy,

Sounds like a interesting project.

I hope you can give more details about its use in the future.

hamilton