I am working with DLP module based on FT245BM device. In my application I need to perform transfers from a DSP to the PC. It consists on blocks of 1488 bytes every 26 mS, and I need to read a LOT of these blocks of data, say more than 10.000. The problem that I am facing is that if I set the Read function to read for more than about
170 blocks of data, the application gets blocked (for this test I set the timeouts to 300000 ms -> timeout of 5 minutes per block of data, which, I admit it, doesn't make much sense). If I change the timeouts to 30 ms, I am able to read about 1000 blocks of data. But no more than that... And I NEED to get the application running this way.
In pseudocode what I do is:
UINT ThreadFunction (LPVOID param) { for (i = 0; i < maxBlocks; i++) { ReadUsbData(pointerToBuffer, 1488); } return 0; }
The problem is when maxBlocks is a big value (>1000). With the timeouts set to 30 miliseconds, the ReadUsbData function returns when the buffer has 1488 bytes, which is what I need.
I would like to know whether it is possible to have a version of the MProg utility which allows me to have Isochronous transfers.
Somewhere I read that it is useful to poll periodically the data which is actually in the buffer, and once it has the amount of data necessary, trigger the ReadUsbData function, to read the buffer. Any opinions about this approach????
All other suggestions/comments/recommendation which you can provide about how to have the trasfer of 1488 bytes every 26 msecs from the peripheral to the PC, regardless of how many of these blocks are being transferred, are VERY welcome.
1) we are using the FT245 devices with continuous data rates to 400 kbps, (communication rate around 500kbps) using the direct driver (not virtual comm) under Windows 2000. We use an additional FIFO buffer that keeps the FTDI device buffer full, but our buffer is hardly used. Your data rate (1488/0.026=57.2Kbps) is significantly less, so you should be able to accomplish your goal with the hardware at hand.
2) The FT245BM has a 384 Byte FIFO TX buffer. Trying to read blocks larger than 384 bytes will only cause the FTDI device to pause while it requests more data. Use a packet size smaller than their buffer.
3) Isochronous transfers can lose data, (not guaranteed) Can your application tolerate that?
4) You can poll the status but you'll be burning a lot of processor cycles. It is better to setup an separate data receiving thread and set an event within that thread that block-waits for that event.
Then, in a loop: while(FOREVER) { WaitForSingleObject(hEvent, INFINITE); ftStatus = pGetStatus(ftHandle, &RxBytes, &TxBytes, &EventDWord); while(ftStatus == FT_OK && (EventDWord & FT_EVENT_RXCHAR) && (RxBytes > 0)) ftStatus = pRead(ftHandle, strbuffer, RxBytes, &iBytesReturned); //Do something with this data (shove it in a circular buffer?) } }
Where pSetEventNotification and pGetStatus are FTDI library functions. This thread stuffs the data into a circular buffer and this circular buffer is access from another thread which communicates with the main application. You'll have to manage the circular buffer pointers, perhaps with criticalsections.
IIRC, the FTDI chips are internally programmed to use bulk transfer mode, and that the mode cannot be changed. This mode guarantees loss-free transfer, but may not be isochronous if retries are needed, or other devices are on the USB.
If you know of a way to change the endpoints for the user comms on the FTD245, I'd be interested in seeing the code.
The rest of the questions about the Win 2K code are above my competence level. I've never had much success with threads in Windows programming, and stick to simple polled code---which doesn't mind an occasional multi-millisecond hiccup by Windows.
Here you have my comments, I hope you can reply back:
I am also using the direct drivers (ft2xx.dll), but with the Classic Interface. Does that make a difference, compared to the Win32 API?
Also an issue is this one: the buffers I am writting to are member variables of an object. I have an array of those objects. Could that be the reason for the blocking of the application in the PC side? Could it be more effective to write to just a big array, instead???
I need to send the packets every 26 ms, 1488 bytes each (from the DSP). How can I make sure (in the PC side) that every packedt will be received correctly, if I use a smaller packet size? Could I do four reads of 372 bytes for each 1488 bytes sent?
I can't tolerate data losses. Anyway, the documentation says that an MProg utility with isochronous fucntions enabled can be obtained from FTDI directly. It also says that you can reprogramm the EEPROM to have isochronous transfers enabled.
The documentation also says that the receiving data buffer can be modified (to change the 4kB default). Could that help? It also talks about the timer latency. Something to say about this?
In this case, then isochronous transfers are not what you want to use, since they are not retried or retransmitted in the case of a lost or corrupt packet.
Isochronous transfers *are* guaranteed bandwidth on the USB bus, but there is no retransmission if a packet is lost/corrupted. Packets are expected at the sink end of the pipe to arrive every 1ms.
It is the responsibility of the data sink to deal with missing transfers.
--
Michael N. Moran (h) 770 516 7918
5009 Old Field Ct. (c) 678 521 5460
Kennesaw, GA, USA 30144
"... abstractions save us time working, but they don't
save us time learning."
Joel Spolsky, The Law of Leaky Abstractions
The Beatles were wrong: 1 & 1 & 1 is 1
I'd like you be more "illustrative", because I feel somewhat lost about thread synchronization, circular buffers (in C++ for PC), and other issues.
Let's start again: I need to send 1488 bytes from device to PC using USB (FT245), every 26 msecs. What is a "framework" to accomplish the task? How to store all of the data? Keep in mind that the transfer is CONTINUOUS, and with no limit on the number of packets to be sent (or which limit is a high number, say 10.000+ packets). I'd like you sketch the steps to take into account for setting up this transfer. Should work for Win98 and/or Win2kPro.
Anyway, I am sending some key code snippets to illustrate the approach I am taking (VC++ 6.0 on Win98SE or Win2kPro). All of your comments are very welcome.
In Main application code: CWinThread* pThread = AfxBeginThread(USBCommThread, &m_CommKernel, THREAD_PRIORITY_TIME_CRITICAL);
The worker thread function is: UINT CMp3FiJaaCHostPCDlg::USBCommThread(LPVOID pParam) { CCommKernel* pCommKernel = (CCommKernel*) pParam; for (int granule = 0; granule < pCommKernel->GetGranules(); granule++) { pCommKernel->GetGranuleDataDSP(granule); } AfxMessageBox("USB Communication finished"); pCommKernel->DataOnMem(); return 0; }
Here, "granules" is the number of packets to be sent/received (10.000+, but could be either more or less)
In CCommKenel: void CCommKernel::GetGranuleDataDSP(int granule) { char* pChar;
The member "m_AudioBuffer" is an array of 10.000+ objects of class CData. Here, 10.000+ is the same number as "granules", and CData is one class defined by me, which has many member variables, most of them are "structs", and also has a member like this: char m_RxBuffer[USB_BUFFER_SIZE];
Here USB_BUFFER_SIZE is the number of bytes to be sent/received by packet, in other words char USB_BUFFER_SIZE = 1488.
Also, when opening the USB device I set the TimeOuts to 30, so I have a timeout fo 30 mSeconds to receive 1488 bytes.
The problem that I am facing is that when granules>=1000 the application seems to get lost. Otherwise, I receive all of the data, flawlessly.
I am using the Direct Drivers with the Classic Interface for programming. Should I move to Win32 API interface for programming?
Please, help me to better strucure my application.
that's about 50kByt/s or 500kBaud. Why not simply use the virtual commport drivers, they can handle up to
3MBaud. You have to do nothing special for it (except maybe increase the serial buffer size) and just use your standard commport component. Stef Mientki
You haven't said anything about what else this application must do. Does it simply write the data to a file? Does it perform some complex analysis of the data? If your goal is to simply suck up the data until a user pushes the "STOP" button, why bother with the threads and complex structures? Simply put the read function into function of the application and wait for a stop signal. I've had good result doing this on a file transfer application that collects many megabytes of data from a microcontroller as fast as it can be sent (about
400KBytes/second for the particular micro).
Here is a code snippet from a USB application written in C++ Builder:
timeout is a variable that gets decremented in a timer handler which is triggered 100 times per second
************************************************** while(timeout > 0){ fts = FT_GetQueueStatus(FTH, &num2read); if((fts == FT_OK) && (num2read > 0)){ FT_Read(FTH,&spbuff[0],num2read,&numread); if(numread > 0){ if(fp != NULL) fwrite(&spbuff[0], numread, 1, fp); bytecount+= numread; EDCount->Text = bytecount; // update display counter timeout = 4; // reset timeout counter to 0.4 }// end of if(numread > 0).... } // end of if(fts == FT_OK....... Application->ProcessMessages(); // catch up on windows message handling } // end of while(....
**************************************************
Note that this function reverses the normal Windoze way of handling things----it gives the USB data collection the top priority and calls the Windows GUI handler when the USB stuff is done, rather than letting the windows gui call the USB handler when it thinks the USB thread priority is appropriate.
The application is a RealTime (of course) MP3 Encoder. The DSP processes the digitez audio and sends the information of MP3 frames to the PC. In the PC I do the final formatting. I needed the PC mainly for storage purposes, so I send the frame information to the PC, and it proccesses these data (offline) and writes the MP3 file. The frame information is 1488 bytes long, and it is ready for transmition to the PC every 1152/44100 = 26mSec. Does it give a better idea of my needs?
Using the GetStatus function I see that the total transfer is 63488 bytes, which means that the buffer is set to 64kB. It is strange, because I set it to be 1536 bytes... I also tried the WaitForSingleEvent function to wait for the data to be received... the application got stuck, like if the event never signaled to ready (this suggestion was form the first reply in this discussion thread).
About you example, some questions:
How big is your spbuff buffer? Mine should be somewhere between 8MB and 16 MB.
What are the settings of your USB device (I mean, timeouts, buffer size, etc... which FTDI driver functions did you use? In which order?)
How do you get sure that all of your data is in the righ order, from the very beginning, till the end of the transfer?
Does the "++" features of programming in (V)C++ make the application less effective? Should it be pure and simplistic "C"?
How could you make sure that you read, say 1500 bytes several times, instead of waiting for 63488 bytes? Why not to use if((fts == FT_OK) && (num2read == 1500)){, for example?
Now that you know what do I need to accomplish, please write back with information about a software framework which I could use to successfully get my application ready, keeping in mind that I can't miss any data.
Please, I need to get this running ASAP, I've tried all the tricks I have imagined, and haven't been able to get it working.
If you want, send to me also a copy of information written in the newsgroup to snipped-for-privacy@ieee.org Maybe it is useful if you want to send some code, which you don't want to make public.
I think the buffer need be only as large as the transmit FIFO on the external device. If the win app doesn't pull the data from the driver fast enough, USB handshaking stops the transfer.
At the FTDI end, I just put bytes into the FIFO until it indicates that it is full.
the normal transfer mode of the FTDI chip guarantees this. Blocks are sent in order---and if a USB block isn't acknowledged, it is automatically retransmitted.
Couldn't say. That is a function of the compiler.
The loop I use reads the data as fast as it arrives---that will generally be determined by the transmitting end of things and the fact that the data gets blocked and transmitted on multiples of
1mSec.
The normal transfer mode guarantees that you won't miss any bytes.
Sounds like you need to hire a consultant! Unfortunately, I'm booked up this month. ;-)
I would normally quote a project like this to take about 4 to six weeks to code and debug. If you'd planned on doing it much more quickly, I would recommend you work on your project scheduling skills. If you're now 5 weeks into the project, you've waited too long to ask for help!
0: 63488 bytes read
1: Don't have data to read yet
2: 380 bytes read
3: Don't have data to read yet
4: Don't have data to read yet
5: Don't have data to read yet
6: Don't have data to read yet
7: Don't have data to read yet
8: 1488 bytes read
9: 2976 bytes read
10: 1488 bytes read
11: 1488 bytes read
12: 1488 bytes read
13: 1488 bytes read . . .
155: 1488 bytes read
156: 1488 bytes read
157: 256 bytes read
158: Don't have data to read yet
159: Don't have data to read yet
160: Don't have data to read yet
161: Don't have data to read yet . . .
297: Don't have data to read yet
298: Don't have data to read yet
299: Don't have data to read yet
My interpretation is this:
The first read of 63488 bytes is because I first started to send data than to receive it (no problem).
The for loop reads blocks of data, sometimes it takes longer time for the PC to read the data than for the DSP to send it, so when the PC goes to read it, it reads two packets of 1488 bytes each, at once.
But what I can't explain myself is why after some reads, when I try to read the buffer, it has no data (I say so because rxBytes returns
0). What happens here?
Three more questions: 1. How to synchronize the packets sent by the DSP to the reads in the PC side? With Sleep? With the FT function related to latency? 2. Why if I use INFINITE in the WaitForSingleObject function it doesn't return, like of never receivves data, even when data actually is being sent? 3. Why didn't I get 1488 bytes in the first transfer, if I set the buffer to be 1436 bytes long (which yields 1488 bytes of effective data)?
In my application I need to have something like what I sketched abouve, but with the loop running FINE for 10.000+ cycles!!!
Why are you sleeping after you read the status?? Is this some peculiarity of threaded programming?
How good is the timing when you request a sleep of 23 (milliseconds, I assume)? Most PC apps that don't use the multimedia timers have lousy timing resolution.
check ftStatus to see if it is successful, since you are not declaring the device handle, maybe it doens't default to zero like the others. ftStatus = FT_SetEventNotification(ftHandle,EventMask,hEvent);
You'd be better off dumping all the data you collect into a pre-allocated array, and then dump the array after you test period is complete. Writing formated strings to a disk file ain't all that quick.
Don't sleep when you are polling. Sleep allows the thread/process to be suspended to allow other tasks to run. I use a higher priority. j = SetThreadPriority( ThreadID, THREAD_PRIORITY_TIME_CRITICAL );
Huh? Why do you have th synchronized the packets?
Possiblely the event is not defined. Test the return parameter.
The USB communication is typically delay because its just getting up to speed.
Because if I don't do so, I get the "Don't have data to read yet" message (which menas that there was no data on the buffer, when I attempted to read the buffer) earlier. It is like the PC was in fact reading the data _faster_ than it is being sent. Of course, as you could see, the Sleep(26) is too much, I should maybe move it to
23msecs, because sometimes I had twice the data in the buffer, as I a previous block wasn't read on time.
This is EXACTLY one of the problems I am facing: how to synchronize the timing for the DSP - FTDI - PC. And the other problem is why does the USB part of the system suddenly stops transfering the data to the PC.
Those are the topics where I need your help, guys.
I am pretty sure that you could provide some clues on how to solve it.
That's the purpose of reading the status. If the status says that there is data available, then you better be able to read some data.
You don't have to read the entire packet at once. You can read whatever is available and stuff it into a buffer until that buffer is the same size as the expected packet.
Of course this assume that the 1st byte is sync'ed with the start of a packet. That's a reason for using using two threads with a circular buffer, 1) to read the USB data and stuff it into the buffer as fast as it is available; 2) to parse the buffer, making sure you locate and sync with the proper "start of packet" sequence, and deliver the properly formated packet to the user application.
My comments are inlined, please read them (and reply back, of course!):
I am using a global ftHandle, previously allocated and (succesfully) open. In other words, my FT_SetEventNotification (without ftHandle) calls the _real_ FT_SetEventNotification with the global ftHandle as a parameter. And returns FT_OK (0), as expected.
Good idea. Mine was a first approach to check what was going on, PACKET BY PACKET.
Will see... I guess the thread you talk about is the one for data transfer only, am I right?
Because the data I send to the PC has some kind of header, and stuff like that. I mean, I must make sure that the transfer collects all the data from the beginning of the first packet, not from somewhere in any packet an continue... I mean, all the packets should be complete!
What should I expect?
Thanks. I'll try your suggestions, which are quite interesting, and will let the group know how does it go.
My comments ar inlined. Please read them, and write back!
My idea was to read packet by packet... It looks liet I'll have to change the paradigm.
What if the data buffer read is longer than the expected packet size? (like reading 63488 bytes that where available, when my packet size is
1488 bytes?)
Of course! That's what I need, because I need COMPLETE packets.
Couldn't you please, provide some rather complete application to illustrate these concepts? (circular buffer in (V)C++, locating and sync'ing with the proper "start of packet" sequence and delivering properly formated packet to the user application?). maybe I'm asking for too much... you tell me.
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.