How to synchronize a streaming system with FPGA+FT2232H FIFO

Hi, I'm having trouble figuring out how to architect a robust system using the FTDI chips in FIFO mode. I'm pretty sure some of you have been through this before.

I'm using the FT2232H in synchronous FIFO, but my concerns are general.

The system has PC FT2232H FPGA, and is used to both stream/receive data between the PC and the FPGA.

I'm wanting to stream continuous data at about 1MB/s each way, WHILE also sending various control commands to the FPGA (low rate).

Now the FPGA will have some kind of state-machine that reads data from the FTDI. It needs to decide if this 'data' is for streaming, or if it's a command to write to a register. I thought of building and sending a fixed 'packet', that always contain say 32 kBytes of streaming data, and 8kBytes of control, or some variant of this.

Now my concern is that there is no synchronization between the FPGA state machine and the FT2232H FIFO. If something goes wrong, the FPGA could start mixing the data.

I'm used to I2C, or SPI, where there is either a CLATCH event, or a START condition, that marks the beginning of a new transfer. Here, there is none of this.

Any idea?

Diego

--------------------------------------- This message was sent using the comp.arch.embedded web interface on

formatting link

Reply to
dlopez
Loading thread data ...

There are three simple solutions:

  1. Do it like Hayes modems. Data mode, Command mode, "+++" and such.

  1. Byte stuffing. Reserve a byte (like 0x00, 0xFF or 0x55) as "Escape" character, so the next byte will be interpreted as a "command".

  2. Packetization. Packet, length, checksum, timeout. The receiver automatically synchronizes to the packet structure.

Vladimir Vassilevsky DSP and Mixed Signal Design Consultant

formatting link

Reply to
Vladimir Vassilevsky

The streaming data is random, so I can't prevent an Escape character like

0x55 to be in there. Does that rule out this solution or I'm missing something?

--------------------------------------- This message was sent using the comp.arch.embedded web interface on

formatting link

Reply to
dlopez

You have to escape the escape character. If one of the commands, for instance 0x55 0x01, meant start-of-message, then another one like 0x55 0x05 would mean "stuff a 0x55 into the incoming data buffer." The transmitter would have examine its data stream, replacing any 0x55 data bytes by the appropriate command.

Mel.

Reply to
Mel

Except that usually an escaped escape character is just the escape character repeated -- i.e. means "Put one escape character in my byte stream".

If your data can be _truely_ random this opens up the possibility for the pathological case where you have a chuck of data that's rich in escape characters that suddenly bogs down your transmission. The longer I stay in this business the more the phrase "that's really unlikely" translates in my head to "that'll get missed by testing and caught by some important customer right before my big vacation".

--
www.wescottdesign.com
Reply to
Tim Wescott

I generally prefer to do this with packetization, although the other protocol types have their place.

Figure out the longest latency you can have between the PC internals issuing a command and the FPGA actually getting it -- this will help to set the largest possible packet. Then figure out the best packet structure for overhead, ease of design, and state machine size in the FPGA. Then implement and go.

Some folks like to have separate checksums for the header and the whole packet -- it lets the receiver figure out early that it's receiving gibberish; it's nice to know you don't have to find a home for those

30000 bytes of data, but it is particularly useful if there's a mechanism for it to send a NACK on the back-channel to abort a long packet.
--
www.wescottdesign.com
Reply to
Tim Wescott

Hi, When you say 'packetization', does it mean you have only and only ONE packet structure? For example:

myPacket = [startByte, dataStream(32 kbytes), dataControl(512 bytes), Checksum, StopByte]

startByte: 0x55 Stream: whatever random data to stream dataControl: say a bunch of [address, data] to be written in the FPGA. CRC: a few bytes here stopByte: 0x99

Then you send this packet over and over, regardless if the PC is needing to send control data or not?

Actually, after re-reading your email, I think you were saying to build different packets, some for only streaming, some for control etc...The streaming one are the longest, so they add latency before the next command could be received.

Are there any hands-on litterature on this type of stuff? I don't want a 'communication systems' texbook:)

Thanks! Diego

--------------------------------------- This message was sent using the comp.arch.embedded web interface on

formatting link

Reply to
dlopez

Good point. Also when data aren't random enough. Therefore it's a good idea NOT to use 0x00 or 0xFF for escape bytes. And avoid 0x55 (and 0xAA) if you suspect people use them for dummy data.

My baptism of fire here (I think I've mentioned this before) was when something in the channel/hardware/driver was injecting extra 0x00 bytes into the data stream. So we escaped the zero data and stripped 0x00 out of the stream. To avoid explosive escaping we xored all the data pre-protocol with

0x55 to make escaped zero values less common.

Mel.

Reply to
Mel

On Fri, 27 Nov 2009 11:32:23 -0600, dlopez wrote: (top posting fixed)

I don't know if there's literature -- I learned this stuff on the job by looking at existing protocols. If someone knows of something I'd like to see it!

I was suggesting a variable-length packet structure -- so at the least you have a length field, a data field (which may be zero bytes if you want to allow that) and a checksum field. Normally you also add a data type field, to tag the 'meaning' of the data, and often you add some sort of a "start of packet" flag if you want the receiver to be able to sleep until it sees a potential new packet.

The minimum that I'd suggest would be

Use a CRC (cyclic redundancy check) instead of a checksum -- it'll be much more likely to catch an error. I'd suggest two bytes each for length, type, and CRC, and one byte for the start of packet -- but a one- byte type field often works well (until you're trying to find a home for the 257th message type -- then it's a disaster). When folks use a start- of-packet byte it's often something obvious like 0x55 or 0xaa, etc.

On a really noisy link, or one that absolutely positively must synchronize quickly, it's often useful to put a CRC on the header:

This adds two bytes to each packet as overhead, but it vastly simplifies sorting through a stream of data for a packet start, as it lets you discard a bad packet a lot quicker.

--
www.wescottdesign.com
Reply to
Tim Wescott

Tim, Thanks a lot for the reply, it's really helpful.

Trying to confirm my understanding:

So let's say for some reason that things are not in-sync (maybe power up...). The FPGA would be reading bytes from the FTDI. It wants to see a byte, so would keep reading until it gets it.

1) If it's a legitimate we're in-sync again.

2) If it was just a random data byte with the same value, the FPGA would start reading the and , which would be 'garbage' but there is no way to know. Then it reads , which would hopefully fail, and start this process again. I guess there IS a slight chance that this CRC passes, so we would keep reading the data and we might be doing some very bad things with it.

Are you saying you should NOT execute any command until the last is read and passes? I guess this requires a buffer in the FPGA that is the length of the longest packet. This could be quite big in the case of the streaming packet...

Diego

--------------------------------------- This message was sent using the comp.arch.embedded web interface on

formatting link

Reply to
dlopez

Yes. Although if the cost of misreading a command is low enough you could maybe just go ahead and execute.

An example of a high-cost command would be one that triggers a charge of dynamite -- you don't want that one executed unless it's really right.

An example of a low-cost command might be a chunk of sound for a speaker

-- get it wrong and at worst there'll be a pop or a second of loud static, but no one dies, no one accidentally buys 1000000 candy bars, etc.

--
www.wescottdesign.com
Reply to
Tim Wescott

You could use HDLC, which does what you want, is standard, and not too complicated.

formatting link

You can use the address byte to switch between data and control streams.

Reply to
Arlet

Hi, This is all getting clearer now. However, I wonder if using this FT2232H IC in synchronous FIFO mode isn't complicating everything. Isn't all of this 'packet' and 'layer' and 'CRC' stuff handled by the clever people who designed USB in the first place?

Then I use a chip which takes care of all of this, but only give me access to a bunch of bytes, with no way to make sure that the receiver is synchronized. I then have to re-do the job of USB by creating my own underlying protocol?

I wonder if the Cypress EZ-USB chip isn't a better option. It seem to support different channels (endpoints in USB terminology). Maybe one could be for data, the other for control?

It just seems like there is a lot of duplicated work going on here, but maybe that's normal with USB.

Diego

--------------------------------------- This message was sent using the comp.arch.embedded web interface on

formatting link

Reply to
dlopez

"Clever people designed USB"

ROTFL

Good one!

--
Grant
Reply to
Grant Edwards

Then transmit "0x55 0x55" instead of 0x55.

Vladimir Vassilevsky DSP and Mixed Signal Design Consultant

formatting link

Reply to
Vladimir Vassilevsky

NO, you are confusing the link transmission method and what you are putting in one end to come out the other end. How the rest of your hardware and/or software deals with the data stream is beyond the scope of USB and the USB interface chip.

Just the same as anybody else who does USB devices like printers, flash drives, hard disk drives, scanners, digital cameras..... Just like the implementation of CD/DVD devices over ATAPI (PATA) or SATA inetrfaces have to.

That is why drivers have to be loaded onto the host system to match the device at the other end of USB.

You are not synchronising the USB receiver, you are synchronising your application, end hardware and how the MPSSE engine inside the FT2232H works.

The inbuilt micros inside the FT2232H, deal with some of this, and as it is you have a DUAL channel device where as far as I can see you are using ONE port in synchronous FIFO mode which is bi-directional, and have another channel you can use in all sorts of ways including GPIO to signal command or data modes, even separate command channel.

USB in a simple description is a shared form of cabling interface, with several bits hidden from normal users.

It appears to me you expect USB and/or a USB device chip to be psychic about how your design works.

Whatever you use, you still have to sort out how to deal with a datastream (data or commands) to the rest of the hardware, even if this was a serial (RS232) connection.

--
Paul Carpenter          | paul@pcserviceselectronics.co.uk
    PC Services
 Click to see the full signature
Reply to
Paul Carpenter

Very clear! Got it!

could

Using the FT2232H in synchronous FIFO mode switches all the ressourcing of the chip to Channel A, and Channel B cannot be used for anything. I guess this means I need to build up some packets and commands like the previous posters recommended.

Yes this makes more and more sense. However it doesn't seem that easy when there are NO extra pins and everything has to be embedded in a continuous stream of bi-directional bytes. I guess this is where the fun begins!

Diego

--------------------------------------- This message was sent using the comp.arch.embedded web interface on

formatting link

Reply to
dlopez

Hi again! I'm working on implementing those ideas and I wonder if usually the packet protocol is implemented the same way from tx to rx.

Say I describe packet this way (from PC to FPGA):

packetWrite is used to write to an 8 bit register in the FPGA. packetStream is used to send 32kBytes of streaming data.

The application would be sending packetStream over and over, and once in a while a packetWrite would get in there.

Now of course comes the 'packetRead', which would read an 8 bit register from the FPGA! So fine, the FPGA gets the 8 bit and now needs to transfer this back to the PC. It also needs to transfer streaming data coming in at the same time. I assume this pretty much means that the FPGA needs to be 'building' packets as well! And at that point might as well use the same protocol? This does complicate the FPGA design quite a bit though, because it now needs to analyze the streaming data to create the CRC...

Any obvious simplifications to this?

Diego

--------------------------------------- This message was sent using the comp.arch.embedded web interface on

formatting link

Reply to
dlopez

In article , snipped-for-privacy@designgame.ca says... ...

If you need the speed or use a different mode like synchronous bitbang FIFO, or a different device.

I personally am moving away from FTDI after too many documentation and other issues.

If you can design assuming a byte stream, then it is easier to change this design, or future designs based on this method to different link layers, ege ethernet, wireless, or some other new interface later.

Remember to get byte order sorted for different platforms and anything representing larger than byte, well documented as to what happens in your device and order to/from USB.

Don't forget to document your command/datastream packets.

--
Paul Carpenter          | paul@pcserviceselectronics.co.uk
    PC Services
 Click to see the full signature
Reply to
Paul Carpenter

Don't forget you can trade bandwidth for information.

So if you have a 2232H, and 1MB/s, you could change to a base rate of

2MB/s and send byte-pairs.

Byte+!Byte is steered a Data Byte Byte+Byte is steered as Control. This also gives some noise protection, as unequal byte-pairs are flagged as errors. (Or used as a Frame Sync/reset)

This is, of course, very simple to code/decode ;)

-jg

Reply to
-jg

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.