Designing a protocol for the first time

Greetings,

I wish to transfer binary data in the simplest way possible from a PC to either an AVR or TMS320F2812 embedded target via USB.

The PC will be running a Python/wx.Python program and use PySerial to send data out through the virtual COMM port produced by using a FTDI FT245BM USB-to-FIFO parallel interface chip on the target. Ideally this program will be useable on Linux as well as Windoze machines.

I estimate that I need up to 120kB/s (without overhead) of data throughput to transfer 30000 points per second of 32-bit words containing vector graphic X,Y,RGB points for a laser show project.

The FTDI chip says it can do 300kB/s with the virtual COMM driver, so this should work unless they are fibbing.

I have gained some experience recently by writing a set of low-level serial driver functions for the F2812. Then I wrote a command parser for it so the F2812 can respond to text commands and send back various reports. This is all via text terminal. That was my first foray into communication interface programming. It was fun and not too difficult.

Now I need to transfer binary data. The project here won't need any human readable command line. I also don't want to increase overhead by having to reserve the 8th bit for example to distinguish control codes from data, or use some text rendition of the data, etc. Just straight binary.

Thus, I suspect the simplest way for the target to behave is to implement a state machine something like this:

data: state:

  1. wait for start code
0x1234 2. (start code received) wait for size field 0x0123 3. (16-bit size field received, specifying 291 words to follow) wait for data 0xdddddddd 4. (data incoming) count data words until 291 0xdddddddd received 0xdddddddd ... 0xdddddddd 5. (291st word received) wait for CRC state 0x3E5B 6. (16-bit CRC received) goto state 1

Now of course, each state transistion can send a code back to the sender telling whether things are proceeding smoothly. Another state transition path to add might be:

data:

  1. wait for start code
0x5678 7. (request for ACK received) send ACK, goto state 1

That way if the receiver is stuck waiting for data and the sender doesn't know what state receiver is in, the sender can request ACK until the data receive counter transitions things back to state 1.

Is there anything more needed than this (aside from additional fields in the protocol)?

Since there is a specific code to tell the receiver that a data package has begun, when the receiver is in state 1, any sort of commands can also be issued by using any codes other than the 0x1234 example.

Any comments would be welcome regarding my budding protocol idea.

Good day!

--
_____________________
Christopher R. Carlen
crobc@bogus-remove-me.sbcglobal.net
SuSE 9.1 Linux 2.6.5
Reply to
CC
Loading thread data ...

X-modem CRC. Then you only have to write it for the target.

Reply to
Jim Stewart

You may be interested in RFC 1549 "PPP in HDLC framing", see

formatting link

The frame format is simple and robust, and has fairly low overhead.

Reply to
Arlet

Le Sun, 02 Mar 2008 19:36:51 -0800, CC a écrit:

I agree with Jim and Arlet although Jim solution is easier to implement

--
HBV
Reply to
Habib Bouaziz-Viallet

r size field

specifying 291

ait for data

until 291

C state

ate 1

nd ACK, goto

Are you not reinventing a wheel that you already have installed? USB already handles ack/nak and CRC protection of your data. Or is there some peculiarity of the FTDI chip here?

Are you sending real-time data? As in "if the sent data is corrupt, we do not have time to ask for a resend, so we will just skip it"? If so, maybe you should skip CRCs altoghether and use USB isochronous mode. If not, you will need some way for the receiving end to tell the transmitter that something went wrong (faulty CRC, not enough bytes received, faulty header data, etc.) and request a retransmission.

Reply to
Hans Odeberg

For a first time protocol all i can say is dont use straight binary, use ascii (start sentinals and end sentials etc) that way you dont have to worry about escape sequences etc.

Reply to
The Real Andy

"CC" a écrit dans le message news: snipped-for-privacy@news5.newsguy.com...

Some ideas

You can implement the protocol you describe but it is rather involved and you will have to validate it. There are tools that can be used for the simulation and validation of protocols. You will need in particular to check all possible circumstances.

Alternatively, you can implement a very simple protocol with:

- A fixed size frame : you know in advance the amount of data you should receive. You can use a single timeout. You will have only to pad the last frame.

- A single frame structure. Fixed size header + fixed size data + fixed size CRC.

- You can compress your data before transmission which will increase throughput.

Reply to
Lanarcam

Thanks for the input.

Since I have total control over the receiving end driver code and can design it to provide a getc() that doesn't process any escape sequences, then as long at the outgoing bytes on from the PC are also not processed by the PC, there is no problem with binary.

As far as I have seen so far, it is the case that outgoing bytes on the PC are emitted directly.

The problem with ASCII is that it will at least double the physical layer throughput needed to acheive a certain high-level data throughput. Ie, if I took each binary byte and sent it as two ASCII text chars, one for each nibble, which is a common way of transmitting binary data in ASCII form, while allowing control chars to be treated as control chars.

--
Good day!

____________________________________
CRC
crobcREMOVETHIS@BOGUSsbcglobal.net
NOTE, delete texts: "REMOVETHIS" and
"BOGUS" from email address to reply.
Reply to
Chris Carlen

Thanks for the input.

Validate it using tools? This isn't a very critical application nor a complex protocol. I would think I could validate it quite easily for the anticipated manners in which things could go right/wrong by hand with creative use of the debugging tools available. Python is interpreted, so it should be very easy to hand send binary data using PySerial and the Python command interpreter. Or using a terminal capable of binary interaction. At the target side, the 'F2812 debugging tools would make this very easy.

Thus, I don't see any need to simulate the protocol using other than the actual hardware and software in this case.

Now the idea of a fixed frame size is interesting. The data is a vector graphic frame. So either it must include a count of the points to render so that the target knows not to attempt to render null data points (because the graphic frames can have variable point counts up to a limit of 1250 points/frame), or it must include a count of the points so that the protocol knows when the end of the data has been reached.

Either way it has to have a count! I will have to think about this more.

That's probably not needed in this case. Remember: simple!

--
Good day!

____________________________________
CRC
crobcREMOVETHIS@BOGUSsbcglobal.net
NOTE, delete texts: "REMOVETHIS" and
"BOGUS" from email address to reply.
Reply to
Chris Carlen

Thanks for your input.

Perhaps you are right. I suppose if a byte is available to read from the FTDI chip, then it is certain to be valid.

But that is at the lowest level of single bytes, since the USB driver in the PC will be emulating a serial COMM port. Thus, I still need some higher level handshaking for the target/PC to even know that one another exists and that they are communicating effectively.

It is real-time in aggregate, but not per byte. The plan is that a full new graphic frame will be sent to the target before the present one has finished rendering. Then the target will simply switch to the next frame when it completes the present one. If a frame fails to transmit correctly, then yes it would be sensible for the target to just repeat rendering the existing frame. A minor visual glitch in the animation would result.

The likelyhood of this sort of trouble is probably extremely low, however.

It does seem that CRC might be unneeded. I will have to get more in depth with the FTDI chip to confirm this. By not doing CRC, I will very quickly get a statistical sense of data transmission reliability since it will be visual! A flipped bit somewhere (particularly in the X,Y coordinates) will cause a very visible glitch in the vector graphic.

I am certain though that I will not have any control over the USB mode if I am using the FTDI virtual COMM port emulation driver. So isochronous mode won't be an option. And I don't want to go there anyway, since that would be more complex. Since the data throughput needs are met in the simplest mode, then that's what I want to use.

--
Good day!

____________________________________
CRC
crobcREMOVETHIS@BOGUSsbcglobal.net
NOTE, delete texts: "REMOVETHIS" and
"BOGUS" from email address to reply.
Reply to
Chris Carlen

:-D

--
Good day!

____________________________________
CRC
crobcREMOVETHIS@BOGUSsbcglobal.net
NOTE, delete texts: "REMOVETHIS" and
"BOGUS" from email address to reply.
Reply to
Chris Carlen

Yeah, I will spend some time looking at Xmodem to see if I should do it that way.

I wonder if there is anything in Python for that? Might have to use some C bindings for Python to use an existing codebase.

--
Good day!

____________________________________
CRC
crobcREMOVETHIS@BOGUSsbcglobal.net
NOTE, delete texts: "REMOVETHIS" and
"BOGUS" from email address to reply.
Reply to
Chris Carlen

I don't know what the big grin means, but I wasn't talking about PPP, but rather about the encapsulation method of PPP over serial lines, which is quite simple.

In fact, it's only slightly more complicated than XMODEM, but more robust.

Reply to
Arlet Ottens

I wasn't laughing at your suggestion. Only that it would take me more time to understand RFC1549 than to implement my simple protocol.

It's just a question of what's minimally necessary.

--
Good day!

____________________________________
CRC
crobcREMOVETHIS@BOGUSsbcglobal.net
NOTE, delete texts: "REMOVETHIS" and
"BOGUS" from email address to reply.
Reply to
Chris Carlen

snip

Your ack scheme will need to be pipelineable, as while USB has fairly high data throughput the latency, so waiting for an ack for the data you've sent will quickly kill you. You need to be able to keep sending, and handle errors after the fact.

You might look at network protocols for an example, and then cut one down to the simplest case.

Reply to
cs_posting

Oh, another thing - USB is packetized. And the virtual serial drivers are not necessarily real smart. If you put bytes into the serial port driver, you may get only one byte of user data per USB packet. So try to pass larger chunks of data to the serial driver (ideally, figure out how much payload it sends at a time and give it that)

Reply to
cs_posting

I get the picture.

Thanks for the input!

--
Good day!

____________________________________
CRC
crobcREMOVETHIS@BOGUSsbcglobal.net
NOTE, delete texts: "REMOVETHIS" and
"BOGUS" from email address to reply.
Reply to
Chris Carlen

If you are going to consider X-modem, try Z-modem. Much better error control, speed, etc. Uses the same transmission paths.

--
 [mail]: Chuck F (cbfalconer at maineline dot net) 
 [page]: 
            Try the download section.
Reply to
CBFalconer

Zmodem (and Xmodem, or Ymodem) are protocols that transmit binary. They require that the full 8 bits of a byte are usable for transmission. Zmodem provides the best results of the three.

--
 [mail]: Chuck F (cbfalconer at maineline dot net) 
 [page]: 
            Try the download section.
Reply to
CBFalconer

Sorry to pick on your idea CBF.

Xmodem is considerably easier to implement than Zmodem. I implemented Xmodem when I was about 10 or 11 years old, first in Basic, and then in C. I remember Zmodem when it was first released, and it was great. Big improvement over Xmodem and Ymodem, lots of extra features and functionality. But it was also a lot more complex. Sliding windows, automatic recovery, big checksums, transfer of filenames, etc etc.

I'm not necessarily advocating anything for your particular application, just wanted to point out that using Zmodem might not be a slam dunk.

Oh, and regarding FTDI chips. Their chips, drivers, and support are all top notch. I'm using their FT232BM chip at 2mbps, and it works reliably. I started using their VCP drivers, which work very well. I eventually switched to using their D2XX .dll drivers, because it offers more fine-grain control.

Be sure to download the datasheets, app notes, and tips on their website, especially when they talk about latency, baud rates, etc.

Hope this helps

Keith

Reply to
Keith M

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.