Suggestions for custom application-layer protocol?

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
I need to implement a simple application layer protocol that will be used to
communicate between an embedded device (single-board computer running Linux)
and a monitoring terminal (running Windows).  There isn't alot of data being
passed around, mostly status information collected by the embedded device,
and control messages from the monitoring terminal.  Messages will consist of
between 1 and 10 fields of data.

I want the protocol to be text based rather than binary since the data
throughput is low.  I also want the protocol to be based on TCP/IP.  Does
anyone here have any suggestions on the design of a simple protocol?  Are
there simple, standard ways of formatting text messages to be sent over a
TCP socket (such as comma-separated)?  I would prefer to avoid the
complexities of XML.  Since either the embedded device or the monitoring
terminal can initiate a message, is it preferable to have TCP servers
running on both sides?  Or is it better to simply leave a TCP connection
open between client and server?  Any suggestions on ultra-simple standard
protocols that do something similar to this?

Any suggestions or pointers to further reading are much appreciated.

Thanks,
Mark



Re: Suggestions for custom application-layer protocol?
Quoted text here. Click to load it

If the throughput is low, binary would be better.

For example, "1.0000236" is four bytes in binary, but 10 bytes as text.


   I also want the protocol to be based on TCP/IP.  Does
Quoted text here. Click to load it

Read the HTTP protocol.  If you really want text-based, HTTP is probably
the way to go.  My suggestion would be just to use CORBA, since it maps
easily into most languages, has a binary representation, and is fairly
standard.

Quoted text here. Click to load it

What sort of complexities do you wish to avoid?  Most of these
complexities exist with all text-based messages.  With XML you have the
advantage that you don't have to write your own parser.

Quoted text here. Click to load it

Use CORBA.  There are lots of good, free ORBs, including MICO, TAO, and
ORBit.  There are also some Java ORBs.

Jon
----
Learn to program using Linux assembly language
http://www.cafeshops.com/bartlettpublish.8640017

Re: Suggestions for custom application-layer protocol?

Quoted text here. Click to load it

I think the OP means that the amount of data is small, so the larger
size of a text based protocol doesn't matter.

Quoted text here. Click to load it

HTTP is a document retrieval protocol, not a two-way message passing
protocol, despite it's frequent abuse as such.

Quoted text here. Click to load it

CORBA is also difficult to use and debug, has a large overhead in
code, and is largely incompatible with itself.  Avoid at all cost.

Quoted text here. Click to load it

Have you ever read the XML specification, and all the other required
documents (DTD spec, etc.)?  Do so, and let's discuss complexities
afterwards.

Quoted text here. Click to load it

With XML you have the disadvantage that you have to write a
complicated parser, or integrate a huge library, for little gain.

Quoted text here. Click to load it

Just keep in mind that most of these are incompatible, often even
between different versions of the same one.

--
Måns Rullgård
snipped-for-privacy@inprovide.com

Re: Suggestions for custom application-layer protocol?
:I need to implement a simple application layer protocol that will be used to
:communicate between an embedded device (single-board computer running Linux)
:and a monitoring terminal (running Windows).  There isn't alot of data being
:passed around, mostly status information collected by the embedded device,
:and control messages from the monitoring terminal.  Messages will consist of
:between 1 and 10 fields of data.

:I want the protocol to be text based rather than binary since the data
:throughput is low.

OK.

:I also want the protocol to be based on TCP/IP.

For the purposes you describe, it sounds as if UDP might be a better
fit. UDP has a lot fewer states to worry about. The main question,
though, would be how you want either end to react if it notices
that a packet has gone missing.

:Does
:anyone here have any suggestions on the design of a simple protocol?  Are
:there simple, standard ways of formatting text messages to be sent over a
:TCP socket (such as comma-separated)?

If the fields are consistant (e.g., if there are 3 fields then
the first 2 are exactly the same as if there were only 2 fields,
or if there are 3 fields they are always the -same- 3 fields)
then you can just list the values with some convenient delimeter character.

If the fields are not consistant (including, e.g., only transfering
data that changed) then keyword/value pairs would be typical.
TEMP346% STATE=stable

The protocol doesn't start to get interesting until you have
"fields" which are lists of values, or some fields may contain
arbitrary text (including your standard delimeter character.)
If it's less complex than that, just go ahead and do whatever seems
natural.

:Since either the embedded device or the monitoring
:terminal can initiate a message, is it preferable to have TCP servers
:running on both sides?  Or is it better to simply leave a TCP connection
:open between client and server?

Sounds like UDP. If you use TCP servers on both sides, then -every-
message requires the full 3-way negotiation. If you leave the
TCP connection open, you have to worry about the possibility of
the connection dropping and needing to detect and recover from that.
[e.g., suppose you have to reboot the Windows machine.]
--
Warning: potentially contains traces of nuts.

Re: Suggestions for custom application-layer protocol?


Quoted text here. Click to load it

    Could you use telnet, plus your own convention for
formatting the messages?

--
snipped-for-privacy@sun.com


Re: Suggestions for custom application-layer protocol?

Quoted text here. Click to load it


Take a look at netcat or nc.
It open a tcp port where you can send any data binary or ascii. You can
find nc for both windows and linux.


--
/ Peter Andersen
/ Skanderborg


Re: Suggestions for custom application-layer protocol?
Quoted text here. Click to load it

Take a look at protocols like SMTP, NNTP, POP3, IMAP, etc.  Another
example to look at is the "AT" command set of modems (although this
doesn't use TCP).  A command is one line of text.  It starts with
some kind of command verb that identifies what follows.  The response
is usually one line of text, and something at the beginning of the
line identifies what kind of response it is (status code) and may
identify whether it's the last line or not.  The status codes are
in groups so a client doesn't have to be aware of all the status
codes (e.g. 4NN is a temporary failure of some kind, 5NN is a
permanent failure).

Spontaneous responses make the protocol a bit more complicated.
You need to be able to identify a spontaneous response (status code)
and perhaps the spontaneous response only says there IS data, and
the client needs to ask for it to actually get it sent.  IMAP
uses tags on requests and replies so you can match them up, and
deals with sponteneous responses.

Use any kind of field separator that's appropriate for the data you
are sending.  POP3 and IMAP use spaces.  Commas might be more
appropriate for certain kinds of data.

Testing is generally easy:  telnet to the appropriate port and
manually type commands.  Read the responses.

                        Gordon L. Burditt

Re: Suggestions for custom application-layer protocol?
snipped-for-privacy@burditt.org (Gordon Burditt) writes:

Quoted text here. Click to load it

This is the best advice.  The design of Internet protocols follows
good principles, and leads to simple implementations.


More complex protocols such as HTTP are needed only when your requests
cannot hold on one command with a few arguments.  Then we format
commands as email headers, with several lines of 'field: value' ended
with an empty line.

Read: http://www.rfc-editor.org

Quoted text here. Click to load it

Note that AT commands are cryptic (not HELP command like in the
Internet protocols, and no mnemonic commands). AT came for the unique
bit pattern A and T have in ASCII which helped the modem to detect and
adjust automatically the bit rate.  You don't need that for a network
protocol.

Quoted text here. Click to load it

--
__Pascal Bourguignon__                     http://www.informatimago.com /

Nobody can fix the economy.  Nobody can be trusted with their finger
We've slightly trimmed the long signature. Click to see the full one.
Re: Suggestions for custom application-layer protocol?


Quoted text here. Click to load it

One option to consider is to embed a web interface into your
application. This has the advantage that you can use any web
browser as the client side of the interface.

There are a variety of very small web servers that are suitable
for embedding. One such that I have used successfully is "pserv".
From the FreeBSD ports description:

| pServ is a small, portable HTTP server. It is written in pure C for speed and
| portability. It runs as a standalone program and does not require inetd.
| It should be small enough to be used in a mobile computer or to be run on your
| obsolete workstation you have somewhere.
|
| This port contains a patched version to support the execution of php scripts.
|
| WWW: http://sourceforge.net/projects/pserv/

-SEan



Re: Suggestions for custom application-layer protocol?

Quoted text here. Click to load it
your
Quoted text here. Click to load it
I don't think this link is even exist.

D

Quoted text here. Click to load it

Re: Suggestions for custom application-layer protocol?

Quoted text here. Click to load it

 Not a terrible idea, a simple HTTP/1.0 server can be pretty small esp. if
you don't mind stopping as soon as something works (the very basics can be
done in < 20 lines of C).

Quoted text here. Click to load it

 Yeh, pretty good ... I stopped looking after seeing 2 major
vulnerabilities on the first google page.
 The code also looked promising ... for more exploits.


 Writing a custom "simple protocol" is likely to be much easier, using
netstrings is probably more likely to make you do the right thing ... but
a simple "CMD arg1 arg2" type telnet/SMTP/NNTP like protocol isn't hard to
get right.

--
James Antill -- snipped-for-privacy@and.org
http://www.and.org/vstr/httpd


Re: Suggestions for custom application-layer protocol?


Quoted text here. Click to load it

Are you commenting on anything beyond the obvious fact
that the code uses strcpy and sprintf?  
 
Quoted text here. Click to load it

None of these are inherently simpler than HTTP, and you
don't get the advantages that a web browser's sophisticated
support for HTML confers.

-SEan









Re: Suggestions for custom application-layer protocol?

Quoted text here. Click to load it

Speaking as one who has done it, adapting HTTP instead of using a custom
protocol has many advantages besides the above. Think of all the proxies
and filters out there, the tools that snoop the wire, making a nice
graphical display sorted into request and response sequences (e.g.
HTTPlook). Consider the existence of client-side libraries ready to use
in any language (libwww or libcurl for C, java.net.* or Jakarta
HttpClient for Java, lots of Perl modules, etc). None of this is
available to a custom protocol, however easy to implement.

--
Henry Townsend

Re: Suggestions for custom application-layer protocol?

Quoted text here. Click to load it

 That isn't enough? See: http://www.and.org/vstr/security
 Programmers _cannot_ get this right.

 If, when I look outside, there is water falling from the sky I do not
need to walk outside to know I'm going to get wet ... and if you are
arguing that the drops of water are small and have large gaps between
them, I am still not going to feel compelled to walk outside to see if
I get wet.

Quoted text here. Click to load it

 Which web browser? As I said, in theory you can get something "simplish"
that looks like a HTTP/1.0 server from the right angle ... and it might
even work with mozilla (as that client is very forgiving), but making it a
real HTTP/1.0 server isn't trivial and supporting HTTP/1.1 is very hard.
 Also if you need state to cross message boundaries you'll have to
implement a lot more code on the server side.

--
James Antill -- snipped-for-privacy@and.org
http://www.and.org/vstr/httpd


Re: Suggestions for custom application-layer protocol?
Quoted text here. Click to load it
to
Linux)
Quoted text here. Click to load it
being
of

Thanks for all the excellent suggestions so far.  I'll read through the RFCs
for various Internet application protocols.  I've also found some good
information in Eric S. Raymond's online book "Art of Unix Programming".  It
would be nice to be able to use a simple standard protocol of some kind.  It
seems strange to have to come up with ways of formatting text fields,
delimiting messages etc since I'm sure this has been done already many times
before.
Thanks,
Mark



Re: Suggestions for custom application-layer protocol?


Quoted text here. Click to load it

    You can take a page from the URL encoding scheme and use a list of
<param_name>=<value> pairs delimited by & signs. Encode all less than signs,
greater than signs, & signs, = signs, percent signs, whitespace, control
characters, and non-ascii characters as '%xy' where x and y are the first
and second hexadecimal digit of the ASCII value.

    So a query might be:

query=get&parameter=uptime<newline>

    And a reply might be:

query=get&parameter=uptime&value=1%20hour,%205%20minutes<newline>

    Specify that each end ignore any parameters they don't understand
(rather than considering it an error), and your protocol is extensible.

    DS



Re: Suggestions for custom application-layer protocol?
:    You can take a page from the URL encoding scheme and use a list of
:<param_name>=<value> pairs delimited by [...]

:    Specify that each end ignore any parameters they don't understand
:(rather than considering it an error), and your protocol is extensible.

I disagree. If you ignore parameters you don't understand,
then you may well be ignorning something that changes the meaning
of something else that you thought you understood.

When I last did this kind of protocol definition work, the first
thing I had the two sides exchange is their respective protocol
revision numbers. Each side would have a range of versions over
which it was able to support backwards compatability; if
the two sides were too far apart, then either end could say
"Sorry, I don't know how to talk to you."

You could offer "ignore what you don't understand" extensibility
within a protocol release provided that the extensions don't
change the meaning of any established parameter, and the
extensions don't command anything critical. For example,
if one added an arrow-shape parameter to a graph, then
likely it would be considered acceptable if the other end
ignored the shape parameter -- but a new parameter that
indicated (e.g.) "Turn off motor #7!" is probably too important
to ignore. Any time there is a notable semantic change or
critical command change, the version number should get a boost.


Note to the original poster:

One of the other issues that I ran into, which might or might not
be relevant to you, is that some of my operations could take
a long time. I needed command response timeouts; and I needed
keep-alives so the one end would know the other end was
working instead of hung up; and I needed a way to interrupt
executing commands. [My code was single threaded and
non-preemptive, so I had to put in co-operative multitasking
hooks.] There's no clean equivilent to ^C over a network connection ;-)
--
  The rule of thumb for speed is:

  1. If it doesn't work then speed doesn't matter.  -- Christian Bau

Re: Suggestions for custom application-layer protocol?



Quoted text here. Click to load it

    That's not a problem. Just have a parameter like 'minimum-version' that
goes in each request. If the minimum version of the query is higher than the
version you are coded to, you know you don't understand the protocol. You
can also use ':' instead of '=' for "critical" parameters, that is, ones
that should cause you to reject the request if you don't understand them.

Quoted text here. Click to load it

    The only problem with this is that you can't easily have multiple
overlapping changes. For example, suppose one person adds a compression
option to the protocol while independently someone adds a security option.
No implementation exists with both options because they're temporarily (or
maybe even fundamentally) incompatible.

    This is a really hard problem. I've seen many "negotiation"
specifications that didn't actually do anything more than announce
capabilities. That's not negotiation.

Quoted text here. Click to load it

    That works fine if and only if one person controls all implementations
of and changes to the protocol.

Quoted text here. Click to load it

    Yeah, we can make this as hard as we want to. ;)

    DS



Re: Suggestions for custom application-layer protocol?

Quoted text here. Click to load it

Right.  Look at HTTP as an example of a protocol that has evolved pretty
gracefully by making use of the "ignore what you don't understand"
philosophy.  Many of the features that eventually became codified in
HTTP/1.1 were originally done as nonstandard extensions in HTTP/1.0.  
For instance, the "Host:" header, which is used by virtual servers, was
nonstandard in 1.0; it would have slowed down growth of the web
significantly if we had to wait for a new version of the standard to be
published before browsers and servers could start using this feature.

What you have to do when you start creating nonstandard extensions,
though, is ensure that they're not critical to operation of the
protocol.  The protocol has to operate properly if the recipient ignores
them.  You need to bump up the protocol version number, and make use of
some form of negotiation, when you introduce new required features.  An
example in HTTP/1.1 is "chunked" encoding -- if a server sends this and
the client doesn't understand it, the document will look all wrong and
the client and server will get out of sync in communication (the
recipient expects the sender to close the connection when it's done).

--
Barry Margolin, snipped-for-privacy@alum.mit.edu
Arlington, MA
We've slightly trimmed the long signature. Click to see the full one.
Re: Suggestions for custom application-layer protocol?
:    The only problem with this is that you can't easily have multiple
:overlapping changes. For example, suppose one person adds a compression
:option to the protocol while independently someone adds a security option.
:No implementation exists with both options because they're temporarily (or
:maybe even fundamentally) incompatible.

:    This is a really hard problem. I've seen many "negotiation"
:specifications that didn't actually do anything more than announce
:capabilities. That's not negotiation.

This discussion reminds me of the old VMS options processing
setup. It was table driven, but it had the flexibility to know
what options took what kind of parameter, and to know what options
were not valid in combination with other options.
--
Warning: potentially contains traces of nuts.

Site Timeline