Your favorite protocol for transmitting big files

Back in the day, this would have been easy to do with a terminal program that supported ZModem or its ancestors YModem and XModem, which pretty much all of them did. Sadly, those protocols seem to have disappeared from the "usual suspects."

It looks like ExtraPuTTY (over at Sourceforge) is a PuTTY fork that includes YModem and XModem.

Yes, nowadays one would use the RS232 link for PPP to gain network access and then say ftp the files.

I am not sure how viable it is to tell a customer to find a terminal program and use it - what PC or tablet will they get with an RS232 port...

Dimiter

------------------------------------------------------ Dimiter Popoff, TGI

------------------------------------------------------

The gizmo has a USB port connected to an FTDI230X

Since support for the BBS protocols is so hard to find nowadays, how about just going with good ol' RealTerm? It supports XON/XOFF for flow control, so you're okay there. Probably even HyperTerminal would do.

Encode the file to be uploaded into Intel Hex format. The downside, of course, it that big files get bigger. But, if you use parity, you'll get a simple check for each character and a simple checksum for each line. You also get a framing check for the proper layout of each line and the address/offset field to check for missing a complete line.

You want Kermit.

You can either have a nice simple implementation which is easy to implement or you can have a higher performing version with sliding windows and large packets.

While developing on ARM boards without JTAG pinouts but which support U-Boot, I use U-Boot's kermit server to receive executable binaries over a serial link (they are deposited at a specific address in memory) and I use C-Kermit running under Linux to send the executable to U-Boot.

It's all automated; U-Boot runs on the board at power up and executes a loadb command followed by a go command; C-Kermit runs a script which waits for the output from the U-Boot loadb command and then sends the file and connects to the serial port so I can see the output.

With kermit running in robust mode, I get about 6 Kbytes/second over a 115,200 bps link with this setup.

For a Windows based terminal emulator with Kermit support, it looks like Tera Term supports Kermit although I have never used that part of Tera Term.

There are multiple versions of Tera Term floating around. The latest version can be found at

Simon.

Add a small header to the file, add a CRC at the end, and just upload the whole thing as one big binary. You can prepare the header+data+CRC beforehand, so all the user has to do is select binary protocol, and send file.

Using teraterm you could even write a script, so all they'd have to do is double click on the script file.

I see you're using USB, so transmit errors should be exceedingly rare, but if there's a problem, you can tell the user, and tell them to try again.

can you please define "transmit".

Is physical access available ?

Is WiFi available ?

Wireless required or not ?

hamilton

What is the application? What is the target cpu? Is it running an OS? What the (likely) host CPU and OS? A windows PC or so? Why are you using serial i/o instead of USB?

xon/xoff doesn't work so well, especially at such high speeds, because of latency in the host OS being enough to let the 115kbps firehose overflow any character fifos. You want ack/nack or a packet protocol, along with hardware (cts/rts) flow control.

I'm going to be a bit paranoid about this, but if the application is even potentially sensitive, I think you should cryptographically sign the transfers instead of using a simple CRC. There are too many stories in the embedded world of designers deciding this was not necessary for their application, and being wrong as it got exposed to the internet or deployed in unexpected ways.

If you're running full-scale OS's at both ends, the easiest approach is run TCP over the USB port (Windows and Linux support this) and then use something like scp.

I am sure you'd like to, but you may *need* to provide a crude application to do this.

"Logging data" is probably important.

This is what things like Python or Tcl are excellent at.

What's the distance from the user to the target? RS232 is iffy over distance.

Assuming nothing is local, I'd be tempted to make it RS485 half duplex from the gizmo to a Cubie or Beagle, then Ethernet back to the network. These work on a Cubie:

The Cubie could mount a filesystem over the network and log to that.

You can have local storage ( SSD or spinning drive ) on the Cubie as well.

That sounds too easy -- I'll try it first.

I suppose I should find a microprocessor that'll support USB the way I want, and see if I can have the thing put up a web page or look like a mass storage device or some such -- but that's another project altogether, and this is mostly a hobby effort, with no payback for my time.

For the people who have asked for more information:

The device logs inertial data during a model airplane flight. The use model is to download right after the flight (there's only room for one flight's worth of data).

The devices uses an FTDI230X USB UART chip on board. So you sit down next to it, plug your laptop into the thing and suck it's brains out. (And yes, there may be better solutions -- this is a first prototype, which kind of grew, and now here I am with a need).

There is no OS in the device beyond a simple task switcher.

Based on suggestions, I may start by trying a simple text upload, and see how far I can get.

How about putting a socketed microsdxc card (3 grams?) on the plane instead of the usb interface. It could hold log info for 100's of flights. To upload it you'd just remove the card and put it in a USB card reader.

Damn. That may solve both my "little file download" problem and my "big file upload" problem.

I need to think about that, hard.

Any idea of how well the typical socket deals with high vibe? Most of the users of this product will be flying electric power, but a sizable fraction of the market I'm addressing (not least of whom is me) will be using this on piston-powered planes; a single-cylinder model airplane engine adds a lot of vibration to an airframe.

A micro sd card-socket may be less sensitive to vibration due to the lesser weight of the card. Any, maybe, there are sockets available that use some additional mechanism to lock the card into place.

No idea. It might indeed be an issue. I think some people have flown video cameras using those cards, though.

AAAAGH! It's just too goddamn big! The board is about 30mm x 45mm, and packed. It'd have to grow to accommodate the microSD board, even taking away the serial flash chip that's on there.

I may circle back to it, though.

It's also possible to implement USB in software, even on small processors, letting you get rid of the FTDI chip on the target side, and use normal USB protocols on the host. Some code for this is at:

Similarly:

I think that for the moment I'm going to continue using the FTDI chip, but keep my options open for hand-rolled USB.

I'd be most open to things that don't require programming on the host side, but I'm not sure if an such thing exists.

Anything I implemented would have to deal with the fact that I'm using the internal RC oscillator in the chip (crystals are big, too), so I'd have to synchronize well. I'm not saying it can't be done -- just that I'd have to do it.