Help Picking Right Programming Tools

No problem in any language these days

Perhaps Python, it is fairly easy to learn and there are a lot of scientific graphics libraries for it. It is free and cross-platform too.

Regarding the single .exe there are probably packaging solutions to do this for you or you can distribute python separately.

Everyone has their favorite tool for this sort of thing, I am sure you could use matlab, C#, java, whatever.

What crates these files? Is it something you have control over (i.e. could it take on this added responsibility)?

So, a block is 96 bytes (32*3) of data and an 8 byte header?

But, the *first* block is NOT prefaced by the "00 FF FF FF FF FF FF 00" BLOCK_HEADER? Instead, the "00 AA AA AA AA AA AA 00" DATASET_HEADER acts to introduce that block? Or, are the 96 blocks of "garbage" in your above example -- after the DATASET_HEADER but before the first BLOCK_HEADER?

(i.e., is the BLOCK_HEADER required before each block? Or, does the DATASET_HEADER also function as a BLOCK_HEADER?)

A dataset contains 2048 (64*32) "measurements" encoded in 6144 bytes (accompanied by 512 bytes of headers).

Do you mean "3 byte measurement in block B1 at offset X1" plus/minus the "3 byte measurement in block B2 at offset X1" plus/minus... ?

I.e., figure out which Bi and Xi to combine into each "plotted value"?

What are the consequences of failing to "keep up" with the data?

Or, of not producing "smooth"/continuous motion in the "strip chart"? (i.e., what if data5 is sluggish to be plotted and appears 1.8 seconds after data4 has been plotted; but, data6 is plotted right on time -- at

What happens if data is *dropped* (i.e., a discontinuity in the plot)?

How important is the "live" aspect of the presentation? Could you, instead, accept an "offline" implementation: gather up all the dataX files and then process them at some later time -- giving the user the ability to scroll through the LONG chart?

You could do it in Python. Here's a script to get you started. It converts your hex file into csv file can pull into Excel.

# hex2csv

import os

def hex2csv(filename): """ Convert an binary hex file to an ascii csv file filename -- The path to the binary hex file generates an ascii csv file """ basename = os.path.splitext(filename)[0] outfilename = basename +'.csv' outfile = open(outfilename, 'wb') infile = open(filename, 'rb') for line in infile: for ch in line: outfile.write( str(int(ord(ch))) + ',') infile.close() outfile.close()

def main(): """ Method called from command line """ print 'Convert a binary hex file to an ascii csv file' filename = raw_input('Enter Path to File: ') print filename hex2csv(filename)

if __name__ == '__main__': main()

Your next steps would be to write programs to break the files into your blo cks, data and whatever. Process the data with numpy. Then you can plot it w ith matplotlib. Then if you need a GUI you can use wxPython or one of the o ther flavors. I would check the answers in each step against what you get p rocessing the data in Excel or one of the freeware spreadsheets.

Have fun.

Data rate is slow enough to allow interpreted BASIC.

The first thing I thought of:

Perl script to watch for new data files, read them, and parse them. Gnuplot (controlled by the Perl script) to plot the data into a graphics file (PNG or similar). Image viewing program that will display the graphics file and refresh the display whenever the file changes.

I am not quite sure which image viewing program to use. IrfanView has many options and may meet the requirements. If you don't mind limiting GUI updates to 1 Hz, you could use a Web browser as the image display program, with a little bit of static HTML to make it refresh the image.

This would give you a graph that updates, but users would not easily be able to pause the updates (while parsing and further plotting continues in the background), or scroll the graph around to examine previous data.

The Perl script can optionally write the parsed data to a CSV or Excel file, if that might be useful later.

For many of the proposed solutions, including mine, this is kind of a tough one. You'll need an interpreter (Perl for me, Python for many others), Gnuplot, and some kind of image viewing program. It is possible to package the installers for all of these into a single .exe. This requires your users to have privileges to install programs, which may not always be available. It also turns the process into "run installer, locate icon for installed script, click on it" rather than "run the provided .exe file and start plotting data immediately".

The leading alternative would be to write a custom application, probably in C++ or C#. There is probably a chart widget that would do the plotting for you; you'd get to roll your own file parsing code.

Matt Roberds

would be straight forward with something like octave

-Lasse

Thanks John. I'll give Python a look.

The data comes over Ethernet from an embedded system. A Java program on the PC parses and saves the files.

I hired a guy to do the plotting program in Java. He came up with a graphics package and wrote the program, but it couldn't keep up.

Yes.

The DATASET_HEADER also functions as a BLOCK_HEADER.

Yes.

Yes, but the choices are plus/minus/skip with ratios 0.25/0.25/0.5.

Yes, Bi, Xi, plus data giving the add/subtract/skip rule (a string with values of +1,-1,0). Each Bi has a unique rule string.

This is a real-time display for human consumption. No terrible consequences for failure other than looking cheesy, as long as it doesn't crash the PC, that is. It is possible to use the system 'blind' without display, but it is a pain. A user could be taking blank data for

Some delay is prob OK. 0.2 seconds would prob be OK.

As long as the system takes back up when the data resumes, it would be acceptable. This assumes that no more than about a second or two of data is lost. Any more would need to signal system failure.

I currently process the data offline using an Octave script. The idea here is to make it a live display.

I want to choose a tool that makes a good trade between ease-of-use and efficiency. I think I need something between Assembler and Java, I just don't know what tool. I don't mind learning something new, I'd just like to ensure that I pick a tool where success is at least possible. I'd also like to pick a tool that I could use for other tasks.

I'm not a windows programmer though, so there is plenty of room for bad assumptions on my part. That's why I'm asking for your help.

Thanks, I'll give Python a look.

Do you really think so, or are you just being humorous?

I have considered trying it in C (or some flavor thereof). I'm worried a bit about the learning curve. Is it much more difficult than programming micros in embedded systems?

Can Octave do real-time display?

I've done both. Writing code for the main algorithms or business logic (in your case, parsing the text file and doing the masking and shifting to extract the individual channel readings from a wad of bits) is almost exactly the same on an MSP430 or a Core i7. On a desktop, you have the luxuries of a fast CPU and tons of RAM, so you can pick your algorithms with that in mind.

IMHO the main difference on a desktop is handling all the user interface stuff. You have to write code to back up every button or checkbox or other control that the user can click. Also, users expect to be able to click on things at random times, rather than following a strict sequence of operations, so you have to either allow for that to happen, or disable certain user controls when you don't want to handle them.

Another difference is that you don't usually have a watchdog on desktop. If your code screws up, your program just hangs or crashes, leaving the user to restart the program. Since about Windows 2000, it's gotten harder and harder for a user program crash to require a reboot of the entire machine, but I'm sure an enterprising programmer could still make that happen.

If you've never done C before, the hardest part of this job IMHO will be getting the parsing of the input data right. It's not too hard to handle it if the input files are all perfectly formed, but it's easy to wrong in a hurry if they are not.

If you want to try this, I would advise getting a version of Microsoft Visual Studio that supports either C++ or C#. (C++ you probably know about. C# is what Microsoft did to Java after the courts told them they couldn't screw up Java and still call it Java.) Sometimes they have limited versions or older versions that can be downloaded for free; I don't know what the current offer is.

Look in the "new project" options of whatever you have; there should be some "application with features X, Y, and Z" templates that will set you up with a skeleton of code to fill in.

I would further suggest to get the mechanism of "wait for new data file, read it, parse it" working correctly first... maybe dump the results as decimal numbers in a text file in a temporary directory. Once that does what you want, work on adding the plotting.

You will find it handy to have a standard set of data0, data1, data2... files for testing. You can write a batch file that will copy these files, one at a time, with pauses in between, to the directory that your plot program is watching.

Matt Roberds

He really thinks so and he is right.

Desktop CPUs are *really* fast, it is just poor programming that makes your computer seem slow sometimes :)

depends on your definition of realtime, but a few update per second isn't a problem. Try something like this for a slowly rolling sine

x=0; while(1) plot(sin((0:.1:pi*2)+0.1*x)) x = x+1; pause(.1) end

-Lasse

Have you tried a faster PC? :> I'm only partially joking. At some point, you will have to decide what sort of hosted environment you expect the code to run in (resources available) and how that affects any "performance guarantees" you make to your user/customer.

OK. Just to clarify that. I.e., the code to look for the start of the next block needs to be smart enough to note that every Nth block will have a *different* header. An obvious bug would be for the code to end up skipping over the first block (because it wasn't prefaced with the BLOCK_HEADER) and, instead, see the *second* block as the first.

Not sure I understand your clarification (I've been up scrubbing the roof -- not a wise thing to do in the middle of the day!) but, I assume, you could express these mathematical relationships relatively easily and repeatably. I.e., take the bytes at block offsets a, b, and c and treat them as a (big/little) endian integer. call that 'fred'. similarly, take the bytes at block offsets d, e and f and treat them as another integer, 'barney'. .... plot (fred * K1) + (barney * K2) +...

Presumably, you would want to code it so that if it couldn't "keep up" with the data, it would discard entire blocks -- or even the balance of a particular *dataset* -- in order to be able to "re-sync" with some future DATASET_HEADER.

Understood. But, it is probably *also* acceptable to just display one dataset out of 2 or 10. I.e., change the design specification to "process the first dataset in the file; display; if another file is available, discard the balance of the current file in favor of *it* (or, one even MORE recent) -- so the display never gets too far out of sync with 'now' ..."

What about a more "stuttered" approach (as above)? Or, even a configuration parameter that allows the user to see some reasonably *static* data before the display is updated (and that data effectively discarded)?

Understood.

Ideally, (IMO) you want something that you can invoke to create a display (a 'plot' with particular axis definitions, etc.) that then sits there waiting for you to push (pipe) data to it. Then, you just have to do the mathematical translations (described above) and format the "output" of your "filter" (the term for the program that resides between your "input" and this "plotter") to be compatible with whatever that "plotter" expects as *its* input.

Does it have to run on windows? Does it have to be a "canned" solution? (I assume it has to be *inexpensive*)