need data capture, stat analysis, graph display software for PC

C# has (in the latest release) good serial suppurt, which shouldnt have a problem linking to serial over bluetooth, and there are a number of decent and FREE 3D graphics interfaces for C# -- DirectX and OpenGL at the low level, Axiom and others at a higher level (though those are more scene oriented)... in any case... C# eliminates the majority of the crap that MFC or any C++ UI will put you through creating a decent UI

We can take this off group if you'd like to discuss it more.

thanks, ive not looked at C# yet maybe its time to, is it similar enough to c++ to pick up quickly ?

Il go look for some online c# references.

its not that far off topic here is it? it is data from an electronic experiment. maybe if we get involved in how to combine al the differnt windows attributes such as scroll copy/paste in one window ..

Colin =^.^=

Would Python be any good at this sort of thing?

martin

If you can get some example code that will write the serial results into a delimited ASCII file, you can do a lot of processing and graphing in Excel. Not elegant, but it saves you from learning new packages or languages

Wim

Serial Port Import to MS Excel:

I haven't tried this one but it sure looks doable.

Hi, thanks, Ive arranged the data in CSV (comma seperated variables), as this seemed most widely used, wich comes in through the com4 port.

Im using bluesoleil software on the PC for the bluetoth dongle, wich works fine with my rs232->bluetooth device except if the remote system gets power cycled or the PC reset for any reason it requires manual intervention to restablish the link. this is quite irritating, all the other classes of bluetooth devices seem to have options to start automatically.

I could save the data in DOS window with 'type com4>file.txt' but thats not so elegent. would be nice to timestamp the data with accurate time too. atm im just loking at it on a dos screen, as hyperterminal realy sucks trying to see large chunks of data.

I would need to do FFT on the data to find the direction, then do FFT on that again to find the cyclcic change in direction, its probably not that difficult to do, in fact I would just do it all in the PIC if it had more than 16k of ram! it has a DSP engine after all ! although ive done it for a limited period but I realy need to do average over fixed time span rather than just a running average so need acess to al the points.

I also need to do standard deviation at both FFT steps and filter out any glitches, again this isnt to dificult in theory as such, but im not sure a spreadhseet would cope with 10e6 data points, I dont realy like the idea of spreadsheets that much tbh, so ive never realy used them. Although Ive programmed in C++ for many many years.

Im not sure if the higher level packages would let you reject points of data or then let you do an FFT on it.

A graphical display of the data would be nice although a final list of the stats would be ok albeit a bit boring for all the work gone into it, the graph would at least give me some visual confirmation of the stats too.

Ive just installed the C# from microsoft as it seems to be free ! C# looks promising too.

Colin =^.^=

and

whatever.

3rps

level.

a

not

Probably not, but 10e4 is easily done

The nice thing of spreadsheets is, that you can create a graph in half a dozen mouse-clicks. You can even do crude DFT's faster then learning an advance math package :-P

data

You can split the processing in several stages: Remove glitches FFT etc. Each of which are very manageble in whatever language. If you wite the intermediate results in a CSV, you can have a quick peek if the result looks halfway decent.

I would suggest to use a standard library for the FFT, so look if there is one for C#

Wim

techincally you could use ANY language that can cope with the serial interface, and you can extend python with C/C++ to pull in the needed graphics capability, but there really isnt a point -- C/C++/C# have the graphics libraries available off the shelf, as well as loads of code available for massaging the data before display -- I dumped C++ for C# after learning Java -- no way I'm going to deal with anything less than a fully object oriented interface ... python leaves something to be desired there, and I dont like its indentation-sensitive syntax

C# is a blend of C++ and Java -- you wont have any problems picking it up very quickly -- plenty of example code out there --

is a good place to start

If you dont want to get hooked in to the MS world -- use Java -- Java3D can handle the data display and the netbeans IDE is pretty comprehensive (you can download it at java.sun.com free)

Ahh you must have found the "express" version or whatever they are calling it lately -- that'll get you there :)

feel free to drop email on me if you have C# specific questions.

yep it was quite surprising to find it, I expected it to be limited. wonder why they giving it for free, must be becuase it ties you into windows easily with .net.

admitedly it wouldnt let me instal it to my e: drive where I had space it still decided to try and plonk 1.3gb on my c drive wich is now full, even though I specified a dir on e:

lets you generate an application in 60 seconds wich is fine if all you want to do is have a button wich changes a label to "hello world" when you press it.

I already have com port open so it puts lines from com4 into where the hello world was when you preess the button lol, so thats 30% of the way there.

I think il just have to massage the data and interpolate between points wich I have to reject as glitches.

There are some FFT libraries too thats another 30%, google came up with one that looked like it would do, dont know if there are any recomendations available? although I dont fully understand the FFT terminology used exactly, I know how to use a DSP to do one though lol. does the PC have real DSP capabilities wich it uses or just a normal but fairly quick FPU? im runing with an AMD64bit CPU.

I have also moved the com port to a new task/thread for reading and handled acess denied exceptions when it wont open. atm im looking for a good way to pass the data from the com task to the processing task. I gues a stream or a pipe or whatever its equivalent is in c#.net Il probably find that too, just take a bit more time looking than the other stuff did so far.

Im sure il find a c# graph library too, again maybe has some prefered ones.

Colin =^.^=

thanks for that, I'm not "a PC programmer" but quite happy writting small stuff for 8 bitters, I was just thinking of a simple solution in the PC world, that would have helped the OP.

martin

FPU based -- no on-board DSP :)

for the data transfer -- look at the System.Collections.Generic namespace -- create a class that represents a block of FFT ready data.. have the com port thread assemble the block, and when ready.. push the block into a typed queue (generics are like C++ templates.. let you create collections of typed objects with full compile time type checking)

use the "lock" statement to wrap your push and pull methods so that access to the queue is serialiized, and optionally create an event to notify the UI code that there is a block of data ready to process. (or use a timer to poll for data -- whatever you are comfy with) Note that C# gets pissy when you attempt to access UI elements from a thread other than the one that created the UI -- look at delegates and the Invoke method to work around the issue if you need the background thread to access the UI.

You might want the FFT and any other data crunching on the com thread or even a seperate cruncher thread so the UI just has to deal with displaying ready-to-render data.

sounds like you are off to a running start, though !! glad to see you are slipping into C# so quickly :)

(ohh and the Mono system for *nix will run dotNet code -- so you really arent all that locked in to MS with C# any more )

The com thread needs to be real quick as it needs to timestamp each line to

10ms resolution if possible, itl just do that and also split it into seperate chanells for fft data and housekeeping data, errors etc. and save it in files. the PIC is short of a RTC with proper time. the fft thread has to read any old data stored in files then add the new data from the com port thread then do number crunching every so often on the data collected so far.

thanks that 'lock' sounds like what I need, of course I could just use the files to transfer data .. just make them each hold chunks of 10 minutes or so they are finished with by the com thread before the fft thread gets to them, but the lock would make me free to make them longer. im sure theres probably simple pipe type class wich uses the thread lock.

yeah thats just like mfc, it used temp data objects wich arnt garanteed to not have been destroyed when other threads are running.

Cool thanks, yeah I get up to speed real quick I gues, but only if i can find the enthusiasm though and then thats either on or off with little inbetween I just bualk at the idea of trying to go through loads of new things trying to evaluate them, ive tried that before and it can realy suck.

Colin =^.^=

I was thinking more realtime than logging to files -- think about having the com thread pump the data to a writer thread in that case :) sounds like you may end up with 3 or 4 threads sharing the load, which is a good thing IMO -- as threads that are idle dont take resources -- event driven makes life much simpler

might also look at the background worker class -- it encapsulates a lot of the crap for starting a worker thread as needed and getting notified when it completes

....

I dragged that very class right of the IDE toolbar into the project. same with the serial class.

3 or 4 threads is probably about right.

I need to save the data regardless anyway. I could have the analysis done after all the data is gathered but realy would like to see how well its progressing so i can catch anything going wrong and not waste days colecting duff data.

I found a Queue wich can transfer objects between threads took me a while to figure out I needed to actualy create an instance of it with new. works but I gues I need to protect it with lock and I need to have something to wait on. I tried the messaging thing but it complains its not installed on my pc. the sockets seem primitive as they stil rely on byte streams, although come to think of it that is actually all what I have anyway.

cant beleive they havnt put in some sort of decent inter thread message queue. although theres a few examples of how to implemet one about. cant even send them as simple windows event messages like with mfc.

Colin =^.^=

ya -- they dont make the collection classes thread-safe for performance reasons -- they figure anyone that needs it thread safe can create a wrapper class derived from the collection and override the key methods to implement locking

If by messaging, you are talking about MSMQ -- thats a bit overkill for this -- just create a protected queue class and go from there -- its easy enough to add events to the class so that the reciever can get notification when data is available. You dont need to "wait" on anything -- just have each class set up its event handlers in the constructor, and leave the instance to sit until an event occurs.

"simple windows event messages" -- why get the OS involved with the notifications ?? I can understand how you would feel that WM is simple given your MFC background, but you are pushing your message out of process when it really doesnt need to. using WM like that was a "lazy" solution for people that didnt want to write thread safe code, or didnt undestand mutual exclusion locks and syncronization. Not that I didnt use it when I was writing win apps in C++, but it really wasnt the right way to solve the problem :)

Re: sockets -- yes they are byte streams at the low level -- all IO is -- even disk -- you can wrap them in a stream reader if you want a string/line oriented interface, or make your own reader to de-block the data any way you want :)

ah ok I guesed it was something a bit more complicated.

done that.

erm easy if you know how I gues, I didnt come accross events so far, im using the workerthread things dragged from the toolbox so I kind bypased the constructor details.

I used System.Threading.AutoResetEvent, to wait for the queue to become non empty, wich is actually easy enough.

but it sounds like these event handlers you mention before do the equivalent of what the old win32 messages do wich is what I was looking for, but just with the strong type control thats needed ?

windows messages wasnt realy a lazy solution as they were the underlying fabric of mfc. they were not only used to talk from task to task but mostly within the same task. although there was the problem of ensuring what was responsible for the allocated memory wich c# takes care of.

I find it hard to beleive the functionality of sscanf isnt in c# I found loads of articles of people trying to convert c++ code into c# and loads of people trying to answer but getting tied into knots with saying you just use regex or tryparse without realy knowing how to exactly, but these are either rather low level or overkill. trying to parse my variables wich are seperated with a variety of delimiters for clarity is irritating without an equivalent.

I tried to make a function template but it wouldnt let me access the right TryPars member from the template. So i have one for each type. it works anyway.

thanks Colin =^.^=

To create an event

1. create a "public delegate ;" outside the class that will have the event... inside the class -- add a "public event ;"

1. any place in the class where you want to fire the event, add "if ( != null) (delegate params);"

2. in any class that you want to handle the event, add "object.

+= handler;"... the IDE will create the handler stub for you if you hit tab twice after the +=

the delegate defines the parameters that will be passed, the event statement defines the variable that will contain the list of handlers to be invoked when the event is fired. Since delegates are typed, you can pass any number/type of parameters that you like -- no limitations :)

Re sccanf == ya -- that doesnt really fit into the java/c# paradigm -- you've probably already found this or something similar -- but here is a regex based sccanf for c#