avionics system testing

NOOOOOOOOOO! NOT LABVIEW AGAIN! AAAUGGGHHH!

(Sorry, couldn't help myself. I'm all better now, really, don't worry. Sure I'm sure. Thanks anyway.)

LabWindows is at least less awful than LabView.

I do all my instrument code in C++, sometimes with a script wrapped round it. If your hardware has drivers with C bindings, use those.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs 
Principal Consultant 
ElectroOptical Innovations LLC 
Optics, Electro-optics, Photonics, Analog Electronics 

160 North State Road #203 
Briarcliff Manor NY 10510 

hobbs at electrooptical dot net 
http://electrooptical.net

I built some control/configuration GUIs in the past using Tcl/Tk backed by C. I hoped for something better. It worked, but the Tcl code was just a nightmare.

Then I discovered Glade + C, and it was close to love at first sight. Glade makes it VERY easy to build a GUI that can not only perform setup but also display results in many formats. Some people like Python for the "back end", but I'm still getting used to it. So, I did all the back end stuff in C to interface to whatever the hardware was. Glade makes an xml file that describes the GUI windows, and links variables and callbacks to the C program. So, you click a button on the GUI and it activates a C function that does whatever is needed. The C program can also send data to text boxes on the GUI, change colors of buttons, and even create graphs (I haven't needed to do any of that, yet.)

One minor downside is that editing a completed GUI window in Glade seems to not work correctly, or I'm doing it wrong. I end up making the changes by editing the xml file in some cases.

Jon

In two different places I've worked, the distinction more or less came down to the availability of people skilled in the "do it yourself" C/C++ solution.

Around 2005 to 2007, I worked with Labview at a university. It was nice if it happened to already have a module that did something you wanted, and it let college students without a lot of coding experience develop a moderately complex embedded system for a car, but boy did we ever have to throw CPU cycles at it.

At a different job in 2009-2012, I worked with Vee, for production testing of radio modems. The engineering group had the experience to automate the testing in C/C++, but not the time, so most of it fell to the production techs. They wrote almost all of it, with some assistance from engineering along the way.

Again, it was nice to be able to just drop in a "talk to the spectrum analyzer" block. Vee can write Excel spreadsheets, and the test program would log various critical measurements and pass/fail status for each test into a spreadsheet, one for each system. When all the testing was complete for one system, that spreadsheet was retained by the QA department as proof of the testing. There was also a calibration procedure for each test station that was run periodically, and the cal results were also saved in spreadsheets. So it was nice to have all the "proof" happen automatically.

The goal was for the production guys to be able to assemble new modules into a system, hook it up to the test equipment, push "start", and then walk away while the tests ran. This "hands off" approach required that each test station have a relay box (locally built) to automatically switch the inputs and outputs of the test equipment to the right points on the system under test. At the end, if the system passed, a green light went on, and they could ship that system. If it didn't pass, a red light went on, and they could then attempt to solve the problem, usually by swapping a module and retesting.

It did work and we tested and shipped product, a lot faster than we could have with manual testing. But developing the Vee code and the test stations took a non-trivial amount of work.

General note: with Labview or Vee, it becomes important to (manually) keep a good change log for the program. You can (and should) put the Labview/Vee program under version control, but IIRC it is a binary blob or XML file that doesn't lend itself well to "diff". If you don't write the notes yourself, it's hard to go back later and see what changed between two versions just by looking at the Labview/Vee program.

Fred Brooks says no. :)

This may be somewhat obvious, but as a friend of mine liked to say, "if you automate a bad process, you will have a badly automated process". You first have to figure out how to do a test manually, twisting knobs and swapping cables around like mad, before you know how to automate it. This lets you figure out the minimum set of equipment you need to do all the tests, and it may suggest a good order for the tests. If you don't mind there being some manual steps in the automated test, you can then start writing code. If you want it to be "hands off" testing, then you might have to build a relay box to handle all the cable/probe swaps, figure out how to talk to that box, and *then* start writing code.

If you want to do "hands off" testing, and you have a complex system, it is helpful if your test procedure can be broken into smaller parts that can be executed independently. This lets you rearrange the test order to catch common problems earlier - it's really frustrating to have a ~1.5 hour automated test fail 5 minutes from the end, requiring the whole thing to be redone. Sometimes you will be in production with a module that has a 1 in 1000 failure rate, so it doesn't matter where in the sequence that you test that module, as long as it gets tested. If there is a design or supplier change for that module that bumps that up to 1 in 10, it's nice to be able to test that module closer to the beginning, so you can find the bad ones without wasting a lot of time. Of course, QA and the system engineer have to be OK with changing the test order.

Matt Roberds

I'm in another business (industrial process control), but your query resonates. I find DIY tools, mostly programs that I code from the ground up in C/C++ or similar languages are a more efficient approach than higher level environments. It's uncommon for me to encounter a testbench requirement where a higher level tool will save me time or money. That said, I have a good toolbox of stuff that I've developed myself, such as loggers with flexible trigger functionality and real time processing functions. One of the advantages of DIY is that you develop a toolset that's specific to your requirements.

[]

We used to work with a less fancy graphical toolkit than Glade, it is called XForms

and in the end building a GUI was as piece of cake.

The graphical stuff was well separated from the interaction with the hardware with a client-server separation. The graphical client was sending TCP/IP packets with a very simple format, handled by a server which interacted with the drivers.

I found this approach extremely handy and scalable. We deployed the system for a large magnetic spectrometer currently installed on the International Space Station and the nice part was that every single client we built worked the same way whether we were commanding the hardware on the ISS or on our lab next door, we only had to change the server to which we were connected.

I feel pretty much comfortable with the toolkit I mentioned, but the driver side poses more of a problem. The card we bought does not have support under linux and I'm unaware how long will it take to get it working.

Any chance someone out there has worked with the comedi libraries? Does anyone knows how long will it take to make it support another card?

Al

Interesting. That sort of lightweight library might come in really handy.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs 
Principal Consultant 
ElectroOptical Innovations LLC 
Optics, Electro-optics, Photonics, Analog Electronics 

160 North State Road #203 
Briarcliff Manor NY 10510 

hobbs at electrooptical dot net 
http://electrooptical.net

Hi Matt,

snipped-for-privacy@att.net wrote: []

There's a great advantage in a DIY, you have control. And if the system is well thought through, it can scale well and provide you great capabilities. Unfortunately it takes time and skills, two resources not always available at hand.

This is my impression as well. When you do not want to dive yourself in a lot of details, you can 'easily' hook up an instrument to a pc and, with a good bucket of money, have a measurement done.

Now the issue comes in two cases:

1. the instrument to plug is rather a set of instruments, not at all standard, which have special requirements in terms of timing and precision

1. you need to get more out of the 'simple' stuff from Labview and you are tangled with an obscure layer of intricacy.

So you confirm that VEE might be easier to handle by people who are not necessarily programmers. This was my impression as well and probably we are in the same situation, people with experience in C/C++ do not have the time. I'm wondering if in the long run this gain in time is really a loss.

I agree that a lot of efforts go in reports as well, in the end having a pass/fail report will alleviate a lot of manual work.

We are building units which are maybe few pieces each type, they are boxes for scientific payloads which evolve constantly and very few can be produced on large amounts. Therefore we cannot really spread the amount of time we spend in setting up the testing environment over the large amount of pieces. OTOH many very interfaces to these boxes are extremely common (1553, CANbus, SpaceWire, ...) therefore a lot can be reused among several projects.

We ought to start thinking diagonally and let our testbenches support several projects. Even further, in our system designs we ought to include our testing capabilities and options already available in house in order to leverage the experience gathered in previous projects. That has not always been the case.

I still believe that for many standard interfaces we should start to build testing 'blocks' which can be easily put together. When a new fancy interface come in we should only need to add another block to our testing environment. A sort of expandable testbed.

This is something I learned at my own expenses. I think not long ago this cartoon was posted here:

Another concept from the same guy is the 'second system effect' which may incur in any system you built, whether or not is the second one!

This is the main reason why we built clients that allowed us to tune every knob, parameter, setting to our finest content. Only after a lot of manipulations we were able to script the sequence. We also built our own scripting language (even though not extremely wise afterall) and built sequences out of sub-sequences. It was very effective in the end and we could really master extremely complex sequences (like charging a cryomagnet in a thermovacuum test).

I concur with that. We were doing all this and created a very solid testing environment. It took time, efforts and skills which I'm not sure my current employer is willing to put on the table. But hey, Rome was not built in a day, and considering that I'm from Rome I have some chance to build it ;-)

In the aerospace world the only fact that you are using HW/SW tools from "leader manufacturers" has its own importance, no matter how/if they actually are the best technical solution. I think that it mainly has to do with the supplier chain (very important in AS9100 quality process): usually the manufacturers and suppliers of "leader tools" are big companies that are less likely to disappear in the next twenty years.

That being said, the only drawback of Labview and Teststand is the price; if the price fits the budget, there isn't any reason not to use them. If you need more control you can still develop a custom VI interfaced to your custom code, but at the same time you don't have to waste time on trivial tasks.

In my experience using Labview/Teststand cuts the development, setup and debug times of at least two thirds, often more.

--
Fletto i muscoli e sono nel vuoto.

I know that comedi exists, but I haven't used it before.

In general, here are some scenarios for supporting a new I/O card under Linux, in increasing order of complexity...

The Acme Model 100 input card is really a General Veeblefetzer Model 10 input card with an Acme sticker on it, and there is already a General Veeblefetzer driver. Sometimes you can force the G.V. driver to load and talk to the Acme card in a configuration file. Or, you might have to modify the "find cards" section of the driver to also look for Acme's PCI ID.

There is a driver for the Acme Model 50 4-input card, and you have the Acme Model 100 6-input card. Everything is almost the same, but on the Model 100, you have two more configuration bytes for the card, and get two more bytes per read. Modify the Model 50 driver yourself to handle two more channels. (You might need the documentation on *both* cards to make sure they are the same.)

There is a driver for the General Veeblefetzer Model 31 card, and it works basically the same way as the Acme Model 100, but all the configuration register addresses are different, and it reports Channel

5 first instead of Channel 0. Copy the G.V. Model 31 driver and use it as a model to write the Acme Model 100 driver. (Again, it helps to have the documentation on *both* cards.)

The Acme Model 100 is brand new and nobody has a driver for it or for anything like it. But, Acme is willing to send you the complete programming instructions - how to write the configuration registers, how to read the data, all that. (They might even send you the source code for the Windows driver.) In that case, you sigh deeply and write a driver from scratch, yourself.

The Acme Model 100 is brand new and nobody has a driver for it or for anything like it. When you ask Acme for the programming instructions, their lawyers tell you not to ask again, or Acme's armed IP protection drone will visit your site. In that case, one option is to load the Windows driver in a virtual machine on Linux, and watch the PCI bus accesses to see how the Windows driver talks to the card. You have to do this in a country where reverse engineering is legal - in other words, not in the US. The other option is to buy a different card from a sane vendor. :)

All of the above assumes that you are going to modify or write the drivers yourself. If you want someone else (the comedi developers?) to write them, this is possible, but they will at least need the programming instructions for the card. Note that they usually can't sign an NDA to get these. Some open-source developers won't read any documentation that is not freely available, while others don't mind reading scanned PDFs or photocopied manuals that mysteriously appear in the post. If all you can give them is the instructions, there will be some back and forth as you test the drivers on the hardware.

It helps a lot if you can send them an actual card to work with. If it has a weird connector, also send them a matching connector or cable, so they can tell if the driver is actually working. You might consider sending a simple way to run the inputs or watch the outputs - for instance, for an A/D card, make up a little harness with a 9 V battery and some potentiometers; for a digital I/O card, some switches and LEDs

- something like that.

If you're going to get outside help with developing the drivers, you may want to run it by the boss and/or the lawyers. In my limited experience, if you say "open source" at a US company, the lawyers just flat say no, while European companies are more open to it. (It also depends a lot on what industry you're in.) The main thing to explain is that if you get help with the I/O card driver, the I/O card driver source will become publically available, but that does NOT mean that you have to release the source code for your entire project to the public.

Matt Roberds

Yes, but they can't be completely clueless. In my first example, the people were electrical engineering students, who at least knew a little bit of coding, plus how to use a voltmeter, oscilloscope, function generator, etc. In the second example, the people were production technicians already - they didn't know a lot of coding, but they were familiar with using voltmeters, oscilloscopes, spectrum analyzers, signal generators, etc. (The reason I point this out is that sometimes salesmen try to tell you that "it's so easy that the secretary can do it!" Which is not true.)

My second example involved production of several hundred systems total. There were times when we needed to produce about 10 complete systems each day to meet the schedule. Each technician kept 2 identical test stands running. So yes, your approach will probably be different. :)

I didn't know about SpaceWire, so I looked it up. DB-9s in spaaaaace!

Make all of it from whatever they made the Coliseum out of. That stuff lasts pretty well. :)

Matt Roberds

First off there is no such thing as a silver bullet for software development. Every tool brings with it advantages and restrictions, Labeview has too many restrictions for my taste.

Along the way i spent over a decade as a test engineer and built many unique test fixtures, test equipment, test methods, and so on. Some were software driven some were not.

Nor have i ever seriously used Labview or VEE. Wrote a lot of pure custom test code though.

For some testing true real time performance is required and python cannot provide that. For that matter very few environments can.

?-)

OTOH, 40 years ago, IBM used to make programmers out of secretaries. Some were quite good. One I worked with in the '90s went on to be quite good (getting her degree at night and last I knew rather high up). To be fair, most were more programming "technicians" (coders) and didn't design the software.

The demand for programmers was too large and it wasn't taught in college (CS was in the graduate school when my brothers were in college). They found smart people where they could and trained them.

Of course today secretaries, well, there aren't any. ;-) Engineers do that work now. :-(

My favorite saying about LabView (and other graphical/diagramming environments) is

"Finally, we can write spaghetti code that _looks_ like spaghetti"

IT's worse than that---automating a bad process is just a way to make mistakes faster, just like with tequila, women and revolvers.

The version of this that I heard was that you needed to have a college degree - not sure if it could be in anything, or if they required some flavor of Bachelor of Science. You could then go to the IBM training class for several months and be fairly sure of a job when you got done.

I heard this from a lady who had done it after getting a bachelor's degree in biology. She took the IBM class and then went to work for American Airlines in Tulsa, working on the SAABRE reservations system. (I think this was in the early to mid 1970s.) She worked there until the late 80s or early 90s, when she was able to go into business for herself, doing something totally unrelated to computers or biology.

What I was originally getting at is that the more exuberant Labview/ Vee/Rational salesmen try to tell you that people with no training at all can now write programs, which I disagree with.

My dad worked at a Very Large Corporation and told me that when he started there in the late 50s, if you were a guy and knew how to type,

*admitting* that you knew how to type was seen as a social error. Important People had secretaries to do that kind of thing for them. This changed as computers and especially e-mail became available, but it was still sort of the culture there in the mid-90s that if you needed to send a (paper) letter outside of the company, you got a secretary to type it for you.

Matt Roberds

Nope. They took secretaries right out of the secretarial pools. Not all, of course. They had to pass the aptitude tests, have recommendations, and all.

There was a dire shortage of programmers. They took people from wherever they could get them. New hires may have been from colleges but I knew many secretaries in the '70s and early '80s who were taken directly from the secretarial pools and moved into programming school. Technicians had the same opportunities and many of them took them up on it. One issue, IIRC, is that if they washed out of the school their old job may not be waiting for them but that may have been incentive. I lived in Po'k (where the training center was) at the time and it was happening all around.

Understood but what do you expect from a salesman? They also said that anyone can design logic with (fill in your HDL language). I've seen what C programmers do with HDLs. ;-)

That's still true today, but they don't type. ;-)

Chances are that if snail mail is required, today, it's pretty formal communication and it's probably the legal department doing it. ;-)