Embedded Scripting -- Tcl? Lua? Thoughts? Suggestions?

I think Blue is better than Red.

Bagels beat donuts hands down! (unless you're *not* in NYC; then damn near ANYTHING beats a bagel!)

Chicago Pizza beats New York Pizza.

42

Connery over all others (Moore, Brosnan, Lazenby, Dalton, Craig)

Anything with butter.

As to *your* question...

What sorts of things does the customer expect to *do* with those scripts? And, (besides space) what other constraints are you needing to place on those actions? E.g., any timeliness constraints? Performance? Numerical precision? etc.

Who is the intended user: *just* the client? Or, will these things be exposed to users?

What sorts of things do the scripts interact with (displays, etc.)? E.g., do you need to be able to manipulate a GUI without constraint? Or, are you just serving up text and/or canned dialogs?

What sort of user input device (keyboard, keypad, etc.)? Other I/O's?

I opted for a (bastardized) Limbo as my scripting language. But, I use it to generate "real" applications and not just automate some preexisting actions. However, it won't fit your memory requirements.

Can you tolerate something as "crude" as a bastardized BASIC? I.e., add "product specific" hooks to it to address some of the special needs (displays, etc.) at a "native" level. A BASIC interpreter/compiler can be really small.

The same can be said of C interpreters, mini-compilers (JIT). But, you'd be targeting a different class of "developer" (user).

--don

I wish everyone was as mature and rational! I'm personally sick and tired of scrotty little scripting languages that grey like Topsy until nobody, including their originators^H^H^H^H^H perpetrators, understood them anymore.

I haven't used it myself, but Lua looks like it might be a good starting point, particularly since it is embeddable in/with C.

ISTR there being an embeddable JavaScript (not Java!) interpreter, which might have the benefit of more familiar syntax.

I used Lua in a project a number of years ago, and would use it again if I needed scripting. It has a few key advantages over TCL. It is a nicer language, it is quite easy to integrate in a C project (it consists of just a few C files), and it's easy to make your own functions callable from Lua. A disadvantage is that there is only one number type, which is floating point by default, but you can change that to integer if it suits you better (and it's certainly better than everything being a string, as it is in TCL).

The product will have a character display, and it'll store text files for later retrieval. It has a measurement thingie (I don't want to get too specific -- but it makes measurements) that'll cough up a number when it's actuated, and it has a sea of (well, around 100) buttons that the user can press.

My intention at the moment is to make the box this big dumb thing with some system code that knows to call a function for each measurement and button press (i.e., when the user presses button 1, the function named "button_1" will get called if it exists). Then I'll encode the actual functionality as a bunch of functions in the language-o-choice.

The nice thing about this is that if a user comes up with some desired customization (which we expect, as we're replacing an existing product for which this was common), we can either tell them to start coding and have fun, or we can charge them $$ to code the thing up to their specifications.

The real driver for wanting to do this as a scripting language is to make it easy for the user to customize. I'm imagining that some users won't want to touch the customization with a 10 foot pole, some will be happy to customize a button or two, some will get their feet wet and then call us, and some will go whole hog with the thing, and do entire systems without us even knowing it's going on.

--

Tim Wescott 
Wescott Design Services 
http://www.wescottdesign.com

Oh, and -- we've booted the idea of BASIC around, too. As far as I'm concerned, the first letter or two of BASIC stands for BArf, and I'll have to write scripts for the thing. So it's not high on our list of preferred options.

--

Tim Wescott 
Wescott Design Services 
http://www.wescottdesign.com

I did a tiny amount of Tcl 10+ years ago, but that was on a Linux desktop with GUI. For a while, sometimes Tcl was used as sort of the Linux equivalent of Visual Basic - for when you wanted a simple GUI but didn't want to spend weeks screwing around with X/(GTK|Qt)/C++. I've never tried embedding Tcl in anything.

I *have* used embedded Lua more recently. Wireshark (a packet sniffer, formerly called Ethereal) understands lots and lots of packet formats, but if you've come up with your own weird thing, it just gives you a hex dump of the payload. You can write your own packet parser in Lua and Wireshark will run it on the payload for you. The basic application is to make the packet more human-readable. You can also use it along with Wireshark's native ability to trigger on or sort by certain data fields to only log packets of interest.

The thing I found odd about the choice of Lua for this application was that *it didn't have any bitwise operators*. You couldn't do stuff like "if(packet[3] & 0x04) { /*...*/ }". I did a lot of multiply and divide by 2, and tests like "if(packet[5] > 127)" to see if the high bit was set. Also, IIRC, Lua's default numeric type was float.

This was a couple of years ago; maybe "full" Lua could do bitwise back then and Wireshark was only implementing a subset, or maybe Lua couldn't do it back then but can do it now.

Wireshark also had the option of writing a packet dissector in C, and then compiling it into a library (.so or .dll depending on platform); this was said to give you faster decoding. Most of the protocols I was looking at sent packets at no more than a few dozen Hz; I think one of them topped out around 1 KHz. The Lua scripts didn't seem to have any problem doing "live" decodes on a new-in-2007 PC.

I know I've seen the option to embed a Lua interpreter in something more "embedded", possibly U-Boot. Lua's Wikipedia page lists a lot of places where it has been embedded.

I will give my standard advice: ship a few example scripts with the product. You can assume that people know how to use arithmetic operators and things like that, but the details of how to get data out of and back into your hardware will be unique to your implementation. I think Wireshark had a limited example in the manual, but since it's popular, I was able to find additional examples online. Most applications won't be able to count on that, though.

Matt Roberds

Presumably, your goal is to make life pleasant for the *customer*, not

*yourself*! :>

I offered BASIC as an option as you can write one in a few KB and is easily understood by "non-programmers". I've worked on instruments where (my) customer (e.g., chemist) was charged with scripting behaviors that *his* customers would blindly invoke (e.g., to determine if the hospital patient has Hep-C, HIV, etc.). So, lots riding on (my) customer "getting it right" as I'm not looking over his shoulder to verify he's considered every exception, boundary case, etc. (All I have to do is ensure the language is well-behaved and has no "gotchas" that are likely to trip someone up two-levels removed from me!

Are you sure you're not thinking of tk?

I looked at a bunch of OTS languages when I had to settle on a scripting (i.e., "application") language. Many are far too cryptic for casual users (I wanted to target "real programmers", hobbyists and "homeowners" -- lots of competing criteria!).

Tcl has some features that are troublesome (e.g., upvar, uplevel) even for "conventional" programmers. Imagine trying to explain them to

*casual* programmers/users.

I didn't like the fact that you could mix and match types without anything alerting you to the fact that you're trying to do something "wrong". E.g., if (5 == "5")...

Likewise, I thought the ability to perform multiple assignments (with a *single* assignment operator) would cause problems. E.g., x, y = 1, 3 -- how would you "read that over the phone"? "x and y get 1 and 3, respectively"? "x gets 1, y gets 3" (then how is that *read* different from x=1 y=3)

Likewise, the potential for confusion when arguments are omitted from function invocations. I.e., it doesn't do much to help protect the programmer from "little (painful) mistakes": "Oh, crap! Foo() takes *3* parameters!)

I've been surgically removing/rewriting features of Limbo (my scripting choice) to try to ensure users can "get it right" without scratching their heads to figure out what's wrong with their code. Too many multiple character operators, etc.

I've been around scripting languages since I built my first one 35 years ago, and I don't think there's a really good solution for your application yet... based on my definition of "good". The problem is two-fold:

• Languages whose parser is very small tend to be hard to read & write (Lisp, Forth) and tend not to have any static type checking, and

• Scripting languages tend to be garbage-collected, which adds to the memory requirement and introduces unpredictable delays.

What I think would work best for embedded application :

• has a static data model (just global and local variables, but no dynamic allocation),

• has a highly humanised syntax that pre-compiles into a byte code format on a host computer, where it can be statically type-checked, but is still easy to execute on the target.

I've been thinking about (and even started) writing such a beast since, oh, about 1995 I think. I actually set out to write an event-driven (actor) language like this, for high-school robotics programs, back when my kids were starting high school. I really don't think it's a very difficult thing to do (just a weekend project for me), but I'm not aware that it's been done. I'd like to hear if anyone knows of an existing candidate that meets these design criteria.

Clifford Heath.

tcl is the language, tk the toolkit - they're very close friends.

The only real use for multiple assignment is swapping values without using an explicit temporary: x, y = y, x

Calling a 3-argument function with 2 arguments should return a function that takes the remaining argument :) (I.e. currying)

Where can I read more about Limbo? Is this the language from Inferno? Hmm, that looks a lot like what I described in my other message :)

They finally added integers to Lua, as of version 5.3, released earlier this year.

OK, embedded enterpreters were one of my favorite subjects for a while, so here goes.

1. The smallest SD cards available these days are in the gigabytes, so

100k is nothing. The interpreter's RAM footprint is probably of more concern on a small embedded system. How much ram are you willing to use? I also have to ask whether you're going to be the one writing the customizations at the user's request, or if the customer him/herself wants to write customizations. You should probably stay away from weird or difficult languages if the customer is doing the customization.

1. TCL and Lua are both quite easy to embed but Lua is a more modern and generally better language, so I'd pick it over Tcl. Lua has more of a programmer community these days (lots of mobile games and apps embed it) and there's a high performance compiler (LuaJIT) available if you want that.
2. However, from my language snob perspective, Lua and Tcl are both somewhat distasteful. Also, you'll probably get much better user uptake from Javascript (another distasteful language) since so many people know it. Javascript will need a lot more machine resources and I don't know what the embedding side is like, but you might want to consider it if you have the CPU and RAM for it.
3. Other alternatives include Lisp dialects--I used to be into these. You might look at GNU Guile (Scheme), Hedgehog Lisp (sort of a functional subset of Scheme), and Lysp
   formatting link
   I guess each of these have drawbacks.
4. You can use Javacard or J2ME in an embedded (non-Linux) environment but I'm not sure why you'd want to. It requires external compilation tools and a somewhat bureaucratic approach to development. I mention it because you brought it up. Main thing it tries to supply (with some success but not total) is security against untrusted, potentially hostile scripts trying to mess with the surrounding application or with other scripts in the same container. I don't know if that's an issue for what you're doing.
5. You can also consider Python though its embedding challenges are harder. I think it can be used without Linux but I've never done it. If you go this route it's probably better to write your outer application in it, calling C modules from Python as necessary, rather than the other way around (writing the outer app in C and loading Python into it).
6. If you're -really- resource constrained or your scripts need realtime guarantees, there is Forth, but I'd stay away from it if you have enough cpu and memory to run something like Lua.

Of all the above, based on the limited info you've given, the quick answer is probably "Lua unless you can afford Javascript or you're a language geek and like Lisp". If you can say more about your application and your hardware constraints, that might help narrow things down.

Yes -- saving a temporary. But, it's a feature that can easily be abused (no direct consequences of doing so -- for the *abuser*).

Limbo suffers from this same problem: x, y := 3; sets *both* x and y to '3'. x, y = 2, 3 is an error.

To it's *credit*, the lua style of multiple assignment can be accomplished using tuples: (x, y) := (2, 3); though tuples are far more useful for other things!

The vita nuova web site has has the two seminal papers on Limbo (A Descent into Limbo; The Limbo Programming Language). I htink there are two (?) texts that also address the language/"system" (as Limbo is intimately tied up in Inferno... where the language ends and the system begins can be debated).

But, it's more fleshy than a scripting language. E.g., contains communication primitives *in* the language, seamlessly supports RPC, multitasking, demand loadable modules, exceptions, GC, etc.

So, it's a good language to develop *real* applications in addition to just hacking together simple little "basic" scripts.

The biggest problem is all the syntax that can easily intimidate a casual user.

E.g., i = 1; i := 1; behave similarly but subtly different. A casual user might be stressed to discover his problem boiled down to a missing/extra colon! Or, "" vs. "=>". Niggly details. I, for example, found it frustrating that you can't declare const lists. Or even *initialize* a list in it's entirety!

(the current compiler leaves a lot to be desired in terms of localizing errors)

FWIW, I think the current license terms are much more generous than the *paid* license I acquired (change in their business model?) So, you can download the environment -- BSD, Windows, OSX, Linux, Slowaris, Plan 9 (of course) -- and play with it. At one point, you could run the VM in a browser (not sure if that is still supported).

I'm trying to stick with an ARM M4 cored processor that has 256kB to

512kB of flash, and maybe 64kB of RAM, and a 72MHz CPU speed. Everything can run at "human time", so delays of no less than 50ms are OK. There is an issue that a really hot operator shouldn't be able to overrun the machine, but the buttons are actuated by a wand (it's complicated -- they just are), which should slow things down.

Even if I loved Lisp (or Forth) more than anything else in the world, I'd still use Lua if it meant better user acceptance. I'll consider Javascript for that very reason, particularly if I can find an open- source interpreter that seems serious and well-supported.

--
www.wescottdesign.com

Hmm, ok, I think Javascript is not practical in something that small. Lua can work but will be a little bit cramped. How much of that flash and ram will actually be available to Lua (i.e. not needed by the non-Lua code)? How complicated do you expect the scripts to be? Are you ok with cross-compiling the scripts? That means loading a script by compiling it to bytecode with a PC application, then loading the bytecode to your embedded board, instead of including the compiler in the board so you can talk to it with a terminal.

has some info. TCL (especially old versions) is probably smaller. I know there are some bigger Cortex M4 parts, like ST has one with 2MB of flash and 256k of ram -- do they cost too much or something?

You'd have to use a bigger processor, but it still may be worth it. I worked on something a couple years ago that could have been done in C on an 8 bitter, but we used an embedded ARM7/Linux system with 64 meg of ram for development convenience, adding maybe $5 to the hardware BOM which was acceptable for what we were doing. I think it was well justified by the flexibility it gave us. We were able to ship a better product faster.

I don't know anything about JS embeddability. I've only programmed it at the application level. I know that various applications other than web browsers do embed JS, but it may be a lot more complicated than using Lua.

Will your scripts have to process much non-numeric data like strings and complicated structures? If not, maybe there's some BASIC dialect that your users would find comfortable, especially if they're hardware types who have been around for a while.

Hmm, this may be of interest:

It claims to run JS in a 64k ram part like yours.

I just came across it and don't know any more. I don't know what dialect of JS it uses. It could be limited in a way that frustrates your users who are used to programming browser JS. I'm sure that fancy modern JS implementations like V8 are way larger.

Now that I think about it, all the GUI stuff seemed to be a combination: Tcl/tk. I guess you can use Tcl on its own without tk.

I don't remember ever crashing Wireshark by screwing up my Lua scripts. Things just wouldn't decode / display on the Wireshark GUI correctly. If that happened, I could always save the captured packets to a file, fix my Lua script, reload the file in Wireshark, and have the script run again.

The Lua interpreter also supported a simple printf-like function that would just display on the Wireshark console and not in the GUI. Wireshark would open a separate 80x25 text window when you used the console. I found this useful for answering "am I pulling the right bytes out of the packet or not", and *then* I could work on pushing those bytes to the GUI in a useful way.

Matt Roberds

My limited experience is that Lua is a more "normal" language, not that Tcl is hard, it's just going to look weirder to most users, and Lua (in its smaller forms) has a smaller footprint, and is a bit faster than Tcl (whether either of those is an issue is obviously application dependent, but Tcl's "everything is a string" metaphor hurts performance some).

Both interface fairly easily to your application, and allow you to add your own commands/functions easily. Both are actively supported, developed, and have widespread use.

If you eventually move to Java, Tcl does have a JVM version, which might be useful for migration purposes.

I really don't think you'd go far wrong with either though.

My vote to Jim-TCL or LUA.

--

Tauno Voipio