Embedded system user interfaces and configuration management

[system configuration]

Ad hoc solutions usually exist because systems are rarely *designed*, from scratch, with anticipation of future needs, dependencies, etc. in mind.

Consider the physical resources made available to this activity (is it a few bytes of NVRAM? Or, a chunk of space on a disk??). It's hardly worthwhile to set aside a lot of resources for configuration if the only thing that needs to be "configured" is the time-of-day clock, etc.

My current approach is to use a RDBMS to store all configuration information. The RDBMS can then enforce "who" can set "what". Server-side stored-procedures are built to handle all of the dependencies/"consequences" of changes to individual "settings". This helps document these dependencies in addition to ensuring that "all the right things" get done each time something is changed. It also makes it really easy to store/restore/update configurations as everything is in one place! And, allows the developer to implement the user interface to that dataset wherever he/she wishes (i.e. as a centralized "configuration program" or in discrete/distinct "settings panels")

Of course, very few projects have the resources available for this sort of approach :<

--don

No Starch Press is coming out with a book called "Linux Appliance Design". I am one of its authors.

The book addresses many of the issues you've raised. It is not a typical "embedded Linux" book because we start with the assumption that the system boots and bash is up and running.

The book's sample appliance is "Laddie", an alarm system that uses the five status lines on a parallel port as inputs from the alarm system hardware. (The book includes a bootable CD that turns your PC into the Laddie appliance.)

The basic layout and chapter list:

- Appliance architecture -- basically what you've posted asking about

- How to talk to a running daemon -- use RTA to make all of your daemons look like a PostgreSQL DB and all of your UIs can talk to the daemons using the PG client libraries. Also, a table is a natural way to view data.

- Responding to appliance events -- we built a daemon to watch syslog (or other event sources) and to respond when certain events occur. Our logger is called "logmuxd" because it can multiplex output to many destinations.

- A daemon to make system data look like it is in a DB -- we wrote a little utility to give a "DB" view of all the system configuration data. We use this, for example, to make the resolv.conf look like they're in a DB.

(All of the UIs below use the PostgreSQL client libraries to talk to the daemons listed above.)

- A Web UI -- PHP, CSS, AJAX

- A Front Panel -- HD44780 two line LCD display, also an AJAX powered web page to look like the front panel

- A CLI -- a simple command line interface using Lex/Yacc; but simple!

- A FrameBuffer UI (with LIRC) -- intro to IR and how LIRC works. Intro to framebuffers and the choices you'll need to make.

- SNMP -- three chapters for SNMP: what is it; how to write a MIB; how to write an agent.

The book is still in edit, but if you're interested I'll be glad to send you (or anyone else who has read this far :) ) the Laddie CD which has all of the source code for the appliance. You might at least want to check out RTA in the projects section of the web site.

Please let me know if you want a CD.

Bob Smith

Yes, this sounds like a different approach then the solutions I've seen up to now - although the first impression from what you wrote here is that it might be a bit heavy for smaller devices with only a few MB's of ram and flash. I am absolutely interested in receiving the CD you mentioned to take a look at your system.

In my original message I also wrote about the problem that a lot of knowledge about subsystems is often spread out through the platform - for example, if a service is configured via the web interface, there is often a lot of specific code hidden in the CGI applications (or even in javescript!) to validate the data, so both the service and the web front-end both need to have knowledge about the data: valid ranges, dependancies, and such. Does your solution address these problems as well ?

Thanks,

_Ico

--
:wq
^X^Cy^K^X^C^C^C^C

Very true, this is one of the reasons for my concerns. But this problem is also hard to address, since things like this just tend to grow and evolve - a company starts with product X version 1.0, with only one feature, the next version does only a little bit more, needs a tiny bit more configuration, et cetera. Thus another monster is born.

What I am actually looking for is a general philosophy, a setup or framework which is flexible enough to grow with the requirements, but which will not grow exponentially more complex when functionality is added to the system.

An analogy would be something like using finite state machines in (embedded) software: Without FSM's, it is very well possible to create complex systems, but each time a function is added, there is a big risk of creating more and more spagetti-code to solve all kinds of dependancies between functions. Coding an application using state machines often looks like a bit more work in the beginning, but tends to be much more modular, extendible and understandable, since every state has well defined bounds, inputs and outputs, and can be easier comprehended, debugged and reviewed. State machines are no magic, you can still mess up ofcourse, but they are a proven technique to keep things modular and maintainable.

Yes, storing all your data in one place has advantages, but also it's cons. One of the problems with this is that there is knowledge about your configuration in more then one place: there is some kind of subsystem that can be configured through the RDBM, but this RDBM also needs to know what data this service needs, what types it's properties are. The frontent (cli/web/whatever) still needs to know how to present those properties to the user, etc.

For example, how would you solve the case ware a configuration item for a service needs to be selected from a list by the user (lets call it an 'enum'), lets say for selecting a compression protocol for some audio stream. Valid choices are PCM, DPCM, MP3 and uLaw. The service is configured through a configfile, where this choice is written to.

Something like this is quite hard to implement properly througout the system. Ofcourse, the service itself knows what the valid choices are, since it needs to parse the configfile and act accordingly. But does your RDBM knows what choices are valid, and allow only these to be stored ? If so, you have knowledge about this enum in two places at least. Assuming a web interface is used for configuration, the user should be presented a list to choose the audio format, so the web application will need to have knowledge about this as well, which makes three duplicates of the same information.

Nightmares really start when settings depend on each other - imaging that for some technical reason the MP3 format can only be used when the samplerate (another configuratble setting) is 22050 hz or higher. This is something the service itself knows, because it can't mix lower sample rates with MP3. But is this something your RDBM needs to know as well ? And how can I present this to the user, since I don't want the user to be able to select MP3 when the samplerate is 8Khz, but I also don't want the user to be able to set the samplerate to 8Khz when he selected MP3 as the encoder. And the hardest part would be to inform the user *why* he is not allowd to set the codec: "Warning: setting A is not compatible with setting B, change setting A first if you want to set B to X !"

Of course, it's all doable - you can just hardcode these dependancies in three different places, but things just grow and get buggier when more features are added when you have to maintain code in three different places to support one single function. "Don't repeat yourself" is a nice philosophy, but is hard to get right!

Shoot the Marketing Staff! :>

Change is inevitable. How you plan for change is what separates the men from the boys :>

Exactly.

Moreover, you can build small FSM's which are *entered* from a larger FSM. Also, you can have multiple FSM's running concurrently and let them "talk" to each other. I.e. the "state language" becomes just another "scripting language".

I store data in exactly *one* place. Part of good database design is never storing duplicate data -- as this would lead to inconsistencies between the datasets.

Each "service" that needs configuration data formally queries the RDBMS for that data. A service is free to *cache* that data locally (so that it doesn't have to keep asking the same query over and over again each time it references that configuration parameter -- i.e. "what color should I make the error messages?").

However, when data in the database is changed, the database issues a signal() to each process that is a *consumer* of that data saying, in effect, "your configuration has changed; please reread the database". This can result in a flurry of queries at each configuration update. But, that's essentially what happens when *any* product's configuration changes -- everything that is configurable needs to know about those changes.

If you store the configuration in a global struct, then there is no cost to the consumers to retrieve this data. But, then you have to worry about which consumers may be actively examining the data when you change it and implement mechanisms to safeguard it's integrity during updates (e.g., with mutex's, etc.). In my scheme, each service knows that it "must" retrieve new data but can do so when it is convenient for that service (a more cooperative, less authoritarian approach)

(trying to avoid getting too deep into database theory, here so I'll just casually describe the mechanism used)

A table (everything is a table!) is created with, for example, two columns ("fields"): the first may be a textual description of the "choice" -- i.e. "PCM", "DPCM", "MP3", "uLaw" in your example. The second (and possibly 3rd, 4th, 5th, etc.) may be a special "code" that is what your software actually *uses*. For example, it may be a magic constant (#define). Or, perhaps a key parameter governing the compressor's operation. It is *whatever* you want/need it to be INTERNALLY. (if you have

5 "variables" that need to be set based on the choice of protocol, you can store the five values for each of those N different protocols... and stuff them in the corresponding variables when that "choice" is selected)

Elsewhere, you have some configuration setting stored in the "main" database (which is also a *table*). The tables are

*linked* in such a way that the value for that "setting" can only be one of those listed in the first column of this other table (just like you can say a field can only contain text, or an integer, or a float, etc., you can also say it is constrained to one of these N values)

The DBMS will *ensure* that this setting never gets any value

*other* than one of those N values.

Your configuration program will query this small table to get the list of choices to display to the user. Based on what he picks, that choice will be stored for the "setting" in question. And, the five (?) other internal parameters will be available for you to copy to whichever "variables" need initialization.

Exactly! To rephrase your statement in *my* reality:

The service knows what the valid choices are -- but, relies on the RDBMS, instead, to *remember* what those actually are! And, what it should *do* for each of those choices (e.g., "set magic_number to M_ULAW if uLaw is selected, etc."). Thereafter, it *relies* on the RDBMS to remember those choices since it will NOT!

And, the configuration program, instead of being manually crafted to agree with the list of choices hard-coded into the application, now ALSO relies on the RDBMS to tell it what those choices are. I.e. adding a new choice to the RDBMS "magically" causes that choice to be available in the configuration program.

See above :>

Perhaps you now see the beauty/elegance of my approach? The web application queries the RDBMS as well and builds it's form based on the results provided *by* the RDBMS.

With careful planning, you can even reduce configuration to a simple "boiler-plate" that is parameterized by the name of the setting to be updated and it fetches all of the choices for the user, etc.

At times, this is a "forced fit" -- some things don't naturally want to fit in this approach. But, you can often rethink the way you specify the data in order to make the fit more intuitive.

E.g., you can design an address book in which you include a place for each person to list their *children*. Or, you can design that same address book with a place for each person to list his/her *parents*! The latter case *seems* clumsy, at first... but, if you think about the implementation, it is much cleaner -- everyone has *exactly* two parents (for the most part :> ) whereas a person may have 0, 1, 2...

10, 12... *children*!

I handle this with an expert system. It embodies this knowledge in it's ruleset. So, it "knows" what is "prohibited". It also knows *consequences* of choices. E.g., "if the user selects MP3, *force* the sample rate to 8KHz" (redefining your criteria, above)

(sigh -- I had hoped not to get dragged into the bowels of this but...)

When the RDBMS is told to update a record (a line in a table), there are "rules"/procedures that are invoked *by* the RDBMS before and/or after (based on how they are defined) the update. So, I can define a trigger that consults the expert system with the *intended* data to be stored in the table. The expert system can then tell me "go" or "no go". In the first case, I then update the table and return "success" to the user (the user in this case is actually the configuration program;

*it* can then figure out how to convey "success" to the HUMAN user). In the latter case, I abort the attempt to update the table and return "failure" to the "user".

In anticipation of other "what if" scenarios you might consider... Each of these potential updates may be part of a larger RDBMS "transaction". Think of them as statements in a subroutine (?). If any of them signal failure, the transaction can automatically be "un-done". As if it *never* occured -- including all of the steps taken (and "successfully" completed!) up to this *failed* step in the transaction.

The beauty is the RDBMS handles all of this for you -- ensuring that *nothing* ever sees the intermediate results of these changes, etc.

I have implemented this in a very heavyweight fashion. (I currently have the resources available to do so) But, I think the same approach can also be applied to a leaner environment. After all, it is just rewrapping *how* you organize your tests and data. It need not be stored in a "real" RDBMS... just something that you *treat* as an RDBMS! Likewise, the "expert system" is just a mechanism that you *treat* as an expert system...

--don

Yes, I've considered that more then once, but since I'm mostly working as a freelancer I'll leave that to the employees; they have the first right, ofcourse.

[ ... ]

Thank you for your very detailed explanation; I agree with most of your approach as this seems like the way to go indeed. This setup looks a lot like the last systems I worked on, but instead of a RDBS we used a simple database we rolled ourselves. This database had knowledge of data typms and valid values, although the logic - the decisions you make in your 'expert system' - was still often in the wrong place, like using javascript for form validation. You can't get everything right the first time. Or the second. The twentieth. Whatever :)

Since I have the chance to make a grand new start for a new project - with a reasonable budget and timespan - I want to make a good start this time. I fully agree with the philosophy that all knowledge should only be defined in only one place, althoug I still have a small problem with using the central DB: there's still a separation of knowledge between the database, and the actual service using this data.

I would like to take this design one step further: what about distributing the database and expert-system logic between the services actually implementing them ? This would require some kind of system-wide inter-process-communication channel (bus?) where services (modules?) can be connected and disconnected on the fly. Every service announces itself on the bus, and can be queried by other subsystems about it's properties, methods, dependencies and such. Modules broadcast events over the bus when something interesting happens, and other modules who know about these kind of events can act upon them. Attaching a module automatically makes it available in the user interface, et cetera.

This would make a much more modular and reusable setup - the developer can enable only the modules needed for a specific platform, and literally *all* information about a module is contained in the module itself - so no need to configure the central database.

The actual implementation of a module interface could be something like a shared library which handles communication with the bus, and provides the necessary hooks for handling configuration setting/getting and event handling/generation. A developer can use this library as part of an application to make it bus-capable right away, or simple wrappers can be created around existing services/daemons which simple create configfiles and restart/reload daemons.

I think this pretty much describes the architecture I'm looking for. I've made some proto's trying to implement something like this, but I tend to get stuck on growing complexity on the bus protocol, data serialisation, etc. I learned from my latest experience that it might be a good idea to use a lightweight scripting language next to plain C to handle the business logic - something like Lua would be feasable, since this behaves quite well on smaller embedded systems which are low on memory.

Anyway, altogether it still is a complex problem, and I'm afraid I won't get away with a simple solution :)

Thanks again for your sharing your ideas,

_Ico

--
:wq
^Xp^Cy^K^X^C^C^C^C

The advantage to doing this with server-side triggers and procedures (i.e. *in* the database) is that it ensures that *everyone* plays by the same rules. I.e. if *any* "program" tries to change the decoder format to MP3, then all of the consequences/prerequisites of this change are enforced *by* the RDBMS. I.e. instead of configuration data being "passive", it appears to be *active*. (e.g., like wrapping monitors around everything!)

Yes. You have to treat the configuration data as "first class objects" and not just casual "settings" private to a particular application. I.e. you need to think about what they actually represent and the consequences/prerequisites/etc. of all that.

The advantage, here, is that other "applications" are now free to use this information to tailor their own interfaces to those applications. E.g., if you have a mixer application, it can define it's interface to your decoder to be inherently compatible with the format of the data that it produces (i.e. if I know the decoder is processing WAV/RIFF/AIFF files, then it would be silly for me to pass MP3's to it and require *it* to convert those prior to processing them... (this is a forced example :< but hopefully you can see where I am headed)

The discipline is one of "figure out what the data model needs to be for the "application"; then, *move* that data model into the database (instead of keeping it "private" to the application)

[of course, you can also let the RDBMS enforce ACL's on that data if you really want to lock it up...]

I think the problem you will run into there (er, *a* problem) is disciplining yourself to implement the same sort of consistent interface to each "module". And, then each module needs to have many of the mechanisms that I have delegated to the RDBMS to enforce (i.e. notifying consumers of that data that the data has changed; enforcing ACLs if you don't want certain "users"/services to be able to manipulate that data; etc.)

And, the power of transaction processing is REALLY hard to implement in a distributed scheme. I.e. if I want to make changes to several different "program configurations" as one "block" and *know* that: + either ALL changes are made or NO changes are made (so that the "changed" configurations remain in a consistent, "agreeable" state -- you haven't changed one program and then failed to change some other *related* program!) + no program sees intermediate results of this "transaction"

I think it would depend on the level of discipline you have (and can impose on others living in your framework). I have to deal with other folks adding to my system. So, I am keenly aware that everything that I put in place to make it easy for folks to do things "the right way" will work to my advantage. If folks feel it is easier to just store two bits of configuration information in a file somewhere, then laziness will have them doing that instead of adhering to these conventions.

You just have to gamble as to how much risk there is for your system to be "ignored" or circumvented. And, the costs that such acts bring to the system overall.

E.g., with the RDBMS, "I" can backup an entire user profile for the user. If someone fails to use my framework to store configuration data, then they will have to create a "backup my configuration" program *and* deal with irate users who

*forget* to manually invoke this tool when they are backing up the rest of their configuration (then, let that third party explain to the user why their application is the only application that "doesn't work right" in this regard :> )

Good luck!

--don

This reminds me of corba, did you have that in mind? You could try a web search with these keywords: corba, data configuration, container/component.

Ico wrote:

The RTA library is about 60 KB when stripped. Add another K or so for each table you make visible to the UIs. (BTW: "table" means an array or linked list of structs. RTA make _YOUR_DATA_ look like it is in a DB. RTA is an interface, not a real DB.) You might get a better of RTA by trying the live demo. Remember, you're talking to a process, not a DB. Click on "mytable".

I am absolutely interested in receiving the CD you

My real email address should be shown. Send me your postal address and I'll mail you a CD. Or, give me an FTP site where I can upload the 30 MB ISO image. You are more than welcome to pass the software around.

In a way, yes. All of the UIs use a PostgreSQL library to read and write data in the various daemons and servers that form the "processing" part of the appliance. When you start a project you carefully and very deliberately define the tables that are visible to the UIs. A division like this lets the UI experts build the UIs without too much knowledge of how the appliance daemons work. The UI codes can test their UI against a dummy PostgreSQL DB, and the real appliance coders can test what they offer the UIs using bash and psql.

The other piece is that RTA lets you define triggers for both read and write. A "trigger" is the DB term for a callback subroutine that is called before the read or write. You can put sanity, security, or "do it" kind of code in the write callbacks. This helps security since it is up to the daemon to protect itself from bad data.

If I lose my mind :) and commit to a second edition, it will contain additional chapters on: - How to update code in the field - How to secure your appliance - How to document your appliance

Bob

Well, Corba would defenitely fit in here, but I think it would only solve a small part of the complete problem. It could provide the communication between modules, doing all the nasty work like the data serialization/deserialization and handling the network stuff, but it would still be only the lowest level of the complete architecture. Most of the complexity would still reside in the system which will be built on top of this RPC layer.

I think any RPC-ish solution could do here, and there are many available alternatives: corba, soap, xmlrpc, json-rpc, dbus, etc. Another poster in this thread even uses the Postgres network protocol for communication between modules, an interesting approach!

--
:wq
^X^Cy^K^X^C^C^C^C

Yes, and this might be one of the defects of many implementations I have seen up to now: configuration and settings are often considered a side effect of the modules/services doing the hard work - the whole system is designed 'code driven', not 'data driven'. The developers focus mainly in getting their service to do the job, and consider configuration a minor detail that can be finished as soon as 'the real work' is done.

Very true!

Yes, this is also one of the often overlooked prerequisites, although this is a very common issue. For example, consider the user changing the IP address and default gateway of a system - they should both reside in the same subnet - which is something the system should enforce - so the only way to change these settings is to do it in one transaction.

Which is always the case with any 'framework' you create for others to use. Other developers will always try to find a balance between how much convenience and how much grief they will get from using such a framework: if it is too strict, people will hate it for limiting them in their work, and if you make it too 'free', things get messy because everybody can do it their own way, lacking coherency throughout the system.

Yes, choosing a RDBMS seems to be a smart approach indeed, your ideas gave me enough to think about for the next few days. Thanks again !

I'll need it :)

--
:wq
^X^Cy^K^X^C^C^C^C

Exactly. Development should be *specification* driven. Figure out what the thing has to *do*. Figure out what it looks like TO THE USER (of which, configuration parameters are a key role), figure out all of the boundary conditions and Things That WILL Go Wrong (TmReg). *Then*, start coding it.

I.e. once you know what it is supposed to look like and how it should behave, you *should* be able to write a User's Manual and have THOUSANDS of them published -- you should have put tht much forethought into it :>

(I write the User Manual *before* I write any software -- though I don't usually have to publish it that early! :> )

Exactly. Or, IP address and netmask -- some values and combinations are "not legal". And, even if the values chosen *are* legal, you need to make sure that they are actually changed concurrently. There are numerous similar examples. And, other, more convoluted examples whereby parameters from different subsystems interact, etc.

Being able to put the "business logic" in the RDBMS and let *it* enforce those rules keeps everything consistent. I.e. *nothing* (meaning "consumers") ever *sees* any of that data until after the RDBMS has blessed it and incorporated it. It's not even

*possible* for other things (programs) to see it since the only place that stores it is the RDBMS! (if you have a second copy of it in a file someplace, then someone could elect to use that copy -- either from ignorance or laziness. unless you apply the same sanity checks, etc. prior to storing it in that file -- and prevent anyone from ALTERING it thereafter -- you run the risk of having two inconsistent copies of data)

Exactly. You ('I') have to offer *value*.

Don't fall into the trap of conventional thinking. E.g., the idea of adding a library to encapsulate each module's interfaces, etc. Why can't that "library" be a set of hooks (stored procedures) and tables in an RDBMS? Functionally equivalent -- just packaged differently (and, if it resides in the RDBMS, then it can avail itself of the mechanisms provided by the RDBMS)!

--don