Shared Memory for Application/Communication decoupling

Op Thu, 07 Dec 2006 23:52:39 +0100 schreef Christian Walter :

By definition, an interface lies _between_ value stores. What do you mean?

I thought that "shared" meant that data is shared, i.e. not duplicated.

Either values can always be read from memory, or values are always consistent. Not both.

What is the advantage here over other communication models?

A firewall tries to prevent bad things from going past it in one or both directions. What are you trying to say here?

It depends.

Yes, in the sense that they will be truly cooperating and not just doing their own thing with some shared data.

I don't see the problem here.

Why?

Absolutely. ;)

Of course each IPC method brings in its own piece of code and using different IPC methods within a single task will introduce a lot of complexity.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/

The fieldbus interface. I don't agree on your definition of an interface. In my opinion a interface is best described as a common boundary between two subsystems. It can be characterized by control properties(control signals), temporal properties(temporal constraints), functional intent(intended functions) and data properties(structure and semantics).

It is duplicated in the sense that when the computational task prepares new values the previous values are still available (In the SHM). At the end of the computation the values in the shared memory are updated in an atomic way. Another approach would also require some double buffering because temporary values must not be communicated to a client.

Your point - But using the shared memory updates can be done in a small step and if the time required for the updates is less than the time required by the communication system consistency can be enforced. Using message passing does not allow for fixed timepoints.

A temporal firewall decouples the two systems from each other [1]. The is currently a lot of research taking place on this and I have borrowed this term from time trigged systems. This is beneficial for a system because it limits error propagation. Of course checking inputs in the value domain is still necessary (and complementary)

[1] ...

They can be truly cooperating in either way.

For example assume there has to be done a big calculation in the application task. First of all the values are subject to change and therefore the temporary values can not be communicated to the communication task. This implies that either

a) there must be a copy of recent (valid) values somewhere b) the task must finish the calculation and then the new values can be communicated.

a imposes the same memory overhead as shared memory and b ist not suitable for our application.

Different IPC mechanisms because of performance. Going with SHM because of the benefits given above (in my opinion).

Thanks a lot for sharing. Still I can't agree on this point because I think the interface requirements are quite different. For example look at other systems where different IPC mechanisms are used when appropriate (EJB - Enterprise Java Beans with Locale and Remote Interface to name only one). Maybe this would be a interesting poll for the ESD magazine - ?Type of IPC mechanisms used today".

Kind regards, Christian Walter

I don't want to get into arguments about semantics, but in general, shared memory techniques are just a nice way of saying 'global data'. If you have > 1 writer, you have to do all the synchronisation and mutual exclusion at low level, where it's difficult to maintian visibility and keep track of.

A messaging based interface is generally more elegant and arguably more reliable, since for any single message, only one task writes and one task reads the data. This means that you can elevate syncronisation issues to a higher, task based level. Message based ipc reinforces an object view of the design at global level.

On modern micros, the overhead is probably insignificant as well. Shared memory / global data is a thing of the past, from the days when micros were not very powerfull...

Regards,

Chris

That is a good description. Anyway I got confused with your use of "side" in that requirement. I think it means this: "Values must be consistent whenever they are communicated across the interface."

In this case, double buffering is not always a doubling of single buffering. Temporary values may be kept in registers and in caches.

Messages can only be transmitted in response to an interrupt (including timers), with a delay that often varies. Reception of messages *can* happen at fixed timepoints (if your API is any good).

No advantage?

Now I understand. Thanks.

That last part (b) puzzles me. If the task must discard the current calculation when new inputs arrive, then you could say that it started too late or that the inputs arrive too often. Similarly, if the task must output a value at a certain time, regardless of the state of the current calculation, then the task waiting for that data is too impatient or its scheduling priority is too high.

Have you considered or heard of "lock-free data exchange" as an alternative?

I didn't say the complexity wouldn't be worth it. ;)

Sometimes you can have the local interface as a subset of the remote interface.

I agree.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/

I agree on that. This certainly will depend on the type of implementation chosen.

In this case temporal decoupling and it is a poll interface for the application task and a push interface for communication (if data flows from external to internal). The reason why this is a beneficial (in my opinion)

• A poll interface is easier for the reader because he can always read data if it wants. The reader wont be delayed by reading data. * A push interface is good for the producer because he can write data at any time he wants. It will not get blocked by writing.

But to a given limit this is also true for a message based interface (If designed and sized correctly(

The task should not discard its current calculation. It should finish it if possible. Let me try to explain my problem in the context of the Modbus fieldbus protocol. In modbus there are registers which in our case represent values measured by our device. In Modbus a PLC can query the values from a device at any time. A typical response time for such devices is in the range of 20-50ms (excluding communication). It is not possible for us to produce and calculate values at this fast rate because of the complexity. It is not a problem if the user reads values at a very fast rate and when we return the same values (because our sampling time is one second). It is only a problem if the user needs to threat our device differently than all other ones. Therefore there must be some buffering at the interface which holds previously calculated values. For a simple temperature sensor it is no problem to calculate values on demand and meet deadlines.

Thanks a lot for that - This sounds interesting and I will dig into it.

Maybe someone catches up on this and decides to write an article. I think this is still a place for a lot of research and it becomes even more important with new multicore architectures.

Kind Regards, Christian Walter

Hello,

You are right with that that this is 'global data'. But I don't see the difficulty for maintenance in this design. Basically I would introduce the following rules for both tasks (Let us assume our deadlines are 100ms).

• No task must lock the shared memory for read/write updates for more than 25ms. * Values which must be consistent must be updated in one block.

At the lower level I would have Lock, Get, Put and Unlock functions where Get and Put could also support automatic locking to ease up things for programmers.

A point against the message based interfaces is that if directly used there is a temporal coupling between the two tasks which must be avoided. Using message passing only to transfer new values and then buffering the data at the communication task provides the same benefits as the shared memory approach (consistency + decoupling) but has the same overhead.

In what case is a filesystem different in your opinion than a shared memory? In that case I would also be wrong to use files to exchange data between applications. Of course using global data between modules and at the interface level is bad because there are a lot of problems with reentrancy and consistency. But applying shared/distributed memory as a design pattern for specific problems is still valuable.

Regards, Christian Walter

But you will need to write and fully debug all the low level stuff, with test harness code, then put it into a library etc. It just seems more work and a less tidy solution than messaging. Most embedded rtos seem to have message based ipc support, so you can make use of a fully debugged code base, rather than write it yourself. The other point is that with a 1 second update, response time in terms of code throughput, however implemented, is unlikely to be an issue. Gut feeling suggests you may be making this more difficult than it is, or needs to be. At worst, you could perhaps encapsulate all the tricky bits into a device driver, with a single, well defined interface.

Areas where shm is usefull, if not madatory, imo, are for things like dma transfer and interrupt handlers. For a dma write, a device transfers data into memory, interrupts / signals on completion, then the initiator reads the data. For interrupt handlers, the interrupt code may typically transfer data into a queue, with the mainline process reading the queue. In both these cases, it's easy to enforce mutual exclusion, since you only have one reader and writer. For the interrupt case, it's trivial to disable interrupts for a couple of lines of code while you update buffer pointers etc. A pretty bulletproof solution with no special cases or loose ends.

I think what i'm trying to get at is that for any system design, you minimise complexity wherever possible and juggle the global view and module relationships to effect this. The other point is that it's best to keep module interelationships at as high a level as possible for visibility. Having high level modules dipping down through the layers to make use of low level primitives is a bad idea - all that stuff should be encapsulated into drivers and hidden from view in a good design.

Agreed, but there will be a temporal displacement between the write to shm and the reader. How this compares to a messaging interface depends on system design.

A filesystem could be considered shared data, but it's a bit out of context for the present discussion, yes ? :-)....

Regards,

Chris

That a good point. I have to mention that we used a custom RTOS in the past (no multitasking) and we where in the need to implement multitasking and IPC primitives. Therefore we already have some working code and experience with that. But of course doing stuff yourself is always bad in terms of cost and work - Specially because management wants to keep down cost and development time.

Right now we are doing a new design and therefore we are evaluating new design options which will allow us to support different fieldbus protocol in the feature.

The other point is

The one second throughput is for our most complicated devices which perform complex operations. We will also have devices which are much easier and the computational part will be a fractional of the fieldbus requirements (100ms).

I think I understand. From this point of view it is bad the the communication task and the application task use a low level primitive like shared memory. Did I understand you correctly that you would recommend introducing another layer which acts as a store with high level functions ( Like UpdateParameter, SetCalibrationSettings, ...). This layer then uses internally a IPC primitive to transfer data between the tasks?

You are right.

Thanks a lot for your input, Christian Walter

Indeed a lot of communication models can effectively be used to implement these requirements.

Now I am beginning to understand. It seems that you left out one component; earlier you spoke of an application task and a communication task. The way I see it, you would benefit by having an arbitration (or distribution) task as well.

Let me illustrate this in a message-passing philosophy. The communication task waits for queries and relays them to the arbitrator. The application task calculates values as fast as it can, and sends each set to the arbitrator. The arbitrator just holds the last set of calculated values it has received, and discards older ones. It waits for relayed queries and responds with the message that it has stored.

Note that appropriate task priorities and a cooperative task scheduler will ensure data consistency and predictable behaviour.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/

That sounds like an excellent idea. It looks simple to implement, uses the standard IPC primitives and fulfills exactly my requirements. I will have to think about your idea with using a coop. scheduler to ensure consistency and what this would mean to our application.

I would really love to see a book about embedded design patterns or best practices. Such things discussed here would certainly be worth added to it together with the problem definition and the solution(s). It would specially be beneficial for people in smaller companies where there are not so many experienced engineers and therefore knowledge is harder accessible.

Thanks for sharing, Christian

--
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ \\\\\\ \\
email: wolti@sil.at      \ Friends don't let friends drink and su(1)
web: www.freemodbus.org  \               -- Kevin Harris

This is what I understood as well. But this is something that should be done at some level no matter what IPC method is used.

"Never let implementation details get in the way of what you want to do!"

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/

I agree fully with Chris's remarks - I'd avoid global data like the plague.

However: I've also done fieldbus work, and I'm aware that one way of mirroring the states of the numerous parameters on numerous controllers is to use shared memory. I don't like this method myself - see above re global data. I much prefer a set of routines for accessing or changing parameter( PARAMETER_NAME ) on controller( index ). The usual arguments against this include a) different data types and b) overhead of the access methods - personally I prefer to solve these issues rather than fall back on global data. That way be dragons.

*However* - if you're *really* stuck for memory, you might consider a block of shared memory as a part of the messaging interface - so long as you forbid your coders from bypassing the messaging interface on pain of death. Having distinct set/read routines is still essential.

BTW: I'm assuming that your comms code builds a mirror of the states of the remote controllers, and your application layer uses the mirrored data. If you interrogate on demand, then the memory issues go away - but the application is kinda slow...

HTH,

Steve

There are already a few books about embedded design patterns and best practices, though finding the good ones is not easy.

About patterns: a computer science teacher named Allen B. Downey wrote the free (GNU FDL) book "The Little Book of Semaphores", containing a lot of (general) patterns and exercises using semaphores, including the common mistakes that students make while trying to solve the problems. As someone who doesn't consider himself a genious and who has succesfully used message-passing in practice, I've found that the basic problems in the book, using lots of explanation, become no-brainers when you are allowed to use messages to solve the problem.

Also some relatively modern pattern books promote locks, mutexes and semaphores for synchronisation as if they're all there is, e.g. "Real-Time Design Patterns" by Bruce Powel Douglass.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/

Do you have URLs handy for those references, especially the GNU book? And, out of curiosity, are you related to THE Dijkstra?

--
Chuck F (cbfalconer at maineline dot net)
   Available for consulting/temporary embedded and systems.

A bit late back to this, but whatever. It can be usefull to view designs in a layered manner, because it makes it easier to separate form from function and to modularise application, data conversion/protocols and hardware specific code into modules that can later can be modified or replaced with minimum impact on the rest of the system. So long as you define the layers and call interfaces carefully and avoid globals, this approach seems to work quite well.

The way I got into all this years ago was to buy books on OS theory which included descriptions and comparisons. For example, cpm80, unix, msdos, vax vms etc. One classic which imo should be on every programmers bookshelf is the IBM systems programming series "Introduction to Operating Sytems" by H M Deitel, Addison Wesley, isbn: 0-201-14473-5. It's probably on second edition by now, but it covers just about every aspect, including chapters on example os's. It has a chapter on concurrent processes and a complete chapter on deadlock and it's avoidance !. Other classic os books are the Tanenbaum book on Minix and the Comer book on Xinu. Abe books probably has s/hand copies at a fraction of new cost as well, so there's no excuse :-). You can't expect to make a success of complex designs without some theoretical background and it's usefull to see how it's all been done before. The same sorts of problems crop up over and over again...

Chris

"The Little Book of Semaphores"

"Real-Time Design Patterns"

Strange, isn't it? Whenever I start talking about semaphores, this question gets asked.

The current answer is: not for at least seven generations. For a price, I can arrange a more definitive answer for you.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/

Hello,

Maybe I was not to clear on the intended function of the device. I am not doing a device which collects data. Its a measurement device and the discussion of shared memory came up because of the design issues I mentioned ( support different fieldbus protocols, temporal decoupling, ...).

Kind Regards, Christian Walter

Hello,

Thanks a lot for all the suggestions. The Semaphore book looks interesting and includes a lot more than things thaught at college classes. I just downloaded a copy and will certainly use it the next time such a question comes up. Having read the reviews and the TOC on the Real-Time Design Patterns" on Amazon I have decided to buy a copy. Besides the time is perfect too for a small Christmas gift for myself ;)

Regards, Christian Walter

Ah, I see. FWIW, this is even more familiar territory for me ;).

You say you're using a custom RTOS with no multitasking. If you're using a roundrobin or some other form of cooperative scheduling, you have synchronism right there. You can control the state the data is in before your application task yields. You can also ensure that all interrogable data is precalculated (or can be quickly precalculated from stable data). To put it another way, ensure that the comms task can only run when the data is stable and self-consistent. If this means building comms data packets on one pass through the loop, and buffering and transmission at a more leisurely pace either by interrupts or by polling, so be it. Similarly, for reception: when the application task runs, any comms-written data should be stable: it'll be able to react to (signalled) changes of data consistently, in one place. And you get your temporal decoupling [1] for free, or more specifically as a key part of the design.

I'd say again: use a messaging system, not global data. Use methods (and hence the opportunity to signal a query or a change) to access internal data. At the very least, ensure that your comms task knows nothing at all about the application task's data structure and use, and vice versa. I repeat: there be dragons.

[1] If you mean as opposed to allowing both your comms task and your application task to read/write shared memory at will, which I can't believe you do. This would not even be a contender for a serious design.

HTH,

Steve