Embedded Linux: share data among different processes

do you need to keep the data from the RS485 or not?

If not a shared memory or (named) pipe or something like this accessed thro ugh a semaphore or a mutex is IMHO better. Some reading:

ng.com/alp-folder/

On the other hand, if you need to keep the data then a DB is the way to go.

Bye Jack

folder/alp-ch05-ipc.pdf

folder/

Since you already using Qt, I can propose a solution which I use quite often. It is even flying in more than 100 helicopters. Yes, penguins can fly :-)

The poller fetches that data from the remote device via RS485 and keeps a for it local copy. If permanent storage is a requirement it can also take care of this. In addition it open a QLocalServer which listens for connections from other processes. The other (multiple) processes connect to this server using a QLocalSocket. They can then poll are are sent the data directly when the poller receives new data. This way no locking is required since the poller does its sending in a single thread. You just have to define a simple protocol. It could be line oriented ASCII.

You can also use QTcpServer/QTcpSocket instead. Then you could even have some processes running on different machines (or virtual ones).

The other - more complicated - option would be to use DBUS.

This Server/Socket combination has the advantage that it easily integrates into the signal/slot mechanism of Qt.

--
Reinhardt

I did that with System V IPC, using message passing for control and shared memory blocks for info common to all the processes. I was able to avoid any locking using the one-writer/many-readers trick. The code was more low-level than many people, perhaps, would like, but it worked well.

Mel.

It depends on how "lightweight" you want the mechanism to be. And, how "elegant".

If you can implement the "Shared Memory" extensions (a kernel option in the *BSD's -- not sure re: Linux) then this is the easiest way forward. Look for shmget(2), shmat(2), shmctl(2), et al.

Use a semaphore to control access (and discipline for thereaders/writers to follow that convention) and keep atomic regions short and sweet.

If the data isn't changing all that often, you can push it to each consumer through pipes and count on them to keep their own copies intact. (here, there be dragons)

The DBMS approach is how I am currently addressing all "persistent data". It's a fair bit heavier-handed (I've got lots of MIPS to spare) but is more elegant; a producer can't inject data into the store unless the data is appropriate for the tables/fields involved (you can't store text somewhere that expects numerics!). Likewise, the consumers need not *check* the data because the checks can be applied on admission to the DB. I.e., if the data is *in* the DB, then you know it satisfied the admission criteria!

However, I don't think SQLite has all of these hooks.

GNeuner is your go-to guy for the DB approach...

This is the classic and probably easiest approach, though not the most economical in terms of machine use. There are other databases besides sqlite that you can also consider. How much data are you talking about? Do you need stuff like persistence across reboots?

1) Qt and HTTP are completely disjoint. Use of Qt is disruptive - it has its own pre-preprocessor and ... keywords specific to Qt. If I could, I would pick one and possibly carve the other off as a separate project.

HTTP is, simply put, a bloated, obese protocol which cannot be put on a decent timeline. It's intentionally entanlged with TCP in very unfortunate ways.

2) You mean you will poll for data over an RS485 link, then have two possible clients on a workstation/phone/tablet/whatever that exploit this data. 2a) Is the RS485 link half or full duplex?

2b) If it is half duplex, do you have an accurate model of the line discipline? For example, with MODBUS ( which is 1/2 dux 485 and address muxed ) there will be hard[1] timers to be managed. I recommend a state machine approach. The port cannot be used by more than one request/response pair at a time and response timeouts dominate latency.

[1] although they can be made to be adaptive and a select()/poll()/epoll() approach can be quite useful. Much depends on whether the responding node identifies itself in responses. But depending on your serial port architecture, this may get weird.

3) SQLite appears to have locking:

3a) SQL is a bloated pain in the neck.

4) A minimal technology for a shared data store under Linux is shared memory. There are shared memory filesystems or you can develop a shared library/object file which enacts an access protocol for the shared data using the System V primitives. See "System V shmem" and "System V semaphores" through Google for details. This isn't difficult but it's fiddly. It's also the highest performing approach. I;d at least consider guarding the shmem with a semaphore, although I've never established that this is completely necessary

5) Consider using pipes for shared data, possibly. Dunno how you integrate pipes with an HTTP daemon.

6) Sockets are also a good thing to consider using, although again, I have never integrated an HTTP server with sockets for another protocol. Your "poller" offers say, a TCP socket ( to be used over localhost ) and does "multiple unicast" or alternatively actual multicast transmission of the entire data store every so often, or differential ( the things that have changed ) transmission as needed.

6a) Multicast constrains you to the use of UDP.

This is a nontrivial project.

All this being said, there may be a good software kit to solve this problem out there; I just don't know what it is.

and

7) I've seen cases where a COTS HTTP server was used to serve up Java applications to do this. That's kind of old school at this writing.

--
Les Cargill

I did something like this (TCP over USB between the base unit and the display) and it wasn't so bad. Yes there was a ton of bloaty system code under the surface (language interpreter, network stack etc.) but who cares? The application itself was reasonably simple. The ARM CPU and 64MB of ram were a few dollars, and the "disk" was a 2GB microSD card that was another couple bucks. The user application was written like any other client-server app. The shared data was managed in the application itself, that ran in a single process (multiple threads), but using a database works perfectly well if you don't mind the overhead. If you have the memory for it, Redis could be a good alternative to SQLite, depending on what the application needs.

SQLite can be run completely in RAM (backed by swap if applicable).

Although not possible out-of-the-box, wIth some hacking it can be made to run from a memory mapped file (for persistence).

If you have relational data, I believe SQLite is the lightest weight file based RDBMS available. There are a handful of RAM-only libraries that accept SQL-like query languages.

If you need only something simple such as key-value pairs, then there are many possible implementations to choose from.

Excellent questions.

George

SQLite "takes care of" that by using a single giant lock. That might work for you, since it sounds like you probably have very easy throughput and latency demands. It would be a good solution if you weren't using Qt. Les Cargill hasn't seen how remarkably lightweight SQLite actually is.

Reinhardt, do you deal with write-blocking if the reader fails? If the write is in the poller thread, any blocking on writes will delay the polling - unless you use non-blocking writes.

I think it's good advice, though I wouldn't personally have chosen Qt.

This application is not demanding enough to need a shared memory solution, with the attendant need for locks and especially, memory barriers, of which most respondents seem ill-educated. In particular, Mel Wilson's "one writer, many reader" is no longer so simple with modern multi-core architectures. Writes do not have to happen in the order they are written, so readers can get confused.

Clifford Heath.

That's only a problem if you "roll your own" locks. IME, the SysV shared memory operations also carry along the SysV semaphores.

And, you *want* the OS to know about any locks you've implemented as, inevitably, you'll want to add another "client" down the road and then have to worry about the various types of deadlock that can occur (that the OS can mitigate, in some instances).

E.g., the "single giant lock" that SQLite uses can similarly be implemented to control ALL accesses to the shared memory segment.

You also have the advantage of *seeing* all of the code that is accessing these shared objects -- instead of wondering what SQLite is doing and when its doing it.

As I said, ill-educated.

As you'll find out when the SysV semaphore grants you access to some shared memory resource that has been written, but only some of the writes have been flushed.

If you're on hardware that can do this, you need to know about it, or leave the work to someone else who does... like the authors of SQLite.

Clifford Heath.

I'd appreciate a definitive reference backing your claim.

And, the folks authoring the semaphore functions *don't*?

Qt internally uses non-blocking read and write. So this is not a problem.

Because of that I prefer to avoid all these locking if it is not needed. Having the data at one place controlled by one thread and send it from there to interested clients via some IPC mechanism make live much easier. And it is also a very effective encapsulation. The other can not even know, how it handled internally in the server. I would use shared memory or similar close coupled methods only if necessary for performance or other requirements.

--
Reinhardt

The semaphore functions are not guaranteed to flush CPU caches or prevent out-of-order execution of writes to the shared memory. Why would they? The semaphore semantics are not intertwined with the memory semantics.

It's common for entry to the kernel to flush the cache, but you need to know - it doesn't happen by magic. But out-of-order execution can still spoil your day.

In some architectures, even a single "increment-memory" instruction can allow another CPU to observe the memory cell to be half-incremented. It's all very troubling...

My mutex functions do exactly that -- to ensure the synchronization primitive works regardless of instruction reordering, the presence of multiple cores, etc. The whole point is to make it possible for a piece of code to be migrated to a different platform (with different hardware capabilities) WITHOUT having to reexamine each potential problem area.

I have no idea how the Linux kernel implements the SysV semaphores (or POSIX semaphores). I will have to look at the FreeBSD and NetBSD sources to see how *they* do. I'd be surprised if a kernel capable of running on such hardware would neglect these sorts of things.

All the more reason to embed these guarantees in these library functions.

You're still missing the point. The synchronisation primitives can fulfil their contract perfectly - with regard to the semaphore operations - but the memory write behaviour is not part of that contract. Only the semaphore operations are in the contract, and nothing about that says that all memory accesses on both sides must be coherent across all processor cores.

Your mutex functions perfectly, I'm sure - but do they guarantee that memory operations either side of the mutex - and not referencing the mutex's memory - is coherent across multiple cores. Note that I am not talking about the memory used to implement the mutex itself, but other memory.

How? If threadX and threadY (operating on the same processor *or* on different cores) can see different orderings, then how is the contract satisfied? It's not an "increment memory" operation...

What value is a synchronization primitive if it doesn't provide synchronization? How is it fulfilling its contract if it *doesn't*?

All synchronization operations MUST perform barriers to memory and operator reordering.

Yes -- as they include memory barriers specific to the particular processor architecture involved. The whole point is NOT to force the developer to have to wonder whether it's a single-threaded environment or multithreaded; uniprocessor or multiprocessor. The function call is a fence.

From a quick peek through some of the online Linux sources, their SysV semaphores also seem to operate in a similar manner. I've not yet looked through the *BSD sources...

Are you claiming the compiler can reorder instructions *across* an opaque system call? E.g., decide to invoke the code at main() AFTER the code at exit()?? (what's to prevent this??)

You're just hiding by wrapping it in an IPC mechanism.

Yes, but it is also considerably more expensive. Every consumer has to be informed of every update to the data in which it has "expressed an interest" (assuming you don't broadcast ALL data to ALL consumers).

E.g., I install triggers in my RDBMS so that any update to a particular (set of) parameter(s) results in an up-call notification of the interested parties that might want to *see* that new value (so, I don't have to broadcast all updates of all data to all consumers).

But, that upcall carries considerable cost (from trigger through upcall to client notification and eventual re-query of the RDBMS). It can effectively bottleneck the RDBMS's operation (cuz another up-call could be enqueued right behind the first one even before the first has finished!).

So, the nature of the problem changes -- to one of deciding *what* information is "genuinely of interest" to each client... AND, how timely that notification should be made (if you can defer a notification, then you might be able to eliminate it in favor of a subsequent notification)

E.g., if some client (having permission to do so!) updates the current timezone, the wall-clock display should *probably* be updated to reflect that. OTOH, it probably can wait a second or two. And, if the timezone is updated AGAIN before the notification is dispatched, the first update can safely be ignored in favor of the second.

OTOH, if *every* update must be conveyed to the client(s), then the notification effectively becomes part of the update transaction and limits how frequently the value can be updated.

IME, people are more accustomed to using shared memory solutions as this mimics the familiar interplay of foreground/background processing; people *seem* to know that some coordination must exist between an ISR and its "clients".

OTOH, sharing memory in an environment with individual, protected memory spaces can be intimidating the first time.

Yes, another problem to solve is the other way: QT and HTTP server could send some control commands to the electronic device (switch on, switch off, change this...).

It is exactly what I will have: one writer (the poller that will continuously fetch updated data from the device) and some readers (at least, HTTP server and QT graphic libraries). Doesn't it needed a syncronization mechanism (semaphore) in this scenario, using shared memory to share common data?