soft real-time on linux

In my experience (on a 400MHz ARM9), if you build the kernel with all the pre-emptable options, and run your task at the highest real-time priority then latency between an external event that causes an interrupt and your user task waking up and running is going to be somwhere in the 100-300 microsecond range. [Assuming the interrupt routine and device driver require minimal processing.]

That, of course, depends on the exact set of drivers you have enabled. All it takes is one long critical section in a poorly written driver to run that into many hundreds of microseconds.

I would be shocked if it ever got up to 1ms.

Unless you disable the rx FIFO, I would guess that the vast majority of the latency is going to be in the UART itself. With the rxFIFO enabled, there's often a delay of several tens of bit-times between the last data byte and the receive interrupt.

I did something like that. We had a microcontroller sending packets of a couple of hundred bytes via a USB-serial link every 5ms containing state of the input hardware, and accepting similar packets meant to control output hardware. We used straight-up Debian Linux on a VIA Epia board (chosen not for real-time ability but for low power consumption, reasonable speed, and lots of Ethernet and USB ports.) Application was written in C with one process for each major sub-department, SysV IPC shared storage for common working data and configuration blocks, and SysV IPC messaging for coordination. The most urgent processes (serial I/O and dispatching for one, and the core of the application for the other) ran with an assigned niceness of -10 to get the dispatches they needed; what was left was enough for the other processes, given the processor we chose. Everything marched to the 5ms beat of the input packets. Testing showed all input being processed and responded to in the next output packet sent 5ms latter.

I've heard since that they've quickened the drum beat to 2ms, but I haven't heard whether they needed to change much code to get that done.

Not very hard requirements for any modern kernel (2.6 and later).

The only thing that I would warn against is the RS-485 data direction (Rx/Tx) control. Do not try to use the RTS pin driven by a user mode driver. Use proper hardware control for the Tx/Rx switching.

Of course with any half duplex protocols, a large pequest/response latency will drop the effective throughput well below the theoretical throughput.

Unless other processes are contending for the cpu, this shouldn't be too much of a problem. On a 300 mhz arm7 board a while back, I remember timing a thread race and finding it could switch threads at around 20 khz. It should be simple enough to set up such a test on your own system.

The Linux thread scheduler is horribly broken, or was last time I used it (a while ago, admittedly). The info pages will tell you all about how to specify priorities, scheduling algorithms, and so forth, but leave out two vital items:

1. You can only hack those things if you're running as root, and
2. You can't have both real-time and normal threads in the same process.

Both (1) and (2) are easy in Windows (or the late lamented OS/2, which is where I learned multithreaded programming, 1992ish).

Note that for my purposes I'd have been totally happy if I could even _reduce_ the priority of some threads in my process to make clear which ones were less important than others. Not trying to hog other users' gjhresources, no sir. Still not doable as an ordinary user.

On the other hand, I think you can set processor affinities so as to reserve one or more cores for just your own code. I hope so, because I'm looking at a similar requirement myself soon. ;)

Cheers

Phil Hobbs

o:

the priority of your processes can be modified with "nice". If you want to change some kernel config of course you need to be root (sam e thing in any other OS)

since threads are created/destroyed dinamically, and the OS can't know what the f*ck is going on in the thread, it's pretty normal that the priority i s process based and not thread based.

If you want to prioritize a particular thread in a process, you need to do it at programming time.

good old stackoverflow:

the (1) is "easy" in win because a normal user is automatically Administrat or, if aou use a user without admin privileges, well you can change the pri ority of your processes, but not others.

Bye Jack

And the (2) is "easy" in Windows because windows is poor at process handling, and compensates by being quite good at thread handling. Linux (like other *nix) has always prioritised good process handling - it does not /need/ such flexibility in thread prioritising, but that is simply not the way you design programs in Linux. If you want a program that involves hard (or as hard as possible) real-time control of a motor, soft real-time logging of the data, and non-real-time display on a gui, you organise it differently on Windows and Linux. With Windows, you put everything in one process and have separate threads, because passing data between processes is inefficient. With Linux, you use three separate processes with different priorities.

Only process-by-process.

Not true in Windows, for instance. It's a serious handicap in my case, where I have compute threads and comms threads in a clusterized simulator: another box can totally stall waiting for a comms thread on the next box to finish its job. It trashes the scaling--Windows boxes scale nearly linearly up to 25 hosts or so, whereas Linux ones start getting less efficient at around 8 hosts.

On a single box, the thread scheduler has a global view of what's going on, so it isn't as big a deal.

Only works as root, and if you want any thread to be real-time, you have to put the whole compute-bound process in the RT class, which brings the system to its knees. What I need is a RT comms thread that is usually blocked on a semaphore and then runs furiously about 1% of the time.

But not the relative priority of threads within the process, whereas you can in Windows. As I said originally, I'd be quite happy to be able just to _reduce_ the priority of the compute threads vs. the comms threads, but _noooooooo_. Broken as designed.

Cheers

Phil Hobbs

Interesting, thanks. For most of my purposes nowadays I can run the simulator on one of my 24-core Opteron servers, running a single kernel instance, which greatly reduces the problem. Back when I had a rack full of pizza-box servers, it was ugly.

Cheers

Phil Hobbs

It has improved quite a bit in the recent kernels. However ...

All of this still is true.

The scheduler historically had problems dealing with overused and underused cores, and it has been greatly improved of late. However control over scheduling has not changed significantly since 2.6.24 (circa 2008).

Have you tried using clone(2)?

Clone is a low level call that can create either new processes or new (kernel) threads in the same process. For new processes it gives quite fine control over the execution environment.

You can, e.g., create a new process that *acts* as though it were a thread: sharing memory (and/or other environment) with the parent, but having a different priority, scheduling policy, process group, etc.

It is the best Linux can manage.

You can choose which cores will execute your code, but it is very difficult to prevent other processes from also using your chosen cores.

You can try to futz with group scheduling to achieve what you want, but that is clumsy and difficult because it can involve manipulating process *owners* as well as parent/child relationships and priorities.

I think the best you can do without inviting a whole lot of headache is to try to refactor your (POSIX?) threaded application into a set of clone processes.

George