Embedded Linux Vs. Real time Linux

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
Hi all,

I have some newbie questions regarding embedded Linux targeted a real
time application.

We are on my job considering the use of Linux in the next generation
of our product line. We are looking into either a ordinary Linux port
(Montavista) or a Real Time port of Linux (Montavista or Wind River).
My questions are:

Will we still be able to write specific program code for some of the
hardware interrupts which still operates completely independant of the
linux kernel? For instance a DMA interrupt that needs real time
handling will it be possible for this interrupt to immediately "de-
schedule" the Linux kernel and service the DMA ISR and then return to
the linux kernel, without the Linux kernel ever knowing that this
interrupt happened?

If this is possible and no problem to implement - what is the use and
idea of a Real Time Linux? As I understand the Real Timelinux kernel
allows for low interrupt latency and preemptive behavior. Also it
implements a defined interface to the Linux application via special
FIFOs.

I had this discussion with one of my coworkers and I insisted on that
this solution would not be a good Idea since the Linux kernel operates
and handles the CPU hardware such as the interrupt controller. Writing
specific code outside the the Linux address space accessing some sort
of shared memory (for data transfer) and shared hardware resources
such as the interrupt controller would in my naive opinion introduce
some conflicts and would require some "tweaking" or modifications of
the Linux kernel? - Is this totally of track and if so....why?

Another question comes to my mind. If an ordinary Linux distribution
is optimized for soft real time applications how is the hardware of
for example a 1Gbit Ethernet mac then serviced ? On high loads holding
relevant packets the Linux kernel (or device driver ?) must service
the MAC hardware very often to offload the data and prevent loss of
packets and data - How is this done fast enough in in a ordinary
Linux?

I hope I'm not talking totally gibberish here. I apologize for the
eventually stupid questions - but hope that some of you experienced
Linux guys can help clearing these questions for me.


Best Regards.

--
Morten M. J  - (Linux nOOb).

Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it

With any type of Linux you need to write a device driver to handle a
hardware interrupt (but a device driver is just a kind of program :) )

Quoted text here. Click to load it

Same is true for DMA, as well, even if no interrupt is used.

Quoted text here. Click to load it

You need to decide if you need hard realtime (a (maybe even some 100 ms)
timeout needs to be met in any case) or a soft realtime specification
(e.g.: as an average, every years once a 100 msec timeout is allowed to
be missed). Kernel 2.6.x offers quite good soft realtime behavior. for
hard realtime, standard Linux is not applicable. Here you need
commercial stuff or add-ins (like RTAI).

Quoted text here. Click to load it

If you use Linux, you need to do everything "the Linux way". There are
lot's of specs that help and restrict.

Quoted text here. Click to load it

I once heard about "realtime Ethernet", but I don't know much about
this. Linux Realtime solutions (e.g. RTAI or PikeOS) provide a basic OS
and Linux runs as a low priority task of same. So all realtime stuff has
priority above all Linux stuff (including Ethernet).

-Michael

Re: Embedded Linux Vs. Real time Linux
Hi Michael,

Quoted text here. Click to load it

Hehe...as I said I'm ultra noob into this brave new world :-D But is the
device driver not a part of the Linux envrioment with defined ways of
communications and scheduling together with the Linux kernel? What I am
talking about is some totally independant code that is able to interrupt all
Linux processes and complete it's own task before re-enabling the Linux
execution.

But how is the Linux/device driver interaction defined? Will a Linux device
driver not make use of Linux system calls in some extent?

Quoted text here. Click to load it

Take for examble my DMA interrupt. When this interrupt is raised I need to
service the DMA and moveing some data from a circular buffer into the
application area where the data will be processed and then trasmitted via
ethernet. I cannot afford to miss data from the DMA since this will cause
huge problems for our system.

So as I see it (and do correct me if i'm the slightest wrong with this) we
need either:

- Some independant code outside the Linux enviroment to service the DMA ISR.
The DMA interrupt will happen without Linux knowing and Linux will be 100%
interrupted and resceduled after ISR completion.

- A Real Time Linux system having a real time extension in the kernel
including a defined interface between the real time part and the kernel.

- Third option?

Quoted text here. Click to load it

What would be the Linux way of the problem above?

Quoted text here. Click to load it

That is exactly what we are considering - but will this be possible with a
standard 2.6 kernel ?

--
MMJ



Re: Embedded Linux Vs. Real time Linux

Quoted text here. Click to load it

I don't understand why you would need to do this.

You can however do whatever you like inside a Kernel device driver. At least
on a single processor system it should still be possible to disable
Interrupts alltogether and running your code exclusively.

Quoted text here. Click to load it

The generic Linux way would be to implement an Interrupt service routine in
your device driver which will afterwords transfer your Data to userland.

If speed is important you might be able to use mmap instead to prevent the
data from getting transmitted between kernel and userland too often.

Quoted text here. Click to load it

Again, I don't see any reason why this code need to run outside the Linux
kernel enviroment.

Quoted text here. Click to load it

Again, I don't see a reason, why this should be necessary

Quoted text here. Click to load it

This is possible to do with the standard Linux Kernel API.

Quoted text here. Click to load it

Some Real Time Linux systems (e.g. RTAI) do it this way.

An ordinary Linux Kernel with Realtime Preemption patches will however
provide ordinary Unix _Userland Applications_ with Realtime capabilities
(http://rt.wiki.kernel.org /).

Sven

--
TCP/IP: telecommunication protocol for imbibing pilsners
                                 (Man-page uubp(1C) on Debian/GNU Linux)

We've slightly trimmed the long signature. Click to see the full one.
Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it


Because we do have some hard real time demands (caused by our hardware
arch - as it is now). We also would like to code our application code in a
Linux enviroment. If we implment the service of the real time demanding
hardware in a ISR compiled into the kernel space - we'll still suffer the
latency that might cause buffer overflow in the hardware.

Quoted text here. Click to load it

Having the the outside code meeting the real time demand of the hardware and
the Linux to mange applications. But as I mentioned I seriously has my
doubts to this soloution in terms of the communications between the to
software entities.

Quoted text here. Click to load it


Yes but what about latency?

Quoted text here. Click to load it

Yea this or the other patches mentioned might be the proper solution. What
latencies might one expect when running a RT linux setup on e.g. a 75Mhz ARM
platform?

--
MMJ



Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it

Hmmm, here are some numbers: PXA270 processor, running at 520MHz and RT
Preempt Linux. About 19us...160us on a system with low load.
With other load running this increases to 500us...750us. This is due to the
fact ARM has its cache on the "wrong side" (=virtual addressbus), so task
switching hurts your system (needs a cache flush every time).

JB

Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it

This can be helped by using µCLinux instead of full Linux (disabling the
MMU).

-Michael

Re: Embedded Linux Vs. Real time Linux
On Fri, 11 Jan 2008 22:35:22 +0100, Michael Schnell
Quoted text here. Click to load it


I can't ever imagine performance being worth running without memory
management any any system more complex than a few thousand lines of code.
Developing software where *any* process can overwrite the memory of any
other process is just nasty.  It gets real cute when a program had an error,
but has exited, and the other clobbered program doesn't develop any problem
until hours later.  

Re: Embedded Linux Vs. Real time Linux
IMHO, this depends.

For systems with user interaction I fully agree.

For embedded systems that contain high security functions, Linux alone
is not recommended anyway. Here a virtualizinfg system (like PikeOS)
should be used that provides a simple (and thus reliable) system besides
Linux that goes on working if Linux crashes. Thus memory protection in
Linux does not help much (other than for debugging purpose).

With a normal embedded system any part of the user software is as
essential as the Kernel. If one of them crashes, the function of the
embedded device is not provided any more Thus memory protection in Linux
does not help much (other than for debugging purpose).

-Michael


Re: Embedded Linux Vs. Real time Linux


Quoted text here. Click to load it

This is not related to memory protection: if your application crashes
on whatever system, it cannot provide the required function anymore,
unless you make your system fault tolerant, for example by restarting
failing applications or rebooting your system (which, in a way, can
also be seen as restarting your application.....).
Memory protection does give your system some protection against
applications going wild. At least you can have an application that can
monitor all other applications, without the risk of being corrupted by
another application.

Kind regards,
    Johan Borkhuis

Re: Embedded Linux Vs. Real time Linux

Quoted text here. Click to load it

Of course. But that is more of a benefit than a restriction.

Quoted text here. Click to load it

The standard Linux Kernel will disable the interrupt now and then, so
the interrupt will not be recognized by the processor. If your hardware
allows multiple interrupt priorities, you could modify the Kernel to
leave one level enabled for your DMA system, but if you do so, there is
no decent possibility (other then a hardware FIFO or similar) to have
your system communicate with the Linux system (to pass the acquired
data), as you can't construct a non-spinning semaphore.

For any decently constructed hardware that creates or needs data bytes
in a speed that is slow enough for moving the stream via Ethernet, with
a decent processor a DMA  enabled device driver should be by far
appropriate (otherwise the Ethernet driver would fail).

If the hardware does not give the processor enough time to allow for
some decent latency between two consecutive DMA actions, the hardware is
erroneously constructed. It should contain some kind of FIFO, double
buffers or a similar construct. With that the processor can react on the
end of the previous DMA block while the next is running.

-Michael

Re: Embedded Linux Vs. Real time Linux

 > Of course. But that is more of a benefit than a restriction.

Indeed it is - I my self would like to everything the right way keeping us
from adding outside code and doing kernel modifications.

Quoted text here. Click to load it

So what you are saying is that the Linux kernel WILL disable global
interrupts from time to time. But I guees the interrupt still will be
pending when the interrupts are reenabled by the kernel ?
This causes a delay on the service time of the DMA and worst case is that
the buffer overflows. Which parameter decides how long the kernel wil
disable interrupts - CPU load? Is it posiible to make any estimation of this
time?

Quoted text here. Click to load it

I'll pass this on to the HW team :-D

--
MMJ



Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it

AFAIK, this is correct (maybe depending on the CPU architecture). You
need to inspect the Kernel sources to be sure.

Quoted text here. Click to load it

Right.


As Linux is not meant to be a realtime OS there are no specs on this.
But of course the time is rather short. But If you can live with that
delay I'm quite sure that doing a standard interrupt driven Linux device
driver will do the work, too.

Quoted text here. Click to load it

What kind of hardware do you (intend to) have ? If you do this with an
FPGA, maybe the hardware can be done/changed appropriately. Anyway if
designing hardware that is to communicate with a processor, the
appropriate design rules should be followed. :)

-Michael

Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it

HW is beeing implmented first off in a FPGA and then moved to a structured
ASIC later on! I think the buffers are rather small because of the memory
amounts in the ASIC. But I'll for sure pass this on.

--
MMJ



Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it

You mentioned DMA. If that is bus master DMA, no buffers in the hardware
are needed, just two "channels" to do double buffering.

If you want to use a DMA controller in the processor chip, maybe you can
use two DMA channels that are activated alternating by the hardware. So
the processor can program one of them while the other is running.

With the project I'm up to start right now, I'll use a processor
programmed in the FPGA, so the cost for the processor chip and PCB space
etc can be used for the FPGA, which makes the solution very scalable.

-Michael

Re: Embedded Linux Vs. Real time Linux
BTW. The NIOS CPU (running in an Altera FPGA) comes with a "scatter" DMA
unit. Same reads its commands from memory and thus can run
uninterruptedly all the time, while the CPU just reads and writes the
shared data and control memory.

-Michael

Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it

Nice feature....to bad we use Xilinx :D Is the NIOS set up for linux with
MMU ?

--
MMJ



Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it

I suppose the Xilinx Microblaze system provides very similar features to
Altera NIOS. Moreover switching the brand is not too hard work until a
PCB is done.

Quoted text here. Click to load it

As of very recently NIOS does come with an MMU that can optionally
enabled. Right now only MMU-less µCLinux exist for NIOS, but I'm very
sure that full Linux will be available very soon (as already is with
Xilinx Microblaze).

-Michael

Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it

If you received multiple interrupts for a single IRQ during the time
the interrupt was disabled this does not have to be true, it is
possible that you only receive one of the interrupts.

Quoted text here. Click to load it

It depends on your definition of short. In some situations it can be
100's of usec, or even msecs. If you need sub-msec responses this is
not acceptable. For a general purpose OS this is pretty good.

The PREEMPT-RT patch tries to get rid of the locks, and has been able
to get a good RT version of Linux, but it is not available on all
architectures. It is mainly developed for x86 platforms, but is
available for some other platforms.

Kind regards,
    Johan Borkhuis

Re: Embedded Linux Vs. Real time Linux
Quoted text here. Click to load it

Is the PREEMPT-RT patch compatible with uCLinux? We're looking at using
a Blackfin  (no MMU) and we have hard realtime latency requirements (DMA
completion interrupts) on the order of 1 to 10 mSec.

Bob

Site Timeline