x86-linux sources of latency

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
Hi
If you're not going to read much, please begin from VVV below

I'm working on a port of currently modula2-based EDM CNC system to
Linux base.
Since this is a port, there's a need to create the same environment as
in original system, which is relatively simple all based on i8254
timer (it's a cool Russian soft, really).
So as for today I've made the following (it's all about linux-2.6.5
kernel):
1) The scheduler is now working from rtc's IRQ8 (at 1024Hz)
      => i8254's channel #0 is free and available
2) local_irq_disable() etc. are reimplemented to mask interrupts (all
but IRQ0) in interrupt controller, so cli/sti sections do not disable
timer's interrupts (this involves several changes in i386 arch files)
Ant, as it seems, at this point the system responds quite
satisfactory. The top criteria is not that hi, stable 1Khz interrupts
(say, +-0.25ms) is enough.
My test suite is just fun (copied from old system's crush-test). I
establish 1-6 Khz IRQ0 and change speaker's gate bit so it gives sound
of given frequency, then I do different actions such as walking
through X, syncing disks etc and if things go bad, I here how the
sound clicks (even short 1-period holes are noticable). Original
2.6.5, ofcourse, is bad-bad, but with my patches I just couldn't break
the sound. Going to make more statistics via Pentium's time stamp
counter.

So. Thanks for reading.
I see, that some or all of this is just what RTAI and RTLinux do
(never looked into), but it's way more interesting to self-code it
and, the main, I get exactly what I need, no more, no less. Maybe this
is interesting for someone else? If so, great!

VVV

Here's what I really need and haven't yet found.
- Information on sources of hardware latencies in PC's, mainly x86 (I
excpect there are I/O issues on buses' level, though I don't know the
arch well enough /yet/).
- Are there other real-time constraints in mainstream Linux kernel
than interrupt disabling?
- As I know, current real-time Linux solutions give 1 ms granularity
(am I wrong? sorry then). What stops developers from going farther?

Thank you and with best regards,
    Dmitry M. Shatrov

Re: x86-linux sources of latency
Quoted text here. Click to load it
Hmm. That sounds like an interesting approach. I assume you are also
enabling kernel preemption?

Quoted text here. Click to load it
Well - contrary to most others - I still think I get better performance
from 2.4.x with both low latency and preemption patches. What I'm
measuring is what shows up at the application level which may be an
apples to oranges comparison to what you measure.

Quoted text here. Click to load it
If your patches are available - I'd like to review them as well. We
don't use either RTAI nor RTLinux since we don't want to run our real
time applications (several millions of lines of code) in kernel mode
(ours is a rehost from a large SGI system).

Quoted text here. Click to load it
I don't certainly know enough details, but am aware of some issues
related to the PCI bus. For example, some of the host bridge settings
will affect both the latency and throughput of operations to PCI
devices. We use a shared memory (like) interface - Scalable Coherent
Interface - where we map large (100's of megabytes) remote memory
regions through a PCI card. By making host bridge configuration changes
- we can improve the performance to copy data from one machine to
another by 20% or more (our best is about 100 Mbyte/sec on a 32 bit 33
Mhz PCI bus). Getting the host bridge really screwed up causes the
performance to drop to perhaps 5 Mbyte/sec.

This will vary by host bridge as well. Our particular application will
use a loopback (a CPU accesses the PCI card, then the card access local
RAM) on occasion and that has been known to break host bridges as well
(the Serverworks 64 bit PCI host bridge was broken about 2 years ago -
haven't tested a more recent version). We had to use VIA chip sets to
get our application to work OK.

Quoted text here. Click to load it
When you get to try to run a real time task (as an application), there
are lots of other activities that can preempt the real time task. For
example - the buffering in the ext3 file system (about 2.6.2 time
period) was giving us > 15 msec latencies when doing something as simple
as:
  cat /dev/mem > /tmp/mj.dat
I has gotten better more recently but I'm still seeing 3 msec or more
latencies with test programs.

Quoted text here. Click to load it
We set HZ to 2400 to be a harmonic frequency of 100 Hz, 80 Hz, 60 Hz, 50
Hz, .... The system appears to run fine (on a 1 Ghz Pentium III, dual
CPU) without a lot of extra overhead. We've been doing this for a couple
years now. We did find some broken drivers in 2.4.x which required
patches or other fixes. Our Gbit Ethernet card (Acenic) for example
would run OK at HZ12%00 but not HZ24%00 until relatively recently. YMMV.

  --Mark

Re: x86-linux sources of latency

[...]
Quoted text here. Click to load it

RTAI allows user space realtime. It's even the default/advised way in
the 3.X releases.

[...]
Quoted text here. Click to load it
You're definitely wrong about this in the context of RTLinux and RTAI; both
have nanosecond timer resolutions.

Herman

--
  K.U.Leuven, Mechanical Eng.,  Mechatronics & Robotics Research Group
    <http://people.mech.kuleuven.ac.be/~bruyninc Tel: +32 16 322480


Re: x86-linux sources of latency

Quoted text here. Click to load it

Just as vanilla 2.6 does. Timer resolution an scheduling granularity
are different, please don't mess them.

Dmitry

Re: x86-linux sources of latency
Quoted text here. Click to load it

Yes, I do enable it. Preemption does good things for userspace and
nothing bad for me.

Quoted text here. Click to load it

Hm. The aims are a bit different. The case with real-time tasks and
user-space processes is that there is no way to make them both hard
real-time and fully functional. For example, we periodically call our
rt-process. We may once (it's even normal) wake it up during a disk
interrupt.. so we can't do anything with filesystems; so on, we may
got woken up in our own (!) sleeping system call.. so we can't call
anything that sleeps.. And we come up with that yes, it is a
user-space process, but the program is limited to some very
restrictive API closely related to a 'real-time solution' being used.
I read that RTAI already allows such processes and that they are
strongly restricted in API, though really user-space.
And we are just as restricted while handling an interrupt that is
desynchronized with any kernel activities. Extereme care is needed for
not to break things.

Quoted text here. Click to load it

They are, I'll just make what is called "a pacth" and post it here in
a day.

Quoted text here. Click to load it

As I understand, you use now 2.4 + low_latency + preempt? It's a
combination that turns Linux into a soft-rt system (as far as I know,
it all began from the wish-letter from linux audio developers) and it
is really better than ordinary 2.6.
What I am going to do is to split into two parts: the first one makes
step-commands in userspace, it's fifo'ed with the second one, which is
in kernel-space and realtime. If such splitting is acceptable for you,
consider trying it.

Quoted text here. Click to load it

This test makes nothing to me. Anyway, I meausure it in an interrupt
and you do in userspace.. apples-oranges, really

Quoted text here. Click to load it

Well, as I mean hart-r.t., I assume that _nothing_ can preempt my task
and that it's guaranteed that the task will be up and running when I
want it.
I've just made a test (TimeStampCounter-based) with 1200Hz IRQ0's
(950-Mhz PC), it showed that there are stable periodic ticks in a
10-microsecond fidelity (+-5).
Interrupts on 50KHz also went fine, still with that 5-microsecond
drift.

Re: x86-linux sources of latency
Quoted text here. Click to load it
But I have had "hard real-time" capabilities as a user-space process
before. On SGI/Irix for example, you get this by using isolated and
restricted (must be both) processors. On a 28 CPU machine, dedicating a
CPU for non real-time tasks is OK, on a cluster of 2 CPU machines, It
seems a waste of 50% of my CPU capability to do that.

I also worked (many years ago) with a Honeywell DPS 6 system where the
tasks were run using the hardware's 32 (or 64?) level priority scheme.
Very nice for a hard real time system. You could put drivers below the
priority level of real time (user-space) tasks if you needed to,
something you can't do with Linux or a stock Unix system.

Quoted text here. Click to load it
Sorry - our application can't do that. It must be able to:
 - talk to the network [works with other real time systems, interface
with separate machines running the GUI display]
 - read / write the disk [simulating a real world hard disk drive]
 - get memory [but over 95% of that is allocated before starting
real-time]
and so on. No restricted API's allowed.

[we seem to agree we have an apples to oranges comparison]
  --Mark

Re: x86-linux sources of latency
Quoted text here. Click to load it
[ ... ]
Quoted text here. Click to load it

If you have hard realtime demands with full functionality, you'll need
an OS that was designed for it from the start, I'm afraid.  If you want
to stay POSIX/UN*Xish, my (biased) opinion is that LynxOS will probably
do what you want.  There are a couple of other OSes out there that also
claim POSIX and realtime, but the other ones I've worked with (eCos and
VxWorks) aren't very POSIX.

Caution: I'm a LynuxWorks shareholder, and worked on LynxOS for 8 years,
so, yes, I'm biased.
--
Steve Watt KD6GGD  PP-ASEL-IA          ICBM: 121W 56' 57.8" / 37N 20' 14.9"
 Internet: steve @ Watt.COM                         Whois: SW32
We've slightly trimmed the long signature. Click to see the full one.
Re: x86-linux sources of latency

Quoted text here. Click to load it

snip


 Beware of system management mode. In this mode the O/S isn't aware anything
happened. In the bios  setup make sure "enable legacy mode USB" is not
enabled ( we had problem with that when I worked at QNX software systems)


Quoted text here. Click to load it


Good luck
Pat



Site Timeline