scientists as superstars

Yebbut :)

Except that C# has a gaping chasm in that security: "unsafe".

I imagine too many programs make use of that "convenience".

I just mean anyone who's had a passing glance at an operating systems course would at least most likely understand.

"That which is not required is forbidden. That which is not forbidden is required."

This is not a problem. Really. Now, if you're debugging whether *memory* works or not, that's a different sorta bear hunt.

But it sorta shocks me how few developers seem to reach for an "instrumentation first" approach.

I guess they guess. :)

Mmmmm.... okay. Good instrumentation may invoke aliasing. but it's not necessary.

I should, at this point... by "instrumentation", I mean anything from free transmission of state to RFC1213 style "pull" approaches to $DEITY only knows. Basically, what used to be known as "telemetry".

Ah, baloney :) ( that bit was intended to be entertaining ) .

I just mean that this whole exersize in pilpul doesn't amount to too much.

Turns out you can, and it's fine.

Extremes are boring.

As should good C programs. Not the same thing as in Java but still. How can you measure something without... measuring something?

Overall, it's not that bad now. It wasn't that bad before. We have continents of embarrassments of options now.

Getting people to pay you for it is another story. Functional correctness is for lawyers now.

--
Les Cargill

We knew that was coming. :)

Oh my. Words fail.

Of coursed they do. Without the proper training, we're all some shade of destructive primate anyway.

--
Les Cargill

I use the clone system call directly in my Forth compiler, resulting in real tasks with shared memory.

My pre-emptive multitasker takes one screen plus one screen it shares with the cooperative multitasker. (A screen is a unit of program size,

16 lines by 64 characters.) The register preservation for system calls is reversely documented. Nobody is interested in what registers those system calls use, it is important which registers will not alter. That is particularly important to know for the clone register call, because registers are the only things that are possibly conserved. Bottom line, in 64 bit linux I use the undocumented feature that R15 is preserved across a clone call.

A similar screen in behalf of windows 32, makes this feature portable, (except for the library used, of course.) My experience with windows 64 is bad. The multitasker of windows contains no 32/64 dependancy (apart from the wordsize obviously ), but it just doesn't work. Microsoft follows the bad habit of the Linux folks to not properly document the system calls (dll) and instead forcing everybody to use "standard" c libraries or even c++ classes.

One example I use it on is a prime counting program. It speeds up approximately with the number of cores. It has been published on comp.lang.forth.

32 bit ciforth works in BSD the same than in linux, as far as I know. I see no reason why you couldn't use a cooperative scheduler within a pre-emptive task under my linux compiler. However that could be messy. So in principle yes.

I guess the real question is whether it can be done with any amount of ease by you, using your undisclosed favourite programming language.

--
This is the first day of the end of your life. 
It may not kill you, but it does make your weaker. 
If you can't beat them, too bad. 
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

Wouldn't the system call setpriority() work, after a thread has bee started?

--
This is the first day of the end of your life. 
It may not kill you, but it does make your weaker. 
If you can't beat them, too bad. 
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

Of course there are. The very minimal thing to do in order to have a computer to play chess is specify the rules. The Google Alpha Zero machine did exactly that, being "programmed" by just specifying the outcome. It beats the best chess engines after chewing a couple of day on those rules. (And no grand master is a match for the best chess engines today.)

--
This is the first day of the end of your life. 
It may not kill you, but it does make your weaker. 
If you can't beat them, too bad. 
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

I hate that mentality. The code should be as good as possible.

1). I will not add superfluous casts to make a pedantic compiler shut up. Those superfluous casts are a liability and may hide future bugs. A warning may set me thinking whether the code can be improved. 2). Some warnings are proper but do not change my mind about the code. It falls in the general concept of tests. Compare an expected outcome with an actual outcome. Some warnings just are part of the expected outcome. (It is known that the double result will fit in a double, so I ignore "possible truncation error". This will require putting a paragraph in the design/spec, why the computation results in a single. Beats adding a cast 1000 to 1.)

Those managers requiring a "clean built" instead of quality deserve to be shot. If you can't convince them by documenting what the warnings mean, and why they result from your code, find a better job.

In my experience objects are all but necessary in order to get complexity under control.

I take a designers stand. Objects or modules do not become fundamentally different by the way they are implemented.

I'v seen collegues being raised on lisp. That is fine, but recursive coding can get you in real trouble in real life. A mental picture of real computers is inexpensible. An interesting example is generating a new error flag each time you communicate. You will never find out that the communicating went wrong for the second time and the internet may be down.

Groetjes Albert

--
This is the first day of the end of your life. 
It may not kill you, but it does make your weaker. 
If you can't beat them, too bad. 
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

As I said way back in the summer, you can't change the priority or scheduling of an individual user thread, and you can't mix user and real-time threads in one process.

Thus you can have the default scheduler, which doesn't scale at all well for this sort of job, or you can run the whole process as real-time and make the box's user interface completely unresponsive any time the real-time job is running.

In Windows and OS/2, you can set thread priorities any way you like. Linux is notionally a multiuser system, so it's no surprise there are limits to how much non-root users can increase priority. I'd be very happy being able to _reduce_ the priority of my compute threads so that the comms threads would preempt them, but nooooooo.

On the plus side, there's been some rumor that the thread scheduler was going to be replaced. None too soon, if so.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs 
Principal Consultant 
ElectroOptical Innovations LLC / Hobbs ElectroOptics 
Optics, Electro-optics, Photonics, Analog Electronics 
Briarcliff Manor NY 10510 

http://electrooptical.net 
http://hobbs-eo.com

Oh, I could code around it, sure, but to fix the problem I'd have to reproduce some major fraction of the OS's thread scheduling. For performance it's very important to balance the workload between cores very accurately, because everything moves at the speed of the slowest one.

It would wind up being some hideous kluge with an N-thread userspace compute process and a 3N-thread real-time comms process sharing memory, with all sorts of mutexes and stuff.

The Linux version is a port of the original Windows/OS2 version, so there's no way I'm going to spend months recoding something that already works. It's way way cheaper to build a bigger cluster. (Better for my soul too--I'd be much less likely to start dreaming about sticking pins in a Linus Torvalds voodoo doll.) ;)

C++.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs 
Principal Consultant 
ElectroOptical Innovations LLC / Hobbs ElectroOptics 
Optics, Electro-optics, Photonics, Analog Electronics 
Briarcliff Manor NY 10510 

http://electrooptical.net 
http://hobbs-eo.com

I don't think that this is true for Red Hat Linux Enterprise Edition. I don't recall if the realtime stuff is added, or built in. People also use MRG with this.

More generally, not all Linux distributions care about such things.

Joe Gwinn

It's in the kernel, so they mostly don't get to care or not care. I've recently tried again with CentOS, which is RHEL EE without the support--no joy.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs 
Principal Consultant 
ElectroOptical Innovations LLC / Hobbs ElectroOptics 
Optics, Electro-optics, Photonics, Analog Electronics 
Briarcliff Manor NY 10510 

http://electrooptical.net 
http://hobbs-eo.com

As a work around can you not put a few ms wait in one of the outer loops so that the core OS gets a turn for the UI updates every second or so?

VBA in Excel fails that way if you don't take countermeasures. DoEvents is the magic incantation that sorts it out there.

It does seem odd that you cannot be user real-time and still at a lower priority than system level real-time. A cooperative approach to letting the OS have a very short timeslice now and then might work...

OS/2 was so good at it that someone wrote a system level driver to implement the 16550 buffered serial chip as a device driver for the original stock Intel 8250 part.

--
Regards, 
Martin Brown

+1

Production code should compile without any errors and warnings. That way if a change breaks something it stands out clearly.

I always used to hate IBM's FORTRAN G for its uncertain message at the end of an entirely successful compile:

NO DIAGNOSTICS GENERATED?

The "?" was a trailing null being translated to "?" on the chain printer and their change procedure made it "too painful" to fix it.

We have generally always learned something when porting code to pedantic compilers on new machines. ISTR the Z80 FORTRAN compiler was one of the pickiest we encountered with none of the common extensions that were tolerated on most of the mainframes at the time.

Hardly. It makes it much easier to spot something that has gone awry if the compiler generates a warning on the latest compile.

I knew one slightly mischievous Modula2 compiler where the default behaviour for an attempt to use an uninitialised variable on the RHS of an equation was to issue a compiler warning and code a hard execution trap. I promoted all such "soft warnings" to hard compile time errors.

They are good for some things but not for everything.

These days compilers do better optimising than most humans can ever hope to since they understand the pipelines better as to what can execute in parallel, speculative execution and how branch prediction interact. However, they are not perfect at it and sometimes do really dumb things.

It is only worth hand tweaking code that has to be either insanely accurate or very very fast (usually only a few lines of source worth). I have found profile directed optimisation rather lacklustre YMMV.

Keeping intermediate results on the FPU stack can gain a worthwhile gain in precision in certain otherwise tricky problems. MS C withdrew convenient access to the hardware supported 80bit reals around v6.0.

Recursion is sometimes a useful tool though.

--
Regards, 
Martin Brown

Let me add that while it does take root permission to turn realtime on, it does not follow that the application must also run as root.

Running a big application as root is a real bad idea, for both robustness and security reasons.

What I've seen done is that during startup, the application startup process or thread is given permission to run sudo, which it uses to set scheduling policies (FIFO) and numerical priority (real urgent) for the main processes and threads before normal operation commences.

The other thing that is normally set up during startup is the establishment of shared memory windows between processes. For may applications, passing data blocks around by pointer passing is the only practical approach.

Joe Gwinn

They're my boxes, and when doing cluster sims they usually run a special OS for the purpose: Rocks 7. It has one head-node and N compute nodes that get network-booted off the head, so I get a clean install every time they boot up. So no worries there.

There's no point in running as root, though, because you can't change the priority of user threads even then--only realtime ones.

The Linux thread scheduler is a really-o truly-o Charlie Foxtrot.

Passing pointers only works on unis or shared-memory SMP boxes. My pre-cluster versions did it that way, but by about 2006 I needed 20+ cores to do the job in a reasonable time.

I was building nanoantennas with travelling wave plasmonic waveguides, coupled to SOI optical waveguides. (They eventually worked really well, but it was a long slog, mostly by myself.)

The plasmonic section was intended to avoid the ~30 THz RC bandwidth of my tunnel junction detectors--the light wave went transverse to the photocurrent, so the light didn't have to drive the capacitance all at once. That saved me over 30 dB of rolloff right there.

The issue was that metals that exhibit plasmons (copper, silver, and gold) exhibit free-electron behaviour in the infrared, i.e. their epsilons are very nearly pure, negative real numbers. That makes the real part of their refractive indices very very small, so you have to take very small time steps or the simulation becomes unstable due to superluminal propagation.

Their imaginary epsilons are very large, so you need very small voxels to represent the fields well. Small voxels and short time steps make for loooong run times, especially when it's wrapped in an optimization loop.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs 
Principal Consultant 
ElectroOptical Innovations LLC / Hobbs ElectroOptics 
Optics, Electro-optics, Photonics, Analog Electronics 
Briarcliff Manor NY 10510 

http://electrooptical.net 
http://hobbs-eo.com

Yeah, probably, but stuff like the disk driver and network stack eventually get hosed too.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs 
Principal Consultant 
ElectroOptical Innovations LLC / Hobbs ElectroOptics 
Optics, Electro-optics, Photonics, Analog Electronics 
Briarcliff Manor NY 10510 

http://electrooptical.net 
http://hobbs-eo.com

I hadn't heard of Rocks 7. I'll look into it.

So, why not use realtime ones? This is done all the time. The application code works the same.

The legacy UNIX "nice" scheduler, retained for backward compatibility, it unsuited for realtime for sure. And for what you are doing it would seem.

The present-day large-scale solution is to use shared memory via Infiniband. Memory is transferred to and from IB to local memory using DMA (Direct Memory Access) hardware.

This is done with a fleet of identical enterprise-class PCs, often from HP or Dell or the like.

Wow! That's worth a bunch of trouble for sure.

Are there any metals that don't exhibit plasmons?

If I recall/understand, the imaginary epsilons are the loss factors, so this is quite lossy.

This sounds like a problem that would benefit from parallelism, and shared memory. And lots of physical memory.

Basically divide the propagation media into independent squares or boxes that overlap somewhat, with inter square/box traffic needed only for the overlap regions. So, there will be an optimum square/box area or volume.

Joe Gwinn

*sigh* One more time, with feelin'.

Because putting compute-bound processes in the RT class detonates the UI and eventually brings the other system services such as disk and network to their knees.

You can call it names all you want, but that doesn't change anything. There's only the one available, and it stinks on ice.

If it has to cross a network it isn't shared memory. Shared memory is a uniprocessor or SMP thing.

Riiiggghhhht. Lots of processors talk to their local memory over Infiniband because of its low latency. Not.

Sure, a good 100 of them. In fact all except copper, silver, and gold AFAIK. The undergraduate "normal conductor" model of metals cannot exhibit plasmons because it predicts that the real and imaginary parts of the refractive index have equal magnitude.

If the imaginary parts of epsilon is less than or comparable with the real part, that's true. However, at 1.5 um, silver has a refractive index of 0.36+j10, corresponding to an epsilon of -100 + j3.5, which is in an entirely different regime from normal conduction. A material with a negative real epsilon at DC would turn to lava very rapidly.

Free-electron metals can propagate surface plasmons over many microns' distance.

Right. And many many threads with sensible priority scheduling that doesn't assume that the Kernel Gods know better than you how your program should run on your own boxes--as in OS/2 or Windows, but not Linux.

I have no idea what that means. If you're interested in how the program actually works, the manual is at

It's written mostly for our own benefit, so it isn't too polished, but you can easily get the gist.

Cheers

Phil Hobbs

--
Dr Philip C D Hobbs 
Principal Consultant 
ElectroOptical Innovations LLC / Hobbs ElectroOptics 
Optics, Electro-optics, Photonics, Analog Electronics 
Briarcliff Manor NY 10510 

http://electrooptical.net 
http://hobbs-eo.com

That is exactly how many realtime systems work - it is assumed that those realtime processes and threads will be designed and used to not eat the computer without the Chief Engineer's permission, a matter settled at design review time.

But "realtime" is not a simple binary property - it has degrees and details.

In Linux, the squeeze-out behaviour is controlled by the scheduling policy, and not directly by the numerical priority (which encoders relative urgency).

.

It sounds like you want to use SCHED_RR, and not SCHED_FIFO.

FIFO is typical in big radars, and RR in small intense systems. In the old days, it was hard frames driven by hardware time interrupts.

SCHED_RR ensures fairness between RT threads, and so no squeeze-out within that family. But one must still meter how much of that is used, of you will squeeze the GUI et al out for sure. But see "Limiting the CPU usage of real-time and deadline processes" in the following.

.

Hmm. Is there any operating system that's really suitable?

More generally, I went through the development of the POSIX standard, and the issues of scheduling policies and numerical priority were well chewed, attempting to accomodate just about everything. And the big radars I work on all use RHEL these days, having ejected all proprietary UNIX flavors for business reasons.

True, but we do it anyway, if the hardware is fast enough.

Well, sort-of. IB is slower than within-box transfer, which is why one uses IB only for lateral communications between solvers running in parallel on the boxes, which have immense amounts of local memory.

These systems are limited by memory system bandwidth, and not so much CPU speed. Usually, the same patch of memory will not support more than one DMA operating at the same time.

All the low-loss metals have those plasmons. Aww.

OK. I'm out of my expertise here.

Hmm. What OS/2 and/or Windows scheduling policy are you happy with?

I scanned the manual. I think that POEMS' Domains and Subdomains are my squares (2D) and boxes (3D). I knew you had to have something of the sort, or we would not be having this discussion.

Joe Gwinn