Spirit rover OS problems - Page 9

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
Re: Spirit rover OS problems - OT: Priority Inversion

Quoted text here. Click to load it

This is what I read: "This problem was not caused by a mistake in the
operating system, such as an incorrectly implemented semaphore, or in the
application. Instead, the software exhibited behavior that is a known
"feature" of semaphores and intertask communication.".

Quoted text here. Click to load it

The article doesn't say that the workaround was not documented. It says, "In
fact, the RTOS used on Pathfinder featured an optional priority-inversion
workaround; the scientists at JPL simply hadn't been aware of that option.".

Quoted text here. Click to load it

I've stated in this newsgroup exactly twice before that I and colleagues
have written IVR software that ran stably on Windows NT (several years ago)
for 1.5 years before rebooting, and then only for good measure. Many people
whom I respect who have experience on both Windows and Linux agree that
there are just as many if not more problems with stability on Linux. Open
Source has its place but it is not the answer to reliability, though it's
nice and helpful to have the source.

Dilton



Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it

But the article *does* say that the priority inheritance option was
intentionally left out by WindRiver to optimize performance.  


Quoted text here. Click to load it

">The Wind River folks, for many of their services, supply global
Quoted text here. Click to load it

It seems pretty clear to me that WindRiver both knew about the potential
problem and how to fix it without making that information available to
the typical user of their products.  

Whether or not this is an important issue is debatable, but the facts
are clear.

--

Rick "rickman" Collins

snipped-for-privacy@XYarius.com
We've slightly trimmed the long signature. Click to see the full one.
Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it
River.
the

rickman,

Please provide that quote from the article, my 40 yr old eyes must be
failing me. :-)

What I got from the article was:

1) NASA determined that priority inversion was the cause of the 1997 probe
failure.
2) NASA had since 1997 to figure out how to fix this.
3) NASA sent the latest probes to Mars on US Tax dollars with this known
problem and it happened again.

Dilton



Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it

My 50 year old eyes found it :)

Did you follow the link titled, "Re: What Really Happened on Mars?"?  In
that one of the engineers on the project explained exactly what happened
and why.  

">3) The priority inversion option was deliberately left out by Wind
River in
Quoted text here. Click to load it


--

Rick "rickman" Collins

snipped-for-privacy@XYarius.com
We've slightly trimmed the long signature. Click to see the full one.
Re: Spirit rover OS problems - OT: Priority Inversion
On Mon, 09 Feb 2004 01:29:43 GMT, "Dilton McGowan II"

Quoted text here. Click to load it

Someone said earlier in this thread that priority inversion was the
cause of the recent Spirit rover problems, and yet, I can't find
anything that backs that up.  I can't even find any news that mentions
"priority inversion" in conjunction with Spirit.

All the news I read says that the problem had something to do with the
flash filesystem.
--
Alex Pavloff - remove BLAH to email
Software Engineer, ESA Technology

Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it

Initial reports: "priority inversion"; later reports: "flash files". My
guess (and that's all) is that both are true - consider that the problem
totally disabled the craft. I believe it was watchdogging into a
semi-shutdown mode, but may be wrong. I'd like to know.

Steve
http://www.fivetrees.com
http://www.sfdesign.co.uk



Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it

It would be pleasant to be able to read that article, but it
apparently contains some evil HTML, at least as far as Netscape
4.75 is concerned.  Without images it appeared to be missing most
of the meat, so I told it to load images, and it never completes
loading.

Show me the code.

--
Chuck F ( snipped-for-privacy@yahoo.com) ( snipped-for-privacy@worldnet.att.net)
   Available for consulting/temporary embedded and systems.
We've slightly trimmed the long signature. Click to see the full one.
Re: Spirit rover OS problems - OT: Priority Inversion

Quoted text here. Click to load it


Reads alright in Konq but  the article does *not* that priority inheritance
option was intentionally left out. What is says is:

This problem was not caused by a mistake in the operating system, such as an
incorrectly implemented semaphore, or in the application. Instead, the
software exhibited behavior that is a known "feature" of semaphores and
intertask communication. In fact, the RTOS used on Pathfinder featured an
optional priority-inversion workaround; the scientists at JPL simply hadn't
been aware of that option. Fortunately, they were able to recreate the
problem on Earth, remotely enable the workaround, and complete the mission
successfully.

HTH

Ian


Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it

Follow the money... I mean links!  

http://catless.ncl.ac.uk/Risks/19.54.html#subj6

--

Rick "rickman" Collins

snipped-for-privacy@XYarius.com
We've slightly trimmed the long signature. Click to see the full one.
Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it

What code?  

Try this link.  http://catless.ncl.ac.uk/Risks/19.54.html#subj6

BTW, I had no trouble viewing either page using Netscape 4.73.  I don't
think your problem was either the browser or the page.  


--

Rick "rickman" Collins

snipped-for-privacy@XYarius.com
We've slightly trimmed the long signature. Click to see the full one.
Re: Spirit rover OS problems - OT: Priority Inversion
On Sun, 08 Feb 2004 17:54:14 GMT, "Dilton McGowan II"

Quoted text here. Click to load it
[Stuff Snipped]

Quoted text here. Click to load it
[More stuf snipped]

Why do press articles alwasy say scientists as in the above paragraph
? It bugs the hell out of me when in all these projects no engineers
are ever acknowledged as having done anything. Hopefully it was not
really the scientists that was doing the engineering anf programming
of the Pathfinder.

Anton Erasmus



Re: Spirit rover OS problems - OT: Priority Inversion
Hi Dilton,

Quoted text here. Click to load it

By the way, to all: I am NOT referring to the above URL. I am
referring to the first reference quoted in it, to which the link is
<http://catless.ncl.ac.uk/Risks/19.54.html#subj6 . This latter
reference appears to be considerably less sanitized than the
embedded.com article.

Quoted text here. Click to load it

This is what *I* read (verbatim):

"Once we understood the problem the fix appeared obvious : change the
creation flags for the semaphore so as to enable the priority
inheritance. The Wind River folks, for many of their services, supply
global configuration variables for parameters such as the "options"
parameter for the semMCreate used by the select service (although this
is not documented and those who do not have vxWorks source code or
have not studied the source code might be unaware of this feature).
However, the fix is not so obvious for several reasons :"

Quoted text here. Click to load it

The article I reference above specifically says that it was
undocumented and its existence could only be learned by studying the
sourcecode.

Quoted text here. Click to load it

Can you point out where I ever said "Linux"? I said open source. I
didn't say Open Source(sm) :) I would point you to eCos, uCos-II, and
now even (kinda) Windows CE as examples of open source applications
that are not Open Source - and specifically not Linux.

I am specifically excluding OSes that are closed-source to the masses
but open-source to the rich illuminati/trough-fillers (potayto,
potahto).

I'm very happy your application ran for so long under Windows NT. This
phenomenon is commonly known as "luck" and is modeled
pseudo-mathematically in many role-playing games with more or less the
same accuracy as any reliability estimate you can give your client.

The problem is that you have no way of guaranteeing that it will
survive a given system loading condition, because without OS source
you have no hope of knowing what the machine is doing at any given
time, nor the tools to find out.

Any professional gambler-cum-mathematician will tell you that a
hundred good hands in a row from an unbiased pack of cards do not
predict the outcome of the next deal. Having the source is like having
X-ray vision at a card game. It doesn't protect you against an
opponent with a world-beating hand, but it lets you plan for it and
limit your losses.

Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it

Hi Lewin,

Ok, you and rickman make your points well. You were both referring to the
article referred to by the article you referred to originally. I read the
article not the reference of the article because you referred to the article
not to a reference contained within the article.



Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it

This is a quote of what I said in my original posting in this thread:

Quoted text here. Click to load it
really

But I did realize after reading some of the replies that people were
just clicking on the embedded.com link and not following my
instructions, which is why I clarified :)

Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it
the
article

Ah. :) I thought you meant the first reference of the article's author
possibly attributing the failure to X; I didn't realize that you literally
meant the first article referenced within the article.



Re: Spirit rover OS problems - OT: Priority Inversion

Quoted text here. Click to load it

Big statement. Our customers get the sources. But I doubt they will
really understand it nor even want to understand it.
(And our source code (though assembly) is shorter than the one from
WindRiver,M$)

But: If something goes wrong, having the source is a plus.
---
42Bastian
Do not email to snipped-for-privacy@yahoo.com, it's a spam-only account :-)
We've slightly trimmed the long signature. Click to see the full one.
Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it

Big statement :) If your customers have $400 million and their
national pride flying 100 million miles away, and a software problem
develops, they will damn well WANT to understand your sourcecode, and
will put however many brains are required into that task. If the
problem cannot be solved, then you better watch out for a
scapegoat-crucifixion squad knocking on your door...

Quoted text here. Click to load it

It's the only way to go. Look, even leaving issues like secrecy and
disclosability aside, if I have a problem with my app on your OS, it
is MUCH MORE EFFICIENT for you to give me the source for the OS than
for me to give you the source to the app(*).

In theory, you have already given me all how-and-why documentation on
your OS as part of the package (and if not - what the hell am I paying
for?). In theory, I have already read and understood all this
documentation. So I'm theoretically way ahead of you - I understand
ALL of my app and ALL of your OS (that you disclosed to me). Whereas
if I give you my app, I need to teach you all about my design
priorities, how I did things, and why I did them that way.

(*) - This statement is hereby limited to the special case of realtime
applications on RTOSes.

Re: Spirit rover OS problems - OT: Priority Inversion
Quoted text here. Click to load it

Right, this special kind of customer should also get a line-by-line
training of the OS (divided by the overall costs it would be nearly
$0)

Quoted text here. Click to load it

Good points.
Actually I don't need to be convinced, so far I had only good
experience giveing the source of any software we produce. But todays
problem is, that many people think of open source == Open Source ==
freeware. Or they buy a product then start changing the code and later
want support for what they might have crippled !

---
42Bastian
Do not email to snipped-for-privacy@yahoo.com, it's a spam-only account :-)
We've slightly trimmed the long signature. Click to see the full one.
Re: Spirit rover OS problems - OT: Priority Inversion
           snipped-for-privacy@yahoo.com "42Bastian Schick" writes:

Quoted text here. Click to load it

I am glad you have also had good experiences giving the source code
to your clients. However, this should only be done within the context
of a suitably trained client who is going to carry on using decent
development practices for anything they are going to change. For the
rest, keep the code away from them but be prepared to help them with
the problem solving (I speak from the context of providing completed
High Integrity Distributed Embedded Control Systems with no client
written code).

Where you are proviing just the underlying OS or base-line software
to which clients are adding the application code then I suppose the
issue gets somewhat muddier.

--
********************************************************************
We've slightly trimmed the long signature. Click to see the full one.
Re: Spirit rover OS problems
On Sat, 7 Feb 2004 10:18:48 -0000, "Steve at fivetrees"

Quoted text here. Click to load it
Uuh.


We at our company (search this group if interesseted which one :-)
wrote an RTOS which is very easy: 6 systemcalls are enough to handle
most problems (no, not 100 parameters :-)

---
42Bastian
Do not email to snipped-for-privacy@yahoo.com, it's a spam-only account :-)
We've slightly trimmed the long signature. Click to see the full one.

Site Timeline