Debugging RTOS problems in embedded systems

Dear All, Most of us involved with RTOS projects in embedded systems need to encounter at one point or other problems pertaining to real time situations:

1)DeadLocks 2)Racing conditions 3)Problems related to re-entrancy. 4)Problems related with priority setting

What are the general debugging techniques used to uncover problems related to above mentioned?I have earlier worked with windriver Tornado and Vxworks,and I got a tool called Windview which will show graphicaly the status of tasks as well as some analysis on timing.One can get an idea,if not exact,on what would have gone wrong in it. But some of the OS are propreitory and further custom OS provided by chip vendors(as is the case in my current product).These OS dont have any tools like windview for debugging.I would like to know what can be done to perform debugging?Clearly adding printf wont help in such situations.

I would also like to learn building tools on my own to debug such situations.

Can of you provide your suggestions or point me to a link ?

Looking farward for your replies and advanced thanks for the same,

Regards, s.subbarayan

P.S:I have crossposted the same query in comp.realtime to attract more inputs.

Reply to
ssubbarayan
Loading thread data ...

... snip ...

No you haven't. You have multiposted, and failed to set follow-ups, both of which are sins.

--
 
 
 
                        cbfalconer at maineline dot net
Reply to
CBFalconer

As you have already noticed the answer is usually RTOS dependent. Some RTOS have better tools available than others. Micrium

formatting link
has recently introduced a general purpose tool uC-Probe that may help. It is supposed to be OS independent. It is an application that communicates with your program and feeds data to a PC. You run your elf file through their product and it learns about all the variables in your code. You can then examine variables in real time. It works directly through some jtag devices or can get the data through a small driver you add to your code.

Scott

Reply to
Not Really Me

Allocate a huge block of RAM for a circular buffer. At each critical section you suspect, or for each thread, write a unique value to the next location in the memory block. When you break on the error, inspect the block of RAM to see who did or didn't do what it was supposed to.

Reply to
Ivanna Pee

Look at the symptoms and the source code and think.

Seriously.

--
Grant Edwards                   grante             Yow! I'm in direct contact
                                  at               with many advanced fun
                               visi.com            CONCEPTS.
Reply to
Grant Edwards

There are the only two debugging tools for everything: the smart head and the assiduous ass. Nothing can help until you know what are you doing. And if you know how the stuff works, there should not be much of a need to debug it.

I have earlier worked with windriver

Those tools are no more then the glossy toys for the beginners. A debug printouts in the critical places and toggling LEDs is sufficient to verify the operation.

VLV

Reply to
Vladimir Vassilevsky

Amongst the many good texts on O/S design and theory, I recommend LaBrosse's "uC/OS-II, The RealTime Kernel"; it includes example scenarios and debugging suggestions in generalized form that pertain to your stated issues.

Regards,

Michael

Reply to
msg

Yes, and I can remove an appendix with a bread knife and a pair of pliers, but that doesn't make those the best tools for the job.

--Gene

Reply to
Gene S. Berkowitz

Op Mon, 02 Jul 2007 07:41:57 +0200 schreef ssubbarayan :

No. Only possible in the classic and error-prone concept of 'shared resources'.

No. Only possible in the classic and error-prone concept of 'non-exclusive ownership of data'.

No. Only possible in the classic and error-prone concept of 'non-readonly data that belongs to all processes'.

Never heard of this.

If you didn't find these problems earlier, then the problem lies in the design phase.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
Reply to
Boudewijn Dijkstra

Search "priority inversion". Most famously it almost took out one of the Mars landers - even tho the problem had been known for decades.

Reply to
Dennis

Ah, but priority inversion I heard of. Priority inversion is not explicitly 'setting' the scheduler priority, as the OP implied, only the apparant execution priority. Another reason not to share resources between tasks.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
Reply to
Boudewijn Dijkstra

It's also possible to introduce bugs by explicitly setting priorities incorrectly. Sometimes the "intuitive" priority settings don't work the way you expect them to. Um, not that _I've_ ever had that problem, of course.

I suppose you can just throw handfulls of CPUs, memory chips, network interfaces, and whatnot at everything. That gets pretty expensive. I can just hear the engineering team explaining the design:

"Of course the device needs 34 Ethernet interfaces: 2 for SNMP (1 for the server, 1 to send alarms), 3 for the web servers (one for each CPU that's running a task than needs to be visible via the web), 1 for the tftp server, 6 for factory testing (one for each CPU), 1 for remote logging, 1 for outbound e-mail, and 2 for each pair of CPUs that need to talk to each other. We're going to be competing against a device with the same features but only 1 CPU and 1 Ethernet interface, so the competitor's costs are about 5% of ours, but we think it's worth it to avoid the evils of shared resources."

--
Grant Edwards                   grante             Yow! OVER the underpass!
                                  at               UNDER the overpass!
                               visi.com            Around the FUTURE and
                                                   BEYOND REPAIR!!
Reply to
Grant Edwards

Please explain how to connect a producer and a consumer without a shared resource.

--
 
 
 
                        cbfalconer at maineline dot net
Reply to
CBFalconer

Preferring intuition over documentation is always like walking on thin ice.

You could also (re)activate your brain and think about realistic ways to allow several tasks to use (not share) a resource. Yes, they have some overhead, but not in the order of handfulls of logic units.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
Reply to
Boudewijn Dijkstra

You're design projects come with documentation specifying how everyting is supposed to be designed? Must be nice.

How is several tasks using a resource not sharing resources.

I'm a little baffled as to how multiple tasks are going to "use a resource" (e.g. CPU, peripheral, data structure, memory pool) without that being "resource sharing".

--
Grant Edwards                   grante             Yow! !  I'm in a very
                                  at               clever and adorable INSANE
                               visi.com            ASYLUM!!
Reply to
Grant Edwards

I sense an interesting (or maybe "interesting") discussion coming...

Reply to
Mike Silva

I think you misspelled "futile"...

--
Grant Edwards                   grante             Yow! Is this going to
                                  at               involve RAW human ecstasy?
                               visi.com
Reply to
Grant Edwards

Full-duplex Ethernet?

But you're still sharing DMA buffers between a task and the Ethernet controller.

I guess the only answer is a giant super-loop with no interrupts or DMA.

--
Grant Edwards                   grante             Yow! Pardon me, but do you
                                  at               know what it means to be
                               visi.com            TRULY ONE with your BOOTH!
Reply to
Grant Edwards

Op Thu, 05 Jul 2007 17:22:09 +0200 schreef CBFalconer :

Without a shared resource, the producer and consumer can simply become sender and receiver of event objects, messages or signals, passed safely by the RTOS, a library or by a framework.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
Reply to
Boudewijn Dijkstra

I see. You want to avoid the whole problem by handing it to a magic OS function, with no understanding of the problems involved. You will go far.

--
 
 
 
                        cbfalconer at maineline dot net
Reply to
CBFalconer

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.