Debugging RTOS problems in embedded systems

Jul 02, 2007 40 Replies

Dear All, Most of us involved with RTOS projects in embedded systems need to encounter at one point or other problems pertaining to real time situations:



1)DeadLocks
2)Racing conditions
3)Problems related to re-entrancy.
4)Problems related with priority setting

What are the general debugging techniques used to uncover problems related to above mentioned?I have earlier worked with windriver Tornado and Vxworks,and I got a tool called Windview which will show graphicaly the status of tasks as well as some analysis on timing.One can get an idea,if not exact,on what would have gone wrong in it. But some of the OS are propreitory and further custom OS provided by chip vendors(as is the case in my current product).These OS dont have any tools like windview for debugging.I would like to know what can be done to perform debugging?Clearly adding printf wont help in such situations.



I would also like to learn building tools on my own to debug such situations.



Can of you provide your suggestions or point me to a link ?



Looking farward for your replies and advanced thanks for the same,



Regards, s.subbarayan



P.S:I have crossposted the same query in comp.realtime to attract more inputs.



... snip ...

No you haven't. You have multiposted, and failed to set follow-ups, both of which are sins.

cbfalconer at maineline dot net

As you have already noticed the answer is usually RTOS dependent. Some RTOS have better tools available than others. Micrium

formatting link
has recently introduced a general purpose tool uC-Probe that may help. It is supposed to be OS independent. It is an application that communicates with your program and feeds data to a PC. You run your elf file through their product and it learns about all the variables in your code. You can then examine variables in real time. It works directly through some jtag devices or can get the data through a small driver you add to your code.

Scott

Allocate a huge block of RAM for a circular buffer. At each critical section you suspect, or for each thread, write a unique value to the next location in the memory block. When you break on the error, inspect the block of RAM to see who did or didn't do what it was supposed to.

Look at the symptoms and the source code and think.

Seriously.

Grant Edwards grante Yow! I'm in direct contact at with many advanced fun visi.com CONCEPTS.

There are the only two debugging tools for everything: the smart head and the assiduous ass. Nothing can help until you know what are you doing. And if you know how the stuff works, there should not be much of a need to debug it.

I have earlier worked with windriver

Those tools are no more then the glossy toys for the beginners. A debug printouts in the critical places and toggling LEDs is sufficient to verify the operation.

VLV

Amongst the many good texts on O/S design and theory, I recommend LaBrosse's "uC/OS-II, The RealTime Kernel"; it includes example scenarios and debugging suggestions in generalized form that pertain to your stated issues.

Regards,

Michael

Yes, and I can remove an appendix with a bread knife and a pair of pliers, but that doesn't make those the best tools for the job.

--Gene

Op Mon, 02 Jul 2007 07:41:57 +0200 schreef ssubbarayan :

No. Only possible in the classic and error-prone concept of 'shared resources'.

No. Only possible in the classic and error-prone concept of 'non-exclusive ownership of data'.

No. Only possible in the classic and error-prone concept of 'non-readonly data that belongs to all processes'.

Never heard of this.

If you didn't find these problems earlier, then the problem lies in the design phase.

Gemaakt met Opera's revolutionaire e-mailprogramma: http://www.opera.com/mail/

Search "priority inversion". Most famously it almost took out one of the Mars landers - even tho the problem had been known for decades.

Ah, but priority inversion I heard of. Priority inversion is not explicitly 'setting' the scheduler priority, as the OP implied, only the apparant execution priority. Another reason not to share resources between tasks.

Gemaakt met Opera's revolutionaire e-mailprogramma: http://www.opera.com/mail/

It's also possible to introduce bugs by explicitly setting priorities incorrectly. Sometimes the "intuitive" priority settings don't work the way you expect them to. Um, not that _I've_ ever had that problem, of course.

I suppose you can just throw handfulls of CPUs, memory chips, network interfaces, and whatnot at everything. That gets pretty expensive. I can just hear the engineering team explaining the design:

"Of course the device needs 34 Ethernet interfaces: 2 for SNMP (1 for the server, 1 to send alarms), 3 for the web servers (one for each CPU that's running a task than needs to be visible via the web), 1 for the tftp server, 6 for factory testing (one for each CPU), 1 for remote logging, 1 for outbound e-mail, and 2 for each pair of CPUs that need to talk to each other. We're going to be competing against a device with the same features but only 1 CPU and 1 Ethernet interface, so the competitor's costs are about 5% of ours, but we think it's worth it to avoid the evils of shared resources."

Grant Edwards grante Yow! OVER the underpass! at UNDER the overpass! visi.com Around the FUTURE and BEYOND REPAIR!!

Please explain how to connect a producer and a consumer without a shared resource.

cbfalconer at maineline dot net

Preferring intuition over documentation is always like walking on thin ice.

You could also (re)activate your brain and think about realistic ways to allow several tasks to use (not share) a resource. Yes, they have some overhead, but not in the order of handfulls of logic units.

Gemaakt met Opera's revolutionaire e-mailprogramma: http://www.opera.com/mail/

You're design projects come with documentation specifying how everyting is supposed to be designed? Must be nice.

How is several tasks using a resource not sharing resources.

I'm a little baffled as to how multiple tasks are going to "use a resource" (e.g. CPU, peripheral, data structure, memory pool) without that being "resource sharing".

Grant Edwards grante Yow! ! I'm in a very at clever and adorable INSANE visi.com ASYLUM!!

I sense an interesting (or maybe "interesting") discussion coming...

I think you misspelled "futile"...

Grant Edwards grante Yow! Is this going to at involve RAW human ecstasy? visi.com

Full-duplex Ethernet?

But you're still sharing DMA buffers between a task and the Ethernet controller.

I guess the only answer is a giant super-loop with no interrupts or DMA.

Grant Edwards grante Yow! Pardon me, but do you at know what it means to be visi.com TRULY ONE with your BOOTH!

Op Thu, 05 Jul 2007 17:22:09 +0200 schreef CBFalconer :

Without a shared resource, the producer and consumer can simply become sender and receiver of event objects, messages or signals, passed safely by the RTOS, a library or by a framework.

Gemaakt met Opera's revolutionaire e-mailprogramma: http://www.opera.com/mail/

I see. You want to avoid the whole problem by handing it to a magic OS function, with no understanding of the problems involved. You will go far.

cbfalconer at maineline dot net

Join the Discussion

Have something to add? Share your thoughts — no account required.

Didn't find your answer?

Ask the community — no account required