For decades I have used the following guidelines:
- Each real-time tasks should spend most of their time idle waiting for an event to process (waiting more than 99 % of the for the highest priority tasks).
- Do as little as possible in the highest priority tasks, i.e. lower priority tasks can execute longer at a time than the high priority tasks.
- At the bottom of the priority list the null tasks (which normally consumes the unused CPU time in an infinite loop) can be replaced by e.g. some round robin scheduler running non-RT tasks such as user interfaces etc.
Unexperienced people seem to increase the priority of those tasks that do not respond fast enough.
When trying to fix messed up messed up RT systems that others have made, the first thing is to look for tasks, for which the priority can be _lowered_, without causing significant performance loss for that task. When less critical tasks priorities have been moved lower, they are no longer competing with the most time critical tasks.
Then it is time to look at those tasks running a long time at each activation (but still needing high priority to avoid e.g. Rx buffer overrun) and move those parts to a lower priority task (which is allowed to execute longer at a time). For instance the received bytes are assembled into a complete message frame in a high priority task, while any application level message data processing is moved to a lower priority task (or even to the null-task).
A large number of priorities is nice, since you can assign a separate priority for each task. Of course, in most cases tasks with similar priorities could operate as well if the internal order is changed.
A moderate level 8-16 of priority levels is usually enough _provided_ that you have a full control of the priority levels. The situation gets complicated, if the OS is using some fixed priorities of its own.
Assuming that the OS has a task receiving messages and at a priority level immediately below it, an other OS tasks that consumes some of these messages addressed to it at a low urgency.
Assume that you want to process at a high urgency some messages received and addressed to you by the high priority task. Your task should run below the producer task, but above the low urgency OS consumer task. In this example, there is no free priority levels between the two OS tasks, so you end up putting your consumer task at the same priority level as the OS consumer task, fighting unnecessary for CPU time.
Please note, I have carefully used the words "addressed to", since otherwise typically in such discussions, someone will sooner or later claim that I am trying to use priorities to affect which task is processing which messages :-).
Absolutely, the OS should make it easy to isolate time critical and non-time critical parts into separate tasks.
The requirements for various RT systems vary so greatly that it is very hard for the OS designer to create a one size fits all solutions. In the worst case, they may even harm the total RT system, when forcing or at least tempting (e.g. providing an easy to use API) to use some solution that is not optimal for a particular RT system.