I wrote a multi-processor serial bus protocol. It was to connect several F340 as slaves to one Cypress FX2LP as master. The master sent out a command byte with the ninth bit set which alerted the slaves to what was happening next. Interested slaves then participated in various ways (but only one sending, of course).
It was unreliable.
Stripped down to one master and one slave, it was still unreliable. Eventually, I had the slave's ISR pulse when it received anything and pulse when it received a command byte (ninth bit set). Because of the nature of the protocol we could also produce a pulse when an unexpected command byte was received.
On the scope we could see those pulses and also the serial line going into the F340. We could easily follow the traffic, bytes being received, and pulses happening. We post triggered on the "unexpected command byte". And there we had it. The software had received again a command byte it had received a few bytes earlier but which had not come in on Rx.
I changed the software in the F340 to use UART0 (no FIFO) and the problem went away.
We have reported this concern (that their FIFO is somehow re-issuing a command byte) to SiLab. They wanted me to move the master software to the PC and use PortMon (I'd have thought a scope trace was much more conclusive). Then they wanted a small test program.
But we have work to do, having lost time on this problem. The email is just a flag in case you see something like this. I had a difficult time nailing this as timing seemed to matter, so adding/subtracting debug made it move around. Eventually logging events to a memory buffer showed what to look for on the scope.
HTH
Bill