After reading the entire thread (with lots of coffee ;) , here are some things.
Even though the ethernet is routed as a differential pair, do you have any stubs for test points? Have you measured the impedance yourself with a decent TDR machine? (Do both single ended and differential TDR for best results).
When you failed, did you get a 3D polar plot for the emissions? That can help isolate where the emissions are coming from. Failing that, do you have a simple passive sniffer (there are schematics available on the net) and a spectrum analyzer?
Is the noise coming from the ethernet device itself? Is it properly decoupled? (You'd get clock noise into the power system).
Is your master oscillator noisy? I notice that for RMII mode, the XT2 pin is a 50MHz clock reference, rather than being able to use a crystal. (See datasheet pages 9 and 11).
formatting link
Did you follow the layout guide? (There are some here who would take exception to some of it's recommendations!)
formatting link
If it is fairly easy to switch to MII mode, you might give it a try. The only thing it seems to save in RMII mode is two transmit MII and two receive MII tracks, at the expense of doubling the signaling frequency on the remaining signals.
I would note that in MII mode, you can use a 25MHz crystal rather than an external 50MHz clock which is single ended by definition (at least at the input to the chip). If the routing of the clock is not dealt with in the same manner as other possible radiators (impedance controlled and terminated) you can expect emissions. You can also expect harmonics (which you seem to be getting). Apart from that, you reduce the signaling frequency on the TX[n] and Rx[n] tracks anyway.
Just something to think about
Cheers
PeteS