If you were implementing a parallel addressed asynchronous bus, possibly using your own protocol, then an off-the-shelf VME rack could speed up design time. Since you want serial I'd say it was OTT.
Personally I would use a 4 wire RS485 along an impedance controlled backplane and use high speed UARTs (or possibly small FPGAs) and a micro on each card to convert to your local SPI interfaces and control the multiplexing. Using 4 wire RS485 means that you don't need to worry about when you turn the drivers on, so it can be treated as if it were a half duplex serial interface from the controller's end.
If you are switching power on and off I agree with the sentiment of having a higher voltage supply on the backplane and regulate locally on the cards.
BTW I believe FireWire is full duplex, as is USB 3.0. USB is possibility, there are 'helper' USB chips and device interface chips, but you do have some software issues and you'd need point-to-point connections via hubs (Linux supports USB though).
Mark.