I see a number of problems.
The cheapest way to manufacture such a device is with single-throw switches. That is, pressing a switch doesn't disconnect anything. Rather, it simply connects the selected input to the output jacks. The previously selected input is disconnected by the mechanical interlock that releases the previously pressed switch. If you forced down two or more input switches, all their signals would be connected to the output.
Unfortunately, switching the selected input _between_ two outputs requires a double-throw switch. And I would bet my life that this unit doesn't contain any.
There is also the fact that the board is hard-wired, and trace-cutting and re-routing will be needed.
I'd suggest this... Buy two of these switches. Connect their _outputs_ (yes, their outputs) together. You then connect the program sources to the inputs of one of them, and the monitors, etc, to the inputs of the other. Select the input on the first, then the display on the second. Easy as eating pie. Four inputs on one, four displays on the other. Any one to any other one.
QED.
I'm waiting to hear people argue about this.