I built and tested this. It seems to work well enough for what I'm going to
> use it for. The only problem is it toggles a few times before it latches,
> but it eventually does latch. This is the fault state where the generator
> feeds back into the dead grid.
the entire point of a transfer switch is to prevent with 100% certainty, (even if the transfer switch faults) output from a generator to back feed into the utility grid- because it's a very dangerous situation.
At a minimum, there is nothing at all in what you made that prevents the generator output from connecting to the utility.
This is done mechanically in the real world with giant switches where the moving contact connects to the load and cannot physically connect to both grid and generator or physically connected the generator to the grid. There's no way it can fail in a dangerous state, even if you spray the control electronics with salt water and then walk up to it and try to run with switch in manual mode.