You still have to charge the gate of the switch through the finite resistance of the pass-gate. This is slower than an active driver, particularly at the low corner.
The alternate is a (comparatively) large driver driving one end of a weaker (order of magnitude) keeper. This is usually faster at low voltages and is the more common latch at the high end.