So after a fair bit of tinkering I finally pushed this out. I'm calling it Hybridlink, and as far as I know it's the first open source firmware replacement for the ST-Link. The whole idea was to have something you can flash yourself and actually read the code for, instead of treating the dongle as a black box.
The write-up ended up as STM32 Tutorial #106, with the fitting title REPLACED ST-LINK. For the demo build I wired up a few green boards back to back and linked them with a multicolored ribbon cable, which honestly looked neater than I expected once it was all sitting on the desk.
There's still plenty I want to clean up in the code, but I'm pretty happy with where it stands. If you've ever been curious about what actually goes on inside one of those little debug probes, this is a fun one to poke thru.