Here are a couple of ideas for parts-count-limited headphone amps that I came up with.
#1, using 5 active devices:
I liked the idea of an inductive loaded output, as John Larkin's amp had. But I don't want to pay for an audio frequency inductor! This one uses a bootstrapped N-channel MOSFET as a constant current source to simulate one. The same bootstrap is used to provide gain to the previous stage. It's class A single ended but also incorporates negative AC feedback and DC stabilization into the same loop.
Unfortunately, this circuit's output swing is limited because the vertical MOSFET models provided by LTSPice all have a pretty high threshold voltage. In a real application it would probably have much better output swing with audio lateral MOSFETS with lower threshold voltages, but the parts values would have to be adjusted.
#2, using 4 active devices:
Class A push-pull. Same thing about MOSFETS with high threshold voltages goes for this circuit. Tweak the R12 and R14 pots until the output is centered and the quiescent current is approximately half of the peak output current. I'm less confident about how well this one would perform; R2 and R3 and R1 and R4 would have to be matched.