The fact that I would only get sound in one ear probably wouldn't
> matter at all since one of my ears is really, really bad and the other
> one is only sort of bad.
You don't have to get sound in only one ear - just parallel the left and right phones. The impedance will be half, but take that into account for your attenuator.
Cheers! Rich