I suspect your feedback resistor is drawing a significant current from the output stage, which loses most of its pull down capability when the output swings below a volt or so. Above about .6 volt output, the pull down is performed by a PNP follower, but below that, there is only a very low current single transistor acting as a current source and losing gain rapidly below one diode drop as the collector base junction goes forward biased.
Take a look at the graph of Output Characteristic Current Sinking at:
True rail to rail opamps usually have a higher current mosfet output transistor that looks like a low resistance when it is on, instead of like a current source.