Thanks for the reply! But I'm not sure I follow. R5 puts a load on the LM317 to keep it happy. The ADJ current gets sunk by U2 input. The idea wasn't that the LM317 would provide the output current, but it would act as a cheapo reference so that 0 volts would give a 4 mA output. LM317s in TO-92 packages are really inexpensive! Unfortunately it's moot, as the OP mentioned below having 4 mA at 0 volts input wasn't really his requirement.
This LM317 model seems to misbehave in the standard voltage regulator circuit too.