We've done this sort of thing already in the early 90's :-)
On a chip Jim has a ton of opportunities that us discrete designers don't have. Such as very good matching between neighboring components. Provided they are oriented in the same direction but that's easy.
You can then run DC or an otherwise calibrated RF signal through a reference chain and error-correct the output. If the loop Jim needs doesn't have to be super-fast one could even servo.
However, 1mV accuracy at 500mV is no small feat at 500MHz.