When you changed from brush to brushless, you also changed servo amps (I am assuming, here). The AMC brushless amp, you say has some non-linearities. You get one angle comparison every 2.5 ms, what is the sampling interval of the servo amp? And, what is the bandwidth? It seems like you may need a much higher bandwidth than the typical servo amp provides. A bandwidth of more than 100 Hz would be of little use in the typical CNC machine tool, but it sounds like a bandwidth of several KHz would be needed here. I am assuming there is some digital circuitry in the servo amp (has to be at least a PWM wave generator, comparator and FET drivers). If there is a microprocessor in the servo amp, it most likely is sampling the velocity command (I guess really a torque command in this case) at a few KHz at most. But, even the random relationship between the PWM wave and the motor commutation signals will produce some torque jitter that may keep your loop from stabilizing. Certainly, for .01 degree P-P at 24,000 RPM, I can't imagine any standard motion control servo amp can handle this job. There are special motors and drives used in polygon spinners and disk drive servo writers that provide much tighter control (maybe this is already what you are using).
I did a PLL motor speed control for a laser photplotter some years ago. It used the age-old Motorola PLL 2-chip set, and the phase comparator fed a linear amplifier (basically an op amp with current-boosting transistors). The speed reference, however, is a 1024 cycle/rev encoder, rather than one pulse per rev. One quirk I had initially was that the rate the motor was spinning only produced a back-EMF of about 3 V, and having the emitter of the low-side Darlington transistor grounded did not provide a current path for braking the motor. So, it introduced a non-linearity in the transfer function, and made the servo loop unstable. Reconnecting the low-side transistor to a -5 V power supply allowed the motor to be effectively braked, thereby making the transfer function linear over a larger range of 2 quadrants (the other 2 are now possible, but not used.) This speed loop has to be pretty stable, as the output of the
1024-rev encoder is then run through an all-digital PLL X20 multiplier which has a pretty narrow lock range. It has never lost lock (which would cause glaring artifacts in the film output). The digital PLL produces the pixel clock of 20480 pixels/rev.Jon