Yes and no -- If you use a PLL with a linear time-invariant loop filter, then it'll certainly take a huge number of seconds to spin up.
If getting down to the best possible number in the least amount of time is important, you can speed things up considerably by using a Kalman filter -- it's almost a classic use case, and you've got an FPGA in the mix. I'd have to, like, expend effort to get exact numbers (and I'd be guessing on the oscillator stability anyway), but you're talking about getting down to less than 10ns RMS error in well under a minute.