Is it normal to just try every part instead of doing the math?

Both of you are right, honestly. Your method is how a lot of real troubleshooting gets done, and his method is how you do it when the drawer is empty. The funny part is that his hour of math landing on the same capacitor you picked in ten minutes is actually the system working as intended. The calculation narrows things down before you ever touch a wire, and the test confirms it. When you've got the candidates in hand, the test is faster and it catches reality, because nameplate formulas are approximations and a motor that's 30 years old doesn't match its own nameplate perfectly anyway.

That said, one thing matters here more than smoothness: getting the size roughly right. A run capacitor that's way off makes the motor run hot and lose torque even if it sounds fine, and by the time you feel the heat you've alredy cooked the windings. So don't judge by ear alone. Check the old dead one's markings if any survive, and aim to land within about ten percent of that value. Voltage rating shold meet or beat the original. If your three spares are all in that neighborhood, then yes, swap them one at a time and pick the one that runs coolest and draws the least current. Also, kill the breaker between swaps, not just the switch. Pump motors on a wet slab are not where you want to get sloppy.

I've done it both ways plenty of times. Calculation when I'm ordering a part, testing when the part's already on the shelf. Your father in law knows that too, which is probly why he was amused instead of annoyed.

One thing I'd add, those drawer spares might not be what thier labels say anyway. Caps lose capacity as they sit, expecially cheap ones. I had a spare that was marked 45uF but tested closer to 30 when I finally checked it, and the motor ran like garbage on it untill I figured that out. A cheap meter with a cap setting pays for itself real quick if you're gonna keep borrowing from the shed. Also worth popping the old dead one open outside, if it's bulged or leaked the mess isn't something you want indoors.

Back in the day we didn't have cap testers in every truck, so the swap-and-feel method was just what you did. I've tuned more than a few motors by touching the case after ten minutes of running. Warm is normal, too hot to keep your hand on is your answer. One thing I'd pass along: if you're going to guess between two values, go a touch under rather than over. Oversizing pushes more current through the windings and that's what cooks them. Keep a paint pen in the shed drawer and write the tested value on each spare once you check it. Saves the next guy the same afternoon.