If that is the case then use the algorithm that I suggested but divide and round the result. I.e. take the state, add two (to avoid the transition on 0 0), then ignore the bottom two bits. This should give you something that will only switch when you're right between detents.
I hit the send button and then thought of a better way:
Use my suggested algorithm, but only 'latch' the data from it when the switch reads 0 0. You may sometimes fail to register a move, but you only _will_ register a move when the switch is right where it belongs, and it can dither around the right answer without you uselessly reloading your synthesizer.
That sounds like a mechanical issue, are you stuck with this encoder ?
ie There are different combinations of encoder out there, with Detent-edge ratios ?
Otherwise, a solution might be to discard some resolution, for stability. - resolve to full 4 quadrants, but only action on some count either side, so it's really a SW backlash. Not as nice to use, but it could tolerate a last-set-click encoder ?
Coincidently, I've just struck a somewhat similar switch-fudge-needed related problem.
The solution is to remove the debounce, but add a speed-related 'gain'- so as the thing spins fast, it uses all the quad edges, and thus gets around the scale fast, and below some set-edge-rate, you change the edge gain, so instead of each edge, you use edge pairs, or even whole cycles.
That means no 'edge afterthought' effects, and you can use the low-spec encoder.
Debounce is not needed on Quad Encoders (fully decoded), but you do need to be quick enough follow them at the highest speed.
I tried the algorithm proposed by Tim Wescott first and it worked much much better. It still skips a staion once in awhile. I put larger dia knob on on it and it's very usable now.
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.