The word "n-coder" is confusing me a bit, because "n" usually represents a natural number. It is also preventing search engines from finding your post. So please say "encoder". IMO "n-coder" is neither cooler nor shorter than "encoder".
What kind of encoder are you using? I assume it is a quadrature, but am not sure.
The outputs of a quadrature decoder look like this, when turned in one direction (use a font with constant width to view)
__ __ __ A __ __ __
__ __ __ B _ __ __ _
Search the web for "quadrature decoder avr"
You are right here ;-)
It is generally a good idea to mention what compiler and microcontroller you are using.
Recent versions of gcc don't need the "inb" function, just write if (!(PIND & 8)) or even better if (!PIND.3)
Same as above for "outp". Better write DDRD = 0;
Can't say more about the code, until I know more about your decoder.
/Jan-Hinnerk