Hi Jason,
First off, that's a pretty cool little algorithm... had to prove to myself that it worked. Let digit = 5 + n, where n = 0..4. We want new_digit = (digit * 2 + shift_in) % 10 = (5 * 2 + n * 2 + shift_in) % 10 = n * 2 + shift_in. If we add three, and wrap around at 16 (since 4-bit representation) then we get ((8 + n) * 2 + shift_in) % 16 = 2 * n + shift_in.
I'm not sure that I've found the problem with your code, but there were a number of stylistic issues and one suspicious fragment I figured I'd point out. I'll add a caveat that I haven't really coded up any VHDL in a year or so, and prior to that my experience was mostly with structural VHDL. And I didn't have a compiler handy to try out my ideas. So I imagine I'll find out I'm wrong and learn something in the process!
(0) I think the first problem is a lack of comments... uncommented code is always wrong :-)
(1) Process #3. I hate variables. Especially variables mixed with signals... you must be very disciplined when using variables (I only use them in for loops...). So I honestly don't know what the code will translate into, as you are assigning a value to variable, then assigning to the variable to a signal, then changing the value of the variable. I'm not sure if the scheduled signal assignment occurs before or after the change in variable value. Also, I'm not sure that the variable turns into a register (i.e. has memory) -- either way though, it's not what you want. If it does become a register, then have 2 24-bit registers (unnecsessary/may not work). If you don't, then I don't know what will happen. Regardless, I think you want to first be adding three, THEN shifting. There are two ways I can think of restructuring this code into something that may work (see below). The first replaces your variable with a signal that is computed combinationally outside of the process. The second is an attempt to still use a variable, based on my limited understanding of this VHDL construct. Note that neither requires the asynchronous condition (though I'm thinking it may need an aclr for bcd_out_s...).
if clk = '1' and clk'event then if load_data = '1' then bcd_out_s '0'); elsif shift_en_s = '1' then bcd_out_s