Others have suggested the crowned roller.
Take a look at how the web is kept centered in the presses. The same methods should work for the belt. It involves adding an extra roller or two to the path so that
You can also use a split roller to do the job. This means that you have to add torque between the sections of the roller to push one part of the belt forwards with respect to the other. This can be made to work on a mechanically "automatic" manner much like the crowned roller.