There are small, local stores of charge, so that when a chip or transistor needs a pulse of current from the supply, it doesn't have to travel through all the inductance of the path back to the supply main storage capacitor or regulator. You don't need one at every load in the system, but the higher the rate of change of current needed by some circuit, the more important it is to have a storage cap nearby.
Remember that such capacitors do not hold the voltage constant, since the only way to draw charge from them is to have their voltage change. But they reduce the voltage sag caused by a sudden current increase or the rise in voltage caused by a sudden current decrease.