Suppose I have a permanent magnet and attract a nail a height h from the ground. If the nail has mass m then I have done mgh of work (potential energy) which must come from the field. If I then attract a second nail the magnetic attraction will be less since I have used energy when attracting the first one - which is still attached. If I keep doing this there is little energy left to lift anything more.
I then pull off the nails which requires energy from me. That energy must be released back into the field since I can then replicate the experiment. Is this right? So energy is stored in permanent magnets.