Looking for a non-permanent way of attaching wires?

You could use a screw terminal:

Why would you want to change the LED? LEDs last a long time.

Perhaps look for a part socket? Sometimes I hackup connectors and turn them into part sockets.

D from BC British Columbia Canada

Any little 2-pin socket for 25 mil square posts. Mouser and Digikey have tons of such things.

John

I has to be fairly small as well for a tight space, forgot to add that...

Also, the positive and negative terminal are spread in different directions, so it would not be a case where I could have them both together in a socket, if that makes sense...

Maybe try something like this??

D from BC British Columbia Canada

Ever heard of crimp wire ends?

No never heard of them, I will look them up on google.

Yes, but even that is overkill.

I would leave the LED legs long, use the lead length to ID the anode lead, and make a mount device that one bends the leads around, that then allows it to be inserted into a socket lined with metal strips, like a cheap battery clip.

You solder your lines to the "battery clip" type socket strips, and all your LED insertions are replaceable from then on.

It is just a .25" x .38" x .125" piece of plastic or balsa wood. The leads make a 180 degree bend at the bottom and point back up toward the LED body. That makes one conductor on each side. Adjust the .38" dimension till you get the desired "contact" length.

The only problem is that all LEDs are not made the same, so you would be tied to a single brand, or have to make a new mount device for each new LED you try.