That is the whole idea of 3D secure. If they give the correct answer to the challenge then either you wrote the keyphrase down and were negligent or you made the transaction. At least that is what they will claim - unfortunately it isn't necessarily true.
In the UK if you dispute an invalid bogus charge and still have physical possesion of the card then they unwind the transaction completely and cancel the card with immediate effect. Their fraud department sometimes spots a suspect transaction and rings you up the same day. I have had it happen a few times over the years. The flip side code is too weak and these days everyone knows how to generate a legal Barclaycard number. A direct brute force attack need only try 1000 combinations to be assured of getting a valid flip side match. And there are plenty of online sites that do not use 3D secure.
The card issuer can ask for the card to be returned if they suspect the owner has physically lost it. This arose in the UK after some payment machines in petrol stations had been "serviced" to skim cards in bulk.
They were horrified to find their customers did have possession of the original cards. Now chip & PIN has forced most skimmers to go overseas and use the magnetic stripe in previous generation cash machines. However, that is about to change as the encryption has been broken - or rather a defect in the verification protocol has been found that allows a PIN is OK signal to be sent no matter what number is typed in!
Regards, Martin Brown