Yes: be prepared for frustration. Wire your board exactly like the Butterfly. Don't use the multiplexed JTAG I/Os for anything else. Be prepared for a lot of weird stuff with order of powering things up and plugging things in. Be prepared to lose contact with the debugger randomly in the middle of a debug session.
I don't even bother any more. I use an AVRISP mk2 to program the chip and debug the old-fashioned way, with a scope, LEDs, RS232 etc.
Have you written code for ISP commands on the ARM ? or are you using open source code ? Have you built an ISP?
I am using ATMEGA2560 , it has three lines for serial programming - PDI , PDO and SCK plus the reset , can I use the AVRISP mk2 to programme it (using the 6 pin header)? Stupid Question I think , but worth asking as I dont have any experience on AVRs
I second that. Put connectors on your board for both ISP and JTAG as there are things one can do the other can not.
The ISP pins are mostly independent of the JTAG pins. You can program via JTAG, you can program via ISP. The CPU config flags in EEPROM can disable JTAG and then only be reached from ISP. With JTAG enabled they can be reached from JTAG but you could screw up and lock yourself out.
IIRC the JTAGICE mkII does not do ISP. The ISP mkII costs all of $35.91, cheap enough that you should not skip it when buying a JTAGICE mkII.
As for the 6 pin ISP header, there is an alternate pinout for 10 pins. Easier to buy 10 pin cable connectors than 6 if it bothers you to plug a
10 pin cable into a 6 pin header. Biggest problem in using 10 pins for ISP is that the header looks same as a JTAG header to production line workers.
The original ISP mkI came with both 6 and 10 pin cables. The 10 pin was not a compatible pinout with the 6.
Double check, and triple check, your pinouts for ISP and JTAG as they connect to different pins on different AVRs. The ATmega64 tripped me the first time as the JTAG documentation named a particular pin function for each pin in the header but there was one line 100 or 200 pages into the datasheet that said, "Oh by the way the JTAG for this AVR connects to these pins rather than those pins used on other AVRs."
Others have mentioned debug sessions on JTAG may run away and require your target and JTAG be power cycled in the correct order to recover. What I have seen is that this is somehow related to the target board but have never figured out why one design works better than others. Happens with JTAGICE, JTAGICE mkII, Dragon, and 3rd party JTAG. I have not had similar problems using DebugWire. DebugWire is only available on smaller parts where JTAG is not available.
Updated my Dragon yesterday to AVR Studio 4.13 Service Pack Build 557 and didn't have the kinds of problems I have had recently with the Dragon.
Wasn't emulating so much as I was only enabling a pin on PORTC and toggling its output while playing with the hardware connected. A time or two AVR Studio claimed to be changing the output bit, but it didn't change on the hardware. Always stuck high if that means anything to anyone.
A power cycle on target and Dragon regained control.
Should point out that static discharges are very likely when fonding hardware like this, whether or not I noticed, no matter the precautions. System seems to set for hours without problems, only requires power cycle reset when I'm poking on it.