The prices Don gave are _old_ prices. Now instead of Z80 one can use Arduino, below 4$ you can get a board and USB cable. Assuming that you have a computer that is all hardware to get started. I like Blue Pils (based on STM32), which are even cheaper than Arduino hardware and more powerful, but there is less support on the net for them. There are a bit more powerful STM32-based board at about
4-5$. For STM-32 board you probably want STLINK debugging dongle which adds something like 2$. There are ESP board and Raspberry Pi Pico, each for few dolars. The above are microcontoller board, for more processing power (and HDMI interface to display) one can get one of single-board computers. To get started you need such board and a SD-card. Small boards are available below 20$, small SD-card can cost 2.5$.Just a little comment: if you need to generate or scan fast digital signals on processor pins, than microcontrollers are in practice faster. OTOH if you need number crunching and say have external source of data like reasonably fast ADC, then single-board computers will win.
Also another comment: programming processor can be done using very little extra hardware (beyond a computer). Many microcontrollers come with "bootloader" in "ROM". When you connect few pins in a special way and turn on the power, bootloader will start and wait for program on say serial port. So USB to serial convertor (available below 2$) and approiate program on the computer is enough to program the chip. Arduino hardware do not have bootloader in ROM, but when you buy a board it will typically come with bootloader burned at start of the flash. And typical Arduino board have USB to serial convertor on the board, so you can program the board from you computer via USB cable. Raspbery Pi Pico contains bootloader which emulates a hard disk. If you copy a file with magic name to this "disk" the effect is to program the chip. Debugging dongles, like STLINK for STM32 chips also support programming.
Above is stuff meant mainly for hobby, but useful also for professionals. If you want to go with stuff blessed by processor manufactures, then there are various developement boards at prices below 20$. You may want Segger debugging dongle which you probably can get below 100$. Osciloscope with bandtwith of 100 MHz or better can sometimes help (but most of the time information collected by programs is enough to resolve problems).
For your audio work you may want fast high resolution ADC, those may be expensive (they are defintely too expensive for me to buy them just for occasional playing).
Of course, beside hardware cost, which may be quite low there is also time. Here much depends on what level you want to work. Arduino may be limiting, but one can do simple things pretty quickly. Manufactures typically provide support libraries and examples, they can save a lot of work. Or you may wish/need to work directly with the chips, expect hundreds of pages of documentation to read in such case.