Got it working.
Using a 8 bit sine table, doing a 16 bits signed Hamming correction, doing most of the FFT in 16 bits signed, and finally the root of the sum of the squares in 32 bits signed, to get it all on a 64 pixels high display.
First I though 'FFT just for fun', but it is actually very accurate and allows you to measure the frequency and amplitude of individual components.
Some remarks for Microchip and their users: First I did some looking on the web for already existing FFT code. I found a person crying the Microchip application note did not work, and they had to BUY FFT code and lost 2 weeks of development time and pissed of a customer. Now that was a challenge, so I decided to write my own FFT code. Had to modify the Microchip math library so it could use signed and unsigned mixed, and used Peter Hemsley's 32 bit signed math routines for the final multiply and square root.
Peter's library does not use the hardware multiply the PIC has, and neither does my modified math library, so both will have to be rewritten (not today eeeek, used my calculator more in the last week then in the whole preceding year). So what do we have: Well RAM is the big limitation, not code space, I have not even filled half the code space! But 256 bytes continuous RAM, plus 96 bytes for variables, and one 32 bit variable takes 4 bytes... Where to store the samples? Where to store the Hamming modified samples? Samples of a 10 bit ADC also take 2 bytes each... Anyways, working back from the end result we have a 128 x 64 (or 64 x 128 if you live on that sort of planet) LCD So any frequency component lower the 1/64 of full scale will display no pixel on the LCD, at least if you use the LCD horizontal, as I do for the scope. So 128 samples (scope) fills a screen, OK all memory full! Part of that memory is also used to 'unplot' the wave form for the scope, so as to speed it up (clear screen between traces makes no sense). So I used those bytes twice (!LOL). Then did the Hamming, as now no unplot is needed in between displaying the scope waveform and the FFT, I have 2 bytes signed for Hamming window corrected samples. Then as there is no need to store the FFT result in RAM, as it only needs to go to the LCD (with it's own RAM), and we calculate one spectral component at the time, that solves the memory problem. Not the variable space problem, but I am still at 96 bytes, and know a few tricks to save some more vars. The resulting FFT is 64 wide on the LCD, and is displayed for the left half over the scope waveform picture, could clear that waveform, but it looks nice this way, and leaves space on the right side for displaying the amplitude and frequency of the component a cursor is on (still have to write that part).
As the whole thing uses the PIC internal 16 MHz oscillator, multiplied by 4 with a PLL, stability (= accuracy) is a point. When sub-sampling a high frequency waveform I noticed the PLL is 'locked', but waves around the centre frequency in an amazing way! You don't not notice this a lot on normal sampling, but sub-sampling increases that effect by say 100x or 1000x and shows your waveform FM modulated... Anyways sub-sampling is an other chapter for some other time. The 18F14K22 does a 128 points FFT in about 1 second, inclusive display, when not using the hardware multiplier it has. I did not try to optimise anything, except the scope acquisition and trigger, so I am sure that speed could be improved.
OK, got it all working last night, what is next is some stuff, like I now have 5 time base ranges, have to make some
1,2 5, ratio ones, some on screen display stuff, moveable cursor, etc.. calibration too.