Accurately recovering a low frequency bias signal from a random signal

Hi,

I have a random voltage signal with a slowly increasing then decreasing voltage riding on top of it (ie gaussian shaped). I'd like to recover the hidden low amplitude gaussian shaped signal from the overall random signal, is a low pass filter or a notch filter the best way to do this?

Also this is a periodic gaussian signal most likely, but I'd like to be able to identify the low amplitude gaussian signal directly without averaging multiple sections of the signal in case it isn't the same gaussian shaped signal in each part, since the shape of the gaussian signal is unknown.

Also regarding phase shift from the filter, I'd like to accurately give the temporal coordinates of the gaussian signal, so will phase shift from the filtering be a problem?

cheers, Jamie

Reply to
Jamie M
Loading thread data ...

That depends on the spectral characteristics of the noise -- if it's white, then a low-pass filter is best, if it's mostly at one frequency then a notch filter is best.

"Wiener filtering" is the keywords you want to look for, but (assuming that you want to minimize mean-squared error) the results boil down to three cases:

1: The desired signal has a significantly higher power spectral density within its bandwidth than the noise, in which case you make a filter matched to the desired signal. 2: The desired signal has a significantly lower power spectral density within its bandwidth than the noise, in which case you are screwed. 3: The desired signal has about the same power spectral density within its bandwidth as the noise, in which case you have a pile-o-math ahead of your, and you may well be screwed anyway.

If the shape is unknown, how can you declare it to be Gaussian in shape?

It will be an issue, and far more so if the signal shape changes. For a signal that is always the same shape you should be able to easily cipher out the delay of the filter and account for it in identifying the time of arrival of the pulse. You will be greatly aided by the fact that if you use a Gaussian-shaped filter it will have a constant phase delay over a fairly wide frequency range.

--

Tim Wescott 
Wescott Design Services 
http://www.wescottdesign.com
Reply to
Tim Wescott

What is "periodic Gaussian"?

A Guassian function exp(-x^2) is precisely aperiodic. You could make a periodic waveform using chunks of it (matched in whatever segments -- discontinuous, piecewise continuous, N continuous derivatives..), but it's no longer unqualified "Gaussian", and necessarily as all* periodic signals are, can be expressed as a harmonic series of sin/cos waves (Fourier series), for which it will be a more effective description (because you're working in the frequency domain, after all).

*Except pathological cases, that fortunately aren't physically realizable as signals.

Tim

-- Seven Transistor Labs Electrical Engineering Consultation Website:

formatting link

Reply to
Tim Williams

Hi,

I think there is a periodic pattern in the signal, and my guess is that it looks gaussian but I don't really know.

cheers, Jamie

Reply to
Jamie M

Hi,

Thanks I am looking into the Wiener filtering looks interesting for recovering signals from noise.

Here is the input data if anyone wants to look for a low frequency signal component! :D

There are 774 samples total and there should be 12 separate possibly gaussian low frequency equally distributed periodic peaks in the data I think.

cheers, Jamie

0.77493332 0.78253332 0.7855999733 0.7933333333 0.7797333333 0.78733332 0.7786666933 0.77653336 0.8029333467 0.79466664 0.7997333333 0.78 0.7589333067 0.74506668 0.7672000133 0.7810666933 0.7984000133 0.8082666533 0.8157333333 0.8198666933 0.81466664 0.8242666667 0.8306666533 0.84 0.8332000267 0.83506668 0.8508000133 0.8469333333 0.8558666933 0.8512 0.8448000133 0.8189333067 0.8142666667 0.8308000133 0.83506668 0.8402666667 0.8427999867 0.8496000133 0.8582667067 0.8657333333 0.8549333733 0.8505333467 0.8521333333 0.84773336 0.8410666933 0.8482666533 0.8569332933 0.8524 0.85613332 0.8452 0.8504767467 0.85285128 0.8791029867 0.8776518667 0.8752773467 0.8693410667 0.8594472 0.84902564 0.8593152533 0.8611620933 0.86841764 0.8706602267 0.86841764 0.87224328 0.8744858667 0.8717156 0.86076636 0.87435392 0.8750135467 0.8802902933 0.8729028133 0.8690771733 0.8627451467 0.85179592 0.8502128667 0.8470468533 0.8527193333 0.8523236 0.8572046133 0.8500810133 0.8610301467 0.8574684 0.8574684 0.8597109867 0.8586556133 0.85997488 0.86538348 0.8628770933 0.8626132 0.8693410667 0.8769923467 0.8855669467 0.88147752 0.8776518667 0.8875457333 0.8883372133 0.8829286267 0.8826648267 0.8817413067 0.8780476133 0.8785752933 0.87409012 0.8690771733 0.86709848 0.8631408933 0.8529832267 0.8500810133 0.84968528 0.8471788 0.8401870933 0.84137436 0.8329316 0.8322720267 0.8281825333 0.8292266 0.81839468 0.8133050133 0.8181336667 0.8195692267 0.83379424 0.8319671867 0.8281825333 0.8229623733 0.8232233867 0.8264860133 0.8319671867 0.8202217467 0.8253114667 0.8187862267 0.8172201467 0.8422770667 0.84345156 0.8506293467 0.8504988133 0.8584596133 0.8734676667 0.8719015733 0.8687694933 0.85898164 0.8511513867 0.8602866667 0.8645933467 0.8660289067 0.8664204 0.8792099067 0.8769912533 0.8681169333 0.8596342 0.8548054933 0.854414 0.8499767867 0.8430600533 0.82779104 0.83314172 0.8097813733 0.79686144 0.81539308 0.81591512 0.8250504533 0.82857408 0.8328807067 0.83235868 0.8194387467 0.8044306933 0.8045612267 0.80416968 0.80051556 0.8045612267 0.8080848 0.8100424 0.8223098133 0.83757888 0.83092312 0.82857408 0.83901444 0.85532752 0.85676304 0.8577967067 0.86516144 0.8562462133 0.8557294667 0.84603904 0.8409999333 0.83906192 0.8402247333 0.8293714933 0.8231695733 0.8232988133 0.8310511333 0.8283378267 0.8447469067 0.84074156 0.8553417733 0.8731722267 0.8913901733 0.8907442133 0.8931990533 0.9109002667 0.9154224533 0.9070240533 0.9061196267 0.90689492 0.9044399867 0.9034063067 0.90004704 0.89371588 0.88854764 0.8871264667 0.88816004 0.8862219333 0.88118292 0.8718801867 0.8685208133 0.8700712133 0.8681332133 0.8886768667 0.8917777733 0.8820873467 0.87976164 0.8857051867 0.8779527733 0.8784696 0.8744642667 0.8627064933 0.8776944 0.8691667733 0.8696836133 0.8509488267 0.8501736267 0.8685208133 0.8784696 0.8917777733 0.89345752 0.87937404 0.88389632 0.88557596 0.8819581867 0.89129992 0.8864370533 0.8852853867 0.8874608267 0.88183028 0.8971864533 0.9083197333 0.91177484 0.9157418267 0.9334015333 0.9337854267 0.9358328667 0.9344252 0.9337854267 0.9310980933 0.9259793467 0.9239318 0.91970892 0.92226824 0.9229081333 0.9234200133 0.9248275733 0.92188436 0.9170215867 0.9258513467 0.9271310133 0.9493975867 0.9405677333 0.9444068267 0.9404397333 0.9366007467 0.9284107733 0.9264912267 0.93020232 0.9258513467 0.9241877867 0.9220123467 0.9156139333 0.9238039067 0.93301764 0.9382642933 0.94031184 0.9470941467 0.9514450267 0.9449186133 0.9444068267 0.9409516267 0.9380084 0.9355769733 0.9382642933 0.93173788 0.9342973067 0.9327616533 0.9369846267 0.9310980933 0.9276428933 0.92162836 0.92175636 0.9234200133 0.9199648133 0.92457168 0.9188131467 0.9170215867 0.9129265067 0.9103671867 0.9121587333 0.9058882667 0.8992935733 0.90271772 0.9028445733 0.89916672 0.8976448267 0.9030981733 0.9005617733 0.9013226667 0.8943474267 0.8928256267 0.89574248 0.8982789733 0.89675708 0.8937133867 0.8900355333 0.8940938267 0.8904159867 0.8848358 0.8830602933 0.8767192 0.87278776 0.8721536133 0.8623883733 0.86188108 0.8617542267 0.85718864 0.8531303467 0.85008664 0.8564277333 0.8493257467 0.8494525867 0.8481842933 0.8469160933 0.8460283467 0.844126 0.8426042 0.8437455467 0.84907204 0.844887 0.8446333067 0.8360094133 0.8322048133 0.8299220267 0.8318242667 0.82941472 0.8303024667 0.8301756267 0.83068292 0.8262441733 0.8196493867 0.81622524 0.8147033467 0.81495704 0.8233272267 0.8229467867 0.8200299333 0.8229467867 0.8225663333 0.82142488 0.82015668 0.8120400933 0.80658676 0.8063331467 0.8117321467 0.8139922 0.8143688533 0.8143688533 0.8131132933 0.8121088 0.8139922 0.8092209333 0.8079653733 0.8112299067 0.8176333867 0.8138666133 0.8033197467 0.7986741067 0.7996785467 0.8000552533 0.7925217467 0.79139172 0.79503292 0.7979207467 0.79628848 0.79628848 0.7974185067 0.79503292 0.80294308 0.7998041333 0.7956606933 0.7946562533 0.7931495333 0.7826027067 0.7785848133 0.77532032 0.7746925467 0.7783337333 0.7651501067 0.7563610933 0.7607556 0.76715904 0.7758225733 0.7702979867 0.7769525867 0.77456696 0.7916428533 0.8062075733 0.81110432 0.81286212 0.8146199333 0.8018130533 0.7986741067 0.8001807867 0.7995530133 0.80696088 0.8089698533 0.8099743467 0.8085932 0.80959768 0.81286212 0.8219023067 0.82102344 0.82624732 0.8236353733 0.8197796667 0.8150532267 0.81169504 0.8053517067 0.7983865333 0.7968939467 0.7874412 0.78806308 0.7828391467 0.78308792 0.7858242667 0.79353572 0.79863524 0.7957745467 0.78756556 0.79229196 0.7983865333 0.8128144 0.8110731067 0.8128144 0.8133119333 0.8136850533 0.82649604 0.8230134533 0.8201527867 0.8238840933 0.82065032 0.82102344 0.8164214 0.8140581733 0.8072174 0.8016203067 0.801496 0.79900836 0.79229196 0.7954014267 0.7912969067 0.78831184 0.79229196 0.7884362 0.7900531333 0.7903018933 0.7819685067 0.7759983333 0.7723913467 0.7727645067 0.77786404 0.7711475733 0.7656749333 0.7563465333 0.7487594267 0.7466449733 0.74502808 0.7427892667 0.7332121067 0.7317195733 0.7334608667 0.7327145733 0.7338339867 0.73582404 0.7394310267 0.7338339867 0.73545092 0.7348290267 0.7338443067 0.7344597333 0.7319980267 0.7287977467 0.7339674267 0.7347059067 0.7354444533 0.73605988 0.7299055333 0.73138256 0.7380292667 0.7386446933 0.73839856 0.7388908667 0.7358136933 0.7209201867 0.7209201867 0.7200585867 0.7179661333 0.7115656 0.7103347067 0.7223972133 0.7222741467 0.7198124 0.7173506533 0.71845844 0.7153812667 0.7168583467 0.7155043733 0.71710448 0.7198124 0.71291952 0.7151350933 0.71427348 0.7098423467 0.7115656 0.7173506533 0.71390424 0.7140273067 0.7233819333 0.7226434 0.7178430133 0.7182122667 0.71845844 0.7182122667 0.7257205733 0.7265821733 0.7243666133 0.7143966 0.70319568 0.7040572933 0.7022109733 0.700857 0.6860866133 0.6876867067 0.68387104 0.68239396 0.68337868 0.68387104 0.6887944933 0.6919947733 0.68707128 0.6833786667 0.6811867867 0.67741192 0.6798473467 0.6747329467 0.6932421333 0.6992088933 0.69726056 0.6919026533 0.6968952533 0.68764068 0.6923897467 0.69872184 0.6970170533 0.7005483733 0.7049321333 0.71491736 0.71491736 0.7189358133 0.7212494533 0.72210184 0.72587676 0.72587676 0.7225889333 0.7212494533 0.72441552 0.7274598 0.7155262 0.7107771333 0.7087070533 0.7154044533 0.71844872 0.7194229067 0.71844872 0.7230760267 0.7186922667 0.7144302533 0.7113859867 0.7128472667 0.7042015467 0.7073675733 0.6999395333 0.69287684 0.6937292267 0.69348568 0.6893454667 0.6903196133 0.69348568 0.6912938 0.69202444 0.7026185067 0.6926332933 0.7027402533 0.7050539333 0.71102068 0.69860004 0.6944598267 0.69287684 0.7031056 0.7072458133 0.7195446667 0.71783988 0.7197882133 0.7161350667 0.7180637333 0.71637616 0.7166172533 0.71239828 0.71770212 0.7191486 0.7213183467 0.72722488 0.7248140533 0.7240907867 0.7310822 0.7303589333 0.7314438 0.7322875867 0.7367476 0.73915844 0.7337340667 0.7368681733 0.7404844 0.7304794533 0.7292740667 0.7255372667 0.7273454 0.7222826533 0.72336752 0.7130010267 0.7052863467 0.7078177467 0.7090231467 0.7061301733 0.70842044 0.7107107067 0.7079382667 0.7092642267 0.7066123067 0.7064917867 0.7099874933 0.70697396 0.70962584 0.7105901867 0.7105901867 0.7049247467 0.7052863467 0.7017906933 0.70396044 0.7032371733 0.6974512 0.69214736 0.69214736 0.6955225467 0.69479928 0.6944376667 0.6938349733 0.6945581867 0.6963663333 0.6949198533 0.6941965867 0.6911830533 0.6931117067 0.6926295333 0.6922679333 0.6981744667 0.6982938133 0.6955483867 0.6929223067 0.6919674 0.68373108 0.67609164 0.6784789733 0.6783596267 0.6789564533 0.67442052 0.6677359733 0.66284196 0.66320008 0.6665423333 0.6657067733 0.6637969067 0.6684521867 0.6591416267 0.6531733067 0.6505472267 0.6518602667 0.6505472267 0.65030848 0.6475630667 0.64171408 0.6374169333 0.6327616 0.63025492 0.6347908533 0.6429077733 0.64434016 0.64219156 0.63872992 0.6425496933 0.6492341867 0.6498310133 0.65245708 0.65245708 0.6475630667 0.6462500267 0.6445789067 0.64171408 0.64087852 0.6392074 0.64004296 0.6458919333 0.6478018 0.6501891333 0.64827928 0.6455338533 0.6384912267 0.6352683333 0.6382524933 0.6438926933 0.6488392133 0.65221736 0.65354444 0.64992504 0.65089024 0.65137284 0.6505282667 0.6536650933 0.6456659333 0.6473373867 0.6468598133 0.6450689333 0.63874124 0.6395769733 0.6386218667 0.6407708533 0.6396963867 0.6380248667 0.6353983067 0.6327716933 0.6246383867 0.6229846533 0.62487464 0.61967716 0.6170784667 0.6171965733 0.6177871867 0.6331432667 0.62487464 0.6253470933 0.61967716 0.6201496667 0.62345712 0.63019016 0.6284183333 0.6293633067 0.62310276 0.6182596933 0.6143615867 0.6124716
Reply to
Jamie M

Which is "riding" on which? Do you mean the DC component is slowly varying? Any idea of the P-P amplitude of the low frequency component? And its frequency?

...Jim Thompson

--
| James E.Thompson                                 |    mens     | 
| Analog Innovations                               |     et      | 
| Analog/Mixed-Signal ASIC's and Discrete Systems  |    manus    | 
| San Tan Valley, AZ 85142     Skype: skypeanalog  |             | 
| Voice:(480)460-2350  Fax: Available upon request |  Brass Rat  | 
| E-mail Icon at http://www.analog-innovations.com |    1962     | 
              
I love to cook with wine.     Sometimes I even put it in the food.
Reply to
Jim Thompson

Apparently the signal is not periodic. If you know that the signal you need to extract is a gaussian function, then why don't you sample and store the signal over the whole duration of the gaussian phenomenon and try to let your computer find the parameters that give the closest approximation to a gaussian shape?

joe

Reply to
joe hey

Hi,

Yep each periodic DC component is slowly varying I think, and there should be a total of 12 cycles in the overall 774 sample data I posted. I don't know if they would be individually gaussian or their amplitude but I think they would look somewhat gaussian, and I think the amplitude is quite small, but the center point of all 12 should be accurately periodic I think, which might make averaging possible over multiple periods, since there is a fixed known location for each of the

12 cycles to synchronize their locations as they are periodic.

cheers, Jamie

Reply to
Jamie M

Just giving that a visual look-see there's more going on than that -- there's clearly a distinct DC bias.

Do you know what the spectra are of the noise, the pulses, and any bias/ wander/whatever? Without that you really can't even start designing filters. You may be able to extract what you need from the data, but you still need to know.

--
www.wescottdesign.com
Reply to
Tim Wescott

12 equally distributed things in 774 samples is 64.5 samples per thing. Add the first 64.5 samples to the second 64.5 samples and the third 64.5 samples etc then look at the data.

I just did it crudely with OpenOffice Calc - used 129 sample chunks then split that into two - no interpolation or nuffink and the data shows what looks like it could be 12 double peaks - take a look...

8.70194252 8.7583963068 8.7444061601 8.7869181335 8.7933093065 8.85274148 8.8562121333 8.83752632 8.8509046265 8.7995587734 8.7496598933 8.7465793068 8.7537914135 8.7344835068 8.7166207069 8.7535699467 8.7539897731 8.7700333199 8.7955810399 8.7784215199 8.7787427468 8.8032783201 8.8471719599 8.8361662398 8.8281742935 8.7563182266 8.7470105199 8.7496541597 8.7531274399 8.72570464 8.7273883333 8.7181052801 8.7209399733 8.7031156133 8.6804912666 8.6860178133 8.6954626267 8.7012629467 8.7000578001 8.7133766667 8.6669223333 8.6854081201 8.6894357069 8.6981012266 8.6804945599 8.6648381999 8.6694525999 8.6664345866 8.6619080666 8.6557276534 8.6494721601 8.6730935734 8.6787560533 8.6757784267 8.6645743069 8.6441281601 8.6332251466 8.6318116664 8.6643856533 8.6680190267 8.6573805466 8.6856666267 8.6611115866 8.7240032532 8.7737129599 8.76797264

Cheers

--
Syd
Reply to
Syd Rumpo

*Head explodes*

Tim

--
Seven Transistor Labs 
Electrical Engineering Consultation 
Website: http://seventransistorlabs.com
Reply to
Tim Williams

Have you done any analyses of your data?

Reply to
John S

In the 1940s a communication system was widely used that consisted of tones mixed with broadcast speech. The tones were barely noticeable to the liste ner. A series of notch or resonant filters recovered the digital data. The system worked well when noise exceeded signal.

sure. But hopefully it's trivial to correct.

NT

Reply to
tabbypurr

Not enough samples. no time info.

- subtracting 0.788708119 average DC bias

- assuming ms time increments gives a waveform. See A.B.S.E. You could filter it, if you can decide what actual information you want to recover.

RL

Reply to
legg

The system worked well when the _overall_ noise exceeded the signal, but not when the noise within the signal bandwidth exceeded the signal.

--
www.wescottdesign.com
Reply to
Tim Wescott

Hi,

I don't see it on alt binaries schematics electronics.

cheers, Jamie

Reply to
Jamie M

Hi,

Yes, I see some interesting peaks after averaging multiple periods, but need to add more periods of the data to know if they will average out or form the bias signal I was looking for. I think it may just average out, but after adding together twelve of these periods I ended up with this waveform:

formatting link

It does have a bit of a gaussian section in the middle, but I need to get more data to see I think.

cheers, Jamie

Reply to
Jamie M

Hi,

Yes, I see some interesting peaks after averaging multiple periods, but need to add more periods of the data to know if they will average out or form the bias signal I was looking for. I think it may just average out, but after adding together twelve of these periods I ended up with this waveform:

formatting link

It does have a bit of a gaussian section in the middle, but I need to get more data to see I think.

cheers, Jamie =============================================================================

It sounds to me like you have a Gaussian pulse arriving every so often, but since I haven't seen you say how these are generated or how you are sampling them it is not clear to me that you know when they will appear. The only way you can signal average in the time domain is if you can synchronize your data collection with the pulse arrival with a time accuracy much better than the width of the pulse. Otherwise they will average to zero along with the noise in the time domain, and you will need to do some kind of periodigram approach where you sum power spectra in the frequency domain so you throw away the meaningless phase information. From random luck you might see a signal to noise improvement from averaging a small number of pulses, but the shape is certain to be distorted and the apparent signal to noise ratio will grow at much less than the square root of the number of scans and will actually reach a maximum and then decrease. For fun a long time ago I did some experiments comparing time domain averaging of a synchronized sine wave with a phase-random sine wave, and I actually saw the s/n reach a maximum at about 75-100 scans added together (with distortion in the peak shape in the final power spectrum), but I'm sure that number was dependent on my actual s/n (about 20 after 100 scans) and all the data collection parameters.

----- Regards, Carl Ijames

Reply to
Carl Ijames

It got through to the news server here.

If still no joy:

formatting link

This assumes 1ms intervals in data.

RL

Reply to
legg

e
y

yes

the bandwidth of each tone was zero. One filter was used per tone, with as narrow a badndwidth as could function at the time. S/N ratios of over 1 wit hin the filter's bandwidth did work, noise fails to build up much oscillati on as it comes & goes with random phase, whereas signal does build up filte r response. So it's inherently slow data rate, but immensely noiseproof.

NT

Reply to
tabbypurr

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.