Keywords: average, mean, RMS. Take N samples of a "noisy" level. Typical scheme seems to be sum(readings)/N. Now, let N be odd, then sort the data and pick the middle value. A minimum seems to be 7 but i prefer 11 as a minimum sample count.
A few outliers have little effect on the result - no matter how outrageous they are; cannot say the same for any of the other methods.
Is there any test or paper that supports this oddball scheme?
"median"
It is called the median and predates the mean as an estimator.
Modern analytical solutions solve it as minimum L1-Norm as opposed to the "normal" approach of least squares which is minimum L2-Norm.
Try Numerical Recipes p699
These methods are usually known as robust estimators.
Regards, Martin Brown
Open up PaintShop, load a picture and select median (smoothing, edge-finding, noise rejecting, etc.) filter. If desired, add noise to the image first to observe the difference.
It provides better smoothing than a linear filtering process (the typical image filter is a couple-pixel-wide FIR), but leaves the image with a somewhat surreal, cartoony character, due to the way the color changes.
Tim
-- Deep Friar: a very philosophical monk. Website:
Try it on a square wave.
John
Robert: it's too bad that you're plonked over here for a few too many posts directly quoted from right-wing rags that call themselves "investment newsletters".
Take N samples of a signal whose noise is exactly equal to 99 1% of the time, and -1 99% of the time.
Do your median scheme.
Do you see a problem?
The nature of the noise dictates the nature of the estimator. As pointed out, different estimators are good in different circumstances. Averaging almost always works well for me; it's good in the case of additive noise with statistics that make it average out to Gaussian. Averaging-with- outliers-tossed can work well, and, as you've found, median filtering can work well.
It all depends on the noise.
Yes, it is proven by showing that the expected value of the sample median is equal to the median of the density function underlying the random variable. In the case where the median is equal to the mean, you have your result. A small write-up here:
n
ut
The problem is that example has no median.
No, for N >= 1 there is always an average, median, mode, etc. (though I admit the mode could be ambiguous in some cases!)
The problem is that those may not be good measures of central tendency for a given population/sample set.
You have to think about what you want the measure to *tell* you. And, how it can be biased in a population.
E.g., the median home value and average home value in any particular neighborhood *tend* to be pretty close. (homes tend to have similar values within a neighborhood... odd to find 7,000 sq ft palaces alongside 600 sq ft "shacks")
The average salary of people riding a *bus* may closely correlate with the median of that same population -- on
*some* busses! On others, it may be wildly different!
Yes, that is another keyword, and seems to apply in this case.
One of the relevant ones to electronics is MTBF for a filament lamp. (and other units subject to infant mortality and limited service life)
Hardly any filament lamps actually fail at their nominal MTBF. A cohort of a few percent die very quickly soon after being fitted and the rest tend to live to a ripe old age. If you swap lamps out too soon to avoid having any in service failures you actually increase the risk of downtime due to the replacement dying suddenly during its burn in phase.
The average in this case significantly underestimates the working life of lamps that survive the first few days - the median is usually a far better estimator where distributions are skewed or subject to outliers.
If you have the set of measured values it is as well to plot the histogram of their distribution to avoid surprises. Crunching it down to a single number can easily lead you astray.
Median and L1-Norm model fitting is generally preferable when the data are extremely noisy and subject to serious interference and outliers. It wasn't popular in the past because apart from a handful of special cases it is much harded to compute. But these days with almost unlimited computing power on your desktop it is not a real problem.
The L2-Norm (aka least squares fit) tends to be dominated by fitting the largest values whether or not they are accurate. Pulse counting devices have systematic errors from dead time at high count rates and so can warp the calibration of otherwise highly linear instruments.
Regards, Martin Brown
Rather close to obscene; not fit for reading by virtually anyone.
That means i should do a noise study to see how "bad" it is and the "spectrum" (distribution) as well.
Thanks.
d, then
be 7
how
ds.
?he
Not sure if that is really good example because in the case reliability statistics what you have is a time dependent failure rate parameter throughout the burn-in phase. In other words, the distribution itself is changing. Once past the burn-in phase, the distribution steady-states to a constant failure rate exponential distribution, in the simplest case, until the wear-out phase wherein the failure rate becomes time-dependent again, i.e. the distribution changes once more. The exponential might look skewed, but in the case of the constant failure rate exponential, the MTBF does coincide with the 50% point of the cumulative and is therefore also the median. So RB's technique of using a sample median as an approximation of population mean is valid and will converge.
"Harder to compute"???? What is so hard to sort a pile of numbers? If the quantity is small, a stupid bubble sort is fast and symple and adding one interchange flag improves it a lot. And the merge-sort is simple, and almost the fastest of them all in almost all cases; again, adding one interchange flag improves it as well.
Sounds like you are talking about the Weibull failure rate curve aka "bathtub" curve.
While you copied the entire previous post, I was unable to see a reference to "harder to compute" anywhere, therein.
Be that as it may...
It is more expensive to find the median of an arbitrary list of values than it is to find the mean. To find the median, you ned to track more state than to find the mean.
E.g., with just two (suitably wide) "registers", you can compute the mean of an arbitrarily long stream of values -- without ever needing to RE-examine a value once it has been "tabulated".
In terms of computational power, the mean requires one addition per datum. By comparison, any sort performs "at least one" comparison on each datum (compare ~= subtraction ~= addition). Plus, of course, the added "state".
*Look* at your algorithm and count operators/registers. You'll find that it's simple, but arithmetic mean is "simpler". THink of how you would process a list of 2^32+1 individual values (or 2^64+1, etc.).
That is one of the handful of special cases. But it is still O(NlogN) vs O(N). If you only want the median value you can do it in O(Nlog(log(N))) or O(N^(2/3)logN) - see Knuth Sorting & Searching for details.
Another is for a robust L1-Norm linear fit of data where a closed form solution exists. But anything more than that and you are into much more complex optimisation codes to solve a non-linear fitting problem
minimise sum_i ABS( data[i] - model[i])
One way it is done in practice is to solve instead
minimise sum_i lim e->0 sqrt( (data[i]-model[i])^2 + e)
For various decreasing values of e and extrapolate to e = 0
You really are out of the stone age aren't you. Compared to the mean which is an O(N) process a moronic bubble sort is O(N^2) and no-one with any sort of clue would ever recommend using it.
Heapsort and quicksort manage O(NlogN) which is respectable but still much slower than a single fast pass over the data for the mean and SD.
Regards, Martin Brown
I know about bubble sort...remember i indicated a *small* group of umbers, and i said an interchange flag improved it (no longer n^2). Merge sort is, i think another name for quicksort.
Have something to add? Share your thoughts — no account required.
Ask the community — no account required