PWL: Data Reduction

I have a string of data (from an IBIS file), voltage-versus-time, a

100ns string, at 100ps intervals... 1000 data points :-(

I'd like to make this into a PWL source, but reduce data points based on areas that stay at essentially a constant slope for awhile.

I can do this by hand, _tediously_ :-(

Anyone know of some kind of algorithm that automates, or at least eases the pain?

Thanks! ...Jim Thompson

--
| James E.Thompson, CTO                            |    mens     |
| Analog Innovations, Inc.                         |     et      |
 Click to see the full signature
Reply to
Jim Thompson
Loading thread data ...

It's not automatic, by any means, but I generally use spreadsheets for this sort of work. I'd do something like copy the value to a column based on a filter (>x% of above or below value, for instance), then use sorts (on the value, then again on time to restore time order) to get the spaces (zeros) out. Then again to do the necessary formatting.

Reply to
krw

=A0 =A0 ...Jim Thompson

=A0 =A0| =A0 =A0mens =A0 =A0 |

=A0 | =A0 =A0 et =A0 =A0 =A0|

=A0|

=A0 =A0 =A0 |

I'm pretty sure I've done PWL input greater than 1000 points in spice. I would just take the raw data and write a simple C or Perl program and convert the raw data to the spice PWL format. I've done this with SOX data. It is way less than a page of code.

Otherwise, you need to fit the curve fit the data to do reduction, and it really is way too much work.

Reply to
miso

=A0 =A0 ...Jim Thompson

=A0 =A0| =A0 =A0mens =A0 =A0 |

=A0 | =A0 =A0 et =A0 =A0 =A0|

=A0|

=A0 =A0 =A0 |

My suggestion - Microsoft Excel running a VBA script. You can stack the time and values in formatted strings and output as

*.txt. Do inline testing on the slope before committing the write (per record). When done, close the file. Then cut & paste into Spice.

I suppose you could also do this with an Excel keystroke macro, but the VB editor is the way to go. Google "Mr. Excel" if you get stuck on any of the VBA syntax, etc... The beauty is once you have it, you can re-use it forever.

Reply to
mpm

Excel is very good for mocking up test data and simple processing of raw data into a more convenient form. You can program macros in a dialect of basic VBA to do fairly complex algorithms.

It is reasonably catholic about accepting CSV files and then you may need to adjust your output format to match PWL expectations.

PS If you want to saty sane use XL2003 the macro recorder works in that and eralier versions. XL2007 is still hopelessly broken.

Regards, Martin Brown

Reply to
Martin Brown

You could try the perl program that appears below, between the two lines of dashes. Change the variable $ELF from 0.01 (that is, 1%) to whatever level of accuracy you want. The main idea of the program is to find end-points of intervals in the original dataset such that points between the end-points are within the specified limit of proportional error, and print out those end-points. The algorithm is not particularly efficient and does not necessarily produce a "best" reduction. [Although it's easy to find a decent approximation, it's hard (probably NP-complete) to find a best reduction, for most values of best.] This algorithm's worst-case time is proportional to the square of the number of points, but for datasets like in the following example it runs in linear time.

Eg: Re datasets, the program takes about 15 milliseconds to process 231 numbers. For data from the first column, it outputs a data set of 65 points when ELF=0.005; 45 points when ELF=0.01; and 31 points when ELF=0.02.

If you have a Windows system and no perl, see for example , , and . The 'Usage' comment in the program is how the program can be run on a linux system.

---------------------------------------- #!/usr/bin/perl

# Program to winnow data by removing points that fall within specified # fraction of linear-interpolated values between remaining points. # (For small values, instead test absolute error; see comments.)

# Usage: ./data-winnow < datasetin > datasetout

# Each line of input should begin with 2 decimal numbers, separated by # one or more blanks. Leading blanks, signs, decimal points are ok.

# Method: Test if points between range-start and last-point-read are # in limits. If not, output range-start and start new range.

$ELF = 0.01; # error-limit fraction ( 0.01 = 1% ) $AEL = 1e-7; # absolute-error-limit, used when abs(y) < eps $eps = 1e-7; # If abs(y) < eps, test abs(y - interpolated y) < AEL $L = 0; # L = input line number $c = -1; # c = conversion state $p = 0; # p = range start point

foreach $s () { # s = string of input ++$L; @nums = split (/ +/, $s); $i = $nums[0]?0:1; # Skip leading blanks if any $x[$L] = 1.0 * $nums[$i]; # Convert string to number $y[$L] = 1.0 * $nums[$i+1];

if ($c > 0) { if (testRange()) { print "$x[$p] $y[$p]\n"; $c = 0; } } if (!$c) { $p = $L-1; } ++$c; } print "$x[$L] $y[$L]\n";

sub testRange { local ($yp=$y[$p], $yl=$y[$L], $a, $i, $u); local ($xp=$x[$p], $xl=$x[$L], $d=$xl-$xp); die "Duplicated x at lines $p and $L ?" if (abs($d) < 1e-10);

for ($i=$p+1; $i < $L; ++$i) { $a = ($x[$i]-$xp)/$d; $u = $y[$i]; $v = $a*$yl + (1-$a)*$yp; if (abs($u) < $eps) { if (abs($u-$v) > $AEL) { return 1; } } else { if (abs(($u-$v)/$u) > $ELF) { return 1; } } } return 0; } # By joe@swo-za, 4 May 2010

------------------------------------------------------------- db340c6484a1cf0d46cdc2d6ce6a9815c3fa8e1e Mcode

Reply to
Joe

I have a data, voltage-versus-time, in 4 columns...

Time DataVersion1 DataVersion2 DataVersion3

Time is in 100ps steps, over a total span of 100ns, thus 1000 data points in each data column.

The end-use software, IBIS, can only handle 100 data points per column.

Snag #1: Data in each column doesn't quite match data in other columns... process variations, temperature, voltage, etc.

Snag #2: Time column must be the _same_ for all three data columns.

Any ideas on how to process, such that enough time points can be eliminated to reduce total time points to 100.

Some kind of simultaneous slope/delta analysis? Maybe some way in Excel?

Thanks!

Fun problem, isn't it ?:-) ...Jim Thompson

--
| James E.Thompson, CTO                            |    mens     |
| Analog Innovations, Inc.                         |     et      |
 Click to see the full signature
Reply to
Jim Thompson

What information do you want to keep, what information can you discard?

If each and every voltage point in there is unique and carries valuable information, then you're screwed. If you have some prior knowledge about the voltages -- e.g. if they're bandlimited to well under 500MHz

-- then there's some grist for the data-reduction mill.

So, tell us more.

This would be a good post for comp.dsp -- this is the sort of question we answer all the time.

--
Tim Wescott
Wescott Design Services
 Click to see the full signature
Reply to
Tim Wescott

It's Spice transient analysis data.

I'm guessing that, at low enough or consistent slopes, you could eliminate intermediate points, since they don't really describe any "significant" change.

("Significant" is a nasty word ;-)

I can do this by eye, very slowly and tediously.

Maybe an Excel take-sum-of-magnitudes-of-all-three-slopes, eliminate data points below a certain slope.

Sorry! I'm flailing for a solution :-) ...Jim Thompson

--
| James E.Thompson, CTO                            |    mens     |
| Analog Innovations, Inc.                         |     et      |
 Click to see the full signature
Reply to
Jim Thompson

So you want to actively analyze the data for information content, and only save the "important" stuff.

Try plotting the data, 1st difference (derivative, slope) of the data, the second difference (2nd derivative, acceleration) of the data, and maybe even the 3rd. I suspect that what you really want to save is data that has a lot of curvature, which would be indicated by the 2nd derivative, or data that has an onset of curvature, which would be indicated by the 3rd.

I'm not sure how you'd wrench Excel into discarding all but the high-curvature data -- it'd be a snap in Scilab or C or Basic or Python or any other real programming language, though.

--
Tim Wescott
Wescott Design Services
 Click to see the full signature
Reply to
Tim Wescott

Yep. Sounds easy when you say it that way ;-)

Good ideas!

Yep, I guess I need to re-remember how to program :-( ...Jim Thompson

--
| James E.Thompson, CTO                            |    mens     |
| Analog Innovations, Inc.                         |     et      |
 Click to see the full signature
Reply to
Jim Thompson

...

The following perl program can do that sort of thing, ie select a subset of data points such that all the original tabulated values of each of the dependent variables remains within some given limits of linearly-interpolated values based on the selected points. See program notes in lines 32-33 re tolerance limits $ELF and $AEL that control how many points get removed. With tolerances as shown, a set of 111 data lines of 5 numbers each (resistances vs temperature for four different models of thermocouples) reduces to a set of 30 data lines. This program does nothing re "Snag #1".

---------cut here---------- #!/usr/bin/perl

# Program to winnow data by removing points that fall within specified # fraction of linear-interpolated values between remaining points. # (For small values, instead test absolute error; see comments.)

# Usage: ./multi-winnow < datasetin > datasetout

# Each line of input should contain decimal numbers, separated from # each other by one or more blanks or tabs. Leading blanks, signs, # decimal points are ok. The first number on each line is treated as # the independent variable, x. Other numbers are treated as dependent # variables.

# Method: Test if points between range-start ($p) and last-point-read # ($L) are within tolerance limits ($ELF and $AEL) on all of the # dependent variables, relative to linearly-interpolated values # between points $p and $L. If not, output point $p and start new # range.

# Change the printf format below as necessary to suit your data; # eg, use %8d for integers, and eg %.4f, %8f, %9.3f, etc for reals. sub outLine { my $p = shift; for ($i=1; $i

Reply to
Joe

...

...

...

Remove line shown below from previously-posted perl program:

Reply to
Joe

Come to think of it, one way to do this would be to start at a point, then just keep skipping samples for as long as a straight line between your start and end sample is within your error budget. As soon as you exceed your error budget, go back on, spit it out, then proceed again.

It may require fiddling to get you down to the 100 points you want, but it'd be a sure way to reduce your number of samples while maintaining a fixed error.

--
Tim Wescott
Wescott Design Services
 Click to see the full signature
Reply to
Tim Wescott

...

...

What you suggest is like the method used in the perl program that I posted 4 hours ago in this thread. It doesn't take much code (a few dozen lines) for the 'greedy' method, and the code is simple enough to be mostly transparent. An interesting question is how lengthy and complicated a program for optimal fit would be, and whether the fit would actually be much better.

Reply to
Joe

It's all important. Do a polynomial curve fit, nab the coefficients, and reconstruct 100 points. Even a little-old-lady Excel driver should be able to do that.

John

Reply to
John Larkin

Use Excel, scatter plot GURRaph (since i am a baer,i can gurrowl). Did you have something else in mind other than value VS time?

Reply to
Robert Baer

Well, Excel does have Visual Basic which can be used for analysis and "pasting" results elsewhere on the spreadsheet, as well as GURRaphing the analysis results.

Reply to
Robert Baer

I hate the military, too many gaywads.

--
I have multiple DUIS, beat my kids, paid $360K for a house worth $275K
got handed a divorce and a Restaining Order to keep away from all of
 Click to see the full signature
Reply to
Keith Thrasher

Great minds think alike. I'm running a simulation right this minute (?) that uses delay lines to create the differences. Maybe in Excel just stagger the column staring points to get deltas as a function of time, the "cherry pick" :-)

(?) 10 minutes or so, and I'll post.

...Jim Thompson

--
| James E.Thompson, CTO                            |    mens     |
| Analog Innovations, Inc.                         |     et      |
 Click to see the full signature
Reply to
Jim Thompson

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.