AI’s Math Tricks Don’t Work for Scientific Computing - Low-precision number formats don’t suit many simulations

I sure hope he ditches denormals!

Cheers

Phil Hobbs

I suspect (as with my dictionary implementation), developers are going to be tasked with embueing their algorithms with more knowledge of the problem domain -- and the actual values to be encountered at specific steps in computations.

Much like selecting algorithms to minimize cancellation instead of resorting to a grade school idea of "how math works".

E.g., I use Big Rationals for user-facing computations. Multiplication and division are pretty inexpensive -- if you defer the reduction/normalization stage until it is worth the savings. As the developer is the only one who knows what operators will be imposed in the future, why prematurely incur a cost if it won't materially improve performance? Do you care if "2.0" is stored as (2,1) vs (800000000000000000000,400000000000000000000)? Does the *user*?

Do not know much about what that guy did. But I noticed I can do most 'scientific things with 32 bits (in asm at that) For example the Fourier transform in

Is that science? Of course when AI wants to do a divide by zero using Albert E.'s brain fog, than it will likely need infinite bits to do the wormhole dance...

My conclusion: 32 bits is enough for most things

John Larkin Highland Tech Glen Canyon Design Center Lunatic Fringe Electronics

I wrote a math package for the 68332, with the usual functions. The format was signed 64 bits, as 32.32.

Ads and subs were fast, as no normalizations were needed. Integer conversions were even faster. Divide was admittedly kinda ugly.

I figured that anything physical can be expressed as 32.32.

John Larkin Highland Tech Glen Canyon Design Center Lunatic Fringe Electronics

A copy of the actual paper is on arXiv here:

Its a log tapered number system. He calls them takums vs posits. Not an easy read. Be interesting to see if it flies in hardware. There has been a *lot* of investment in IEEE754 FP already!

CDC7600 et all got it about right 60 bit reals were good enough for most purposes. float32 was always somewhat lacking in precision. Far too easy to have underflows and overflows in quite modest computations.

In the past I stuck with legacy compiler versions that supported x87 native 80bit FP (gcc still does today, ICX can be forced to).

I have my own DIY lightweight float128 class that exploits fused multiply and add to provide fast high dynamic range bigger floats without the overheads of a full multiprecision math library.

Denorms are not all *that* bad - some modern CPUs can even process them at full speed - though many are still glacially slow and in the past they used to be even slower (you can set DAZ flag now if you don't care). Often they were handled by a trap and tediously slow microcode.

Intel ICX compiler defaults to round denorms as zero.

But it is worth testing. For my Ph.D. work I found myself accumulating some sums with double precision numbers - rounding errors had a nasty way of accumulating when I was adding up hundreds of experimental observations.

FFTs are relatively forgiving where numerical precision is concerned. The basis functions are perfectly orthogonal summed over the domain.

Even something as simple as solving a cubic equation x^3 + ax^2 + bx + c can easily go wrong when computing in float32 since it involves computing a^6. You can work around this lack of dynamic range but it is painful!

Double precision also helps a lot when accumulating summations of reals or even in FFTs to do recurrence relations for sin/cos(n*w*t)

Almost all modern FP libraries today promote the float32 argument to double and do the computation in double precision rounding the result back to float at the end. It avoids a lot of messing about ensuring nothing overflows during the intermediate calculations.

CDC7600 60 bits really was good enough for most orbital dynamics computations which is why astronomical codes used them (and BMEWS too).

Today's CPUs double precision 64bit and float 32 bit have essentially the same performance unless you are vectorising or using huge arrays so that unless you *really* know what you are doing double precision is preferred for most routine scientific calculations. The exception is bulk raw data where you seldom have more than 4 significant figures.

The problem with any fixed precision math "library" is the user (developer?) has to be cognizant of those limitations while using the library.

If doing "one-off" calculations, he may be observant enough to notice things aren't going as expected and question his approach, operator ordering, etc.

But, if embedding a calculation in a piece of code, he may never see some corner condition where his approach s**ts the bed.

The Sun has an apparent diameter of 864,400 miles. Imagine a strap hugging it at its equator. That strap would be 864,400 * 2 * pi miles (5,431,185.37952603 to 15 digits) long; 864,400 * 2 * pi * 5,280 * 12 INCHES (344,119,905,646.769 to 15 digits) long!

Cut that band and insert an additional 6 inch length (344,119,905,652.769).

What's the gap between that lengthened band and the equatorial surface? We both know the answer to be (6/2*pi) inches yet a naive "software implementation" (with doubles) will expose a discrepancy compared to "pen and paper".

C has implicit promotions for floats that sometimes bites developers who read what they want instead of what the code actually *says*.

I had to change the title line. Eternal September rejected it for some reason.

John Larkin Highland Tech Glen Canyon Design Center Lunatic Fringe Electronics

I think he does.

Unless you use power-of-two scaling, allowing bit shifts to do the job.

I've done much the same, but usually more like 16.16 - these computers were tiny.

Joe

The standard dodge was to scale things such that the numerical values were close to unity, so even the sixth power was still close enough to unity.

But it's slow.

Yep.

Also when carrying data from place to place before modern fiber-optic transmission systems - Even 32 bit was overkill for ADC output data, so kit was best to send that in integer form, and convert to floats as late as possible in the process.

Joe

The problem with that is you have to send along all of the factors that allow those integers to be ACCURATELY mapped to the quantities they represent. Or, arrange for all of that information to already

Converting to a float/engineering units AT the acquisition site lets you encapsulate all of that hardware/domain/application specific information AT the acquisition point so you don't have to expose all of that detail. Imagine replacing a 10b device with a 12b -- how much tinkering will you have to do if the far end was expecting 10b data and now sees 12b?

Same problem here. Same solution!

Probably the apostrophes.

Joe

I was PWMing a heater and wanted to adjust for the unregulated supply voltage, which was the only divide in that system. So it didn't have to be very good. That was inside the more serious temperature control loop.

I could have done a lookup table, I guess.

Since I was running realtime control loops, the package threw no exceptions. It always returned a legal-format value and made its best guess.

But it was a joy to code in assembler.

John Larkin Highland Tech Glen Canyon Design Center Lunatic Fringe Electronics

That is traditional.

This was the processor I was using for realtime, and also for my original toaster Mac, which had a socket for the coprocessor chip. Apple sold and installed these for a very high price, so I bought the chip from Digikey or then equivalent, and installed it myself. It wasn't very hard, but needed care to not bust those pins.

The realtime use did not have the coprocessors, because FP was still far too slow, and achieved far more precision than needed.

Yes. I knew the fellow who developed the Instruction Set Architecture (ISA) for the 6800 and 68000-series processors. They were based on the DEC PDP-11 ISA.

Joe

In those days, we were happy to get anything this fancy to work, even if it would have to be totally replaced to add anything.

Joe

John Larkin snipped-for-privacy@Glen--Canyon.com wrote: |-------------------------------------------------------------| |"I figured that anything physical can be expressed as 32.32."| |-------------------------------------------------------------|

A lot but not all. (S.