AI’s Math Tricks Don’t Work for Scientific Computing - Low-precision number formats don’t suit many simulations

Neural networks for generative AI tend to be massively "overparamterized", so relatively large differences in vector scaling/rounding often don't make that much different in an output.

Activation functions also tend to be nonlinear and naturally behave kind of like mu-law/A-law compression, so you can bit-reduce the internal representation similar to why you can reduce the internal representation in a system that includes companding.

So why use e.g. int16 vectors when int8 or int4 (or remarkably sometimes even int2) can do well enough

I've cut more corners than most folks. But, in hindsight, it was wasted effort. Hardware has always been cheap -- even when it wasn't! Development (and re-development) has always been expensive. And, there are costs associated with "reputation" (you don't want to be known for a particular bug in your product).

But, managers are short-sighted; they see the cost of the BoM and panic over trying to shave a few dollars off -- at the expense of man-months of time.

I can recall trying to take a few hundred bytes out of a 12KB memory image to save the cost of an "extra" EPROM. Of course, once that was done, ADDING any functionality looked prohibitively expensive ("Well have to ADD another EPROM...")

I originally started my current project with that "low cost" mindset... use lots of PIC-ish motes feeding a *big* machine. And, the big machine keeps getting bigger (more complex, expensive) in an attempt to keep the motes dirt cheap.

But, when you look at *real* costs, the savings are illusions. Especially as you start thinking about how you're going to market "add ons" (does the user have to upgrade the "big machine" if he wants add-ons X, Y or Z? Or, if he has too many W's??)

Instead, add capability as you add functionality. Let the hardware make your job easier and more reliable.

I think that the advantage of Takums for AI in particular is that one can do arithmetic on a random collection of Takums of varying representation size (in bits) directly, without needing to convert to and from a common representation. And things like overflow and underflow don't exist. I don't know if such things as signed Infinity or indeterminate or NaN can be represented in any natural way.

Joe

For a few years I worked with the inventor of the ARM Thumb instruction set, Paul Denman. The motivation of that invention was exactly what you refer to - shaving off some memory cost in the processors used in FAX machines. US patent 5784585 awarded 21 July 1998 John

And now days, the evolved ARM ISA has won the world, displacing Intel.

Joe

Yes, but the world has changed. My first commercial product had 128 bytes of RAM. Essentially every byte was effectively a union -- of all of the possible uses that were being made of that byte (based on which part of the code was executing). The stack was defined as a specific number of singleton *bytes* -- the number derived from a static analysis of the deepest call stack plus the worst case pile-up of ISRs.

It took 4 hours to turn the crank to evaluate a change in the software. And, that was done on the actual hardware (no emulators, etc.). You probed specific pins in the design waiting for your code to twiddle them to indicate its progress.

This was a horrid way of using developer's time! It did nothing to improve the quality of the product. And, being the *sole* entry in that market, likely did nothing to increase sales from what they would have been at a higher sell price.

Now, I build more elaborate runtimes to support (and constrain!) the application. Online diagnostics. Hot plugging of software updates. Runtime invariants. etc. Things that consume resources but enhance the quality of the delivered code. Because my time is worth far more than the added hardware costs.

Most of the "old guard" made horrible decisions in the MCU market.

I still have a copy of the 86010 databook. And, GI before the PIC's fame (notoriety?).

That was handled in proces control equipment by using the ADC MSB-aligned. Full scale was defined as 1.0, and 10b would give FFC0, 12 bits then FFF0. Just the resolution improved, not the scale factor changed. No tinkering. I've designed ADC and DAC boards that worked that way.

Arie

But you can't report the available resolution or calibration factors per input point. If you don't expose those data, then calibration doesn't know "how close is close".

You want to hide that information *in* the acquisition node, not have to drag it around and require its "consumers" to apply those factors. No one *wants* to bear the cost of NBS traceable devices!

This also gives you freedom to change the measurement technology without having to alert the application to such a change.

The PDP-11 was revolutionary. You could

ASR PC

namely arithmetic shift the progam counter if you wanted. It was that general.

I wrote two RTOS's for the 11. It was beautiful.

John Larkin Highland Tech Glen Canyon Design Center Lunatic Fringe Electronics

The CompSci term is "orthogonal", which is the ideal for all languages natural and invented. It's parallel to composability. One achieves great expressive power, plus the ability to speak complete nonsense. But that's a semantic issue...

I wrote lots of code for those 68xxx computers in the mid-1980s, but we bought the real time operating system MTOS (Multi-Tasking OS) from a small outfit (two guys) called IPI (Industrial Programming Inc), now long gone. Its claim to fame was the ability to work with multiple SBCs and global memory boards in parallel, in VMEbus crate slots.

.

There are also schemes such as FDTD (finite difference time domain) EM simulation, where the roundoff error is pretty nearly constant--all the numerical noise flows out of the simulation domain at the speed of light! (Providing you use the right sort of absorbing boundaries, of course.)

My clusterized simulator is all done in single precision floats, which makes a big difference in speed.

Cheers

Phil Hobbs

Don Y snipped-for-privacy@foo.invalid wrote: |-------------------------------------------------------------------| |"[. . .] there are costs associated with "reputation" | |(you don't want to be known for a particular bug in your product)."| |-------------------------------------------------------------------|

Willy H. Gates III wanted to be known for DONKEY.BAS, but its flickering is not precisely a bug. Donkeys are also not precisely bugs.

|--------------------| |"But, managers are "| idiots. |--------------------|

|----------------------------------------------------------------| |"But, when you look at *real* costs, the savings are illusions."| |----------------------------------------------------------------|

True. (S.

Customers/users occasionally welcome design bugs -- if they can be exploited to their advantage.

Pinball machines (back in the electro-mechanical era) used to "randomly" generate a 2 digit value (rightmost of which was always zero, of course) which would be compared to the rightmost two digits of a player's score at the end of the game. If the two agreed, a free game was awarded.

Of course, rather than a true random number generator, the number generated was driven by observable events during game play -- like the number of times a particular target was struck.

Being able to count such events, knowing where the "number generator" was at the start of the game AND the sequence of "random" values that it would provide, you could predict the setting of the generator at any given time. So, just prior to the end of the game, deliberately tilt the game when your score coincides with the "predicted" value of the number generator -- and you can continue with a new game.

The physical clearance between some targets and the cover glass on some machines is close enough that you can sit on the glass to deflect it just enough to inhibit those targets "resetting".

Coin mechanisms can be tricked into accepting pennies in lieu of quarters.

Some video games allow the player to pass THROUGH solid objects.

In-band tone signalling for telephones gave rise to "Cap'n Crunch" and the whole phreaking movement. (TAP, anyone?)

One time I hit 53 goals on that soccer pinball at "The Pinball Museum". It made the top ten all time.

At the back or top of the field was a keeper, after activating some bumpers, he would wobble more-or-less back and forth or "randomly", then hitting shots from the flippers to past the keeper to the goal resulted a score, also the machine went "Gooooaaall." So, it was a matter of luck, and reflexes.

"Lock is lit."

Don Y snipped-for-privacy@foo.invalid wrote: |-----------------------------------------------------------------------------| |"Customers/users occasionally welcome design bugs -- if they can be exploited| |to their advantage." | |-----------------------------------------------------------------------------|

This is not so much a bug, but GNAT (the GNU Ada Translator) Version

|-----------------------------------------------------------------------------| |"Of course, rather than a true random number generator, the number | |generated was driven by observable events during game play -- like | |the number of times a particular target was struck." | |-----------------------------------------------------------------------------|

Don Y is reminding me about Retrogamer Keith Ainsworth who memorized winning moves for Pacman.

|-----------------------------------------------------------------------------| |"Some video games allow the player to pass THROUGH solid objects." | |-----------------------------------------------------------------------------|

Jet Set Willy is so buggy that it cannot be completed. Cf.

-->

Software (47)

A computer-game-programming professional told me about a game which was released in America which was not released here. He said that it is not the most fun game because it is accidentally excessively physically realistic. E.g. strapping a rifle onto a player's back means that a player cannot walk thru a doorway.

For a good laugh read a webpage on the Daily WTF about a buggy truck videogame where the posiitoning is imprecise that the truck can be driven off the ground and be trapped in an 'inexistent' space.

|-----------------------------------------------------------------------------| |"In-band tone signalling for telephones gave rise to "Cap'n Crunch" | |and the whole phreaking movement. (TAP, anyone?)" | |-----------------------------------------------------------------------------|

Cáritas in Portugal used to use a touch-tone landline telephone which is so buggy that though it can successfully telephone, it cannot control a touch-tone menu, so Cáritas used to hold a cell phone up to this landline telephone's microphone to successfully choose options in a touch-tone menu! Sic! (S.

When a bug causes grief, it makes your product memorable -- for all the wrong reasons.

Robotron has (had?) a bug where the explosion code for enforcers improperly clipped the "shrapnel" to the display portion of memory (frame buffer). So, shrapnel could clobber data structures that the software relied on in the *adjacent* "program memory".

Players HATED this as it would usually manifest when you were in the heat of a really intense level and the machine would spontaneously reset and repaint the Willy balls. Terminating your game, depriving you of any "high score" recognition, etc.

Such problems persist in current day products. E.g., I can drive on roadways that my GPS doesn't yet recognize as such. It will sometimes show me driving on "unpaved areas" OR suddenly "snapping" to a nearby road that it DOES know about. In each case, one has to wonder what other results (distance/time to destination, best route, etc.) are compromised.

A friend bought me a "pocket dialer" in the early 80's. It was a credit-card sized device with an LCD display and keypad -- with a small speaker on the rear side. Hold it over the mouthpiece of a phone and dial a number

I reciprocated with a Demon Dialer

I had the same feelings of joy programming the 68020 in assembly language. It had a symmetrical and complete set of op-codes unlike what I had used before.

This was after suffering through the contortions of programming the 8085, Z-80 and Z-80 variants (HD64180, Z80180) and in my recollection, the truly awful 8051.

Back then I thought a 16Mhz 68020 was quite fast.

My favorite scope has a 68020 in it.

Cheers

Phil Hobbs