Artificial neurons that behave like real brain cells

Sigh. The problem with this kind of thing for 'AI' is that brains and LLMs are about connectivity. In semiconductor terms, each neuron has a fanout of about 1000. The GPU folks handle that because everything is just an enormous matrix that's being streamed from fast memory, and if you're multiplying a 1 zillion x 1 zillion matrix then achieving a fanin of 1000 is easy to arrange - you just have 1000 non-zero input cells for each row or column.

But with memristors there's no fanout - if you do it in fixed wires you have have an enormous place and route problem (and an inability to change the connections), and if you do it via a massive in-memory routing table then that's going to eat your power even if your actual neuron is very efficient.

Not to say that better models of neurons aren't useful for eg medical purposes or biological research, but not for today's 'AI'.

Theo

Natural neurones grow their synapses - in our brain that keeps on happening until around 18 years of age - and then proceed to edit out a lot of them - which keeps on going until about age 25.

One of wife's colleagues who used ECG to find out what was happening in the human brain when it processed speech wouldn't look at left-handers like me because the processing happens in different areas in our brain - it is the same processing, but while right-handers typically put the various stages of processing in much the same locations in their heads, left handers mix it up more, and differently, from one to the next.

It doesn't seem to make any difference to the results. Maybe it did back when the speech processing system was evolving to cope with the human languages we were in the process of evolving, some 50,000 years ago.

Growing the connections would be a slow way of setting up the models.

Even single cells with no neurons have very complex behavior and learning. It's absurd to claim that some cartoon gadget approximates a neuron, but people keep doing it.

John Larkin Highland Tech Glen Canyon Design Center Lunatic Fringe Electronics

This ------------------------------------------------^^^^^^^^^^^^^^^^^^^^^^

I think the brain gains efficiency by knowing (learning) which connections don't bear fruit -- so, why support them?

A machine has to have ALL those (potential) connections in place and

"Dedicated appliances"

While that's true, it's a game of how much detail you can throw away for what percentage of an approximation. Doing a full model of cell chemistry (ion potentials etc) is very complex and time consuming, and if you can model that by a row in a matrix then you can make the problem much easier. The question is whether that gives you sufficient accuracy to give something of interest, or whether the problem is sufficiently awkward that you have to model it in full detail.

Sort of like is it linear in which case the first step of the iterative solver is enough, or is it chaotic in which case you have to follow the solver all the way to convergence?

The 'AI' folks seem to think they can manage with throwing away a lot of bits of precision. Remains to be seen whether that applies to biological systems too.

Theo

Exactly.

I recall my housemaster

The Chinese have apparently demonstrated that even *more* can be discarded than Western researches have assumed -- without "noticeable" impact on results.

(As with Lettvin, they have yet to answer whether or not there are practical consequences to these approximations)

"All hail, great and glorious, Grand Wazoo!" RIP, Jerry.

Some optimisations are possible. When I was working on such things around 1990 I found that while training of ANNs needed floating point machines (Sun workstations at that time) to avoid running into numerical accuracy problems, the final trained neural network could be implemented on 16-bit integer hardware. Carefully crafted assembler on a TMS320C50 DSP gave very high performance for the power used. Each artificial neuron in the network (which was a multi-layer perceptron) multiplied an input vector by a fixed vector of weights and generated the result in the 32-bit accumulator of the DSP. This process generates more bits of precision than one starts with in each element of the input vector, so appropriately scaled fixed point processing works just fine and is very power efficient. The application was a cochlear implant speech processor - battery powered and wearable. The Harvard DSP architecture is very well suited to implementing artificial neurons because it allows a pair of numbers to be fetched and multiplied together, the result to be added to an accumulator and index pointers to be incremented and checked against bounds all in one machine cycle. John

As I mentioned earlier, the structure of an artificial neuron is effectively an integrator which has more bits of precision than the individual inputs (of which there are many). One of the problems I had with developing ANN code was that bugs made relatively small differences to the final output. I once spent many days trying to find such a bug because it made a detectable but very small difference to the output. I had forgotten to add an extra assembler instruction to flush the pipeline at the end of a long multiply-accumulate sequence. However, during the training process, much more precision was needed to avoid settling on local minima in the optimisation. Added noise was useful at times to shake things up and help to find a more global minimum. High precision was needed to avoid going in the wrong direction. John

Of course! But, you're now dealing with a specific problem, not "general learning".

---------------------------------^^^^^^^^

But, the same network likely couldn't be used to "learn" how to control the body's muscles for locomotion. A "brain" has to be able to meet ANY of the needs with which the organism is challenged. I.e., my reference to "dedicated appliances", above.

Even "deep learning" limits the number of layers of perceptrons to something "small"; do we know how many layers the brain employs to "solve" a particular task? How does a brain implement "sequencing" as not all "solutions" manifest instantly as a sum-of-products? The brain seems to be aware of time -- and, factors that into its processing.

Perceptrons are also *binary* actors; I (speaking solely based on "feel") suspect our nervous system relies on more than just "did it fire" but likely also addresses the *type* of response to propagate to next layers in the network.

Also, this *suggests* (much research is conducted HOPING to extrapolate to organic equivalents, but without guarantees that this would be possible/applicable) that learning requires more functionality than "remembering". I.e., a (organic) network, once trained, can continue to function despite deteriorating effects of aging, injury, etc.

I have no idea whether the brain uses different areas for learning versus acting on that learning later. Its an interesting question though.

Relative timing can be important. For example the area of the brainstem where signals from each ear come together is associated with the ability to determine the direction of a sound source. Microsecond differences in timing seem to be detectable.