Spiking Neural Networks Place Data In Time

Artificial neural networks have found a variety of commercial applications, from facial recognition to recommendation engines. Compute-in-memory accelerators seek to improve the computational efficiency of these networks by helping to overcome the von Neumann bottleneck.

But the success of artificial neural networks also highlights their inadequacies. They replicate only a small subset of the cognitive capabilities of biological brains. Even those tasks they can tackle — often poorly — need vast data centers for tasks that untrained humans find trivial.

In biological brains, synapses define the strength of connections between neurons. If two neurons often fire together, the connection between them becomes strong. Otherwise, it weakens or breaks entirely. Artificial neural networks emulate this behavior with an array of weights — the response to a stimulus is a weighted sum across the input nodes. Compute-in-memory accelerators seek to optimize these weighted sums.

The synapses are only half the story, though. Rather than doing “computation” in the von Neumann sense, biological brains create and accumulate sequences of electrical spikes. For example, the rods and cones in the eye emit electrical signals in response to light falling on the retina. This “spike train” both captures a portion of the image and places it in a temporal sequence. Successive layers of neurons assemble signals from the full retina into complex images, combine them with auditory and other sensory inputs, then use knowledge and experience to evaluate the situation and take action if needed. Synapses allow the brain to prioritize inputs, but neurons are responsible for actually sending and receiving signals. Spiking neural network designs seek to emulate this signal processing behavior.

Instead of accumulating weighted sums, spiking neural network models track one or more time-dependent state variables as they evolve in response to input spikes. Biologically, neural currents are due to the movement of ions (mostly Na+) through channels in the cellular membrane. So a simple model might have two state variables, membrane potential and current, related by a single linear differential equation. When the relevant state variable reaches a threshold, the neuron fires. More complex models might involve more variables, reflecting changes in firing rate due to membrane leakage, recovery time after a spike, and so on.

Though such simple models are easy to implement, they fail to capture the richness of biological behavior. Neuroscientists use experimental techniques like functional magnetic resonance imaging (fMRI) to observe the activation of specific regions in biological brains in response to a stimulus. While directly decoding neurological signals is only possible in very limited circumstances, it’s clear that different cognitive tasks involve different patterns of neurological activation. Each neuron is connected to potentially thousands of its peers and is able to both receive and send signals across each connection. In human brains, more than two hundred known neurotransmitters help define neural connectivity. “Neuromorphic” artificial intelligence models face a conflict between, on one hand, the complexity of biological systems and, on the other hand, the need to operate within reasonable limitations on circuit area, power consumption, and so on. More biologically accurate models are more difficult to implement.

Introducing the neuristor
In 2012, Matthew D. Pickett and colleagues at HP Labs proposed a “neuristor” device to capture three fundamental elements of biological neural behavior. The first, already mentioned, is threshold-driven electrical spiking. Spikes arise from the potential difference between the interior of the neuron and the surrounding intracellular fluid. Once the potential difference exceeds a threshold, the neuron fires, sending current spikes to adjacent neurons. In biological neurons, Nodes of Ranvier along the neural fibers recharge the action potential and facilitate lossless propagation of these spikes, the second fundamental characteristic, and a prerequisite for fine motor control at extremities. Finally, each spike “discharges” the neuron for a period of time — the third fundamental characteristic is the need for a recovery period between spikes.

The simplest mathematical models and hardware implementations that can capture these most basic behaviors are “integrate and fire” models. A neuron collects input spikes from upstream neurons and fires a spike downstream whenever those inputs exceed a threshold. The next step up in sophistication, “leaky integrate and fire” models, introduce a decay period. The potential difference at the cellular membrane dissipates over time. To cause a downstream spike, the rate of inputs must be high enough to raise the potential difference more rapidly than it leaks away. These models introduce a memory effect. That is, the state variable evolves over time, and the spiking behavior depends on that history.

Integrate and fire models, leaky or not, still fall short of the complex activation patterns seen in biological brains. In particular, biological neurons exhibit both excitatory and inhibitory behavior. A signal from neuron A can make neuron B either more or less likely to fire. A third neuron, C, might control the interaction between A and B.

A 2017 survey by Catherine Schuman and colleagues at Oak Ridge National Laboratory observed that the Hodgkins-Huxley model is one of the simplest models able to produce biologically plausible results. It describes the change in potential across the cellular membrane in terms of sodium and potassium ion flows, using four state variables described by four differential equations. This model is a starting point for many attempts to build artificial spiking neural networks.

Neurons and synapses have different roles and require different circuit elements. While compute-in-memory accelerators might use RRAM and other non-volatile memory elements to store synaptic weights, the dynamic fire-and-recover behavior of neurons requires different characteristics. And CMOS-based designs can replicate Hodgkins-Huxley behavior, but Wei Yi and colleagues at HRL Laboratories argued that doing so requires a very large circuit footprint. A CMOS-based design can be scalable or biologically plausible, they said, but not both.

If not CMOS, what?
CMOS devices are also inherently deterministic, while real neuron behavior is stochastic. In biological neurons, random fluctuations in the membrane potential arise from the stochastic behavior of ion channels, stochastic synaptic transmission, crosstalk between neighboring neurons, and so on. This natural randomness leads to variations in spike periodicity that some advanced models, like Bayesian inference, seek to exploit.

As an alternative to CMOS, Pickett pointed out that the Hodgkins-Huxley model is mathematically equivalent to a system with two Mott insulators and two parallel capacitors. The second part of this article looks at system designs based on the Hodgkins-Huxley model and the differences in learning behavior between conventional and spiking neural networks.

Read Part 2 of this story.

Related
Spiking Neural Networks: Research Projects Or Commercial Products?
Opinions differ widely, but in this space that isn’t unusual.
Scaling Up Compute-In-Memory Accelerators
New research points to progress and problems in a post-von Neumann world.
Compute-In Memory Accelerators Up-End Network Design Tradeoffs
Compute paradigm shifting as more data needs to be processed more quickly.
Are Better Machine Training Approaches Ahead?
Why unsupervised, reinforcement and Hebbian approaches are good for some things, but not others.
Manufacturing Bits: May 5,2020
Spiking neural network radar chip