Biologically inspired: How neural networks are finally maturing

More than two decades ago, neural networks were widely seen as the next generation of computing, one that would finally allow computers to think for themselves.

Now, the ideas around the technology, loosely based on the biological knowledge of how the mammalian brain learns, are finally starting to seep into mainstream computing, thanks to improvements in hardware and refinements in software models.

Computers still can’t think for themselves, of course, but the latest innovations in neural networks allow computers to sift through vast realms of data and draw basic conclusions without the help of human operators.

“Neural networks allow you to solve problems you don’t know how to solve,” said Leon Reznik, a professor of computer science at the Rochester Institute of Technology.

Slowly, neural networks are seeping into industry as well. Micron and IBM are building hardware that can be used to create more advanced neural networks.

On the software side, neural networks are slowly moving into production settings as well. Google has applied various neural network algorithms to improve its voice recognition application, Google Voice. For mobile devices, Google Voice translates human voice input to text, allowing users to dictate short messages, voice search queries and user commands even in the kind of noisy ambient conditions that would flummox traditional voice recognition software.

Neural networks could also be used to analyze vast amounts of data. In 2009, a group of researchers used neural network techniques to win the Netflix Grand Prize.

At the time, Netflix was holding a yearly contest to find the best way to recommend new movies based on its data set of approximately 100 million movie ratings from its users. The challenge was to come up with a better way to recommend new movie choices to users than Netflix’s own recommendation system. The winning entry was able to improve on Netflix’s internal software, offering a more accurate predictor of what movies Netflix may want to see.

Neural networking vs. computing
As originally conceived, neural networking differs from traditional computing in that, with conventional computing, the computer is given a specific algorithm, or program, to execute. With neural networking, the job of solving a specific problem is largely left in the hands of the computer itself, Reznick said.

To solve a problem such as finding a specific object against a backdrop, neural networks use a similar, though vastly simplified, approach to how a mammalian cerebral cortex operates. The brain processes sensory and other information using billions of interconnected neurons. Over time, the connections among the neurons change, by growing stronger or weaker in a feedback loop, as the person learns more about his or her environment.

An artificial neural network (ANN) also uses this approach of modifying the strength of connections among different layers of neurons, or nodes in the parlance of the ANN. ANNs, however, usually deploy a training algorithm of some form, which adjusts the nodes to extract the desired features from the source data. Much like humans do, a neural network can generalize, slowly building up the ability to recognize, for instance, different types of dogs, using a single image of a dog.

There are numerous efforts under way to try to replicate, at high fidelity, how the brain operates in hardware, such as the EU’s Human Brain Project (see accompanying story: “Bringing brains to computers”). Researchers in the field of computer science, however, are borrowing the ideas from biology to build systems that, over time, may learn in the same way brains do, even if their approach differs from that of biological organisms.

Evolution of neural networking
Although investigated since the 1940s, research into ANNs, which can be thought of as a form of artificial intelligence (AI), hit a peak of popularity in the late 1980s.

“There was a lot of great things done as part of the neural network resurgence in the late 1980s,” said Dharmendra Modha, an IBM Research senior manager who is involved in a company project to build a neuromorphic processor. Throughout the next decade, however, other forms of closely related AI started getting more attention, such as machine learning and expert systems, thanks to a more immediate applicability to industry usage.

Nonetheless, the state-of-the-art in neural networks continued to evolve, with the introduction of powerful new learning models that could be layered to sharpen performance in pattern recognition and other capabilities,

“We’ve come to the stage where much closer simulation of natural neural networks is possible with artificial means,” Reznick said. While we still don’t know entirely how the brain works, a lot of advances have been made in cognitive science, which, in turn, are influencing the models that computer scientists are using to build neural networks.

“That means that now our artificial computer models will be much closer to the way natural neural networks process information,” Reznick said.

The continuing march of Moore’s Law has also lent a helping hand. Over the past decade, the microprocessor fabrication process has provided the density needed to run large clusters of nodes even on a single slice of silicon, a density that would not have been possible even a decade ago.

“We’re now at a point where the silicon has matured and technology nodes have gotten dense enough where it can deliver unbelievable scale at really low power,” Modha said.

Harnessing processors
Reznick is leading a number of projects to harness today’s processors in a neural network-like fashion. He is investigating the possibility of using GPUs (graphics processing units), which thanks to their large number of processing cores, are inherently adapt at parallel computing. He is also investigating how neural networking could improve intrusion detection systems, which are used for detect everything from trespassers on a property to malicious hackers trying to break into a computer system.

Today’s intrusion detection systems work in one of two ways, Reznick explained. They either use signature detection, in which they recognize a pattern based on a pre-existing library of patterns. Or they look for anomalies in a typically static backdrop, which can be difficult to do in scenarios with lots of activity. Neural networking could combine the two approaches to strengthen the ability of the system to detect unusual deviations from the norm, Reznick said

One hardware company investigating the possibilities of neural networking is Micron. The company has just released a prototype of a DDR memory module with a built-in processor, called Automata.

While not a replacement for standard CPUs, a set of Automata modules could be used to watch over a live stream of incoming data, seeking anomalies or patterns of interest. In addition to these spatial characteristics, they can also watch for changes over time, said Paul Dlugosch, director of Automata processor development in the architecture development group of Micron’s DRAM division.

“We were in some ways biologically inspired, but we made no attempt to achieve a high fidelity model of a neuron. We were focused on a practical implementation in a semiconductor device, and that dictated many of our design decisions,” Dlugosch said.

Nonetheless, because they can be run in parallel, multiple Automata modules, each serving as a node, could be run together in a cluster for doing neural network-like computations. The output of one module can be piped into another module, providing the multiple layers of nodes needed for neural networking. Programming the Automata can be done through a compiler that Micron developed that uses either an extension of the regular expression language or its own Automata Network Markup Language (ANML).

Another company investigating this area is IBM. In 2013, IBM announced it had developed a programming model for some cognitive processors it built as part of the U.S. Defense Advanced Research Projects Agency (DARPA) SyNAPSE (Systems of Neuromorphic Adaptive Plastic Scalable Electronics) program.

IBM’s programming model for these processors is based on reusable and stackable building blocks, called corelets. Each corelet is in fact a tiny neural network itself and can be combined with other corelets to build functionality. “One can compose complex algorithms and applications by combining boxes hierarchically,” Modha said.

“A corelet equals a core. You expose the 256 wires emerging out of the neurons, and expose 256 axioms going into the core but inside of the code is not exposed. From the outside perspective, you only see these wires,” Modha said.

In early tests, IBM taught one chip how to play the primitive computer game Pong, to recognize digits, to do some olfactory processing, and to navigate a robot through a simple environment.

While it is doubtful that neural networks would ever replace standard CPUs, they may very well end up tackling certain types of jobs difficult for CPUs alone to handle.

“Instead of bringing sensory data to computation, we are bringing computation to sensors,” Modha said. “This is not trying to replace computers, but it is a complementary paradigm to further enhance civilization’s capability for automation.”