Artificial neural networks

What it is

An artificial neural network is a computing system made of many simple connected units, usually called neurons. Each unit combines the signals arriving from other units and passes on an output. The network's behaviour is set by the strengths of the connections between units, known as weights.

How it works

Consider a single unit with two inputs. It multiplies each input by its weight, adds the results, and produces a signal only when the total passes a threshold. Changing one weight changes which inputs the unit responds to. For example, a unit can be set up to respond when its first input is active and its second is not. A network stacks many such units in layers, and its overall behaviour depends on all their weights together. Learning means adjusting those weights so that the outputs match the desired answers more often.

Where it came from

The idea starts in 1943. Warren McCulloch and Walter Pitts proposed a mathematical model of an all-or-none neuron that fires only when enough excitatory inputs are active and no inhibitory input is active. Networks of these units can carry out propositional logic. The authors also argued that such networks, given a tape and scanners, can compute the same numbers as a Turing machine. Their analysis assumed that the structure of the net does not change over time, so it offered no learning procedure.^[1]

Learning came with Frank Rosenblatt's perceptron, described in Psychological Review in 1958. Its retina of sensory units feeds association units, which feed response units, and the values of the connections change during training. The system therefore learns to classify stimuli by trial and error.^[2]

In a July 1958 demonstration funded by the U.S. Office of Naval Research, a perceptron simulated on an IBM 704 computer learned, after 50 trials, to distinguish punch cards marked on the left from those marked on the right.^[3]

Hardware followed. The Mark I Perceptron, custom-built for image recognition and completed around 1960, took its input from a 20×20 grid of cadmium sulfide photocells, 400 pixels in all. Its adjustable weights were stored in potentiometers, which electric motors turned during learning. The machine is now in the Smithsonian Institution.^[4]

Where it is used today

Today's networks are much larger. They usually have many layers of units and are trained on large amounts of data, and they are widely used for tasks such as image recognition and speech and language processing. A network with one or more hidden layers between input and output, known as a multilayer perceptron, is more powerful than a single-layer model. The line of work built on such networks led to today's deep learning.^[4]

Limits

A single-layer perceptron can only learn patterns that are linearly separable. It cannot, for example, compute the XOR (exclusive-or) function, which outputs true only when exactly one of its two inputs is true.^[4] Minsky and Papert's 1969 book Perceptrons analysed these limits and was followed by a long decline in funding and interest in neural networks, which lasted until a resurgence in the 1980s.^[4]