Hopfield network

What it is

A Hopfield network is a form of recurrent neural network — and, in the description Wikipedia gives, a spin glass system — that can serve as a content-addressable memory ^[1]. The distinction matters: ordinary computer memory is read by address, whereas a content-addressable memory is read by content. You hand the network a fragment, and it returns the whole stored item.

The model is named for John Hopfield, whose 1982 paper applied statistical mechanics to the study of a network with binary activation functions ^[1].

How it works

In the original model the units take binary activation values; in a 1984 paper Hopfield extended the model to continuous activation functions ^[1]. Patterns are stored with a Hebbian learning algorithm, and a trained network can recover a stored state from a distorted input ^[1].

Recall happens through dynamics rather than lookup. Certain inputs are fixed, and the network is left to evolve so as to minimise an energy function, moving towards local minima that correspond to the stored patterns ^[1]. The constraint that the weights are symmetric guarantees that the energy function decreases monotonically under the activation rules ^[1], so the state does not wander: it always runs downhill on the energy surface.

Picture the state of the network as a point on a landscape whose valleys are the stored patterns. A corrupted input — a copy of a stored pattern with some units flipped to the wrong value — places the point somewhere on a slope, and the downhill motion carries it into the nearest valley. The 1982 abstract makes the same point in its own language: the model produces a content-addressable memory that correctly yields an entire memory from any subpart of sufficient size, and the algorithm for time evolution is based on asynchronous parallel processing ^[2].

Where it came from

The paper was titled "Neural networks and physical systems with emergent collective computational abilities" and appeared in Proceedings of the National Academy of Sciences 79(8):2554–2558 ^[1]. The Caltech repository record gives the same volume, issue and pages, and notes that the article was contributed by John J. Hopfield on January 15, 1982 ^[3]. That record also states that work at the California Institute of Technology was supported in part by National Science Foundation Grant DMR-8107494 ^[3]. PubMed indexes the paper in Proc Natl Acad Sci U S A, 1982 Apr, under PMID 6953413 and DOI 10.1073/pnas.79.8.2554 ^[2].

The abstract states the underlying claim: computational properties can emerge as collective properties of systems having a large number of simple equivalent components, and it describes the physical meaning of content-addressable memory through a phase space flow of the state of a system ^[2].

According to Wikipedia, the model became a standard model for the study of neural networks through statistical mechanics ^[1].

What it was used for

In 1985 Hopfield and Tank presented an application of the network to the travelling-salesman problem, after which it was widely used for optimisation ^[1]. In 2024 John J. Hopfield and Geoffrey E. Hinton were awarded the Nobel Prize in Physics for foundational contributions to machine learning such as the Hopfield network ^[1].

Limitations

Classical Hopfield networks are rarely used in modern machine learning, for reasons Wikipedia lists: binary-valued neurons, limited scalability, and incompatibility with gradient-based learning ^[1]. Their importance is therefore conceptual as much as practical: they made it possible to treat a network of simple units as a physical system with an energy, and to ask what such a system computes.