NVIDIA GeForce GTX 580

What it is

The GeForce GTX 580 is a consumer graphics card that NVIDIA announced on 9 November 2010. It is built on the Fermi architecture, which NVIDIA described at launch as having been re-engineered "from the transistor-level up" for better power efficiency ^[1]. Like any graphics card, its job is to run many small arithmetic operations at once: work that games need for shading and geometry, and work that later turned out to suit neural networks as well.

Launch and specifications

NVIDIA's launch release cited 512 cores and 16 PolyMorph engines, and claimed up to 35% better performance per watt and up to 30% more speed than the GeForce GTX 480 ^[1]. The cores are the parallel arithmetic units, while the PolyMorph engines handle geometry processing for DirectX 11. The card went on sale immediately at an estimated price of $499, through partners such as ASUS, EVGA, Gigabyte, MSI and Zotac ^[1]. Marketing pitched it at DirectX 11 games, and the launch release also listed support for CUDA, NVIDIA's platform for general-purpose computing on GPUs ^[1]. CUDA is the piece that connects the gaming product to the research story: it is the interface through which a programmer can write general numerical code and have the graphics card execute it.

Two cards, one network

In 2012, AlexNet — the convolutional network by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton that won ILSVRC-2012 — was trained on two GTX 580 cards with 3 GB of memory each ^[2]. The paper reports that training took five to six days for roughly 90 passes through the 1.2 million training images ^[2]. While a network trains, its parameters and the intermediate values it computes have to sit in memory, so 3 GB per card set a hard ceiling on how large the model could be. The authors worked around it by splitting the network across the two GPUs, putting half of the kernels on each card and letting the GPUs communicate only in certain layers ^[2]. The paper notes that the GPUs could read from and write to each other's memory directly, without going through host memory, which made this cross-GPU parallelisation practical ^[2].

Where it sits in the story

The AlexNet authors wrote that the network's size was limited mainly by the memory available on current GPUs, and that their results could be improved simply by waiting for faster GPUs and bigger datasets ^[2]. NVIDIA's own corporate timeline lists 2012 as the year it "sparked the era of modern AI by powering the breakthrough AlexNet neural network" ^[3]. For a consumer graphics card that is a small footnote in a product line; read the other way, it is a record of how a piece of gaming hardware on sale for an estimated $499 became, two years after launch, the machine on which a landmark result was produced ^[1][2].