MNIST database of handwritten digits

The problem it addressed

Handwritten digit recognition was already used as a test case for document recognition and pattern recognition. A concrete earlier resource was a dataset of digits from the US Postal Service constructed in 1988, with 7,291 training images and 2,007 test images; it was used to train and benchmark the 1989 LeNet.^[1] Such work needed data with a clear split between examples used for fitting a model and examples used for testing it. MNIST was assembled to provide that kind of shared reference set for handwritten digits, and its canonical description appears in the paper "Gradient-Based Learning Applied to Document Recognition" by Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, published in the Proceedings of the IEEE, 86(11):2278-2324, November 1998.^[2]

How the database was made

MNIST contains 60,000 training images and 10,000 testing images of handwritten digits. The images are black and white, taken from NIST data and normalized to fit into a 28×28 pixel bounding box; they were also anti-aliased.^[1] It was not a completely new collection of handwriting. The creators made it by re-mixing samples from NIST's original datasets. In those original NIST datasets, the training set was taken from American Census Bureau employees and the testing set was taken from American high school students. For MNIST, half of the training set and half of the test set came from NIST's training dataset, while the other halves came from NIST's testing dataset.^[1] This mixing matters: the final MNIST training and test sets are each built from both source pools, rather than one being only census employees and the other only students.^[1]

Results and reported performance

The creators' original paper reported a support-vector machine error rate of 0.8% on MNIST.^[1] That number is a useful anchor because it states performance on the fixed test set rather than on an arbitrary collection of digits. The database also carries a caution that is easy to overlook: the original MNIST dataset contains at least four wrong labels.^[1] A small amount of label noise means that a model's errors are not all alike; some may come from ambiguous or mislabeled examples, and near-perfect scores need to be read with that in mind. The earlier USPS dataset, by contrast, had 7,291 training and 2,007 test images and was used to train and benchmark the 1989 LeNet.^[1]

Why it matters

MNIST's design makes comparisons straightforward. A fixed 60,000/10,000 split, a uniform 28×28 input format, and a small number of digit classes make it possible to describe an experiment precisely: train on the training set, test on the test set, report the error rate.^[1] The support-vector machine result of 0.8% gave later work a concrete reference point in the creators' original paper.^[1] The dataset also connects to a longer line of document-recognition research: the USPS digits had already been used for LeNet in 1989, and the canonical MNIST paper appeared in 1998 under the title "Gradient-Based Learning Applied to Document Recognition."^[1][2]

Limits and later role

MNIST is a benchmark, not a complete model of handwriting. Its images are normalized into a 28×28 bounding box, so variation in scale and position is reduced before a model sees the digit.^[1] Its labels are not perfectly clean, with at least four known wrong labels in the original dataset.^[1] Those limits do not make it useless; they define what a score means. A result on MNIST says how well a method separates the digit classes under the database's fixed preparation and split, and the 0.8% SVM error rate in the original paper is one such point of comparison.^[1] In that role, MNIST serves as a compact common ground for work on document recognition and on methods that learn from labeled images.