IBM Research begins the Jeopardy! challenge that produces Watson

Background

In 1997 IBM's chess system Deep Blue defeated Garry Kasparov. IBM says that since then it had been looking for a new human-versus-machine challenge, and that in 2006 David Ferrucci pitched the idea of a computer that could beat a human at Jeopardy! ^[1]

Jeopardy! is a long-running televised quiz show. What makes it hard for a machine is not one narrow specialty but the breadth of the material: a player has to handle questions from many unrelated fields, which is why the system that eventually played it is described as an open-domain question-answering system ^[2]. Answering a question in the open domain means locating candidate answers in a large body of text and judging how much confidence to place in each, with no fixed list of topics to prepare for.

The project begins

IBM's Chinese-language history timeline dates the start of IBM Research's work on a computer system that could compete with Jeopardy! champions to 2007 ^[2]. The stated goal was not a demonstration but a system that could hold its own against the show's best players ^[2].

What the challenge demanded

IBM Research says the Jeopardy Challenge helped address the requirements that led to the design of the DeepQA architecture and the implementation of Watson ^[3]. After three years of intense research and development by a core team of about 20 researchers, Watson was performing at human expert levels in precision, confidence and speed on the Jeopardy quiz show ^[3]. IBM puts the overall effort at five years of work to perfect the question-answering system ^[1].

Precision, confidence and speed are the three qualities a quiz player is judged on: whether the answer is right, whether the player knows how sure to be before buzzing in, and whether the answer comes fast enough to beat the others. A machine can fail on any one of them.

The team and the machine

Ferrucci led the research team and assembled more than two dozen scientists, engineers and programmers at the Thomas J. Watson Research facility in Yorktown Heights, New York ^[1]. The original Watson was a room-sized computer: 10 racks holding 90 servers with a total of 2,880 processor cores, all running the DeepQA software ^[1]. It was named after IBM's first CEO, Thomas J. Watson Sr., and it competed without being connected to the internet ^[1].

The matches

Four years after the work started, in 2011, the Watson open-domain question-answering system defeated the two highest-ranked players in two nationally televised Jeopardy! matches ^[2].

Why it matters

Two things came out of the challenge: the DeepQA architecture and Watson itself ^[3]. A televised quiz show did the job of an engineering specification. It set a target outside judges could recognise, beating the champions, and it named the qualities that had to be good enough: precision, confidence and speed ^[3]. In IBM's account the system had reached human expert levels on those measures by the end of the research phase ^[3].

For anyone following AI, the episode also marks a shift in what was used as a yardstick. A board game has fixed rules and a bounded set of moves; a general-knowledge quiz does not. Turning an open-domain quiz into a research target meant that the hard part was no longer search over a game tree but finding and ranking answers drawn from a very wide range of subjects, and knowing when to trust them.