The Netflix Prize (2006-2009)

Background

Netflix's recommendations rested on collaborative filtering, the idea that one person's taste can be predicted from the ratings of many other users. The company's own algorithm, Cinematch, was the baseline that contestants had to beat ^[1]. In October 2006 Netflix released a data set of 100 million anonymous movie ratings and challenged the data mining, machine learning and computer science communities to beat Cinematch's accuracy ^[2]. Announcing the contest, Netflix vice president of recommendation systems Jim Bennett said that 100 million movie ratings from about 500,000 subscribers were being released with personal information removed ^[3]. A Netflix announcement dated 2 October 2006 described an award of $1 million to the first person or team achieving the accuracy goals, with details at www.netflixprize.com ^[4].

The data

The ratings came from over 480 thousand randomly chosen anonymous subscribers, covered nearly 18 thousand movie titles and had been collected between October 1998 and December 2005; over 3 million of the most recent ratings were withheld as a qualifying set ^[2]. In exact terms, the training set held 100,480,507 ratings that 480,189 users gave to 17,770 movies ^[1]. The qualifying set contained 2,817,131 triplets, split into a quiz set of 1,408,342 ratings that produced the leaderboard scores and a test set of 1,408,789 that decided the winners ^[1].

How it worked

Contestants had to predict all 3 million withheld ratings. The error was computed immediately on a fixed but unknown half — the quiz set — and posted to a public leaderboard, while the unreported test half was used to identify the winners ^[2]. Submissions were scored by root mean squared error (RMSE): on the quiz set a trivial algorithm scored 1.0540, against 0.9514 for Cinematch ^[1].

Prizes

The grand prize of $1,000,000 required a further 10% improvement over Cinematch, that is an RMSE of 0.8572 on the test set ^[1]. As long as no team won the grand prize, a progress prize of $50,000 was awarded each year for the best result so far ^[1]. NBC News reported that if there was no winner, Netflix would pay $50,000 annually to the entrant making the most progress toward the goal ^[3]. Winners of the various prizes were required to document and publish their approaches publicly ^[2].

The response

By June 2007, over 20 thousand teams from over 152 countries had registered, 2 thousand had submitted prediction sets, and over 650 teams had already exceeded Cinematch's accuracy ^[2]. Clearing the baseline was therefore common; what stayed out of reach was the full 10%, and no team claimed the grand prize until the competition's final award ^[1].

The result

On 21 September 2009 the grand prize of US$1,000,000 was given to the team BellKor's Pragmatic Chaos, whose best entry beat Netflix's own algorithm by 10.06% ^[1]. The competition had run from 2006 to 2009 ^[1].

Why it mattered

The contest turned an internal engineering problem into a public benchmark: the data, the metric and the leaderboard were the same for everyone, so very different methods could be compared directly on one scale ^[1][2]. The rule that winners document and publish their approaches meant the results did not stay inside Netflix but became part of the public record on recommender systems ^[2]. For participants, the 100 million ratings also provided a shared test bed at a scale that let thousands of teams work on the same prediction task ^[2].