Cinematch

What it was

Cinematch was Netflix's own algorithm for predicting the rating a user would give a film ^[1]. Its predictions appeared on the company's website as red stars ^[2]. Accuracy was measured as root mean squared error (RMSE): the typical size of the gap between the rating the system predicted and the rating the subscriber actually gave, expressed in the same units as the ratings themselves ^[2].

How it worked

Netflix described Cinematch in public as a set of straightforward statistical linear models combined with a lot of data conditioning — the general practice of cleaning and preparing raw data before fitting a model ^[1]. A paper by Netflix engineers James Bennett and Stan Lanning gives a more detailed picture: the system analysed the accumulated movie ratings weekly, using a variant of Pearson correlation to find films that were predictive of one another, and then computed a multivariate regression in real time as the user rated films ^[2]. Pearson correlation here is a standard way of measuring how closely two quantities move together, so the first step amounts to finding films whose ratings rise and fall in similar patterns. When no personalised prediction was available for a given user and film, the system fell back on the average rating for that film ^[2].

The data behind it was large. Netflix said it had collected more than 1.9 billion ratings from more than 11.7 million subscribers on more than 85 thousand titles since October 1998, and that Cinematch was producing several hundreds of millions of personalised predictions per day ^[2].

How accurate it was

Using only the training data, Cinematch scored an RMSE of 0.9514 on the quiz data and 0.9525 on the test set ^[1]. For comparison, a trivial algorithm that simply predicts each movie's average grade from the training data produced an RMSE of 1.0540 ^[1]. Bennett and Lanning put the quiz figure at 0.9514, which they described as a 9.6% improvement over simply predicting individual movie averages ^[2]. In other words, most of the value came from knowing the film and the user's history at all; the remaining gap — a few hundredths of a rating point — was what the harder modelling work had to attack.

The Netflix Prize

By 2006 Cinematch's performance had plateaued, and Netflix built its public competition around that fact: the Netflix Prize was judged on improvement over Cinematch ^[1]. NBC News reported that the company was offering a $1 million reward for improving the accuracy of its movie recommendation system by at least 10 percent ^[3]. Netflix CEO Reed Hastings framed the ambition with a comparison: "Right now, we're driving the Model T version of what is possible," he said, and "We want to build a Ferrari" ^[3].

Where it sits

Cinematch is the reference point against which the Netflix Prize was defined: a working, large-scale recommender whose error rate was published so that outside teams could try to beat it ^[1][2]. Its own numbers — 0.9514 on the quiz set against 1.0540 for plain movie averages — fixed the scale on which progress in the contest was measured ^[1]. Because the target was stated as a relative improvement over Cinematch, every later result in that line of work is expressed as a distance from what this system could already do.