Merck Molecular Activity Challenge on Kaggle

Background

Kaggle is an online platform for predictive-modelling competitions. An organisation posts a problem together with a data set, competitors submit predictions, and the submissions are scored on a leaderboard. In August 2012 the platform ran the Merck Molecular Activity Challenge, which opened on 17 August 2012 and closed on 17 October 2012 ^[1].

The sponsor was Merck, a pharmaceutical company; the problem it posed was one of the everyday bottlenecks of drug discovery, where the useful question about a molecule is often not only what it looks like but what it does inside a living system.

What the challenge asked for

The task was to predict biological activities of molecules, on- and off-target, from numerical descriptors generated from their chemical structures ^[1]. Two ideas sit behind that sentence. A molecule's on-target activity is what it does to the biological target it is meant to act on; its off-target activity is what it does to other targets, which is where many unwanted effects come from. Measuring either for every candidate molecule is slow and expensive, so predicting activity from structure alone is attractive. A descriptor, meanwhile, is a number computed from a chemical structure: a way of turning a drawing of atoms and bonds into a fixed-length list of values that a statistical model can consume.

The contest was built on 15 molecular activity data sets, each for a biologically relevant target ^[1]. Competitors therefore had to produce models that held up across many different targets rather than being tuned to a single one.

Scale and prizes

Prizes totalled $40,000, with $22,000 going to first place ^[1]. The challenge drew 374 entrants forming 236 teams ^[1].

The result

First place on Kaggle's results went to George Dahl, Ruslan Salakhutdinov, Navdeep Jaitly, Chris Jordan-Squire and Geoffrey Hinton ^[1]. The New York Times reported in November 2012 that a team of graduate students studying with Geoffrey Hinton had won the top prize in the contest sponsored by Merck ^[2]. According to the newspaper, the team entered at the last minute and designed its software with no specific knowledge of how the molecules bind to their targets ^[2].

Why it mattered

Kaggle chief executive Anthony Goldbloom said it was the first time deep learning had won, and that it had won on a data set on which it would not have been expected to win ^[2]. Deep learning here means neural networks with many layers that build their own internal representations from data, rather than relying on features that domain experts have designed by hand. A chemistry contest is a natural place to expect the opposite: predictors shaped by knowledge of chemistry and of how molecules interact with targets.

What the reported result showed was narrower and more concrete than any general claim about either approach. A team that came in late, and that wrote its software without knowledge of how molecules bind, produced predictions good enough for first place across the challenge's 15 target data sets ^[2][1]. The value of the episode for a reader today is as a documented early case in which a general-purpose learning method, given only descriptor numbers as input, came out ahead in a field where expert-designed features had been the norm — and where the organisers of the contest themselves had not expected that outcome ^[2].

It is also a small landmark in how scientific prediction problems are posed in public: a sponsor supplies data and a metric, many teams attack the same problem independently, and the leaderboard records who did best. The number of entrants — 374 people in 236 teams — is part of the record of how such open competitions grew ^[1].