DARPA Robotics Challenge Trials

A trial built around a disaster scenario

The DARPA Robotics Challenge Trials took place on 20 and 21 December 2013 at the Homestead-Miami Speedway in Homestead, Florida ^[1]. The premise was simple to state and hard to carry out: a robot enters a disaster zone and does useful physical work there while the person controlling it stays out of harm's way ^[1]. That last condition is what makes the problem difficult. If the operator cannot walk into the building, the machine has to be steady enough to cross rubble, capable enough to handle tools and doors, and connected enough that a human somewhere else can still follow what it is doing and tell it what to do next. The trials were arranged so that these abilities could be measured in a common way, letting very different machines be compared on the same set of jobs.

Eight tasks, sixteen teams

DARPA said 16 teams demonstrated their prototype robots' ability to perform critical real-world disaster-response skills across eight tasks ^[1]. The eight tasks simulated what a robot might have to do to enter and work inside a disaster zone while its operator stayed out of harm's way ^[1]. Each task therefore stood in for a piece of the overall job — a movement, an obstacle, a manipulation — rather than for a whole disaster. Scoring across eight separate tasks, with points accumulated into a single total, gives a picture of where a machine is competent and where it is not; a single demonstration of one impressive skill would not show that. The format also puts pressure on teams to make their robots work across a range of situations rather than optimising for one.

The scoreboard

SCHAFT of Tokyo scored the highest, with 27 points out of a possible 32 ^[1]. IHMC Robotics scored 20 points, Tartan Rescue 18, Team MIT 16 and RoboSimian 14 ^[1]. Gill Pratt, the DARPA programme manager for the event, said it exceeded his expectations multiple times over, pointing out that the top four teams each scored half the available points or more ^[1]. That observation is a useful way to read the table: the spread between first and fifth place was wide, but the leading group was not scraping by on a handful of points — roughly half the total or better was reached by four different teams, each with its own design and its own engineering choices.

The machines behind the numbers

NBC News reported that Team SCHAFT's humanoid S-One won top scores in four of the eight tasks ^[2]. The same report noted that Team MIT wrote its own software to run on the Atlas humanoid built by Boston Dynamics ^[2]. Taken together, the two details point at something structural in this kind of competition: the physical robot and the control software are separate pieces of work, and a team can be competitive by being strong on either side. A team that builds its own machine and a team that writes software for someone else's machine can end up on the same scoreboard, measured by the same tasks.

What the trials left behind

DARPA said it selected eight teams to receive up to $1 million in funding to continue their work ^[1]. The decision indicates what the trials were for: not to crown a winner and stop, but to find out which approaches were worth carrying further and to keep those teams going. For anyone following the field, the event also left a comparable record — one set of tasks, one scoring scheme, a ranked list of teams and the specific skills their robots managed to demonstrate.