Apache Hadoop

What it is

Apache Hadoop is a collection of open-source software utilities for reliable, scalable distributed computing^[1]. It provides a framework for distributed storage and processing of big data using the MapReduce programming model^[1]. "Distributed" here means splitting a data set and a computation across many machines; what Hadoop addresses is how to do that without losing data or correctness when machines misbehave, and how to grow to more machines when needed.

How it works

The core consists of two parts: a storage part, the Hadoop Distributed File System (HDFS), and a processing part that uses the MapReduce programming model^[1].

HDFS cuts large files into blocks, places the blocks on different machines and keeps multiple copies of each. If a machine goes away, the data can be read from another copy. MapReduce is a convention for writing parallel programs: the programmer expresses a computation as a map stage, applied to each piece of data independently, and a reduce stage that groups and combines the intermediate results produced by the map stage. Because the map stage does not require the pieces to know about one another, the work can be handed out to many machines and run at the same time, with the results collected at the end.

One design assumption runs through the whole framework: all the modules are designed on the assumption that hardware failures are common and should be handled automatically by the framework^[1]. In a large cluster some machine is always failing, so rather than demanding unusually reliable hardware, the software treats failure as normal — a failed task is run again elsewhere, a lost copy of a block is restored from the remaining copies. That assumption explains much of how Hadoop behaves.

Hadoop is implemented in Java and released under the Apache License 2.0^[1]. Doug Cutting and Mike Cafarella are listed as the original authors and the Apache Software Foundation as the developer, with the first release dated 1 April 2006^[1].

Where it came from

Hadoop grew out of a web-search project. In 2002 Cutting and Cafarella began the Nutch web-search project and based its distributed platform on Google's GFS and MapReduce papers; the code that came out of that work became HDFS and Hadoop's MapReduce^[2]. In 2006 Yahoo! hired Cutting and adopted Hadoop, and the storage and processing parts of Nutch were spun out to form Hadoop as an open-source Apache project^[2].

So the two things Hadoop is known for — the distributed file system and the parallel processing model — were first written so that a search engine could index the web across many machines, and were named and released on their own once a large web company wanted the same capability for its own data.

Where it sits

Hadoop's significance is that it turned the approach described in Google's GFS and MapReduce papers into open-source software that anyone could download and set up^[1][2]. A team no longer had to design a distributed storage and batch-processing system from scratch in order to work with data larger than one machine could hold.

For someone learning the field today, Hadoop remains a good reference point. Two ideas in it are worth understanding first: the separation into a distributed storage layer and a batch processing layer, and the decision to design for failure rather than against it. Reading HDFS as "store several copies and let the framework repair them" and MapReduce as "map each piece, then reduce the results" is a more useful start than memorising component names.