MapReduce: Simplified data processing on large clusters

MapReduce: Simpli ed Data Processing on Large ClustersJeffrey Dean and Sanjay Ghemawatjeff@google.com, sanjay@google.comGoogle, Inc.Abstract given day, etc. Most such computations are conceptu-ally straightforward. However, the input data is usuallyMapReduce is a programming model and an associ- large and the computations have to be distributed acrossated implementation for processing and generating large hundreds or thousands of machines in order to nish indata sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par-key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handlepairs, and a reduce function that merges all intermediate failures conspire to obscure the original simple compu-values associated with the same key. Many tation with large amounts of complex code to deal withreal world tasks are expressible in this model, as shown these issues.in the paper. As a reaction to this complexity, we designed a newPrograms written in this functional style are automati- abstraction that allows us to express the simple computa-cally parallelized and executed on a large cluster of com- tions we were trying to perform but hides the messy de-modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distributiondetails of partitioning the input data, scheduling the pro- and load balancing in a library. Our ...