Difference between revisions of "MapReduce"

#ENCODING
import io
import sys
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-16')
#MONGO
from pymongo import MongoClient
client = MongoClient()
client.progzoo.authenticate('scott','tiger')
db = client['progzoo']
#PRETTY
import pprint
pp = pprint.PrettyPrinter(indent=4)

Page Under Construction

Introducing the MapReduce function

The MapReduce function is an aggregate function that consists of two functions: Map and Reduce. As the name would suggest, the map is always performed before the reduce.

The map function takes data and breaks it down into tuples (key/value pairs) for each element in the dataset
The reduce function then takes the result of the map function and simply reduces it in to a smaller set of tuples by merging all values with the same key.

Map is used to deal with "embarassingly parallel problems" where a task can be broken down into subtasks that can then be ran simultaneously without affecting each other. Instead of just processing elements one by one, all elements can all be dealt with at the same time in parallel. This allows for massively reduced processing times as well as large scalability across multiple servers, making it an attractive solution to handling Big Data.

This is a feature more suited for the shell or a Node.JS implementation, as here we will need to use JavaScript code inside Pymongo. Also note that the Mongo shell version of this is mapReduce, whereas PyMongo use map_reduce()

For this example our MapReduce takes the form:

db.<collection>.map_reduce(
    <map function>,
    <reduce function>,
    <out collection>
)

from bson.code import Code
pp.pprint(
    db.world.map_reduce(
        Code("function(){emit(this.continent, this.population)}"), 
        Code("function(key, values){"
                  "    var total = 0;"
                  "    for (var i = 0; i < values.length; i++){"
                  "        total += values[i];"
                  "    }"
                  "    return total;"
                  "}"),
        {"inline":1},
    )
)

from bson.code import Code
pp.pprint(
    db.world.map_reduce(
        Code("function(){emit(this.continent, this.population)}"), 
        Code("function(key, values){"
                  "    var total = 0;"
                  "    for (var i = 0; i < values.length; i++){"
                  "        total += values[i];"
                  "    }"
                  "    return total/values.length;"
                  "}"),
        {"inline":1},
    )
)