Difference between revisions of "MapReduce"
m |
|||
| Line 19: | Line 19: | ||
The reduce function then takes the result of the map function and simply reduces it in to a smaller set of tuples by merging all values with the same key.<br/><br/> | The reduce function then takes the result of the map function and simply reduces it in to a smaller set of tuples by merging all values with the same key.<br/><br/> | ||
Map is used to deal with [https://en.wikipedia.org/wiki/Embarrassingly_parallel "embarassingly parallel problems"] where a task can be broken down into subtasks that can then be ran simultaneously without affecting each other. Instead of just processing elements one by one, all elements can all be dealt with at the same time in parallel. This allows for massively reduced processing times as well as large scalability across multiple servers, making it an attractive solution to handling [https://en.wikipedia.org/wiki/Big_data Big Data].<br/><br/> | Map is used to deal with [https://en.wikipedia.org/wiki/Embarrassingly_parallel "embarassingly parallel problems"] where a task can be broken down into subtasks that can then be ran simultaneously without affecting each other. Instead of just processing elements one by one, all elements can all be dealt with at the same time in parallel. This allows for massively reduced processing times as well as large scalability across multiple servers, making it an attractive solution to handling [https://en.wikipedia.org/wiki/Big_data Big Data].<br/><br/> | ||
| − | This is a feature more suited for the shell or a Node.JS implementation, as here we will need to use JavaScript code inside Pymongo. Also note that the Mongo shell | + | This is a feature more suited for the shell or a Node.JS implementation, as here we will need to use JavaScript code inside Pymongo. Also note that the Mongo shell uses the camelCase 'mapReduce()', whereas PyMongo uses the Python naming scheme: map_reduce()<br/><br/> |
For this example our MapReduce takes the form:<br/> | For this example our MapReduce takes the form:<br/> | ||
<pre>db.<collection>.map_reduce( | <pre>db.<collection>.map_reduce( | ||
| Line 30: | Line 30: | ||
<div class=q data-lang="py3">In this example we will be returning the population of all the continents.<br/> | <div class=q data-lang="py3">In this example we will be returning the population of all the continents.<br/> | ||
<code>emit(k,v)</code> lets us pick the fields we want to turn into tuples, where k is the key and v is the value. Our keys will be the continents and our values will be the population<br/> | <code>emit(k,v)</code> lets us pick the fields we want to turn into tuples, where k is the key and v is the value. Our keys will be the continents and our values will be the population<br/> | ||
| − | In our reduce we sum all the values associated with | + | In our reduce we sum all the values associated with each key. |
Finally we specify that we want the "out" part of the mapreduce to be inline rather than a collection, allowing us to print it to screen. | Finally we specify that we want the "out" part of the mapreduce to be inline rather than a collection, allowing us to print it to screen. | ||
<pre class=def> | <pre class=def> | ||
| Line 50: | Line 50: | ||
</div> | </div> | ||
| − | <div class=q data-lang="py3">This is all very good, but what if we only want the results of a MapReduce? Whilst we would normally write our results to another collection, for these examples we can create a temporary results set like so | + | <div class=q data-lang="py3">This is all very good, but what if we only want the results of a MapReduce? Whilst we would normally write our results to another collection, for these examples we can create a temporary results set like so using python to select the results.<br/><br/> |
| + | Alternatively, if you want more information about the timing you can use <code>verbose=1</code> | ||
<p class="strong">Find the average population of each country by continent</p> | <p class="strong">Find the average population of each country by continent</p> | ||
<pre class=def> | <pre class=def> | ||
| Line 99: | Line 100: | ||
from bson.code import Code | from bson.code import Code | ||
temp = db.world.map_reduce(map=Code("function(){emit(this.continent, this.population)}"), reduce=Code("function(key, values){var total = 0;for (var i = 0; i < values.length; i++){total += values[i];}return total/values.length;}"),out={"inline":1});import operator;pp.pprint(list(sorted(temp["results"],key=operator.itemgetter('value'), reverse=True))) | temp = db.world.map_reduce(map=Code("function(){emit(this.continent, this.population)}"), reduce=Code("function(key, values){var total = 0;for (var i = 0; i < values.length; i++){total += values[i];}return total/values.length;}"),out={"inline":1});import operator;pp.pprint(list(sorted(temp["results"],key=operator.itemgetter('value'), reverse=True))) | ||
| + | </div> | ||
| + | </div> | ||
| + | |||
| + | <div class=q data-lang="py3">Until now we've been working on all the data, let's apply a mapreduce to a more specific set of elements by using <code>query</code><br/> | ||
| + | <p class="strong">Find the GDP for each continent, but only include countries that start with the letter A or B.</p> | ||
| + | <pre class=def> | ||
| + | from bson.code import Code | ||
| + | temp = db.world.map_reduce( | ||
| + | query={"name": {"$regex":"^(A|B)"}}, | ||
| + | map=Code("function(){emit(this.continent, this.gdp)}"), | ||
| + | reduce=Code("function(key, values){" | ||
| + | " var total = 0;" | ||
| + | " for (var i = 0; i < values.length; i++){" | ||
| + | " total += values[i];" | ||
| + | " }" | ||
| + | " return total/values.length;" | ||
| + | "}"), | ||
| + | out={"inline":1}, | ||
| + | ) | ||
| + | |||
| + | import operator | ||
| + | pp.pprint(list( | ||
| + | temp["results"] | ||
| + | )) | ||
| + | </pre> | ||
| + | <div class="ans"> | ||
| + | from bson.code import Code;temp = db.world.map_reduce(query={"name": {"$regex":"^(A|B)"}},map=Code("function(){emit(this.continent, this.gdp)}"),reduce=Code("function(key, values){var total = 0;for (var i = 0; i < values.length; i++){total += values[i];}return total/values.length;}"),out={"inline":1});import operator;pp.pprint(list(temp["results"])) | ||
</div> | </div> | ||
</div> | </div> | ||
Revision as of 11:15, 21 July 2015
#ENCODING
import io
import sys
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-16')
#MONGO
from pymongo import MongoClient
client = MongoClient()
client.progzoo.authenticate('scott','tiger')
db = client['progzoo']
#PRETTY
import pprint
pp = pprint.PrettyPrinter(indent=4)
Page Under Construction
Introducing the MapReduce function
The MapReduce function is an aggregate function that consists of two functions: Map and Reduce. As the name would suggest, the map is always performed before the reduce.
The map function takes data and breaks it down into tuples (key/value pairs) for each element in the dataset
The reduce function then takes the result of the map function and simply reduces it in to a smaller set of tuples by merging all values with the same key.
Map is used to deal with "embarassingly parallel problems" where a task can be broken down into subtasks that can then be ran simultaneously without affecting each other. Instead of just processing elements one by one, all elements can all be dealt with at the same time in parallel. This allows for massively reduced processing times as well as large scalability across multiple servers, making it an attractive solution to handling Big Data.
This is a feature more suited for the shell or a Node.JS implementation, as here we will need to use JavaScript code inside Pymongo. Also note that the Mongo shell uses the camelCase 'mapReduce()', whereas PyMongo uses the Python naming scheme: map_reduce()
For this example our MapReduce takes the form:
db.<collection>.map_reduce(
map=<function>,
reduce=<function>,
out=<collection>
)
emit(k,v) lets us pick the fields we want to turn into tuples, where k is the key and v is the value. Our keys will be the continents and our values will be the population
In our reduce we sum all the values associated with each key.
Finally we specify that we want the "out" part of the mapreduce to be inline rather than a collection, allowing us to print it to screen.
from bson.code import Code
pp.pprint(
db.world.map_reduce(
map=Code("function(){emit(this.continent, this.population)}"),
reduce=Code("function(key, values){"
" var total = 0;"
" for (var i = 0; i < values.length; i++){"
" total += values[i];"
" }"
" return total;"
"}"),
out={"inline":1},
)
)
Alternatively, if you want more information about the timing you can use verbose=1
Find the average population of each country by continent
from bson.code import Code
temp = db.world.map_reduce(
map=Code("function(){emit(this.continent, this.population)}"),
reduce=Code("function(key, values){"
" var total = 0;"
" for (var i = 0; i < values.length; i++){"
" total += values[i];"
" }"
" return total/values.length;"
"}"),
out={"inline":1},
)
pp.pprint(list(
temp["results"]
))
from bson.code import Code; temp = db.world.map_reduce(map=Code("function(){emit(this.continent, this.population)}"), reduce=Code("function(key, values){ var total = 0; for (var i = 0; i < values.length; i++){total += values[i];}return total/values.length;}"),out={"inline":1});pp.pprint(list(temp["results"]))
Find the average population of each country by continent then sort by value in descending order.
from bson.code import Code
temp = db.world.map_reduce(
map=Code("function(){emit(this.continent, this.population)}"),
reduce=Code("function(key, values){"
" var total = 0;"
" for (var i = 0; i < values.length; i++){"
" total += values[i];"
" }"
" return total/values.length;"
"}"),
out={"inline":1},
)
import operator
pp.pprint(list(
sorted(temp["results"],key=operator.itemgetter('value'), reverse=True)
))
from bson.code import Code temp = db.world.map_reduce(map=Code("function(){emit(this.continent, this.population)}"), reduce=Code("function(key, values){var total = 0;for (var i = 0; i < values.length; i++){total += values[i];}return total/values.length;}"),out={"inline":1});import operator;pp.pprint(list(sorted(temp["results"],key=operator.itemgetter('value'), reverse=True)))
queryFind the GDP for each continent, but only include countries that start with the letter A or B.
from bson.code import Code
temp = db.world.map_reduce(
query={"name": {"$regex":"^(A|B)"}},
map=Code("function(){emit(this.continent, this.gdp)}"),
reduce=Code("function(key, values){"
" var total = 0;"
" for (var i = 0; i < values.length; i++){"
" total += values[i];"
" }"
" return total/values.length;"
"}"),
out={"inline":1},
)
import operator
pp.pprint(list(
temp["results"]
))
from bson.code import Code;temp = db.world.map_reduce(query={"name": {"$regex":"^(A|B)"}},map=Code("function(){emit(this.continent, this.gdp)}"),reduce=Code("function(key, values){var total = 0;for (var i = 0; i < values.length; i++){total += values[i];}return total/values.length;}"),out={"inline":1});import operator;pp.pprint(list(temp["results"]))