Difference between revisions of "MAPREDUCE Elite"

Revision as of 16:48, 29 July 2015

#ENCODING
import io
import sys
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-16')
#MONGO
from pymongo import MongoClient
client = MongoClient()
client.elite.authenticate('scott','tiger')
db = client['elite']
#PRETTY
import pprint
pp = pprint.PrettyPrinter(indent=4)
#JS
from bson.code import Code

Introducing the elite database **WORK IN PROGRESS

These questions will introduce the "elite" database, which contains data about the video game Elite Dangerous
from bson.code import Code has been added to the setup so that you no longer need to include it in your answer.

There are two collections, commodities and systems. Inside systems there is are nested documents called stations
A system has many stations, and a station has many trade listings

Keys used in this database.

    commodities: 
        _id, average_price, category, name
    systems: 
        _id, allegiance, faction, government, name, population, primary_economy, security, state, stations, updated_at, x, y, z

    systems.stations: 
        allegiance, distance_to_star, economies, export_commodities,has_blackmarket, has_commodities, has_rearm, has_repair,
        has_shipyard, has_outfitting, faction, government, listings, max_landing_pad, name, state, type, updated_at

    systems.stations.listings: 
        buy_price, collected_at, demand, commodity, sell_price, supply, update_count

Read more about the structure here: Elite Document Structure

Questions

The commodities collection contains the name and average_price of each commodity.

There are 99 unique commodities and 15 categories.

Find the average price of each category, round to the nearest whole number

pp.pprint(
    db.commodities.find_one()
)

from bson.code import Code;temp = db.commodities.map_reduce( map=Code("function(){emit(this.category,this.average_price)}"), reduce=Code("""function(key,values){var total = 0;for (var i = 0; i < values.length; i++){total += values[i];}return Math.round(total/values.length);} """),out={"inline":1} );pp.pprint(temp['results'])

Each system has an allegiance. There are three main factions: The Federation, The Empire, and The Alliance.

The Alliance consists of independent systems, though an independent system is not necessarily part of The Alliance. In such a case their allegiance will be stored as "Independent"
There are a few systems that come under Anarchy. Non-populated systems without stations do not have an allegiance, and should be ignored.

Show the amount of systems following each type of allegiance.

temp = db.systems.map_reduce( query={"allegiance": {"$ne":None}}, map=Code("function(){emit(this.allegiance, 1)}"), reduce=Code("""function(key,values){var total = 0;values.forEach(function(value){total += value;});return total;} """), out={"inline":1} );pp.pprint(temp['results'])

What are the populations of the three main factions?

temp = db.systems.map_reduce(query={"allegiance":{"$in":["Alliance","Empire","Federation"]}}, map=Code("function(){emit(this.allegiance,this.population)}"), reduce=Code("""function(key,values){var total = 0;values.forEach(function(value){total += value;});return total;} """), out={"inline":1} );pp.pprint(temp['results'])

Harder Questions

How much Hydrogen Fuel is owned by stations in systems that are allied with the three main factions? Limit your query to the first 5000 stations.

The amount of stations and the amount of listings aren't fixed, you'll need to query to ensure that they exist and find a way of iterating through them in your map stage.

temp = db.systems.map_reduce( limit=5000, query={"allegiance":{"$in":["Alliance","Empire","Federation"]},"stations.listings.commodity":"Hydrogen Fuel","stations.listings.supply":{"$exists":1}}, map=Code("""function(){ for(var i in this.stations) for(var j in this.stations[i].listings) emit(this.allegiance,this.stations[i].listings[j].supply) }"""), reduce=Code("""function(k,vs){var t = 0;vs.forEach(function(v){t += v});return t;} """), out={"inline":1} ); pp.pprint(temp['results']);

A power_control_faction or Power is an individual or organisation who is in control of a system.

These powers have allegiances, but the systems they control do not nescessarily have the same allegiance as they do.

At the time of writing Zemina Torval is allied with the Empire and controls 47 systems.

    {   '_id': 'Zemina Torval',
        'value': {   'alliance': 0.0,
                     'anarchy': 0.0,
                     'empire': 39.0,
                     'federation': 3.0,
                     'independent': 5.0}}]

Show the allegiance of each of the power's systems

temp = db.systems.map_reduce(query={"power_control_faction":{"$exists":1}},map=Code("""function(){switch(this.allegiance){case "Alliance":emit(this.power_control_faction,{alliance:1,anarchy:0,empire:0,federation:0,independent:0});break;case "Anarchy":emit(this.power_control_faction,{alliance:0,anarchy:1,empire:0,federation:0,independent:0});break; case "Empire":emit(this.power_control_faction,{alliance:0,anarchy:0,empire:1,federation:0,independent:0});break;case "Federation":emit(this.power_control_faction,{alliance:0,anarchy:0,empire:0,federation:1,independent:0});break;case "Independent":emit(this.power_control_faction,{alliance:0,anarchy:0,empire:0,federation:0,independent:1});break;}}"""),reduce=Code("""function(k,vs){var a=vs[0];for(var i=1;i<vs.length;i++){var b=vs[i];a.alliance+=b.alliance;a.anarchy+=b.anarchy;a.empire+=b.empire;a.federation+=b.federation;a.independent+=b.independent;}return a}"""),out={"inline":1}); pp.pprint(temp['results']);

Our dataset doesn't contain the allegiance of a power, following on from the previous question:

Using python, guess the Power's allegiance based on the amount of systems they control.

Zemina Torval: Empire(39.0)

Show the allegiance of each of the power's systems

temp = db.systems.map_reduce(query={"power_control_faction":{"$exists":1}},map=Code("""function(){switch(this.allegiance){case "Alliance":emit(this.power_control_faction,{alliance:1,anarchy:0,empire:0,federation:0,independent:0});break;case "Anarchy":emit(this.power_control_faction,{alliance:0,anarchy:1,empire:0,federation:0,independent:0});break; case "Empire":emit(this.power_control_faction,{alliance:0,anarchy:0,empire:1,federation:0,independent:0});break;case "Federation":emit(this.power_control_faction,{alliance:0,anarchy:0,empire:0,federation:1,independent:0});break;case "Independent":emit(this.power_control_faction,{alliance:0,anarchy:0,empire:0,federation:0,independent:1});break;}}"""),reduce=Code("""function(k,vs){var a=vs[0];for(var i=1;i<vs.length;i++){var b=vs[i];a.alliance+=b.alliance;a.anarchy+=b.anarchy;a.empire+=b.empire;a.federation+=b.federation;a.independent+=b.independent;}return a}"""),out={"inline":1}); pp.pprint(temp['results']); for power in temp['results']:

   max = 0;
   key = "";
   #print(power['value'])
   for id in power['value']:
       if (power['value'][id] > max): 
           max = power['value'][id]
           key = id
   print(power['_id']+": "+key+"("+str(max)+")")

@@ Line 73: / Line 73: @@
 ==Harder Questions==
+<div class=q data-lang="py3">How much Hydrogen Fuel is owned by stations in systems that are allied with the three main factions? Limit your query to the first 5000 stations.
+<div class="hint" title="hint">
+The amount of stations <b>and</b> the amount of listings aren't fixed, you'll need to <code>query</code> to ensure that they exist and find a way of iterating through them in your <code>map</code> stage.
+</div>
+<pre class=def>
+</pre>
+<div class="ans">
+temp = db.systems.map_reduce( limit=5000, query={"allegiance":{"$in":["Alliance","Empire","Federation"]},"stations.listings.commodity":"Hydrogen Fuel","stations.listings.supply":{"$exists":1}}, map=Code("""function(){ for(var i in this.stations) for(var j in this.stations[i].listings) emit(this.allegiance,this.stations[i].listings[j].supply) }"""), reduce=Code("""function(k,vs){var t = 0;vs.forEach(function(v){t += v});return t;} """), out={"inline":1} );
+pp.pprint(temp['results']);
+</div>
+</div>
 <div class=q data-lang="py3">A <code>power_control_faction</code> or <b>Power</b> is an individual or organisation who is in control of a system.<br/>
 These powers have allegiances, but the systems they control do not nescessarily have the same allegiance as they do.
@@ Line 94: / Line 106: @@
 </div>
-<div class=q data-lang="py3">How much Hydrogen Fuel is owned by stations in systems that are allied with the three main factions? Limit your query to the first 5000 stations.
+<div class=q data-lang="py3">Our dataset doesn't contain the allegiance of a power, following on from the previous question:
-<div class="hint" title="hint">
+<p class="strong">Using python, guess the Power's allegiance based on the amount of systems they control.
-The amount of stations <b>and</b> the amount of listings aren't fixed, you'll need to <code>query</code> to ensure that they exist and find a way of iterating through them in your <code>map</code> stage.
+<div class="hint" title="Example"><code>Zemina Torval: Empire(39.0)</code></div>
-</div>
+<p class="strong">Show the allegiance of each of the power's systems</p>
 <pre class=def>
 </pre>
 <div class="ans">
-temp = db.systems.map_reduce( limit=5000, query={"allegiance":{"$in":["Alliance","Empire","Federation"]},"stations.listings.commodity":"Hydrogen Fuel","stations.listings.supply":{"$exists":1}}, map=Code("""function(){ for(var i in this.stations) for(var j in this.stations[i].listings) emit(this.allegiance,this.stations[i].listings[j].supply) }"""), reduce=Code("""function(k,vs){var t = 0;vs.forEach(function(v){t += v});return t;} """), out={"inline":1} );
+temp = db.systems.map_reduce(query={"power_control_faction":{"$exists":1}},map=Code("""function(){switch(this.allegiance){case "Alliance":emit(this.power_control_faction,{alliance:1,anarchy:0,empire:0,federation:0,independent:0});break;case "Anarchy":emit(this.power_control_faction,{alliance:0,anarchy:1,empire:0,federation:0,independent:0});break; case "Empire":emit(this.power_control_faction,{alliance:0,anarchy:0,empire:1,federation:0,independent:0});break;case "Federation":emit(this.power_control_faction,{alliance:0,anarchy:0,empire:0,federation:1,independent:0});break;case "Independent":emit(this.power_control_faction,{alliance:0,anarchy:0,empire:0,federation:0,independent:1});break;}}"""),reduce=Code("""function(k,vs){var a=vs[0];for(var i=1;i<vs.length;i++){var b=vs[i];a.alliance+=b.alliance;a.anarchy+=b.anarchy;a.empire+=b.empire;a.federation+=b.federation;a.independent+=b.independent;}return a}"""),out={"inline":1});
 pp.pprint(temp['results']);
+for power in temp['results']:
+    max = 0;
+    key = "";
+    #print(power['value'])
+    for id in power['value']:
+        if (power['value'][id] > max):
+            max = power['value'][id]
+            key = id
+    print(power['_id']+": "+key+"("+str(max)+")")
 </div>
 </div>