Cookies help us deliver our services. By using our services, you agree to our use of cookies. More information

Difference between revisions of "AGGREGATE examples"

From NoSQLZoo
Jump to: navigation, search
m
 
(13 intermediate revisions by 2 users not shown)
Line 1: Line 1:
==Introducing the aggregation framework==
+
==aggregate==
These examples introduce the aggregation framework and its operators. Again we will be using the collection <code>world</code>
+
The aggregate function takes a list of operations - this is the pipeline.
 +
 
 +
The data passes through each stage of the pipeline in turn.
 +
 
 +
Pipeline stages can include:
 +
* '''$group''' This is the aggregate special sauce; you specify the _id value, the output from this stage includes one entry for each distinct _id value. In SQL you would use a GROUP BY clause
 +
* '''$match''' - this acts as a filter, some data items pass through this stage, some do not. Similar to the WHERE clause in SQL
 +
* '''$project''' - this can be used to transform each element. Rather like the values on the SELECT line of an SQL query
 +
* '''$limit'''
 +
* '''$sort'''
 +
* '''$skip'''
 
<div class='extra_space' style='width:1em; height:6em;'></div>
 
<div class='extra_space' style='width:1em; height:6em;'></div>
 +
==$group==
 +
<div class="q" data-lang="mongo">
 +
'''$group''' allows you to collect group items that share common features
 +
* _id - this determines the values to be grouped
 +
* $continent and $population refers to keys 'continent' and 'population' available in each item
 +
* $sum is an aggregating function, it takes many values in and returns a single value. Other examples of aggregating functions are $min $max, $avg
 +
<p class="strong">List the continents</p>
 +
<pre class="def"><nowiki>
 +
db.world.aggregate(
 +
    {$group: {_id: "$continent"}, pop:{$sum:'$population'}}
 +
);</nowiki></pre>
 +
</div>
  
 +
==$match==
 
<div class="q" data-lang="mongo">
 
<div class="q" data-lang="mongo">
<code>$match</code> performs queries in a similar way to <code>find()</code>
+
'''$match''' performs queries in a similar way to <syntaxhighlight lang="JavaScript" inline>find()</syntaxhighlight>
<p class=strong>Show all the details for France</p>
+
<p class="strong">Show all the details for France</p>
 
<pre class="def"><nowiki>
 
<pre class="def"><nowiki>
 
db.world.aggregate([
 
db.world.aggregate([
 
     {$match: {name: "France"}}
 
     {$match: {name: "France"}}
 
]);</nowiki></pre>
 
]);</nowiki></pre>
<div class="ans"><nowiki>db.world.aggregate([{$match:{name:"France"}}]);</nowiki></div>
+
<pre class="ans"><nowiki>db.world.aggregate([{$match:{name:"France"}}]);</nowiki></pre>
 
</div>
 
</div>
  
<div class="q" data-lang="mongo"><code>limit</code> sets the amount of documents to be handed to the next stage in the pipeline.
+
==$limit==
<p class=strong>Return the first document</p>
+
<div class="q" data-lang="mongo">'''$limit''' sets the amount of documents to be handed to the next stage in the pipeline.
 +
<p class="strong">Return the first two document</p>
 
<pre class="def"><nowiki>
 
<pre class="def"><nowiki>
 
db.world.aggregate([
 
db.world.aggregate([
     {$limit: 1}
+
     {$limit: 2}
 
]);</nowiki></pre>
 
]);</nowiki></pre>
<div class="ans"><nowiki>db.world.aggregate([{"$limit":1}]);</nowiki></div>
+
<pre class="ans"><nowiki>db.world.aggregate([{"$limit":2}]);</nowiki></pre>
 
</div>
 
</div>
  
 +
==$project==
 
<div class="q" data-lang="mongo">
 
<div class="q" data-lang="mongo">
<code>$project</code> selects what fields to display.<br/>
+
'''$project''' selects what fields to display.<br/>
It can also has the ability to create new fields and to compare fields against each other without using <code>$where</code>
+
It can also has the ability to create new fields and to compare fields against each other without using '''$where'''
<p class=strong>Show the name and population density of all Asian countries. (population/area)</p>
+
<p class="strong">Show the name and population density of all Asian countries. (population/area)</p>
Note that "density" is a new field, made from the result of dividing two existing fields, and that <code>$divide</code> is an aggregate function.
+
Note that "density" is a new field made from the result of dividing two existing fields, and that '''$divide''' is an aggregate function.
<div class=hint title="Dealing with division by 0">
+
<div class="hint" title="Dealing with division by 0">
To avoid diving by 0 insert a <code>$match</code> to remove any countries with 0 area (Vatican City), then pipe these results through to <code>$project</code><br/>
+
To avoid diving by 0 insert a '''$match''' to remove any countries with no area (Vatican City), then pipe these results through to '''$project'''<br/>
There is no need to check if values are <code>null</code>, MongoDB will ignore these documents.
+
There is no need to check if values are <syntaxhighlight lang="JavaScript" inline>null</syntaxhighlight>, MongoDB will ignore these documents.
 
</div>
 
</div>
 
<pre class="def"><nowiki>
 
<pre class="def"><nowiki>
Line 39: Line 64:
 
         density: {$divide: ["$population", "$area"]}
 
         density: {$divide: ["$population", "$area"]}
 
     }}
 
     }}
])</nowiki></pre>
+
]);</nowiki></pre>
 
<pre class="ans"><nowiki>db.world.aggregate([{"$match":{"area":{"$ne":0},"continent":"Asia"}},{"$project":{"_id":0,"name":1,"density":{"$divide":["$population","$area"]}}}]);</nowiki></pre>
 
<pre class="ans"><nowiki>db.world.aggregate([{"$match":{"area":{"$ne":0},"continent":"Asia"}},{"$project":{"_id":0,"name":1,"density":{"$divide":["$population","$area"]}}}]);</nowiki></pre>
 
</div>
 
</div>
  
 +
==aggregate composition==
 
<div class="q" data-lang="mongo">
 
<div class="q" data-lang="mongo">
Because aggregate is a pipeline stages may be repeated, and stages don't have to be used in a specific order.
+
You can have several pipeline stages, the data flows through each one in turn.
<p class=strong>Show the name of Asian countries with a density that's over 500 people per km<sup>2</sup>. (population/area)</p>
+
<p class="strong">Show the name of Asian countries with a density that's over 500 people per km<sup>2</sup>. (population/area)</p>
 
<pre class="def"><nowiki>
 
<pre class="def"><nowiki>
 
db.world.aggregate([
 
db.world.aggregate([
 
   {$match: {area: {$ne: 0}, continent: "Asia"}},
 
   {$match: {area: {$ne: 0}, continent: "Asia"}},
   {$project:{
+
   {$project: {
 
       _id: 0,
 
       _id: 0,
 
       name: 1,
 
       name: 1,
Line 59: Line 85:
 
</div>
 
</div>
  
 +
==$sort==
 
<div class="q" data-lang="mongo">
 
<div class="q" data-lang="mongo">
<code>$sort</code> allows ordering of the results set, where 1 is ascending and -1 is descending.<br/>
+
'''$sort''' allows ordering of the results set, where 1 is ascending and -1 is descending.<br/>
Note that not including <code>$match</code> is the same as <code>{"$match":{}}</code>
+
Note that not including '''$match''' is the same as <syntaxhighlight lang="JavaScript" inline>{"$match":{}}</syntaxhighlight>
<p class=strong>Show the name of all countries in descending order.</p>
+
<p class="strong">Show the name of all countries in descending order.</p>
 
<pre class="def"><nowiki>
 
<pre class="def"><nowiki>
 
db.world.aggregate([
 
db.world.aggregate([
Line 72: Line 99:
 
         "name":-1
 
         "name":-1
 
     }}   
 
     }}   
])</nowiki></pre>
+
]);</nowiki></pre>
<div class="ans"><nowiki>db.world.aggregate([{"$project":{"_id":0,"name":1,}},{"$sort":{"name":-1}}])</nowiki></div>
+
<pre class="ans"><nowiki>db.world.aggregate([{"$project":{"_id":0,"name":1,}},{"$sort":{"name":-1}}])</nowiki></pre>
 
</div>
 
</div>
  
 
==Grouping==
 
==Grouping==
Grouping provides accumulator operations such as <code>$sum</code><br />
+
Grouping provides accumulator operations such as '''$sum'''<br />
All groups must have an <code>_id</code>. To see why this is useful imagine the following:<br/><br/>
+
All groups must have an '''_id'''. To see why this is useful imagine the following:<br/><br/>
  
So far you've been using the collection <code>world</code><br/>As every country has a continent, it would make sense to have countries as a nested document inside continents: e.g:
+
So far you've been using the '''world''' collection</code><br/>
+
As every country has a continent, it would make sense to have countries as a nested document inside continents: e.g:
<pre>
+
<syntaxhighlight lang="JavaScript">
 
[     
 
[     
     {"name":"Africa",
+
     {"name": "Africa",
     "countries":[
+
     "countries": [
         {"name":"Algeria",   "capital":"Algiers",   ...},
+
         {"name": "Algeria",     "capital": "Algiers",   ...},
         {"name":"Angola",     "capital":"Luanda",   ...},
+
         {"name": "Angola",     "capital": "Luanda",     ...},
         {"name":"Benin",     "capital":"Porto-Novo",...}.
+
         {"name": "Benin",       "capital": "Porto-Novo", ...}.
 
         {...},
 
         {...},
 
         ...
 
         ...
 
     ]},
 
     ]},
     {"name":"Asia",
+
     {"name": "Asia",
     "countries":[
+
     "countries": [
         {"name":"Afghanistan","capital":"Kabul", ...},
+
         {"name": "Afghanistan", "capital": "Kabul", ...},
         {"name":"Azerbaijan", "capital":"Baku", ...},
+
         {"name": "Azerbaijan", "capital": "Baku",   ...},
         {"name":"Bahrain",   "capital":"Manama",...},
+
         {"name": "Bahrain",     "capital": "Manama", ...},
 
         {...},
 
         {...},
 
         ...
 
         ...
Line 103: Line 130:
 
     ...
 
     ...
 
]
 
]
</pre>
+
</syntaxhighlight>
  
The <code>world</code> collection isn't like this however. It uses the following structure, which has a redundancy where <code>continent</code> is repeated for each country.
+
The '''world''' collection isn't like this however. It uses the following structure, which has a redundancy where '''continent''' is repeated for each country.
  
<pre>
+
<syntaxhighlight lang="JavaScript">
 
[  
 
[  
     {"name":"Afghanistan","capital":"Kabul", "continent":"Asia", ...},
+
     {"name": "Afghanistan", "capital": "Kabul",   "continent": "Asia",   ...},
     {"name":"Albania",   "capital":"Tirana", "continent":"Europe, ...},
+
     {"name": "Albania",     "capital": "Tirana", "continent": "Europe", ...},
     {"name":"Algeria",   "capital":"Algiers","contiennt":"Africa",...},
+
     {"name": "Algeria",     "capital": "Algiers", "contiennt": "Africa", ...},
 
     {...},
 
     {...},
 
     ...
 
     ...
 
]
 
]
</pre>
+
</syntaxhighlight>
The code to group by continent is <code> "_id":"$continent"</code><br/>  
+
The code to group by continent is <syntaxhighlight lang="JavaScript" inline>"_id":"$continent"</syntaxhighlight><br/>  
If instead the question was to group by country the code would be <code>"_id":"$name"</code>.<br/>
+
If instead the question was to group by country the code would be <syntaxhighlight lang="JavaScript" inline>"_id": "$name"</syntaxhighlight>.<br/>
To operate over the whole document (which would have the same effect as <code>"_id":"$name"</code>) <code>"_id":"null"</code> or <code>"_id":None</code> can be used.
+
To operate over the whole document (which would have the same effect as <syntaxhighlight lang="JavaScript" inline>"_id": "$name"</syntaxhighlight>) <syntaxhighlight lang="JavaScript" inline>"_id": "null"</syntaxhighlight> or <syntaxhighlight lang="JavaScript" inline>"_id": None</syntaxhighlight> can be used.
 
==group operators==
 
==group operators==
 
<div class="q" data-lang="mongo">  
 
<div class="q" data-lang="mongo">  
<code>$max</code> and <code>$min</code> can be used to get the largest and smallest values in a group.
+
'''$max''' and '''$min''' can be used to get the largest and smallest values in a group.
 
<p class="strong">Get the smallest and largest GDPs of each continent.</p>
 
<p class="strong">Get the smallest and largest GDPs of each continent.</p>
<pre class="def">
+
<pre class="def"><nowiki>
 
db.world.aggregate([
 
db.world.aggregate([
 
     {$group: {
 
     {$group: {
Line 136: Line 163:
 
     }}
 
     }}
 
]);</nowiki></pre>
 
]);</nowiki></pre>
<div class="ans"><nowiki>db.world.aggregate([{"$group":{'_id':'$continent','min':{"$min":"$gdp"},'max':{"$max":"$gdp"}}},{"$project":{"_id":1,"min":1,"max":1}}]);</nowiki></div>
+
<pre class="ans"><nowiki>db.world.aggregate([{"$group":{'_id':'$continent','min':{"$min":"$gdp"},'max':{"$max":"$gdp"}}},{"$project":{"_id":1,"min":1,"max":1}}]);</nowiki></pre>
 
</div>
 
</div>
  
 
<div class="q" data-lang="mongo">
 
<div class="q" data-lang="mongo">
Some other useful aggregate functions to know are <code>$sum</code> and average: <code>$avg</code><br/>
+
Some other useful aggregate functions to know are '''$sum''' and average: '''$avg'''<br/>
This example combines all the material in these examples.  
+
The example below  combines previous example material.
<p class=strong>Order the continents in descending order by total GDP, Include the average GDP for each country.</p>
+
<p class="strong">Order the continents in descending order by total GDP, Include the average GDP for each country.</p>
<pre class="def">
+
<pre class="def"><nowiki>
 
db.world.aggregate([
 
db.world.aggregate([
 
     {$match: {}},
 
     {$match: {}},
Line 161: Line 188:
 
     }}
 
     }}
 
]);</nowiki></pre>
 
]);</nowiki></pre>
<div class="ans"><nowiki>db.world.aggregate([{"$group":{"_id":"$continent","Total GDP":{"$sum":"$gdp"},"Average GDP":{"$avg":"$gdp"}}},{"$sort":{"Total GDP":-1}},{"$project":{"Area":"$_id","Total GDP":1,"Average GDP":1,"_id":0}}]);</nowiki></div>
+
<pre class="ans"><nowiki>db.world.aggregate([{"$group":{"_id":"$continent","Total GDP":{"$sum":"$gdp"},"Average GDP":{"$avg":"$gdp"}}},{"$sort":{"Total GDP":-1}},{"$project":{"Area":"$_id","Total GDP":1,"Average GDP":1,"_id":0}}]);</nowiki></pre>
 
</div>
 
</div>
  
 
<div class="q" data-lang="mongo">
 
<div class="q" data-lang="mongo">
 
Using Conditions<br/><br/>
 
Using Conditions<br/><br/>
<code>$cond</code> is similar to a <code>CASE</code> statement in other languages.<br/>
+
'''$cond''' is similar to a '''CASE''' statement in other languages.<br/>
It has the form <code>"$cond": [{<comparison> :[<field or value>,<field or value>]},<true case>,<false case>]</code><br/><br/>
+
It has the form <syntaxhighlight lang="JavaScript" inline>"$cond": [{<comparison>: [<field or value>, <field or value>]}, <true case>, <false case>]</syntaxhighlight><br/><br/>
 
<pre class="def"><nowiki>
 
<pre class="def"><nowiki>
 
db.world.aggregate([
 
db.world.aggregate([
Line 183: Line 210:
 
         area: 1
 
         area: 1
 
     }}
 
     }}
]);
+
]);</nowiki></pre>
</nowiki></pre>
+
</div>

Latest revision as of 20:22, 1 April 2022

aggregate

The aggregate function takes a list of operations - this is the pipeline.

The data passes through each stage of the pipeline in turn.

Pipeline stages can include:

  • $group This is the aggregate special sauce; you specify the _id value, the output from this stage includes one entry for each distinct _id value. In SQL you would use a GROUP BY clause
  • $match - this acts as a filter, some data items pass through this stage, some do not. Similar to the WHERE clause in SQL
  • $project - this can be used to transform each element. Rather like the values on the SELECT line of an SQL query
  • $limit
  • $sort
  • $skip

$group

$group allows you to collect group items that share common features

  • _id - this determines the values to be grouped
  • $continent and $population refers to keys 'continent' and 'population' available in each item
  • $sum is an aggregating function, it takes many values in and returns a single value. Other examples of aggregating functions are $min $max, $avg

List the continents

db.world.aggregate(
    {$group: {_id: "$continent"}, pop:{$sum:'$population'}}
);

$match

$match performs queries in a similar way to find()

Show all the details for France

db.world.aggregate([
    {$match: {name: "France"}}
]);
db.world.aggregate([{$match:{name:"France"}}]);

$limit

$limit sets the amount of documents to be handed to the next stage in the pipeline.

Return the first two document

db.world.aggregate([
    {$limit: 2}
]);
db.world.aggregate([{"$limit":2}]);

$project

$project selects what fields to display.
It can also has the ability to create new fields and to compare fields against each other without using $where

Show the name and population density of all Asian countries. (population/area)

Note that "density" is a new field made from the result of dividing two existing fields, and that $divide is an aggregate function.

To avoid diving by 0 insert a $match to remove any countries with no area (Vatican City), then pipe these results through to $project
There is no need to check if values are null, MongoDB will ignore these documents.

db.world.aggregate([
    {$match: {area: {$ne: 0}, continent: "Asia"}},
    {$project: {
        _id: 0,
        name: 1,
        density: {$divide: ["$population", "$area"]}
    }}
]);
db.world.aggregate([{"$match":{"area":{"$ne":0},"continent":"Asia"}},{"$project":{"_id":0,"name":1,"density":{"$divide":["$population","$area"]}}}]);

aggregate composition

You can have several pipeline stages, the data flows through each one in turn.

Show the name of Asian countries with a density that's over 500 people per km2. (population/area)

db.world.aggregate([
   {$match: {area: {$ne: 0}, continent: "Asia"}},
   {$project: {
       _id: 0,
       name: 1,
       density: {$divide: ["$population", "$area"]}
   }},
   {$match: {density: {$gt: 500}}}
]);
db.world.aggregate([{"$match":{"area":{"$ne":0},"continent":"Asia"}},{"$project":{"_id":0,"name":1,"density":{"$divide":["$population","$area"]}}},{"$match":{"density":{"$gt":500}}}]);

$sort

$sort allows ordering of the results set, where 1 is ascending and -1 is descending.
Note that not including $match is the same as {"$match":{}}

Show the name of all countries in descending order.

db.world.aggregate([
    {"$project":{
        "_id":0,
        "name":1,
    }},
    {"$sort":{
        "name":-1
    }}  
]);
db.world.aggregate([{"$project":{"_id":0,"name":1,}},{"$sort":{"name":-1}}])

Grouping

Grouping provides accumulator operations such as $sum
All groups must have an _id. To see why this is useful imagine the following:

So far you've been using the world collection
As every country has a continent, it would make sense to have countries as a nested document inside continents: e.g:

[    
    {"name": "Africa",
     "countries": [
         {"name": "Algeria",     "capital": "Algiers",    ...},
         {"name": "Angola",      "capital": "Luanda",     ...},
         {"name": "Benin",       "capital": "Porto-Novo", ...}.
         {...},
         ...
    ]},
    {"name": "Asia",
     "countries": [
         {"name": "Afghanistan", "capital": "Kabul",  ...},
         {"name": "Azerbaijan",  "capital": "Baku",   ...},
         {"name": "Bahrain",     "capital": "Manama", ...},
         {...},
         ...
    ]},
    {...},
    ...
]

The world collection isn't like this however. It uses the following structure, which has a redundancy where continent is repeated for each country.

[ 
    {"name": "Afghanistan", "capital": "Kabul",   "continent": "Asia",   ...},
    {"name": "Albania",     "capital": "Tirana",  "continent": "Europe",  ...},
    {"name": "Algeria",     "capital": "Algiers", "contiennt": "Africa", ...},
    {...},
    ...
]

The code to group by continent is "_id":"$continent"
If instead the question was to group by country the code would be "_id": "$name".
To operate over the whole document (which would have the same effect as "_id": "$name") "_id": "null" or "_id": None can be used.

group operators

$max and $min can be used to get the largest and smallest values in a group.

Get the smallest and largest GDPs of each continent.

db.world.aggregate([
    {$group: {
        _id: '$continent',
        min: {$min: "$gdp"},
        max: {$max: "$gdp"}
    }},
    {$project: {
        _id: 1,
        min: 1,
        max: 1
    }}
]);
db.world.aggregate([{"$group":{'_id':'$continent','min':{"$min":"$gdp"},'max':{"$max":"$gdp"}}},{"$project":{"_id":1,"min":1,"max":1}}]);

Some other useful aggregate functions to know are $sum and average: $avg
The example below combines previous example material.

Order the continents in descending order by total GDP, Include the average GDP for each country.

db.world.aggregate([
    {$match: {}},
    {$group: {
        _id:"$continent",
        "Total GDP": {"$sum": "$gdp"},
        "Average GDP": {"$avg": "$gdp"}
    }},
    {$sort: {
        "Total GDP":-1
    }},
    {$project:{
        "Area": "$_id",
        "Total GDP": 1,
        "Average GDP": 1,
        _id: 0
    }}
]);
db.world.aggregate([{"$group":{"_id":"$continent","Total GDP":{"$sum":"$gdp"},"Average GDP":{"$avg":"$gdp"}}},{"$sort":{"Total GDP":-1}},{"$project":{"Area":"$_id","Total GDP":1,"Average GDP":1,"_id":0}}]);

Using Conditions

$cond is similar to a CASE statement in other languages.
It has the form "$cond": [{<comparison>: [<field or value>, <field or value>]}, <true case>, <false case>]

db.world.aggregate([
    {$group: {
        _id: {
            $cond: [{"$eq": ["$continent", "Eurasia"]}, "Europe", "$continent"]
        },
        area: {$sum: "$area"}
    }},
    {$sort: {
        area: -1
    }},
    {$project: {
        _id: 1,
        area: 1
    }}
]);