---
layout: default
title: Aggregate Plugins
---
[DDL]: /mcollective/reference/plugins/ddl.html
[Examples]: https://github.com/puppetlabs/marionette-collective/tree/master/plugins/mcollective/aggregate

## Overview
MCollective Agents return data and we try to provide as much usable user
interface for free. To aid in this we require agents to have [DDL] files that
describe the data that the agent returns.

DDL files are used to configure the client but also to assist with user
interface generation.  They are used to ask questions that an action needs but
also to render the results when the replies come in.  For example we turn
*:freecpu* into "Free CPU" when displaying the data based on the DDL.

Previously if data that agents returned required any summarization this had to
be done using a custom application.  Here is an example from *mco nrpe*:

{% highlight console %}
% mco nrpe check_load
.
.
Finished processing 25 / 25 hosts in 556.48 ms

              OK: 25
         WARNING: 0
        CRITICAL: 0
         UNKNOWN: 0
{% endhighlight %}

Here to get the summary of results displayed in a way that has contextual
relevance to the nrpe plugin a custom application had to be written and anyone
who interacts with the agent using other RPC clients would not get the benefit
of this summary.

By using aggregate plugins and updating the DDL we can now provide such a
summary in all result sets and display it using the *mco rpc* application and
any calls to *printrpc*.

{% highlight console %}
% mco rpc nrpe runcommand command=check_load
Discovering hosts using the mongo method .... 25

 * [============================================================> ] 25 / 25

Summary of Exit Code:

            OK : 25
       WARNING : 0
       UNKNOWN : 0
      CRITICAL : 0


Finished processing 25 / 25 hosts in 390.70 ms
{% endhighlight %}

Here you get a similar summary as before, all that had to be done was a simple
aggregate plugin be written and distributed with your clients.

The results are shown as above using *printrpcstats* but you can also get access to
the raw data so you can decide to render it in some other way - perhaps using a
graph on a web interface.

We provide a number of aggregate plugins with MCollective and anyone can write
more.

For examples that already use functions see the *rpcutil* agent - its
*collective_info*, *get_fact*, *daemon_stats* and *get_config_item* actions all
have summaries applied.

*NOTE:* This feature is available since version 2.1.0

## Using existing plugins

### Updating the DDL
At present MCollective supplies 3 plugins *average()*, *summary()* and *sum()*
you can use these in any agent, here is an example from the *rpcutil* agent DDL
file:

{% highlight ruby %}
action "get_config_item", :description => "Get the active value of a specific config property" do
    output :value,
           :description => "The value that is in use",
           :display_as => "Value"

    summarize do
        aggregate summary(:value)
    end
end
{% endhighlight %}

We've removed a few lines from this example DDL block leaving only the relevant
lines.  You can see the agent outputs data called *:value* and we reference that
output in the summary function *summary(:value)*, the result would look like
this:

### Viewing summaries on the CLI
{% highlight console %}
% mco rpc rpcutil get_config_item item=collectives
.
.
dev8
   Property: collectives
      Value: ["mcollective", "uk_collective"]

Summary of Value:

      mcollective = 25
    uk_collective = 15
    fr_collective = 9
    us_collective = 1

Finished processing 25 / 25 hosts in 349.70 ms
{% endhighlight %}

You can see that the value in this case contains arrays, the *summary()*
function produce the table in the output showing the data distribution.

### Producing summaries in your own clients
You can enable the same display in your own code, here is ruby code that has the
same affect as the CLI call above:

{% highlight ruby %}
require 'mcollective'

include MCollective::RPC

c = rpcclient("rpcutil")

printrpc c.get_config_item(:item => "collectives")

printrpcstats :summarize => true
{% endhighlight %}

Without passing in the *:summarize => true* you would not see the summaries

### Getting access to the raw summary results
If you wanted to do something else entirely like produce a graph on a web page
of the summaries you can get access to the raw data, here's some ruby code to
show all computed summaries:

{% highlight ruby %}
require 'mcollective'

include MCollective::RPC

c = rpcclient("rpcutil")
c.progress = false

c.get_config_item(:item => "collectives")

c.stats.aggregate_summary.each do |summary|
  puts "Summary of type: %s" % summary.result_type
  puts "Display format: '%s'" % summary.aggregate_format
  puts
  pp summary.result
end
{% endhighlight %}

As you can see you will get an array of summaries this is because each DDL can
use many aggregate calls, this would be an array of all the computed summaries:

{% highlight console %}
Summary of type: collection
Display format: '%13s = %s'

{:type=>:collection,
 :value=>
  {"mcollective"=>25,
   "fr_collective"=>9,
   "us_collective"=>1,
   "uk_collective"=>15},
 :output=>:value}
{% endhighlight %}

There are 2 types of result *:collection* and *:numeric*, in the case of numeric
results the :value would just be a number.

The *aggregate_format* is either a user supplied format or a dynamically
computed format to display the summary results on the console.  In this case
each pair of the hash should be displayed using the format to produce a nice
right justified list of keys and values.

## Writing your own function
We'll cover writing your own function by looking at the Nagios one from earlier
in this example.  You can look at [the functions supplied with
MCollective][Examples] for more examples using other types than the one below.

First lets look at the DDL for the existing *nrpe* Agent:

{% highlight ruby %}
action "runcommand", :description => "Run a NRPE command" do
    input :command,
          :prompt      => "Command",
          :description => "NRPE command to run",
          :type        => :string,
          :validation  => '\A[a-zA-Z0-9_-]+\z',
          :optional    => false,
          :maxlength   => 50

    output :output,
           :description => "Output from the Nagios plugin",
           :display_as  => "Output",
           :default     => ""

    output :exitcode,
           :description  => "Exit Code from the Nagios plugin",
           :display_as   => "Exit Code",
           :default      => 3

    output :perfdata,
           :description  => "Performance Data from the Nagios plugin",
           :display_as   => "Performance Data",
           :default      => ""
end
{% endhighlight %}

You can see it will return an *:exitcode* item and from the default value you
can gather this is going to be a number.  Nagios defines 4 possibly exit codes
for a Nagios plugin and we need to convert this *:exitcode* into a string like
WARNING, CRITICAL, UNKNOWN or OK.

Usually when writing any kind of summarizer for an array of results your code
might contain 3 phases.

Given a series of Nagios results like this:

{% highlight ruby %}
[
 {:exitcode => 0, :output => "OK", :perfdata => ""},
 {:exitcode => 2, :output => "failure", :perfdata => ""}
]
{% endhighlight %}

You would write a nagios_states() function that does roughly this:

{% highlight ruby %}
def nagios_states(results)
  # set initial values
  result = {}
  status_map = ["OK", "WARNING", "CRITICAL", "UNKNOWN"]
  status_map.each {|s| result[s] = 0}

  # loop over all the data, increment the count for OK etc
  results.each do |result|
    status = status_map[result[:exitcode]]
    result[status] += 1
  end

  # return the result hash, {"OK" => 1, "CRITICAL" => 1, "WARN" => 0, "UNKNOWN" => 0}
  return result
end
{% endhighlight %}

You could optimise the code but you can see there are 3 major stages in the life
of this code.

 * Set initial values for the return data
 * Loop the data building up the state
 * Return the data.

Given this, here is our Nagios exitcode summary function, it is roughly the same
code with a bit more boiler plate to plugin into mcollective, but the same code
can be seen:

{% highlight ruby %}
module MCollective
  class Aggregate
    class Nagios_states<Base
      # Before function is run processing
      def startup_hook
        # :collection or :numeric
        @result[:type] = :collection

        # set default aggregate_format if it is undefined
        @aggregate_format = "%10s : %s" unless @aggregate_format

        @result[:value] = {}

        @status_map = ["OK", "WARNING", "CRITICAL", "UNKNOWN"]
        @status_map.each {|s| @result[:value][s] = 0}

      end

      # Determines the average of a set of numerical values
      def process_result(value, reply)
        if value
          status = @status_map[value]
          @result[:value][status] += 1
        else
          @result["UNKNOWN"] += 1
        end
      end

      # Post processing hook that returns the summary result
      def summarize
        result_class(@result[:type]).new(@result, @aggregate_format, @action)
      end
    end
  end
end
{% endhighlight %}

This shows that an aggregate function has the same 3 basic parts.  First we set
the initial state using the *startup_hook*.  We then process each result as it
comes in from the network using *process_result*.  Finally we turn that into a
the result objects that you saw earlier in the ruby client examples using the
*summarize* method.

### *startup_hook*
Each function needs a startup hook, without one you'll get exceptions.  The
startup hook lets you set up the initial state.

The first thing to do is set the type of result this will be.  Currently we
support 2 types of result either a plain number indicated using *:numeric* or a
complex *:collection* type that can be a hash with keys and values.

Functions can take display formats in the DDL, in this example we set
*@aggregate_format* to a *printf* default that would display a table of results
but we still let the user supply his own format.

We then just initialize the result hash to and build a map from the English
representation of the Nagios status codes.

### *process_result*
Every reply that comes in from the network gets passed into your
*process_result* method.  The first argument will be just the single value the
DDL indicates you are interested in but you'll also get the whole rely so you
can get access to other reply values and such.

This gets called each time, we just look at the value and increment each Nagios
status or treat it as an unknown - in case the result data is missformed.

### *summarize*
The summarize method lets you take the state you built up and convert that into
an answer.  The summarize method is optional what you see here is the default
action if you do not supply one.

The *result_class* method accepts either *:collection* or *:numeric* as
arguments and it is basically a factory for the correct result structure.

### Deploy and update the DDL
You should deploy this function into your *libdir/aggregate* directory called
*nagios_states.rb* on the client machines - no harm deploying it everywhere
though.

Update the DDL so it looks like:

{% highlight ruby %}
action "runcommand", :description => "Run a NRPE command" do
    .
    .
    .

    if respond_to?(:summarize)
        summarize do
            aggregate nagios_states(:exitcode)
        end
    end
end
{% endhighlight %}

Add the last few lines - we check that we're running in a version of MCollective
that supports this feature and then we call our function with the *:exitcode*
results.