3 title: Aggregate Plugins
5 [DDL]: /mcollective/reference/plugins/ddl.html
6 [Examples]: https://github.com/puppetlabs/marionette-collective/tree/master/plugins/mcollective/aggregate
9 MCollective Agents return data and we try to provide as much usable user
10 interface for free. To aid in this we require agents to have [DDL] files that
11 describe the data that the agent returns.
13 DDL files are used to configure the client but also to assist with user
14 interface generation. They are used to ask questions that an action needs but
15 also to render the results when the replies come in. For example we turn
16 *:freecpu* into "Free CPU" when displaying the data based on the DDL.
18 Previously if data that agents returned required any summarization this had to
19 be done using a custom application. Here is an example from *mco nrpe*:
21 {% highlight console %}
25 Finished processing 25 / 25 hosts in 556.48 ms
33 Here to get the summary of results displayed in a way that has contextual
34 relevance to the nrpe plugin a custom application had to be written and anyone
35 who interacts with the agent using other RPC clients would not get the benefit
38 By using aggregate plugins and updating the DDL we can now provide such a
39 summary in all result sets and display it using the *mco rpc* application and
40 any calls to *printrpc*.
42 {% highlight console %}
43 % mco rpc nrpe runcommand command=check_load
44 Discovering hosts using the mongo method .... 25
46 * [============================================================> ] 25 / 25
56 Finished processing 25 / 25 hosts in 390.70 ms
59 Here you get a similar summary as before, all that had to be done was a simple
60 aggregate plugin be written and distributed with your clients.
62 The results are shown as above using *printrpcstats* but you can also get access to
63 the raw data so you can decide to render it in some other way - perhaps using a
64 graph on a web interface.
66 We provide a number of aggregate plugins with MCollective and anyone can write
69 For examples that already use functions see the *rpcutil* agent - its
70 *collective_info*, *get_fact*, *daemon_stats* and *get_config_item* actions all
71 have summaries applied.
73 *NOTE:* This feature is available since version 2.1.0
75 ## Using existing plugins
78 At present MCollective supplies 3 plugins *average()*, *summary()* and *sum()*
79 you can use these in any agent, here is an example from the *rpcutil* agent DDL
83 action "get_config_item", :description => "Get the active value of a specific config property" do
85 :description => "The value that is in use",
86 :display_as => "Value"
89 aggregate summary(:value)
94 We've removed a few lines from this example DDL block leaving only the relevant
95 lines. You can see the agent outputs data called *:value* and we reference that
96 output in the summary function *summary(:value)*, the result would look like
99 ### Viewing summaries on the CLI
100 {% highlight console %}
101 % mco rpc rpcutil get_config_item item=collectives
105 Property: collectives
106 Value: ["mcollective", "uk_collective"]
115 Finished processing 25 / 25 hosts in 349.70 ms
118 You can see that the value in this case contains arrays, the *summary()*
119 function produce the table in the output showing the data distribution.
121 ### Producing summaries in your own clients
122 You can enable the same display in your own code, here is ruby code that has the
123 same affect as the CLI call above:
126 require 'mcollective'
128 include MCollective::RPC
130 c = rpcclient("rpcutil")
132 printrpc c.get_config_item(:item => "collectives")
134 printrpcstats :summarize => true
137 Without passing in the *:summarize => true* you would not see the summaries
139 ### Getting access to the raw summary results
140 If you wanted to do something else entirely like produce a graph on a web page
141 of the summaries you can get access to the raw data, here's some ruby code to
142 show all computed summaries:
145 require 'mcollective'
147 include MCollective::RPC
149 c = rpcclient("rpcutil")
152 c.get_config_item(:item => "collectives")
154 c.stats.aggregate_summary.each do |summary|
155 puts "Summary of type: %s" % summary.result_type
156 puts "Display format: '%s'" % summary.aggregate_format
162 As you can see you will get an array of summaries this is because each DDL can
163 use many aggregate calls, this would be an array of all the computed summaries:
165 {% highlight console %}
166 Summary of type: collection
167 Display format: '%13s = %s'
174 "uk_collective"=>15},
178 There are 2 types of result *:collection* and *:numeric*, in the case of numeric
179 results the :value would just be a number.
181 The *aggregate_format* is either a user supplied format or a dynamically
182 computed format to display the summary results on the console. In this case
183 each pair of the hash should be displayed using the format to produce a nice
184 right justified list of keys and values.
186 ## Writing your own function
187 We'll cover writing your own function by looking at the Nagios one from earlier
188 in this example. You can look at [the functions supplied with
189 MCollective][Examples] for more examples using other types than the one below.
191 First lets look at the DDL for the existing *nrpe* Agent:
194 action "runcommand", :description => "Run a NRPE command" do
196 :prompt => "Command",
197 :description => "NRPE command to run",
199 :validation => '\A[a-zA-Z0-9_-]+\z',
204 :description => "Output from the Nagios plugin",
205 :display_as => "Output",
209 :description => "Exit Code from the Nagios plugin",
210 :display_as => "Exit Code",
214 :description => "Performance Data from the Nagios plugin",
215 :display_as => "Performance Data",
220 You can see it will return an *:exitcode* item and from the default value you
221 can gather this is going to be a number. Nagios defines 4 possibly exit codes
222 for a Nagios plugin and we need to convert this *:exitcode* into a string like
223 WARNING, CRITICAL, UNKNOWN or OK.
225 Usually when writing any kind of summarizer for an array of results your code
226 might contain 3 phases.
228 Given a series of Nagios results like this:
232 {:exitcode => 0, :output => "OK", :perfdata => ""},
233 {:exitcode => 2, :output => "failure", :perfdata => ""}
237 You would write a nagios_states() function that does roughly this:
240 def nagios_states(results)
243 status_map = ["OK", "WARNING", "CRITICAL", "UNKNOWN"]
244 status_map.each {|s| result[s] = 0}
246 # loop over all the data, increment the count for OK etc
247 results.each do |result|
248 status = status_map[result[:exitcode]]
252 # return the result hash, {"OK" => 1, "CRITICAL" => 1, "WARN" => 0, "UNKNOWN" => 0}
257 You could optimise the code but you can see there are 3 major stages in the life
260 * Set initial values for the return data
261 * Loop the data building up the state
264 Given this, here is our Nagios exitcode summary function, it is roughly the same
265 code with a bit more boiler plate to plugin into mcollective, but the same code
271 class Nagios_states<Base
272 # Before function is run processing
274 # :collection or :numeric
275 @result[:type] = :collection
277 # set default aggregate_format if it is undefined
278 @aggregate_format = "%10s : %s" unless @aggregate_format
282 @status_map = ["OK", "WARNING", "CRITICAL", "UNKNOWN"]
283 @status_map.each {|s| @result[:value][s] = 0}
287 # Determines the average of a set of numerical values
288 def process_result(value, reply)
290 status = @status_map[value]
291 @result[:value][status] += 1
293 @result["UNKNOWN"] += 1
297 # Post processing hook that returns the summary result
299 result_class(@result[:type]).new(@result, @aggregate_format, @action)
306 This shows that an aggregate function has the same 3 basic parts. First we set
307 the initial state using the *startup_hook*. We then process each result as it
308 comes in from the network using *process_result*. Finally we turn that into a
309 the result objects that you saw earlier in the ruby client examples using the
313 Each function needs a startup hook, without one you'll get exceptions. The
314 startup hook lets you set up the initial state.
316 The first thing to do is set the type of result this will be. Currently we
317 support 2 types of result either a plain number indicated using *:numeric* or a
318 complex *:collection* type that can be a hash with keys and values.
320 Functions can take display formats in the DDL, in this example we set
321 *@aggregate_format* to a *printf* default that would display a table of results
322 but we still let the user supply his own format.
324 We then just initialize the result hash to and build a map from the English
325 representation of the Nagios status codes.
328 Every reply that comes in from the network gets passed into your
329 *process_result* method. The first argument will be just the single value the
330 DDL indicates you are interested in but you'll also get the whole rely so you
331 can get access to other reply values and such.
333 This gets called each time, we just look at the value and increment each Nagios
334 status or treat it as an unknown - in case the result data is missformed.
337 The summarize method lets you take the state you built up and convert that into
338 an answer. The summarize method is optional what you see here is the default
339 action if you do not supply one.
341 The *result_class* method accepts either *:collection* or *:numeric* as
342 arguments and it is basically a factory for the correct result structure.
344 ### Deploy and update the DDL
345 You should deploy this function into your *libdir/aggregate* directory called
346 *nagios_states.rb* on the client machines - no harm deploying it everywhere
349 Update the DDL so it looks like:
352 action "runcommand", :description => "Run a NRPE command" do
357 if respond_to?(:summarize)
359 aggregate nagios_states(:exitcode)
365 Add the last few lines - we check that we're running in a version of MCollective
366 that supports this feature and then we call our function with the *:exitcode*