3 title: Writing SimpleRPC Clients
5 [SimpleRPCIntroduction]: index.html
6 [WritingAgents]: agents.html
7 [RPCUtil]: /mcollective/reference/plugins/rpcutil.html
8 [WritingAgentsScreenCast]: http://mcollective.blip.tv/file/3808928/
9 [RubyMixin]: http://juixe.com/techknow/index.php/2006/06/15/mixins-in-ruby/
10 [OptionParser]: http://github.com/puppetlabs/marionette-collective/blob/master/lib/mcollective/optionparser.rb
11 [AppPlugin]: ../reference/plugins/application.html
13 As pointed out in the [SimpleRPCIntroduction] page you can use the _mco rpc_ CLI
14 to call agents and it will do it's best to print results in a sane way. When
15 this is not enough you can write your own clients.
17 Simple RPC clients can do most of what a normal [client][WritingAgents] can do
18 but it makes a lot of things much easier if you stick to the Simple RPC
21 This guide shows how to write standalone scripts to interact with your
22 collective. There is a single executable system. You can apply most of
23 the techniques documented here to writing plugins for that application system.
24 See the full reference for the plugin system [here][AppPlugin]. You should try
25 to write your general agent CLIs using this plugin system rather than the stand
26 alone scripts detailed below as that promote a unified interface that behave in a
29 We've recorded a [tutorial that will give you a quick look at what is involved
30 in writing agents and a very simple client][WritingAgentsScreenCast].
32 We'll walk through building a ever more complex example of Hello World here.
35 The client is mostly a bunch of helper methods that you use as a [Ruby
36 Mixin][RubyMixin] in your own code, it provides:
38 * Standard command line option parsing with help output
39 * Ability to add your own command line options
40 * Simple access to agents and actions
41 * Tools to help you print results
42 * Tools to print stats
43 * Tools to construct your own filters
44 * While retaining full power of _MCollective::Client_ if you need the additional feature sets
45 * And being as simple or as complex to match your level of code proficiency
47 We'll write a client for the _Helloworld_ agent that you saw in the
48 [SimpleRPCIntroduction].
50 ## Call an Agent and print the result
51 A basic hello world client can be seen below:
53 {% highlight ruby linenos %}
58 include MCollective::RPC
60 mc = rpcclient("helloworld")
62 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
69 Save this into _hello.rb_ and run it with _--help_, you should see the standard basic help including filters for discovery.
71 If you've set up the Agent and run the client you should see something along these lines:
76 Finished processing 44 hosts in 375.57 ms
79 While it ran you would have seen a little progress bar and then just the summary line. The idea is that if you're talking to a 1000 machine there's no point in seeing a thousand _OK_, you only want to see what failed and this is exactly what happens here, you're only seeing errors.
81 If you run it with _--verbose_ you'll see a line of text for every host and also a larger summary of results.
83 I'll explain each major line in the code below then add some more features from there:
86 include MCollective::RPC
88 mc = rpcclient("helloworld")
91 The first line pulls in the various helper functions that we provide, this is the Mixin we mentioned earlier.
93 We then create a new client to the agent "helloworld" that you access through the _mc_ variable.
96 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
101 To call a specific action you simply have to do _mc.echo_ this calls the _echo_ action, we pass a _:msg_ parameter into it with the string we want echo'd back. The parameters will differ from action to action. It returns a simple array of the results that you can print any way you want, we'll show that later.
103 _printrpc_ and _printrpcstats_ are functions used to print the results and stats respectively.
109 This cleanly disconnects the client from the middleware, some middleware tools like ActiveMQ will log confusing exceptions if you do not do this. It's good form to always disconnect but isn't strictly required.
111 ## Adjusting the output
113 ### Verbosely displaying results
114 As you see there's no indication that discovery is happening and as pointed out we do not display results that are ok, you can force verbosity as below on individual requests:
117 mc = rpcclient("helloworld")
119 mc.discover :verbose => true
121 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC"), :verbose => true
124 Here we've added a _:verbose_ flag and we've specifically called the discover method. Usually you don't need to call discover it will do it on demand. Doing it this way you'll always see the line:
126 {% highlight console %}
127 Determining the amount of hosts matching filter for 2 seconds .... 44
130 Passing verbose to _printrpc_ forces it to print all the results, failures or not.
132 If you just wanted to force verbose on for all client interactions, do:
135 mc = rpcclient("helloworld")
138 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
141 In this case everything will be verbose, regardless of command line options.
143 ### Disabling the progress indicator
144 You can disable the twirling progress indicator easily:
147 mc = rpcclient("helloworld")
151 Now whenever you call an action you will not see the progress indicator.
153 ### Saving the reports in variables without printing
154 You can retrieve the stats from the clients and also get text of reports without printing them:
157 stats = mc.echo(:msg => "Welcome to MCollective Simple RPC").stats
159 report = stats.report
162 _report_ will now have the text that would have been displayed by 'printrpcstats' you can also use _no_response_report_ to get report text for just the list of hosts that didnt respond.
164 If you didn't want to just print the results out to STDOUT you can also get them back as just text:
167 report = rpcresults mc.echo(:msg => "Welcome to MCollective Simple RPC")
171 ## Applying filters programatically
172 You can pass filters on the command line using the normal _--with-`*`_ options but you can also do it programatically. Here's a new version of the client that only calls machines with the configuration management class _/dev_server/_ and the fact _country=uk_
175 mc = rpcclient("helloworld")
177 mc.class_filter /dev_server/
178 mc.fact_filter "country", "uk"
180 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
183 You can set other filters like _agent`_`filter_, _identity`_`filter_ and _compound`_`filter_.
185 The fact_filter method supports a few other forms in adition to above:
188 mc.fact_filter "country=uk"
189 mc.fact_filter "physicalprocessorcount", "4", ">="
192 This will limit it to all machines in the UK with more than 3 processors.
194 ## Resetting filters to empty
195 If while using the client you wish to reset the filters to an empty set of filters - containing only the agent name that you're busy addressing you can do it as follows:
198 mc = rpcclient("helloworld")
200 mc.class_filter /dev_server/
205 After this code snippet the filter will only have an agent filter of _helloworld_ set.
207 ## Processing Agents in Batches
208 By default the client will communicate with all machines at the same time.
209 This might not be desired as you might affect a DOS on related components.
211 You can instruct the client to communicate with remote agents in batches
212 and sleep between each batch.
214 Any client application has this capability using the _--batch_ and _--batch-sleep-time_
215 command line options.
217 You can also enable this programatically either per client or per request:
220 mc = rpcclient("helloworld")
222 mc.batch_sleep_time = 5
224 mc.echo(:msg => "hello world")
228 mc = rpcclient("helloworld")
230 mc.echo(:msg => "hello world", :batch_size => 10, :batch_sleep_time => 5)
233 By default batching is disabled and sleep time is 1
235 Setting the batch_size to 0 will disable batch mode in both examples above,
236 effectively overriding what was supplied on the command line.
238 ## Forcing Rediscovery
239 By default it will only do discovery once per script and then re-use the results, you can though force rediscovery if you had to adjust filters mid run for example.
242 mc = rpcclient("helloworld")
244 mc.class_filter /dev_server/
245 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
249 mc.fact_filter "country", "uk"
250 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
253 Here we make one _echo_ call - which would do a discovery - we then reset the client, adjust filters and call it again. The 2nd call would do a new discovery and have new client lists etc.
255 ## Supplying your own discovery information
257 A new core messaging mode has been introduced that enables direct non filtered communicatin to specific nodes. This has enabled us to provide an discovery-optional
258 mode but only if the collective is configured to support direct messaging.
261 mc = rpcclient("helloworld")
263 mc.discover(:nodes => ["host1", "host2", "host3"]
265 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
268 This will immediately, without doing discovery, communicate just with these 3 hosts. It will do normal failure reporting as with normal discovery based
269 requests but will just be much faster as the 2 second discovery overhead is avoided.
271 The goal with this feature is for cases such as deployment tools where you have a known expectation of which machines to deploy to and you always want
272 to know if that fails. In that use case a discovery based approach is not 100% suitable as you won't know about down machines. This way you can provide
273 your own source of truth.
275 When using the direct mode messages have a TTL associated with them that defaults to 60 seconds. Since 1.3.2 you can set the TTL globally in the configuration
276 file but you can also set it on the client:
279 mc = rpcclient("helloworld")
282 mc.discover(:nodes => ["host1", "host2", "host3"]
284 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
287 With the TTL set to 3600 if any of the hosts are down at the time of the request the request will wait on the middleware and should they come back up
288 before 3600 has passed since request time they will then perform the requested action.
290 ## Only sending requests to a subset of discovered nodes
291 By default all nodes that get discovered will get the request. This isn't always desirable maybe you want to deploy only to a random subset of hosts or maybe you have a service exposed over MCollective that you want to treat as a HA service and so only speak with one host that provides the functionality.
293 You can limit the hosts to talk to either using a number or a percentage, the code below shows both:
296 mc = rpcclient("helloworld")
298 mc.limit_targets = "10%"
299 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
302 This will pick 10% of the discovered hosts - or 1 if 10% is less than 1 - and only target those nodes with your request. You can also set it to an integer.
304 There are two possible modes for choosing the targets. You can configure a global default method but also set it on your client:
307 mc = rpcclient("helloworld")
309 mc.limit_targets = "10%"
310 mc.limit_method = :random
311 printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
314 The above code will force a _random_ selection, you can also set it to _:first_
316 ## Gaining access to the full MCollective::Client
317 If you wanted to work with the Client directly as in [WritingAgents] after perhaps setting up some queries or gathering data first you can gain access to the client, you might also need access to the options array that was parsed out from the command line and any subsequent filters that you added.
320 mc = rpcclient("helloworld")
326 The first call will set up the CLI option parsing, create clients etc, you can then just grab the client and options and go on as per [WritingAgents]. This is a much quicker way to write full power clients, by just by short-circuiting the options parsing etc.
328 ## Dealing with the results directly
329 The biggest reason that you'd write custom clients is probably if you wanted to do custom processing of the results, there are 2 options to do it.
331 <a name="Results_and_Exceptions"> </a>
333 ### Results and Exceptions
334 Results have a set structure and depending on how you access the results you will either get Exceptions or result codes.
336 |Status Code|Description|Exception Class|
337 |-----------|-----------|---------------|
339 |1|OK, failed. All the data parsed ok, we have a action matching the request but the requested action could not be completed.|RPCAborted|
340 |2|Unknown action|UnknownRPCAction|
341 |3|Missing data|MissingRPCData|
342 |4|Invalid data|InvalidRPCData|
343 |5|Other error|UnknownRPCError|
345 Just note these now, I'll reference them later down.
347 ### Simple RPC style results
348 Simple RPC provides a trimmed down version of results from the basic Client library. You'd choose to use this if you just want to do simple things or maybe you're just learning Ruby. You'll get to process the results _after_ the call is either done or timed out completely.
350 This is an important difference between the two approaches, in one you can parse the results as it comes in, in the other you will only get results after processing is done. This would be the main driving facter for choosing one over the other.
352 Here's an example that will print out results in a custom way.
355 mc.echo(:msg => "hello world").each do |resp|
356 printf("%-40s: %s\n", resp[:sender], resp[:data][:msg])
360 This will produce a result something like this:
362 {% highlight console %}
363 dev1.you.net : hello world
364 dev2.you.net : hello world
365 dev3.you.net : hello world
368 The _each_ in the above code just loops through the array of results. Results are an array of Hashes, the data you got for above has the following structure:
370 {% highlight console %}
372 :sender=>"dev1.your.net",
373 :data=>{:msg => "hello world"},
376 :sender=>"dev2.your.net",
377 :data=>{:msg => "hello world"},
381 The _:statuscode_ matches the table above so you can make decisions based on each result's status.
383 ### Gaining access to MCollective::Client#req results
384 You can get access to each result in real time, in this case you will need to handle the exceptions in the table above and you'll get a different style of result set. The result set will be exactly as from the full blown client.
386 In this mode there will be no progress indicator, you'll deal with results as and when they come in not after the fact as in the previous example.
389 mc.echo(:msg => "hello world") do |resp|
391 printf("%-40s: %s\n", resp[:senderid], resp[:body][:data])
393 puts "The RPC agent returned an error: #{e}"
398 The output will be the same as above
400 In this mode the results you get will be like this:
403 {:msgtarget=>"/topic/mcollective.helloworld.reply",
404 :senderid=>"dev2.your.net",
405 :msgtime=>1261696663,
406 :hash=>"2d37daf690c4bcef5b5380b1e0c55f0c",
407 :body=>{:statusmsg=>"OK", :statuscode=>0, :data=>{:msg => "hello world"}},
408 :requestid=>"2884afb0b52cb38ea4d4a3146d18ef5f",
409 :senderagent=>"helloworld"}
412 Note how here we need to catch the exceptions, just handing _:statuscode_ will not be enough as the RPC client will raise exceptions - all descendant from _RPCError_ so you can easily catch just those.
414 You can additionally gain access to a SimpleRPC style result in addition to the more complex native client results:
417 mc.echo(:msg => "hello world") do |resp, simpleresp|
419 printf("%-40s: %s\n", simpleresp[:sender], simpleresp[:data][:msg])
421 puts "The RPC agent returned an error: #{e}"
426 You can still use printrpc to print these style of results and gain advantage of the DDL and so forth:
429 mc.echo(:msg => "hello world") do |resp, simpleresp|
433 puts "The RPC agent returned an error: #{e}"
438 You will need to handle exceptions yourself but you have a simpler result set to deal with
440 ## Adding custom command line options
441 You can look at the _mco rpc_ script for a big sample, here I am just adding a simple _--msg_ option to our script so you can customize the message that will be sent and received.
443 {% highlight ruby linenos %}
446 require 'mcollective'
448 include MCollective::RPC
450 options = rpcoptions do |parser, options|
451 parser.define_head "Generic Echo Client"
452 parser.banner = "Usage: hello [options] [filters] --msg MSG"
454 parser.on('-m', '--msg MSG', 'Message to pass') do |v|
459 unless options.include?(:msg)
460 puts("You need to specify a message with --msg")
464 mc = rpcclient("helloworld", :options => options)
466 mc.echo(:msg => options[:msg]).each do |resp|
467 printf("%-40s: %s\n", resp[:sender], resp[:data][:msg])
471 This version of the code should be run like this:
473 {% highlight console %}
480 Documentation for the Options Parser can be found [in it's code][OptionParser].
482 And finally if you add options as above rather than try to parse it yourself you will get help integration for free:
484 {% highlight console %}
486 Usage: hello [options] [filters] --msg MSG
488 -m, --msg MSG Message to pass
491 -c, --config FILE Load configuratuion from file rather than default
492 --dt SECONDS Timeout for doing discovery
494 -t, --timeout SECONDS Timeout for calling remote agents
495 -q, --quiet Do not be verbose
496 -v, --verbose Be verbose
497 -h, --help Display this screen
500 --wf, --with-fact fact=val Match hosts with a certain fact
501 --wc, --with-class CLASS Match hosts with a certain configuration management class
502 --wa, --with-agent AGENT Match hosts with a certain agent
503 --wi, --with-identity IDENT Match hosts with a certain configured identity
506 ## Disabling command line parsing and supplying your options programatically
508 Sometimes, perhaps when embedding an MCollective client into another tool like Puppet, you do not want MCollective to do any command line parsing as there might be conflicting command line options etc.
510 This can be achieved by supplying an options hash to the SimpleRPC client:
513 include MCollective::RPC
515 options = MCollective::Util.default_options
517 client = rpcclient("test", {:options => options})
520 This will create a RPC client for the agent test without any options parsing at all.
522 To set options like discovery timeout and so forth you will need use either the client utilities or manipulate the hash upfront, the client utility methods is the best. The code below sets the discovery timeout in a way that does not require you to know any internal structures or the content of the options hash.
525 options = MCollective::Util.default_options
527 client = rpcclient("test", {:options => options})
528 client.discovery_timeout = 4
531 Using this method of creating custom options hashes mean we can make internal changes to MCollective without affecting your code in the future.
533 ## Sending SimpleRPC requests without discovery and blocking
535 Usually this section will not apply to you. The client libraries support sending a request without waiting for a reply. This could be useful if you want to clean yum caches but don't really care if it actually happens everywhere.
537 You will loose these abilities:
539 * Knowing if your request was received by any agents
540 * Any stats about processing times etc
541 * Any information about the success or failure of your request
543 The above should make it clear already that this is a limited use case, it's essentially a broadcast request with no feedback loop.
545 The code below will send a request to the _runonce_ action for an agent _puppetd_, once the request is dispatched I will have no idea if it got handled etc, my code will just continue onwards.
548 p = rpcclient("puppetd")
550 p.identity_filter "your.box.com"
551 reqid = p.runonce(:forcerun => true, :process_results => false)
554 This will honor any attached filters set either programatically or through the command line, it will send the request but will
555 just not handle any responses. All it will do is return the request id.
557 ## Doing your own discovery
558 For web applications you'd probably use cached copied of Registration data to do discovery rather than wait for MC to do discovery between each request.
560 To do this, you'll need to construct a filter and a list of expected hosts and then do a custom call:
563 puppet = rpcclient("puppetd")
565 printrpc puppet.custom_request("runonce", {:forcerun => true}, "your.box.com", {"identity" => "your.box.com"})
568 This will do a call with exactly the same stats, block and other semantics as a normal call like:
571 printrpc puppet.runonce(:forcerun => true)
574 But instead of doing any discovery it will use the host list and filter you supplied in the call.
576 ## The _rpcutil_ Helper Agent
578 A helper agent called [_rpcutil_][RPCUtil] is included that helps you gather stats, inventory etc about the running daemon.