3 title: Writing SimpleRPC Agents
5 [WritingAgents]: /mcollective/reference/basic/basic_agent_and_client.html
6 [SimpleRPCClients]: /mcollective/simplerpc/clients.html
7 [ResultsandExceptions]: /mcollective/simplerpc/clients.html#Results_and_Exceptions
8 [SimpleRPCAuditing]: /mcollective/simplerpc/auditing.html
9 [SimpleRPCAuthorization]: /mcollective/simplerpc/authorization.html
10 [DDL]: /mcollective/reference/plugins/ddl.html
11 [WritingAgentsScreenCast]: http://mcollective.blip.tv/file/3808928/
12 [RPCUtil]: /mcollective/reference/plugins/rpcutil.html
13 [ValidatorPlugins]: /mcollective/reference/plugins/validator.html
15 Simple RPC works because it makes a lot of assumptions about how you write agents, we'll try to capture those assumptions here and show you how to apply them to our Helloworld agent.
17 We've recorded a [tutorial that will give you a quick look at what is involved in writing agents][WritingAgentsScreenCast].
19 ## Conventions regarding Incoming Data
21 As you've seen in [SimpleRPCClients] our clients will send requests like:
24 mc.echo(:msg => "Welcome to MCollective Simple RPC")
27 A more complex example might be:
30 exim.setsender(:msgid => "1NOTVx-00028U-7G", :sender => "foo@bar.com")
33 Effectively this creates a hash with the members _:msgid_ and _:sender_.
35 Your data types should be preserved if your Security plugin supports that - the default one does - so you can pass in Arrays, Hashes, OpenStructs, Hashes of Hashes but you should always pass something in and it should be key/value pairs like a Hash expects.
37 You cannot use the a data item called _:process_results_ as this has special meaning to the agent and client. This will indicate to the agent that the client is'nt going to be waiting to process results. You might choose not to send back a reply based on this.
40 Here's our sample *Helloworld* agent:
42 {% highlight ruby linenos %}
45 class Helloworld<RPC::Agent
50 reply[:msg] = request[:msg]
58 Strictly speaking this Agent will work but isn't considered complete - there's no meta data and no help.
60 A helper agent called [_rpcutil_][RPCUtil] is included that helps you gather stats, inventory etc about the running daemon. It's a full SimpleRPC agent including DDL, you can look at it for an example.
63 The agent name is derived from the class name, the example code creates *MCollective::Agent::Helloworld* and the agent name would be *helloworld*.
65 <a name="Meta_Data_and_Initialization"> </a>
67 ### Meta Data and Initialization
68 Simple RPC agents still need meta data like in [WritingAgents], without it you'll just have some defaults assigned, code below adds the meta data to our agent:
70 **NOTE**: As of version 2.1.1 the _metadata_ section is deprecated, all agents must have DDL files with this information in them.
72 {% highlight ruby linenos %}
75 class Helloworld<RPC::Agent
76 metadata :name => "helloworld",
77 :description => "Echo service for MCollective",
78 :author => "R.I.Pienaar",
81 :url => "http://projects.puppetlabs.com/projects/mcollective-plugins/wiki",
88 reply[:msg] = request[:msg]
95 The added code sets our creator info, license and version as well as a timeout. The timeout is how long MCollective will let your agent run for before killing them, this is a very important number and should be given careful consideration. If you set it too low your agents will be terminated before their work is done.
97 The default timeout for SimpleRPC agents is *10*.
100 Actions are the individual tasks that your agent can do:
102 {% highlight ruby linenos %}
104 validate :msg, String
106 reply[:msg] = request[:msg]
110 Creates an action called "echo". They don't and can't take any arguments.
113 In the past you had to copy an agent only to machines that they should be running on as
114 all agents were activated regardless of dependencies.
116 To make deployment simpler agents support the ability to determine if they should run
117 on a particular platform. By default SimpleRPC agents can be configured to activate
121 plugin.helloworld.activate_agent = false
124 You can also place the following in _/etc/mcollective/plugins.d/helloworld.cfg_:
127 activate_agent = false
130 This is a simple way to enable or disable an agent on your machine, agents can also
131 declare their own logic that will get called each time an agent gets loaded from disk.
136 class Helloworld<RPC::Agent
139 File.executable?("/usr/bin/puppet")
146 If this block returns false or raises an exception then the agent will not be active on
147 this machine and it will not be discovered.
149 When the agent gets loaded it will test if _/usr/bin/puppet_ exist and only if it does
150 will this agent be enabled.
152 ## Help and the Data Description Language
153 We have a separate file that goes together with an agent and is used to describe the agent in detail, a DDL file for the above echo agent can be seen below:
155 **NOTE**: As of version 2.1.1 the DDL files are required to be on the the nodes before an agent will be activated
157 {% highlight ruby linenos %}
158 metadata :name => "echo",
159 :description => "Echo service for MCollective",
160 :author => "R.I.Pienaar",
163 :url => "http://projects.puppetlabs.com/projects/mcollective-plugins/wiki",
166 action "echo", :description => "Echos back any message it receives" do
168 :prompt => "Service Name",
169 :description => "The service to get the status for",
171 :validation => '^[a-zA-Z\-_\d]+$',
176 :description => "The message we received",
177 :display_as => "Message"
181 As you can see the DDL file expand on the basic syntax adding a lot of markup, help and other important validation data. This information - when available - helps in making more robust clients and also potentially auto generating user interfaces.
183 The DDL is a complex topic, read all about it in [DDL].
186 If you've followed the conventions and put the incoming data in a Hash structure then you can use a few of the provided validators to make sure your data that you received is what you expected.
188 If you didn't use Hashes for input the validators would not be usable to you. In future validation will happen automatically based on the [DDL] so I strongly suggest you follow the agent design pattern shown here using hashes.
190 In the sample action above we validate the *:msg* input to be of type *String*, here are a few more examples:
192 {% highlight ruby linenos %}
193 validate :msg, /[a-zA-Z]+/
194 validate :ipaddr, :ipv4address
195 validate :ipaddr, :ipv6address
196 validate :commmand, :shellsafe
197 validate :mode, ["all", "packages"]
200 The table below shows the validators we support currently
202 |Type of Check|Description|Example|
203 |-------------|-----------|-------|
204 |Regular Expressions|Matches the input against the supplied regular expression|validate :msg, /\[a-zA-Z\]+/|
205 |Type Checks|Verifies that input is of a given ruby data type|validate :msg, String|
206 |IPv4 Checks|Validates an ip v4 address, note 5.5.5.5 is technically a valid address|validate :ipaddr, :ipv4address|
207 |IPv6 Checks|Validates an ip v6 address|validate :ipaddr, :ipv6address|
208 |system call safety checks|Makes sure the input is a string and has no ><backtick, semi colon, dollar, ambersand or pipe characters in it|validate :command, :shellsafe|
209 |Boolean|Ensures a input value is either real boolean true or false|validate :enable, :bool|
210 |List of valid options|Ensures the input data is one of a list of known good values|validate :mode, \["all", "packages"\]|
212 All of these checks will raise an InvalidRPCData exception, you shouldn't catch this exception as the Simple RPC framework catches those and handles them appropriately.
214 We'll make input validators plugins so you can provide your own types of validation easily.
216 Additionally if can escape strings being passed to a shell, escaping is done in line with the _Shellwords#shellescape_ method that is in newer version of Ruby:
218 {% highlight ruby linenos %}
219 safe = shellescape(request[:foo])
222 As of version 2.2.0 you can add your own types of validation using [Validator Plugins][ValidatorPlugins].
224 ## Agent Configuration
226 You can save configuration for your agents in the main server config file:
229 plugin.helloworld.setting = foo
232 In your code you can retrieve the config setting like this:
235 setting = config.pluginconf["helloworld.setting"] || ""
238 This will set the setting to whatever is the config file of "" if unset.
240 ## Accessing the Input
241 As you see from the echo example our input is easy to get to by just looking in *request*, this would be a Hash of exactly what was sent in by the client in the original request.
243 The request object is in instance of *MCollective::RPC::Request*, you can also gain access to the following:
245 |Property|Description|
246 |--------|-----------|
247 |time|The time the message was sent|
248 |action|The action it is directed at|
249 |data|The actual hash of data|
250 |sender|The id of the sender|
251 |agent|Which agent it was directed at|
253 Since data is the actual Hash you can gain access to your input like:
265 Accessing it via the first will give you full access to all the normal Hash methods where the 2nd will only give you access to *include?*.
267 ## Running Shell Commands
269 A helper function exist that makes it easier to run shell commands and gain
270 access to their _STDOUT_ and _STDERR_.
272 We recommend everyone use this method for calling to shell commands as it forces
273 *LC_ALL* to *C* as well as wait on all the children and avoids zombies, you can
274 set unique working directories and shell environments that would be impossible
275 using simple _system_ that is provided with Ruby.
277 The simplest case is just to run a command and send output back to the client:
280 reply[:status] = run("echo 'hello world'", :stdout => :out, :stderr => :err)
283 Here you will have set _reply`[`:out`]`_, _reply`[`:err`]`_ and _reply`[`:status`]`_ based
284 on the output from the command
286 You can append the output of the command to any string:
291 status = run("echo 'hello world'", :stdout => out, :stderr => err)
294 Here the STDOUT of the command will be saved in the variable _out_ and not sent
295 back to the caller. The only caveat is that the variables _out_ and _err_ should
296 have the _<<_ method, so if you supplied an array each line of output will be a
297 single member of the array. In the example _out_ would be an array of lines
298 while _err_ would just be a big multi line string.
300 By default any trailing new lines will be included in the output and error:
303 reply[:status] = run("echo 'hello world'", :stdout => :out, :stderr => :err)
304 reply[:stdout].chomp!
305 reply[:stderr].chomp!
308 You can shorten this to:
311 reply[:status] = run("echo 'hello world'", :stdout => :out, :stderr => :err, :chomp => true)
314 This will remove a trailing new line from the _reply`[`:out`]`_ and _reply`[`:err`]`_.
316 If you wanted this command to run from the _/tmp_ directory:
319 reply[:status] = run("echo 'hello world'", :stdout => :out, :stderr => :err, :cwd => "/tmp")
322 Or if you wanted to include a shell Environment variable:
325 reply[:status] = run("echo 'hello world'", :stdout => :out, :stderr => :err, :environment => {"FOO" => "BAR"})
328 The status returned will be the exit code from the program you ran, if the program
329 completely failed to run in the case where the file doesn't exist, resources were
330 not available etc the exit code will be -1
332 You have to set the cwd and environment through these options, do not simply
333 call _chdir_ or adjust the _ENV_ hash in an agent as that will not be safe in
334 the context of a multi threaded Ruby application.
336 ## Constructing Replies
339 The reply data is in the *reply* variable and is an instance of *MCollective::RPC::Reply*.
342 reply[:msg] = request[:msg]
346 As pointed out in the [ResultsandExceptions] page results all include status messages and the reply object has a helper to create those.
350 validate :msg, String
351 validate :msg, /[a-zA-Z]+-[a-zA-Z]+-[a-zA-Z]+-[a-zA-Z]+/
352 reply.fail "No such message #{request[:msg]}", 1 unless have_msg?(request[:msg])
354 # check all the validation passed before doing any work
355 return unless reply.statuscode == 0
357 # now remove the message from the queue
362 The number in *reply.fail* corresponds to the codes in [ResultsandExceptions] it would default to *1* so you could just say:
365 reply.fail "No such message #{request[:msg]}" unless have_msg?(request[:msg])
368 This is hypothetical action that is supposed to remove a message from some queue, if we do have a String as input that matches our message id's we then check that we do have such a message and if we don't we fail with a helpful message.
370 Technically this will just set *statuscode* and *statusmsg* fields in the reply to appropriate values.
372 It won't actually raise exceptions or exit your action though you should do that yourself as in the example here.
374 There is also a *fail!* instead of just *fail* it does the same basic function but also raises exceptions. This lets you abort processing of the agent immediately without performing your own checks on *statuscode* as above later on.
376 ## Actions in external scripts
377 Actions can be implemented using other programming languages as long as they support JSON.
381 implemented_by "/some/external/script"
385 The script _/some/external/script_ will be called with 2 arguments:
387 * The path to a file with the request in JSON format
388 * The path to a file where you should write your response as a JSON hash
390 You can also access these 2 file paths in the *MCOLLECTIVE_REPLY_FILE* and *MCOLLECTIVE_REQUEST_FILE* environment variables
392 Simply write your reply as a JSON hash into the reply file.
394 The exit code of your script should correspond to the ones in [ResultsandExceptions]. Any text in STDERR will be
395 logged on the server at *error* level and used in the text for the fail text.
397 Any text to STDOUT will be logged on the server at level *info*.
399 These scripts can be placed in a standard location:
403 implemented_by "script.py"
407 This will search each configured libdir for _libdir/agent/agent_name/script.py_. If you specified a full path it will not try to find the file in libdirs.
409 ## Sharing code between agents
410 Sometimes you have code that is needed by multiple agents or shared between the agent and client. MCollective has
411 name space called *MCollective::Util* for this kind of code and the packagers and so forth supports it.
413 Create a class with your shared code given a name like *MCollective::Util::Yourco* and save this file in the libdir in *util/yourco.rb*
415 A sample class can be seen here:
428 You can now use it in your agent or clients by first loading it from the MCollective lib directories:
431 MCollective::Util.loadclass("MCollective::Util::Yourco")
433 helpers = MCollective::Util::Yourco.new
438 You can write a fine grained Authorization system to control access to actions and agents, please see [SimpleRPCAuthorization] for full details.
441 The actions that agents perform can be Audited by code you provide, potentially creating a centralized audit log of all actions. See [SimpleRPCAuditing] for full details.
444 You can write to the server log file using the normal logger class:
447 Log.debug ("Hello from your agent")
450 You can log at levels *info*, *warn*, *debug*, *fatal* or *error*.
453 As of version 2.2.0 there is a system wide Cache you can use to store data that might be costly to create on each request.
455 The Cache is thread safe and can be used even with multiple concurrent requests for the same agent.
457 Imagine your agent interacts with a customer database on the node that is slow to read data from but this data does not
458 change often. Using the cache you can arrange for this be read only every 10 minutes:
461 action "get_customer_data" do
462 # Create a new cache called 'customer' with a 600 second TTL,
463 # noop if it already exist
464 Cache.setup(:customer, 600)
467 customer = Cache.read(:customer, request[:customerid])
469 customer = Cache.write(:customer, request[:customerid], get_customer(request[:customerid])
472 # do something with the customer data
476 Here we setup a new cache table called *:customer* if it does not already exist, the cache has a 10 minute validity.
477 We then try to read a cached customer record for *request\[:customerid\]* and if it's not been put in the cache
478 before or if it expired I create a new customer record using a method called *get_customer* and then save it
481 If you have critical code in an agent that can only ever be run once you can use the Mutex from the same cache
482 to synchronize the code:
485 action "get_customer_data" do
486 # Create a new cache called 'customer' with a 600 second TTL,
487 # noop if it already exist
488 Cache.setup(:customer, 600)
490 Cache(:customer).synchronize do
491 # Update customer record
496 Here we are using the same Cache that was previously setup and just gaining access to the Mutex protecting the
497 cache data. The code inside the synchronize block will only be run once so you won't get competing updates to
500 If the lock is held too long by anyone the mcollectived will kill the threads in line with the Agent timeout.
503 We provide a few hooks into the processing of a message, you've already used this earlier to <a href="#Meta_Data_and_Initialization">set meta data</a>.
505 You'd use these hooks to add some functionality into the processing chain of agents, maybe you want to add extra logging for audit purposes of the raw incoming message and replies, these hooks will let you do that.
507 |Hook Function Name|Description|
508 |------------------|-----------|
509 |startup_hook|Called at the end of the initialize method of the _RPC::Agent_ base class|
510 |before_processing_hook(msg, connection)|Before processing of a message starts, pass in the raw message and the <em>MCollective::Connector</em> class|
511 |after_processing_hook|Just before the message is dispatched to the client|
514 Called at the end of the _RPC::Agent_ standard initialize method use this to adjust meta parameters, timeouts and any setup you need to do.
516 This will not be called right when the daemon starts up, we use lazy loading and initialization so it will only be called the first time a request for this agent arrives.
518 ### *before_processing_hook*
519 Called just after a message was received from the middleware before it gets passed to the handlers. *request* and *reply* will already be set, the msg passed is the message as received from the normal mcollective runner and the connection is the actual connector.
521 You can in theory send off new messages over the connector maybe for auditing or something, probably limited use case in simple agents.
523 ### *after_processing_hook*
524 Called at the end of processing just before the response gets sent to the middleware.
526 This gets run outside of the main exception handling block of the agent so you should handle any exceptions you could raise yourself. The reason it is outside of the block is so you'll have access to even status codes set by the exception handlers. If you do raise an exception it will just be passed onto the runner and processing will fail.