--- /dev/null
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright 2010, 2011 Puppet Labs
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
--- /dev/null
+The Marionette Collective
+=========================
+
+The Marionette Collective aka. mcollective is a framework to build server orchestration or parallel job execution systems.
+
+For full information, wikis, ticketing and downloads please see http://marionette-collective.org/
--- /dev/null
+# Rakefile to build a project using HUDSON
+
+begin
+ require 'rdoc/task'
+rescue LoadError
+ require 'rake/rdoctask'
+end
+
+require 'rake/packagetask'
+require 'rake/clean'
+require 'find'
+require 'rubygems/package_task'
+
+PROJ_DOC_TITLE = "The Marionette Collective"
+PROJ_VERSION = "2.3.1"
+PROJ_RELEASE = "2"
+PROJ_NAME = "mcollective"
+PROJ_RPM_NAMES = [PROJ_NAME]
+PROJ_FILES = ["#{PROJ_NAME}.init", "COPYING", "doc", "etc", "lib", "plugins", "ext", "bin"]
+PROJ_FILES.concat(Dir.glob("mc-*"))
+RDOC_EXCLUDES = ["mcollective/vendor", "spec", "ext", "website", "plugins"]
+
+ENV["RPM_VERSION"] ? CURRENT_VERSION = ENV["RPM_VERSION"] : CURRENT_VERSION = PROJ_VERSION
+ENV["BUILD_NUMBER"] ? CURRENT_RELEASE = ENV["BUILD_NUMBER"] : CURRENT_RELEASE = PROJ_RELEASE
+ENV["DEB_DISTRIBUTION"] ? PKG_DEB_DISTRIBUTION = ENV["DEB_DISTRIBUTION"] : PKG_DEB_DISTRIBUTION = "unstable"
+
+CLEAN.include(["build", "doc"])
+
+def announce(msg='')
+ STDERR.puts "================"
+ STDERR.puts msg
+ STDERR.puts "================"
+end
+
+def init
+ FileUtils.mkdir("build") unless File.exist?("build")
+end
+
+def safe_system *args
+ raise RuntimeError, "Failed: #{args.join(' ')}" unless system *args
+end
+
+spec = Gem::Specification.new do |s|
+ s.name = "mcollective-client"
+ s.version = PROJ_VERSION
+ s.author = "R.I.Pienaar"
+ s.email = "rip@puppetlabs.com"
+ s.homepage = "https://docs.puppetlabs.com/mcollective/"
+ s.summary = "Client libraries for The Marionette Collective"
+ s.description = "Client libraries for the mcollective Application Server"
+ s.files = FileList["{bin,lib}/**/*"].to_a
+ s.require_path = "lib"
+ s.test_files = FileList["spec/**/*"].to_a
+ s.has_rdoc = true
+ s.executables = "mco"
+ s.default_executable = "mco"
+ s.add_dependency "systemu"
+ s.add_dependency "json"
+ s.add_dependency "stomp"
+ s.add_dependency "i18n"
+
+ excluded_files = ["bin/mcollectived", "lib/mcollective/runner.rb", "lib/mcollective/vendor/json", "lib/mcollective/vendor/systemu", "lib/mcollective/vendor/i18n", "lib/mcollective/vendor/load"]
+
+ excluded_files.each do |file|
+ s.files.delete_if {|f| f.match(/^#{file}/)}
+ end
+end
+
+Gem::PackageTask.new(spec) do |pkg|
+ pkg.need_tar = false
+ pkg.need_zip = false
+ pkg.package_dir = "build"
+end
+
+desc "Build documentation, tar balls and rpms"
+task :default => [:clean, :doc, :package]
+
+# task for building docs
+rd = Rake::RDocTask.new(:doc) { |rdoc|
+ rdoc.rdoc_dir = 'doc'
+ rdoc.title = "#{PROJ_DOC_TITLE} version #{CURRENT_VERSION}"
+ rdoc.options << '--line-numbers' << '--main' << 'MCollective'
+
+ RDOC_EXCLUDES.each do |ext|
+ rdoc.options << '--exclude' << ext
+ end
+}
+
+desc "Run spec tests"
+task :test do
+ sh "cd spec;rake"
+end
+
+desc "Create a tarball for this release"
+task :package => [:clean, :doc] do
+ announce "Creating #{PROJ_NAME}-#{CURRENT_VERSION}.tgz"
+
+ FileUtils.mkdir_p("build/#{PROJ_NAME}-#{CURRENT_VERSION}")
+ safe_system("cp -R #{PROJ_FILES.join(' ')} build/#{PROJ_NAME}-#{CURRENT_VERSION}")
+
+ announce "Setting MCollective.version to #{CURRENT_VERSION}"
+ safe_system("cd build/#{PROJ_NAME}-#{CURRENT_VERSION}/lib && sed -i -e s/@DEVELOPMENT_VERSION@/#{CURRENT_VERSION}/ mcollective.rb")
+
+ safe_system("cd build && tar --exclude .svn -cvzf #{PROJ_NAME}-#{CURRENT_VERSION}.tgz #{PROJ_NAME}-#{CURRENT_VERSION}")
+end
+
+desc "Creates the website as a tarball"
+task :website => [:clean] do
+ FileUtils.mkdir_p("build/marionette-collective.org/html")
+
+ Dir.chdir("website") do
+ safe_system("jekyll ../build/marionette-collective.org/html")
+ end
+
+ unless File.exist?("build/marionette-collective.org/html/index.html")
+ raise "Failed to build website"
+ end
+
+ Dir.chdir("build") do
+ safe_system("tar -cvzf marionette-collective-org-#{Time.now.to_i}.tgz marionette-collective.org")
+ end
+end
+
+desc "Creates a RPM"
+task :rpm => [:clean, :doc, :package] do
+ announce("Building RPM for #{PROJ_NAME}-#{CURRENT_VERSION}-#{CURRENT_RELEASE}")
+
+ sourcedir = `rpm --eval '%_sourcedir'`.chomp
+ specsdir = `rpm --eval '%_specdir'`.chomp
+ srpmsdir = `rpm --eval '%_srcrpmdir'`.chomp
+ rpmdir = `rpm --eval '%_rpmdir'`.chomp
+ rpmdist = ''
+
+ `which lsb_release`
+ if $?.success?
+ lsbdistrel = `lsb_release -r -s | cut -d . -f1`.chomp
+ lsbdistro = `lsb_release -i -s`.chomp
+ case lsbdistro
+ when 'CentOS'
+ rpmdist = ".el#{lsbdistrel}"
+ when 'Fedora'
+ rpmdist = ".fc#{lsbdistrel}"
+ end
+ end
+
+ `which rpmbuild-md5`
+ rpmcmd = $?.success? ? 'rpmbuild-md5' : 'rpmbuild'
+
+ safe_system %{cp build/#{PROJ_NAME}-#{CURRENT_VERSION}.tgz #{sourcedir}}
+ safe_system %{cat ext/redhat/#{PROJ_NAME}.spec|sed -e s/%{rpm_release}/#{CURRENT_RELEASE}/g | sed -e s/%{version}/#{CURRENT_VERSION}/g > #{specsdir}/#{PROJ_NAME}.spec}
+
+ if ENV['SIGNED'] == '1'
+ safe_system %{#{rpmcmd} --sign -D 'version #{CURRENT_VERSION}' -D 'rpm_release #{CURRENT_RELEASE}' -D 'dist #{rpmdist}' -D 'use_lsb 0' -ba #{specsdir}/#{PROJ_NAME}.spec}
+ else
+ safe_system %{#{rpmcmd} -D 'version #{CURRENT_VERSION}' -D 'rpm_release #{CURRENT_RELEASE}' -D 'dist #{rpmdist}' -D 'use_lsb 0' -ba #{specsdir}/#{PROJ_NAME}.spec}
+ end
+
+ safe_system %{cp #{srpmsdir}/#{PROJ_NAME}-#{CURRENT_VERSION}-#{CURRENT_RELEASE}*.src.rpm build/}
+
+ safe_system %{cp #{rpmdir}/*/#{PROJ_NAME}*-#{CURRENT_VERSION}-#{CURRENT_RELEASE}*.rpm build/}
+end
+
+desc "Create the .debs"
+task :deb => [:clean, :doc, :package] do
+ announce("Building debian packages")
+
+ FileUtils.mkdir_p("build/deb")
+ Dir.chdir("build/deb") do
+ safe_system %{tar -xzf ../#{PROJ_NAME}-#{CURRENT_VERSION}.tgz}
+ safe_system %{cp ../#{PROJ_NAME}-#{CURRENT_VERSION}.tgz #{PROJ_NAME}_#{CURRENT_VERSION}.orig.tar.gz}
+
+ Dir.chdir("#{PROJ_NAME}-#{CURRENT_VERSION}") do
+ safe_system %{cp -R ext/debian .}
+ safe_system %{cp -R ext/debian/mcollective.init .}
+ safe_system %{cp -R ext/Makefile .}
+
+ File.open("debian/changelog", "w") do |f|
+ f.puts("mcollective (#{CURRENT_VERSION}-#{CURRENT_RELEASE}) #{PKG_DEB_DISTRIBUTION}; urgency=low")
+ f.puts
+ f.puts(" * Automated release for #{CURRENT_VERSION}-#{CURRENT_RELEASE} by rake deb")
+ f.puts
+ f.puts(" See http://marionette-collective.org/releasenotes.html for full details")
+ f.puts
+ f.puts(" -- The Marionette Collective <mcollective-dev@googlegroups.com> #{Time.new.strftime('%a, %d %b %Y %H:%M:%S %z')}")
+ end
+
+ if ENV['SIGNED'] == '1'
+ if ENV['SIGNWITH']
+ safe_system %{debuild -i -k#{ENV['SIGNWITH']}}
+ else
+ safe_system %{debuild -i}
+ end
+ else
+ safe_system %{debuild -i -us -uc}
+ end
+ end
+
+ safe_system %{cp *.deb *.dsc *.diff.gz *.orig.tar.gz *.changes ..}
+ end
+end
+
+desc "Update the website error code reference based on current local"
+task :update_msgweb do
+ mcollective_dir = File.join(File.dirname(__FILE__))
+
+ $:.insert(0, File.join(mcollective_dir, "lib"))
+
+ require 'mcollective'
+
+ messages = YAML.load_file(File.join(mcollective_dir, "lib", "mcollective", "locales", "en.yml"))
+
+ webdir = File.join(mcollective_dir, "website", "messages")
+
+ I18n.load_path = Dir[File.join(mcollective_dir, "lib", "mcollective", "locales", "*.yml")]
+ I18n.locale = :en
+
+ messages["en"].keys.each do |msg_code|
+ md_file = File.join(webdir, "#{msg_code}.md")
+
+ puts "....writing %s" % md_file
+
+ File.open(md_file, "w") do |md|
+ md.puts "---"
+ md.puts "layout: default"
+ md.puts "title: Message detail for %s" % msg_code
+ md.puts "toc: false"
+ md.puts "---"
+ md.puts
+ md.puts "Example Message"
+ md.puts "---------------"
+ md.puts
+ md.puts " %s" % (MCollective::Util.t("%s.example" % msg_code, :raise => true) rescue MCollective::Util.t("%s.pattern" % msg_code))
+ md.puts
+ md.puts "Additional Information"
+ md.puts "----------------------"
+ md.puts
+ md.puts MCollective::Util.t("%s.expanded" % msg_code, :raise => true)
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env ruby
+
+require 'mcollective'
+require 'pp'
+
+oparser = MCollective::Optionparser.new({:verbose => true}, "filter")
+
+options = oparser.parse{|parser, options|
+ parser.define_head "Call an agent parsing an argument to it"
+ parser.banner = "Usage: mc-call-agent [options] --agent agent --argument arg"
+
+ parser.on('-a', '--agent AGENT', 'Agent to call') do |v|
+ options[:agent] = v
+ end
+
+ parser.on('--arg', '--argument ARGUMENT', 'Argument to pass to agent') do |v|
+ options[:argument] = v
+ end
+}
+
+if options[:agent] == nil || options[:argument] == nil
+ puts("Please use either --agent or --argument")
+ exit 1
+end
+
+begin
+ options[:filter]["agent"] << options[:agent]
+
+ client = MCollective::Client.new(options[:config])
+ client.options = options
+
+ c = 0
+
+ stats = client.discovered_req(options[:argument], options[:agent]) do |resp|
+ next if resp == nil
+
+ c += 1
+
+ if options[:verbose]
+ puts("#{resp[:senderid]}>")
+ pp resp[:body]
+ else
+ puts if c % 4 == 1
+ printf("%-30s", resp[:senderid])
+ end
+ end
+
+ client.disconnect
+rescue Exception => e
+ STDERR.puts "Could not call remote agent: #{e}"
+ exit 1
+end
+
+client.display_stats(stats)
--- /dev/null
+#!/usr/bin/env ruby
+
+require 'mcollective'
+
+Version = MCollective.version
+known_applications = MCollective::Applications.list
+
+# links from mc-ping to mc will result in ping being run
+if $0 =~ /mc\-([a-zA-Z\-_\.]+)$/
+ app_name = $1
+else
+ app_name = ARGV.first
+ ARGV.delete_at(0)
+end
+
+if known_applications.include?(app_name)
+ # make sure the various options classes shows the right help etc
+ $0 = app_name
+
+ MCollective::Applications.run(app_name)
+else
+ puts "The Marionette Collective version #{MCollective.version}"
+ puts
+ puts "usage: #{$0} command <options>"
+ puts
+ puts "Known commands:"
+ puts
+
+ known_applications.sort.uniq.in_groups_of(3) do |apps|
+ puts " %-20s %-20s %-20s" % [apps[0], apps[1], apps[2]]
+ end
+
+ puts
+ puts "Type '#{$0} help' for a detailed list of commands and '#{$0} help command'"
+ puts "to get detailed help for a command"
+ puts
+end
--- /dev/null
+#!/usr/bin/env ruby
+
+require 'mcollective'
+require 'getoptlong'
+
+opts = GetoptLong.new(
+ [ '--help', '-h', GetoptLong::NO_ARGUMENT ],
+ [ '--config', '-c', GetoptLong::REQUIRED_ARGUMENT],
+ [ '--pidfile', '-p', GetoptLong::REQUIRED_ARGUMENT]
+)
+
+configfile = "/etc/mcollective/server.cfg"
+pid = ""
+
+opts.each do |opt, arg|
+ case opt
+ when '--help'
+ puts "Usage: mcollectived.rb [--config /path/to/config] [--pidfile /path/to/pid]"
+ exit
+ when '--config'
+ configfile = arg
+ when '--pidfile'
+ pid = arg
+ end
+end
+
+config = MCollective::Config.instance
+
+config.loadconfig(configfile) unless config.configured
+
+MCollective::Log.info("The Marionette Collective #{MCollective::VERSION} started logging at #{config.loglevel} level")
+
+if config.daemonize
+ MCollective::Log.debug("Starting in the background (#{config.daemonize})")
+
+ if MCollective::Util.windows?
+ require 'mcollective/windows_daemon'
+
+ MCollective::WindowsDaemon.daemonize_runner
+ else
+ require 'mcollective/unix_daemon'
+
+ MCollective::UnixDaemon.daemonize_runner(pid)
+ end
+else
+ MCollective::Log.debug("Starting in the foreground")
+ runner = MCollective::Runner.new(configfile)
+ runner.run
+end
--- /dev/null
+mcollective (2.3.1~mos6.1+1) trusty; urgency=low
+
+ * Pick the sources from packages/precise/mcollective.
+ * Fix dependency on rubygems (which is buiting in ruby >= 1.9).
+ * Install *.rb files into the correct directory.
+
+ -- Alexei Sheplyakov <asheplaykov@mirantis.com> Wed, 07 Jan 2015 11:32:22 +0000
--- /dev/null
+Source: mcollective
+Section: utils
+Priority: extra
+Maintainer: Riccardo Setti <giskard@debian.org>
+Build-Depends: debhelper (>= 7), dpatch, cdbs
+Standards-Version: 3.8.0
+Homepage: http://marionette-collective.org/
+
+Package: mcollective
+Architecture: all
+Depends: ruby (>= 1.8.1), mcollective-common (= ${source:Version})
+Description: build server orchestration or parallel job execution systems
+ The Marionette Collective aka. mcollective is a framework
+ to build server orchestration or parallel job execution systems.
+
+Package: mcollective-client
+Architecture: all
+Depends: ruby (>= 1.8.1), mcollective-common (= ${source:Version})
+Description: build server orchestration or parallel job execution systems
+ The Marionette Collective aka. mcollective is a framework
+ to build server orchestration or parallel job execution system
+
+Package: mcollective-common
+Replaces: mcollective (<< 2.0.0-1)
+Breaks: mcollective (<< 2.0.0-1), mcollective-client (<< 2.0.0-1)
+Architecture: all
+Depends: ruby (>= 1.8.1), rubygems | ruby (>= 1.9.1)
+Description: build server orchestration or parallel job execution systems
+ The Marionette Collective aka. mcollective is a framework
+ to build server orchestration or parallel job execution systems.
+ .
+ Common files for mcollective packages.
+
+#Package: mcollective-doc
+#Architecture: all
+#Section: doc
+#Description: Documentation for mcollective
+# The Marionette Collective aka. mcollective is a framework
+# to build server orchestration or parallel job execution systems.
+# .
+# Documentation package.
+
--- /dev/null
+This package was debianized by Riccardo Setti <giskard@debian.org> on
+Mon, 04 Jan 2010 17:09:50 +0000.
+
+It was downloaded from http://code.google.com/p/mcollective
+
+Upstream Author:
+ R.I.Pienaar <rip@devco.net>
+
+Copyright:
+
+ Copyright 2009 R.I.Pienaar <rip@devco.net>
+
+License:
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+The Debian packaging is (C) 2010, Riccardo Setti <giskard@debian.org> and
+is licensed under the Apache License v2.
+
--- /dev/null
+usr/bin/mco usr/bin/
+usr/sbin/mc-* usr/sbin/
+etc/mcollective/client.cfg etc/mcollective
+usr/share/mcollective/plugins/mcollective/application usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/aggregate usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/pluginpackager usr/share/mcollective/plugins/mcollective
--- /dev/null
+usr/lib/ruby/1.8/* usr/lib/ruby/vendor_ruby
+etc/mcollective/*.erb etc/mcollective
+usr/share/mcollective/plugins/mcollective/agent usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/audit usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/connector usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/data usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/facts usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/registration usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/security usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/validator usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/discovery usr/share/mcollective/plugins/mcollective
--- /dev/null
+usr/share/doc/mcollective/* usr/share/doc/mcollective-doc/
--- /dev/null
+#!/bin/sh
+#
+# mcollective Application Server for STOMP based agents
+#
+#
+# description: mcollective lets you build powerful Stomp compatible middleware clients in ruby without having to worry too
+# much about all the setup and management of a Stomp connection, it also provides stats, logging and so forth
+# as a bonus.
+#
+### BEGIN INIT INFO
+# Provides: mcollective
+# Required-Start: $remote_fs
+# Required-Stop: $remote_fs
+# Default-Start: 2 3 4 5
+# Default-Stop: 0 1 6
+# Short-Description: Start daemon at boot time
+# Description: Enable service provided by mcollective.
+### END INIT INFO
+
+# check permissions
+
+uid=`id -u`
+[ $uid -gt 0 ] && { echo "You need to be root to run file" ; exit 4 ; }
+
+
+# PID directory
+pidfile="/var/run/mcollective.pid"
+
+name="mcollective"
+mcollectived=/usr/sbin/mcollectived
+daemonopts="--config=/etc/mcollective/server.cfg"
+
+
+# Source function library.
+. /lib/lsb/init-functions
+
+# Check that binary exists
+if ! [ -f $mcollectived ]
+then
+ echo "mcollectived binary not found"
+ exit 5
+fi
+
+# See how we were called.
+case "$1" in
+ start)
+ echo "Starting daemon: " $name
+ # start the program
+ if [ -f $pidfile ]; then
+ if [ -f $(cat /proc/$(cat $pidfile)/exe > /dev/null) ] ; then
+ echo MCollective appears to be running
+ exit 1
+ else
+ /sbin/start-stop-daemon --start -b --quiet --oknodo -m --pidfile $pidfile --exec $mcollectived -- $daemonopts
+ [ $? = 0 ] && { exit 0 ; } || { exit 1 ; }
+ fi
+ else
+ /sbin/start-stop-daemon --start -b --quiet --oknodo -m --pidfile $pidfile --exec $mcollectived -- $daemonopts
+ fi
+ log_success_msg "mcollective started"
+ ;;
+ stop)
+ echo "Stopping daemon: " $name
+ /sbin/start-stop-daemon --stop -q --pidfile $pidfile
+ if [ -f $pidfile ]; then
+ rm -f $pidfile
+ fi
+ [ $? = 0 ] && { exit 0 ; } || { exit 1 ; }
+ log_success_msg "mcollective stopped"
+ ;;
+ restart)
+ echo "Restarting daemon: " $name
+ $0 stop
+ sleep 2
+ $0 start
+ [ $? = 0 ] && { echo "mcollective restarted" ; exit 0 ; }
+ ;;
+ status)
+ status_of_proc -p ${pidfile} ${mcollectived} ${name} && exit 0 || exit $?
+ ;;
+ *)
+ echo "Usage: mcollectived {start|stop|restart|condrestart|status}"
+ exit 2
+ ;;
+esac
--- /dev/null
+usr/sbin/mcollectived usr/sbin
+etc/mcollective/facts.yaml etc/mcollective
+etc/mcollective/server.cfg etc/mcollective
+etc/init.d etc/
+etc/mcollective/ssl
--- /dev/null
+pluginsdir.dpatch
--- /dev/null
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## conffile.dpatch by <giskard@debian.org
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: default config options
+
+@DPATCH@
+diff -urNad mcollective-0.4.1~/etc/client.cfg mcollective-0.4.1/etc/client.cfg
+--- mcollective-0.4.1~/etc/client.cfg 1970-01-01 00:00:00.000000000 +0000
++++ mcollective-0.4.1/etc/client.cfg 2010-01-06 14:37:24.000000000 +0000
+@@ -0,0 +1,19 @@
++libdir = /usr/share/mcollective/plugins
++logger_type = console
++loglevel = warn
++
++# Plugins
++securityprovider = psk
++plugin.psk = unset
++
++connector = activemq
++plugin.activemq.pool.size = 1
++plugin.activemq.pool.1.host = stomp1
++plugin.activemq.pool.1.port = 6163
++plugin.activemq.pool.1.user = mcollective
++plugin.activemq.pool.1.password = marionette
++
++# Facts
++factsource = yaml
++plugin.yaml = /etc/mcollective/facts.yaml
++
+diff -urNad mcollective-0.4.1~/etc/client.cfg.dist mcollective-0.4.1/etc/client.cfg.dist
+--- mcollective-0.4.1~/etc/client.cfg.dist 2010-01-02 14:54:25.000000000 +0000
++++ mcollective-0.4.1/etc/client.cfg.dist 1970-01-01 00:00:00.000000000 +0000
+@@ -1,19 +0,0 @@
+-libdir = /usr/libexec/mcollective
+-logger_type = console
+-loglevel = warn
+-
+-# Plugins
+-securityprovider = psk
+-plugin.psk = unset
+-
+-connector = stomp
+-plugin.stomp.host = stomp.your.com
+-plugin.stomp.port = 6163
+-plugin.stomp.user = mcollective
+-plugin.stomp.password = marionette
+-
+-# Facts
+-factsource = yaml
+-plugin.yaml = /etc/mcollective/facts.yaml
+-
+diff -urNad mcollective-0.4.1~/etc/facts.yaml mcollective-0.4.1/etc/facts.yaml
+--- mcollective-0.4.1~/etc/facts.yaml 1970-01-01 00:00:00.000000000 +0000
++++ mcollective-0.4.1/etc/facts.yaml 2010-01-06 14:37:27.000000000 +0000
+@@ -0,0 +1,2 @@
++---
++mcollective: 1
+diff -urNad mcollective-0.4.1~/etc/facts.yaml.dist mcollective-0.4.1/etc/facts.yaml.dist
+--- mcollective-0.4.1~/etc/facts.yaml.dist 2010-01-02 14:54:25.000000000 +0000
++++ mcollective-0.4.1/etc/facts.yaml.dist 1970-01-01 00:00:00.000000000 +0000
+@@ -1,2 +0,0 @@
+----
+-mcollective: 1
+diff -urNad mcollective-0.4.1~/etc/server.cfg mcollective-0.4.1/etc/server.cfg
+--- mcollective-0.4.1~/etc/server.cfg 1970-01-01 00:00:00.000000000 +0000
++++ mcollective-0.4.1/etc/server.cfg 2010-01-06 14:37:50.000000000 +0000
+@@ -0,0 +1,20 @@
++libdir = /usr/share/mcollective/plugins
++logfile = /var/log/mcollective.log
++loglevel = info
++daemonize = 1
++
++# Plugins
++securityprovider = psk
++plugin.psk = unset
++
++connector = activemq
++plugin.activemq.pool.size = 1
++plugin.activemq.pool.1.host = stomp1
++plugin.activemq.pool.1.port = 6163
++plugin.activemq.pool.1.user = mcollective
++plugin.activemq.pool.1.password = marionette
++
++# Facts
++factsource = yaml
++plugin.yaml = /etc/mcollective/facts.yaml
++
+diff -urNad mcollective-0.4.1~/etc/server.cfg.dist mcollective-0.4.1/etc/server.cfg.dist
+--- mcollective-0.4.1~/etc/server.cfg.dist 2010-01-02 14:54:25.000000000 +0000
++++ mcollective-0.4.1/etc/server.cfg.dist 1970-01-01 00:00:00.000000000 +0000
+@@ -1,20 +0,0 @@
+-libdir = /usr/libexec/mcollective
+-logfile = /var/log/mcollective.log
+-loglevel = info
+-daemonize = 1
+-
+-# Plugins
+-securityprovider = psk
+-plugin.psk = unset
+-
+-connector = stomp
+-plugin.stomp.host = stomp.your.com
+-plugin.stomp.port = 6163
+-plugin.stomp.user = mcollective
+-plugin.stomp.password = marionette
+-
+-# Facts
+-factsource = yaml
+-plugin.yaml = /etc/mcollective/facts.yaml
+-
--- /dev/null
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## initlsb.dpatch by <giskard@debian.org
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: Init func.
+
+@DPATCH@
+diff -urNad mcollective-0.4.1~/mcollective.init mcollective-0.4.1/mcollective.init
+--- mcollective-0.4.1~/mcollective.init 2010-01-02 14:54:25.000000000 +0000
++++ mcollective-0.4.1/mcollective.init 2010-01-07 10:52:38.000000000 +0000
+@@ -8,6 +8,15 @@
+ # much about all the setup and management of a Stomp connection, it also provides stats, logging and so forth
+ # as a bonus.
+ #
++### BEGIN INIT INFO
++# Provides: scriptname
++# Required-Start: $remote_fs $syslog
++# Required-Stop: $remote_fs $syslog
++# Default-Start: 2 3 4 5
++# Default-Stop: 0 1 6
++# Short-Description: Start daemon at boot time
++# Description: Enable service provided by daemon.
++### END INIT INFO
+
+ RUBYLIB=/usr/local/lib/site_ruby/1.8:$RUBYLIB
+ export RUBYLIB
+@@ -104,7 +113,10 @@
+ echo "mcollectived: service not started"
+ exit 1
+ fi
+- ;;
++ ;;
++ force-reload)
++ echo "not implemented"
++ ;;
+ *)
+ echo "Usage: mcollectived {start|stop|restart|condrestart|status}"
+ exit 1
--- /dev/null
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## makefile.dpatch by <giskard@debian.org
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: Add Makefile for install task
+
+@DPATCH@
+diff -urNad mcollective-0.4.1~/Makefile mcollective-0.4.1/Makefile
+--- mcollective-0.4.1~/Makefile 1970-01-01 00:00:00.000000000 +0000
++++ mcollective-0.4.1/Makefile 2010-01-06 13:43:56.000000000 +0000
+@@ -0,0 +1,38 @@
++#!/usr/bin/make -f
++
++DESTDIR=
++
++clean:
++
++install: install-bin install-lib install-conf install-plugins install-doc
++
++install-bin:
++ install -d $(DESTDIR)/usr/sbin
++ cp mc-* $(DESTDIR)/usr/sbin
++ cp mcollectived.rb $(DESTDIR)/usr/sbin/mcollectived
++
++install-lib:
++ install -d $(DESTDIR)/usr/lib/ruby/1.8/
++ cp -a lib/* $(DESTDIR)/usr/lib/ruby/1.8/
++
++install-conf:
++ install -d $(DESTDIR)/etc/mcollective/
++ install -d $(DESTDIR)/etc/init.d
++ cp etc/* $(DESTDIR)/etc/mcollective/
++ cp mcollective.init $(DESTDIR)/etc/init.d/mcollective
++
++install-plugins:
++ install -d $(DESTDIR)/usr/share/mcollective/
++ cp -a plugins $(DESTDIR)/usr/share/mcollective/
++
++install-doc:
++ install -d $(DESTDIR)/usr/share/doc/
++ cp -a doc $(DESTDIR)/usr/share/doc/mcollective
++
++uninstall:
++ rm -f $(DESTDIR)/usr/sbin/mcollectived
++ rm -rf $(DESTDIR)/usr/lib/ruby/1.8/mcollective*
++ rm -rf $(DESTDIR)/usr/share/mcollective
++ rm -rf $(DESTDIR)/etc/mcollective
++
++.PHONY: clean install uninstall
--- /dev/null
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## pluginsdir.dpatch by <gisakrd@debian.org>
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: fix plugins dir
+
+@DPATCH@
+diff -urNad mcollective-1.1.4~/etc/client.cfg.dist mcollective-1.1.4/etc/client.cfg.dist
+--- mcollective-1.1.4~/etc/client.cfg.dist 2011-04-06 14:13:08.829462165 -0700
++++ mcollective-1.1.4/etc/client.cfg.dist 2011-04-06 14:12:53.129384114 -0700
+@@ -1,6 +1,6 @@
+ main_collective = mcollective
+ collectives = mcollective
+-libdir = /usr/libexec/mcollective
++libdir = /usr/share/mcollective/plugins
+ logger_type = console
+ loglevel = warn
+
+diff -urNad mcollective-1.1.4~/etc/server.cfg.dist mcollective-1.1.4/etc/server.cfg.dist
+--- mcollective-1.1.4~/etc/server.cfg.dist 2011-04-06 14:12:30.889527230 -0700
++++ mcollective-1.1.4/etc/server.cfg.dist 2011-04-06 14:12:23.779407065 -0700
+@@ -1,6 +1,6 @@
+ main_collective = mcollective
+ collectives = mcollective
+-libdir = /usr/libexec/mcollective
++libdir = /usr/share/mcollective/plugins
+ logfile = /var/log/mcollective.log
+ loglevel = info
+ daemonize = 1
+
--- /dev/null
+#!/usr/bin/make -f
+
+DEB_MAKE_CLEAN_TARGET :=
+DEB_MAKE_INSTALL_TARGET := install DESTDIR=$(CURDIR)/debian/tmp
+
+include /usr/share/cdbs/1/rules/debhelper.mk
+include /usr/share/cdbs/1/rules/dpatch.mk
+include /usr/share/cdbs/1/class/makefile.mk
+DEB_MAKE_INVOKE = $(DEB_MAKE_ENVVARS) make -f ext/Makefile -C $(DEB_BUILDDIR)
+
+install/mcollective::
+ mv $(CURDIR)/debian/tmp/etc/mcollective/server.cfg.dist $(CURDIR)/debian/tmp/etc/mcollective/server.cfg
+ mv $(CURDIR)/debian/tmp/etc/mcollective/client.cfg.dist $(CURDIR)/debian/tmp/etc/mcollective/client.cfg
+ mv $(CURDIR)/debian/tmp/etc/mcollective/facts.yaml.dist $(CURDIR)/debian/tmp/etc/mcollective/facts.yaml
+# dh_installinit -pmcollective -o
+
+binary-fixup/mcollective::
+ chmod 640 $(CURDIR)/debian/mcollective/etc/mcollective/server.cfg
--- /dev/null
+main_collective = mcollective
+collectives = mcollective
+libdir = /usr/libexec/mcollective
+logger_type = console
+loglevel = warn
+
+# Plugins
+securityprovider = psk
+plugin.psk = unset
+
+connector = activemq
+plugin.activemq.pool.size = 1
+plugin.activemq.pool.1.host = stomp1
+plugin.activemq.pool.1.port = 6163
+plugin.activemq.pool.1.user = mcollective
+plugin.activemq.pool.1.password = marionette
+
+# Facts
+factsource = yaml
+plugin.yaml = /etc/mcollective/facts.yaml
+
--- /dev/null
+<%= metastring %>
+QUERY FUNCTION INPUT:
+
+% if entities[:data][:input][:query]
+ Description: <%= entities[:data][:input][:query][:description] %>
+ Prompt: <%= entities[:data][:input][:query][:prompt] %>
+ Type: <%= entities[:data][:input][:query][:type] %>
+% if entities[:data][:input][:query][:type] == :string
+ Validation: <%= entities[:data][:input][:query][:validation] %>
+ Length: <%= entities[:data][:input][:query][:maxlength] %>
+% elsif entities[:data][:input][:query][:type] == :list
+ Valid Values: <%= entities[:data][:input][:query][:list].join(", ") %>
+% end
+% if entities[:data][:input][:query][:default]
+ Default Value: <%= entities[:data][:input][:query][:default] %>
+% end
+% else
+ This plugin does not take any input
+% end
+
+QUERY FUNCTION OUTPUT:
+
+% entities[:data][:output].keys.sort.each do |output|
+ <%= output %>:
+ Description: <%= entities[:data][:output][output][:description] %>
+ Display As: <%= entities[:data][:output][output][:display_as] %>
+
+% end
--- /dev/null
+<%= metastring %>
+DISCOVERY METHOD CAPABILITIES:
+% [["Filter based on configuration management classes", :classes],
+% ["Filter based on system facts", :facts],
+% ["Filter based on mcollective identity", :identity],
+% ["Filter based on mcollective agents", :agents],
+% ["Compound filters combining classes and facts", :compound]].each do |cap|
+% if entities[:discovery][:capabilities].include?(cap.last)
+ <%= cap.first %>
+% end
+% end
+
--- /dev/null
+---
+mcollective: 1
--- /dev/null
+<%= meta[:name] %>
+<%= "=" * meta[:name].size %>
+
+<%= meta[:description] %>
+
+ Author: <%= meta[:author] %>
+ Version: <%= meta[:version] %>
+ License: <%= meta[:license] %>
+ Timeout: <%= meta[:timeout] %>
+ Home Page: <%= meta[:url] %>
+% if requirements[:mcollective]
+
+ Requires MCollective <%= requirements[:mcollective] %> or newer
+% end
+% unless @usage == ""
+
+<%= Util.align_text(@usage, nil, 3) if @usage != "" %>
+% end
--- /dev/null
+Help for message: <%= msg_code %>
+
+Example Message:
+
+<%= Util.align_text(msg_example, nil, 3) %>
+
+Message Detail:
+
+<%= Util.align_text(msg_detail, nil, 3) %>
+
+For more information please see http://docs.puppetlabs.com/mcollective/messages/<%= msg_code %>.html
--- /dev/null
+<%= metastring %>
+ACTIONS:
+========
+ <%= entities.keys.sort.join(", ") %>
+
+% entities.keys.sort.each do |action|
+ <%= action %> action:
+ <% (action.size + 8).times do %>-<% end %>
+ <%= entities[action][:description] %>
+
+ INPUT:
+% if entities[action][:input].size > 0
+% entities[action][:input].keys.sort.each do |input|
+ <%= input %>:
+ Description: <%= entities[action][:input][input][:description] %>
+ Prompt: <%= entities[action][:input][input][:prompt] %>
+ Type: <%= entities[action][:input][input][:type] %>
+ Optional: <%= !!entities[action][:input][input][:optional] %>
+% if entities[action][:input][input][:type] == :string
+ Validation: <%= entities[action][:input][input][:validation] %>
+ Length: <%= entities[action][:input][input][:maxlength] %>
+% elsif entities[action][:input][input][:type] == :list
+ Valid Values: <%= entities[action][:input][input][:list].join(", ") %>
+% end
+% if entities[action][:input][input][:default]
+ Default Value: <%= entities[action][:input][input][:default] %>
+% end
+
+% end
+% else
+ This action does not have any inputs
+% end
+
+ OUTPUT:
+% entities[action][:output].keys.sort.each do |output|
+ <%= output %>:
+ Description: <%= entities[action][:output][output][:description] %>
+ Display As: <%= entities[action][:output][output][:display_as] %>
+% if entities[action][:output][output][:default]
+ Default Value: <%= entities[action][:output][output][:default] %>
+% end
+
+% end
+% end
--- /dev/null
+main_collective = mcollective
+collectives = mcollective
+libdir = /usr/libexec/mcollective
+logfile = /var/log/mcollective.log
+loglevel = info
+daemonize = 1
+
+# Plugins
+securityprovider = psk
+plugin.psk = unset
+
+connector = activemq
+plugin.activemq.pool.size = 1
+plugin.activemq.pool.1.host = stomp1
+plugin.activemq.pool.1.port = 6163
+plugin.activemq.pool.1.user = mcollective
+plugin.activemq.pool.1.password = marionette
+
+# Facts
+factsource = yaml
+plugin.yaml = /etc/mcollective/facts.yaml
+
--- /dev/null
+#!/usr/bin/make -f
+
+DESTDIR=
+
+build:
+
+clean:
+
+install: install-bin install-lib install-conf install-plugins install-doc
+
+install-bin:
+ install -d $(DESTDIR)/usr/sbin
+ install -d $(DESTDIR)/usr/bin
+ cp bin/mc-* $(DESTDIR)/usr/sbin
+ cp bin/mco $(DESTDIR)/usr/bin
+ cp bin/mcollectived $(DESTDIR)/usr/sbin/mcollectived
+
+install-lib:
+ install -d $(DESTDIR)/usr/lib/ruby/1.8/
+ cp -a lib/* $(DESTDIR)/usr/lib/ruby/1.8/
+
+install-conf:
+ install -d $(DESTDIR)/etc/mcollective/
+ install -d $(DESTDIR)/etc/init.d
+ cp -r etc/* $(DESTDIR)/etc/mcollective/
+ cp mcollective.init $(DESTDIR)/etc/init.d/mcollective
+ rm $(DESTDIR)/etc/mcollective/ssl/PLACEHOLDER
+ rm $(DESTDIR)/etc/mcollective/ssl/clients/PLACEHOLDER
+
+install-plugins:
+ install -d $(DESTDIR)/usr/share/mcollective/
+ cp -a plugins $(DESTDIR)/usr/share/mcollective/
+
+install-doc:
+ install -d $(DESTDIR)/usr/share/doc/
+ cp -a doc $(DESTDIR)/usr/share/doc/mcollective
+
+uninstall:
+ rm -f $(DESTDIR)/usr/sbin/mcollectived
+ rm -rf $(DESTDIR)/usr/lib/ruby/1.8/mcollective*
+ rm -rf $(DESTDIR)/usr/share/mcollective
+ rm -rf $(DESTDIR)/etc/mcollective
+
+.PHONY: build clean install uninstall
--- /dev/null
+Makefile
+blib
+pm_to_blib
--- /dev/null
+#!perl
+use strict;
+use ExtUtils::MakeMaker;
+WriteMakefile(
+ NAME => "MCollective::Action",
+ PREREQ_PM => {
+ "JSON" => 0,
+ },
+);
--- /dev/null
+package MCollective::Action;
+use strict;
+use warnings;
+use JSON;
+
+=head1 NAME
+
+MCollective::Action - helper class for writing mcollective actions in perl
+
+=head1 SYNOPSIS
+
+
+In your mcollective agent
+
+ action "echo" do
+ validate :message, String
+
+ implemented by "/tmp/echo.perl"
+ end
+
+And C</tmp/echo.perl>
+
+ #!/usr/bin/env perl
+ use strict;
+ use MCollective::Action;
+
+ my $mc = MCollective::Action->new;
+ $mc->reply->{message} = $mc->request->{message};
+ $mc->reply->{timestamp} = time;
+ $mc->info("some text to log on the server");
+
+
+=head1 DESCRIPTION
+
+mcollective version 1.X introduced a mechanism for writing agent actions as
+external commands. This module provides a convenient api for writing them in
+perl which performs some of the boilerplate for you.
+
+=head2 METHODS
+
+=over
+
+=item new
+
+create a new MCollection::Action helper object
+
+=cut
+
+sub new {
+ my $class = shift;
+ my $self = bless {
+ request => {},
+ reply => {},
+ }, $class;
+ $self->_load;
+ return $self;
+}
+
+=item request
+
+returns a hash reference containing the request
+
+=cut
+
+
+sub request { $_[0]->{request} }
+
+
+=item reply
+
+returns a hash reference you should populate with your reply
+
+=cut
+
+sub reply { $_[0]->{reply} }
+
+
+sub _load {
+ my $self = shift;
+ my $file = $ENV{MCOLLECTIVE_REQUEST_FILE};
+ open my $fh, "<$file"
+ or die "Can't open '$file': $!";
+ my $json = do { local $/; <$fh> };
+ $self->{request} = JSON->new->decode( $json );
+ delete $self->request->{data}{process_results};
+}
+
+sub DESTROY {
+ my $self = shift;
+ $self->_save;
+}
+
+sub _save {
+ my $self = shift;
+ my $file = $ENV{MCOLLECTIVE_REPLY_FILE};
+ open my $fh, ">$file"
+ or die "Can't open '$file': $!";
+ print $fh JSON->new->encode( $self->reply );
+}
+
+=item info($message)
+
+report a message into the server log
+
+=cut
+
+sub info {
+ my ($self, $message) = @_;
+ print STDOUT $message, "\n";
+}
+
+=item error($message)
+
+report an error into the server log
+
+=cut
+
+
+sub error {
+ my ($self, $message) = @_;
+ print STDERR $message, "\n";
+}
+
+=item fail
+
+reports an error and exits immediately
+
+=cut
+
+sub fail {
+ my ($self, $message) = @_;
+ $self->error( $message );
+ exit 1;
+}
+
+1;
+
+__END__
+
+=back
+
+=head1 AUTHOR
+
+Richard Clamp <richardc@unixbeard.net>
+
+=head1 COPYRIGHT
+
+Copyright 2011, Richard Clamp. All Rights Reserved.
+
+This program is free software; you can redistribute it
+and/or modify it under the same terms as Perl itself.
+
+=head1 SEE ALSO
+
+http://docs.puppetlabs.com/mcollective/
+
+=cut
+
--- /dev/null
+#!perl
+use strict;
+use Test::More;
+use JSON;
+use File::Temp;
+
+my $class = "MCollective::Action";
+use_ok( $class );
+
+my $infile = File::Temp->new;
+my $outfile = File::Temp->new;
+
+$ENV{MCOLLECTIVE_REQUEST_FILE} = $infile->filename;
+$ENV{MCOLLECTIVE_REPLY_FILE} = $outfile->filename;
+print $infile JSON->new->encode({ red => "apples", blue => "moon" });
+close $infile;
+{
+ my $mc = $class->new;
+ isa_ok( $mc, $class );
+ is( $mc->request->{red}, "apples", "apples are red" );
+ $mc->reply->{potato} = "chips";
+}
+
+my $json = do { local $/; <$outfile> };
+ok( $json, "Got some JSON" );
+my $reply = JSON->new->decode( $json );
+
+is( $reply->{potato}, "chips", "Got the reply that potato = chips" );
+
+done_testing();
--- /dev/null
+A simple helper to assist with writing MCollective actions in PHP.
+
+Given an action as below:
+
+<pre>
+action "echo" do
+ validate :message, String
+
+ implemented_by "/tmp/echo.php"
+end
+</pre>
+
+The following PHP script will implement the echo action externally
+replying with _message_ and _timestamp_
+
+<pre>
+<?php
+ require("mcollective_action.php");
+
+ $mc = new MCollectiveAction();
+ $mc->message = $mc->data["message"];
+ $mc->timestamp = strftime("%c");
+ $mc->info("some text to info log on the server");
+?>
+</pre>
+
+Calling it with _mco rpc_ results in:
+
+<pre>
+$ mco rpc test echo message="hello world"
+Determining the amount of hosts matching filter for 2 seconds .... 1
+
+ * [ ============================================================> ] 1 / 1
+
+
+nephilim.ml.org : OK
+ {:message=>"hello world", :time=>"Tue Mar 15 19:20:53 +0000 2011"}
+</pre>
--- /dev/null
+<?php
+class MCollectiveAction {
+ public $infile = "";
+ public $outfile = "";
+ public $request = array();
+
+ function __construct() {
+ if (!isSet($_ENV["MCOLLECTIVE_REQUEST_FILE"])) {
+ throw new Exception("no MCOLLECTIVE_REQUEST_FILE environment variable");
+ }
+
+ if (!isSet($_ENV["MCOLLECTIVE_REPLY_FILE"])) {
+ throw new Exception("no MCOLLECTIVE_REPLY_FILE environment variable");
+ }
+
+ $this->infile = $_ENV["MCOLLECTIVE_REQUEST_FILE"];
+ $this->outfile = $_ENV["MCOLLECTIVE_REPLY_FILE"];
+
+ $this->readJSON();
+ }
+
+ function __destruct() {
+ $this->save();
+ }
+
+ function readJSON() {
+ $this->request = json_decode(file_get_contents($this->infile), true);
+ unset($this->request["data"]["process_results"]);
+ }
+
+ function save() {
+ file_put_contents($this->outfile, json_encode($this->request["data"]));
+ }
+
+ // prints a line to STDERR that will log at error level in the
+ // mcollectived log file
+ function error($msg) {
+ fwrite(STDERR, "$msg\n");
+ }
+
+ // prints a line to STDOUT that will log at info level in the
+ // mcollectived log file
+ function info($msg) {
+ fwrite(STDOUT, "$msg\n");
+ }
+
+ // logs an error message and exits with RPCAborted
+ function fail($msg) {
+ $this->error($msg);
+ exit(1);
+ }
+
+ function __get($property) {
+ if (isSet($this->request[$property])) {
+ return $this->request[$property];
+ } else {
+ throw new Exception("No $property in request");
+ }
+ }
+
+ function __set($property, $value) {
+ $this->request["data"][$property] = $value;
+ }
+}
+?>
--- /dev/null
+A simple helper to assist with writing MCollective actions in Python.
+
+Given an action as below:
+
+<pre>
+action "echo" do
+ validate :message, String
+
+ implemented_by "/tmp/echo.py"
+end
+</pre>
+
+The following Python script will implement the echo action externally
+replying with _message_ and current _time_.
+
+<pre>
+#!/usr/bin/env python
+
+import sys
+import time
+import mcollective_action as mc
+
+if __name__ == '__main__':
+ mc = mc.MCollectiveAction()
+ request = mc.request()
+ mc.message = request['data']['message']
+ mc.time = time.strftime('%c')
+ mc.info("Some text to info log in the server")
+
+ sys.exit(0)
+</pre>
+
+Calling it with _mco rpc_ results in:
+
+<pre>
+$ mco rpc test echo message="Hello World"
+Determining the amount of hosts matching filter for 2 seconds .... 1
+
+ * [ ============================================================> ] 1 / 1
+
+
+host.example.com : OK
+ {:message=>"Hello World", :time=>"Tue Mar 15 19:20:53 +0000 2011"}
+</pre>
+
+This implementation was successfully tested with Python 2.4 and 2.6.
--- /dev/null
+#!/usr/bin/env python
+
+import sys
+import time
+import mcollective_action as mc
+
+if __name__ == '__main__':
+ mc = mc.MCollectiveAction()
+ request = mc.request()
+ mc.message = request['data']['message']
+ mc.time = time.strftime('%c')
+ mc.info("An example echo agent")
+
+ sys.exit(0)
+
+# vim: set ts=4 sw=4 et :
--- /dev/null
+#!/usr/bin/env python
+
+import os
+import sys
+
+class Error(Exception):
+ pass
+
+class MissingModule(Error):
+ pass
+
+class MissingFiles(Error):
+ pass
+
+class MissingEnvironemntVariable(Error):
+ pass
+
+class FileReadError(Error):
+ pass
+
+class JSONParsingError(Error):
+ pass
+
+try:
+ import simplejson as json
+except ImportError:
+ raise MissingModule('Unable to load JSON module. Missing module?')
+
+class MCollectiveAction(object):
+
+ _environment_variables = [ 'MCOLLECTIVE_REQUEST_FILE',
+ 'MCOLLECTIVE_REPLY_FILE' ]
+
+ def __init__(self):
+ self._info = sys.__stdout__
+ self._error = sys.__stderr__
+
+ for entry in '_reply', '_request':
+ self.__dict__[entry] = {}
+
+ self._arguments = sys.argv[1:]
+
+ if len(self._arguments) < 2:
+ try:
+ for variable in self._environment_variables:
+ self._arguments.append(os.environ[variable])
+ except KeyError:
+ raise MissingEnvironemntVariable("Environment variable `%s' "
+ "is not set." % variable)
+
+ self._request_file, self._reply_file = self._arguments
+
+ if len(self._request_file) == 0 or len(self._reply_file) == 0:
+ raise MissingFiles("Both request and reply files have to be set.")
+
+ def __setattr__(self, name, value):
+ if name.startswith('_'):
+ object.__setattr__(self, name, value)
+ else:
+ self.__dict__['_reply'][name] = value
+
+ def __getattr__(self, name):
+ if name.startswith('_'):
+ return self.__dict__.get(name, None)
+ else:
+ return self.__dict__['_reply'].get(name, None)
+
+ def __del__(self):
+ if self._reply:
+ try:
+ file = open(self._reply_file, 'w')
+ json.dump(self._reply, file)
+ file.close()
+ except IOError, error:
+ raise FileReadError("Unable to open reply file `%s': %s" %
+ (self._reply_file, error))
+
+ def info(self, message):
+ print >> self._info, message
+ self._info.flush()
+
+ def error(self, message):
+ print >> self._error, message
+ self._error.flush()
+
+ def fail(self, message, exit_code=1):
+ self.error(message)
+ sys.exit(exit_code)
+
+ def reply(self):
+ return self._reply
+
+ def request(self):
+ if self._request:
+ return self._request
+ else:
+ try:
+ file = open(self._request_file, 'r')
+ self._request = json.load(file)
+ file.close()
+ except IOError, error:
+ raise FileReadError("Unable to open request file `%s': %s" %
+ (self._request_file, error))
+ except json.JSONDecodeError, error:
+ raise JSONParsingError("An error occurred during parsing of "
+ "the JSON data in the file `%s': %s" %
+ (self._request_file, error))
+ file.close()
+
+ return self._request
+
+# vim: set ts=4 sw=4 et :
--- /dev/null
+A simple helper to assist with writing MCollective actions in Python.
+
+Given an action as below:
+
+<pre>
+action "echo" do
+ validate :message, String
+
+ implemented_by "/tmp/echo.py"
+end
+</pre>
+
+The following Python script will implement the echo action externally
+replying with _message_ and _timestamp_
+
+<pre>
+#!/bin/env python
+import mcollectiveah
+import time
+
+mc = mcollectiveah.MCollectiveAction()
+mc.reply['message'] = mc.request['message']
+mc.reply['timestamp'] = time.strftime("%c")
+mc.reply['info'] = "some text to info log in the server"
+</pre>
+
+Calling it with _mco rpc_ results in:
+
+<pre>
+$ mco rpc test echo message="hello world"
+Determining the amount of hosts matching filter for 2 seconds .... 1
+
+ * [ ============================================================> ] 1 / 1
+
+
+nephilim.ml.org : OK
+ {:message=>"hello world", :time=>"Tue Mar 15 19:20:53 +0000 2011"}
+</pre>
--- /dev/null
+#!/bin/env python
+# -*- coding: utf-8 -*- vim: set ts=4 et sw=4 fdm=indent :
+import os, sys
+
+try:
+ import simplejson
+except ImportError:
+ sys.stderr.write('Unable to load simplejson python module.')
+ sys.exit(1)
+
+class MCollectiveActionNoEnv(Exception):
+ pass
+class MCollectiveActionFileError(Exception):
+ pass
+
+class MCollectiveAction(object):
+ def __init__(self, *args, **kwargs):
+ try:
+ self.infile = os.environ['MCOLLECTIVE_REQUEST_FILE']
+ except KeyError:
+ raise MCollectiveActionNoEnv("No MCOLLECTIVE_REQUEST_FILE environment variable")
+ try:
+ self.outfile = os.environ['MCOLLECTIVE_REPLY_FILE']
+ except KeyError:
+ raise MCollectiveActionNoEnv("No MCOLLECTIVE_REPLY_FILE environment variable")
+
+ self.request = {}
+ self.reply = {}
+
+ self.load()
+
+ def load(self):
+ if not self.infile:
+ return False
+ try:
+ infile = open(self.infile, 'r')
+ self.request = simplejson.load(infile)
+ infile.close()
+ except IOError, e:
+ raise MCollectiveActionFileError("Could not read request file `%s`: %s" % (self.infile, e))
+ except simplejson.JSONDecodeError, e:
+ infile.close()
+ raise MCollectiveActionFileError("Could not parse JSON data in file `%s`: %s", (self.infile, e))
+
+ def send(self):
+ if not getattr(self, 'outfile', None): # if exception was raised during or before setting self.outfile
+ return False
+ try:
+ outfile = open(self.outfile, 'w')
+ simplejson.dump(self.reply, outfile)
+ outfile.close()
+ except IOError, e:
+ raise MCollectiveActionFileError("Could not write reply file `%s`: %s" % (self.outfile, e))
+
+ def error(self, msg):
+ """Prints line to STDERR that will be logged at error level in the mcollectived log file"""
+ sys.stderr.write("%s\n" % msg)
+
+ def fail(self, msg):
+ """Logs error message and exitst with RPCAborted"""
+ self.error(msg)
+ sys.exit(1)
+
+ def info(self, msg):
+ """Prints line to STDOUT that will be logged at info level in the mcollectived log file"""
+ sys.stdout.write("%s\n" % msg)
+
+ def __del__(self):
+ self.send()
--- /dev/null
+#!/bin/env python
+# -*- coding: utf-8 -*- vim: set ts=4 et sw=4 fdm=indent :
+import unittest, tempfile, simplejson, os, random
+
+import mcollectiveah
+
+class TestFunctions(unittest.TestCase):
+ def test_raise_environ(self):
+ try:
+ del os.environ['MCOLLECTIVE_REQUEST_FILE']
+ del os.environ['MCOLLECTIVE_REPLY_FILE']
+ except: pass
+ self.assertRaises(mcollectiveah.MCollectiveActionNoEnv, mcollectiveah.MCollectiveAction)
+
+ def test_raise_file_error(self):
+ os.environ['MCOLLECTIVE_REQUEST_FILE'] = '/tmp/mcollectiveah-test-request.%d' % random.randrange(100000)
+ os.environ['MCOLLECTIVE_REPLY_FILE'] = '/tmp/mcollectiveah-test-reply.%d' % random.randrange(100000)
+
+ self.assertRaises(mcollectiveah.MCollectiveActionFileError, mcollectiveah.MCollectiveAction)
+
+ os.unlink(os.environ['MCOLLECTIVE_REPLY_FILE'])
+
+ def test_echo(self):
+ tin = tempfile.NamedTemporaryFile(mode='w', delete=False)
+ self.data = {'message': 'test'}
+
+ simplejson.dump(self.data, tin)
+ os.environ['MCOLLECTIVE_REQUEST_FILE'] = tin.name
+ tin.close()
+
+ tout = tempfile.NamedTemporaryFile(mode='w')
+ os.environ['MCOLLECTIVE_REPLY_FILE'] = tout.name
+ tout.close()
+
+ mc = mcollectiveah.MCollectiveAction()
+ mc.reply['message'] = mc.request['message']
+ del mc
+
+ tout = open(os.environ['MCOLLECTIVE_REPLY_FILE'], 'r')
+ data = simplejson.load(tout)
+ tout.close()
+
+ self.assertEqual(data, self.data)
+
+
+ os.unlink(os.environ['MCOLLECTIVE_REQUEST_FILE'])
+ os.unlink(os.environ['MCOLLECTIVE_REPLY_FILE'])
+
+if __name__ == '__main__':
+ unittest.main()
--- /dev/null
+Summary: Apache ActiveMQ
+Name: activemq
+Version: 5.3.0
+Release: 1%{?dist}
+License: Apache
+Group: Network/Daemons
+Source0: apache-activemq-%{version}-bin.tar.gz
+Source1: wlcg-patch.tgz
+BuildRoot: %{_tmppath}/%{name}-%{version}-root
+BuildArch: noarch
+Requires: tanukiwrapper >= 3.2.0
+
+#%define buildver 5.1.0
+
+%define homedir /usr/share/%{name}
+%define libdir /var/lib/%{name}
+%define libexecdir /usr/libexec/%{name}
+%define cachedir /var/cache/%{name}
+%define docsdir /usr/share/doc/%{name}-%{version}
+
+%description
+ApacheMQ is a JMS Compliant Messaging System
+
+%package info-provider
+Summary: An LDAP information provider for activemq
+Group:grid/lcg
+%description info-provider
+An LDAP infomation provider for activemq
+
+%package meta
+Summary: A metapackage
+Group:grid/lcg
+Requires: activemq = ${version}-${release}, activemq-info-provider = ${version}-${release}
+%description meta
+A metapackage
+
+%prep
+%setup -q -a1 -n apache-activemq-%{version}
+
+%build
+install --directory ${RPM_BUILD_ROOT}
+
+%install
+rm -rf $RPM_BUILD_ROOT
+install --directory ${RPM_BUILD_ROOT}%{homedir}
+install --directory ${RPM_BUILD_ROOT}%{homedir}/bin
+install --directory ${RPM_BUILD_ROOT}%{docsdir}
+install --directory ${RPM_BUILD_ROOT}%{libdir}/lib
+install --directory ${RPM_BUILD_ROOT}%{libexecdir}
+install --directory ${RPM_BUILD_ROOT}%{libdir}/webapps
+install --directory ${RPM_BUILD_ROOT}%{cachedir}
+install --directory ${RPM_BUILD_ROOT}%{cachedir}/data
+install --directory ${RPM_BUILD_ROOT}/var/log/%{name}
+install --directory ${RPM_BUILD_ROOT}/var/run/%{name}
+install --directory ${RPM_BUILD_ROOT}/etc/%{name}
+install --directory ${RPM_BUILD_ROOT}/etc/init.d
+install --directory ${RPM_BUILD_ROOT}/etc/httpd/conf.d
+
+# Config files
+install conf/activemq.xml ${RPM_BUILD_ROOT}/etc/%{name}
+install conf/credentials.properties ${RPM_BUILD_ROOT}/etc/%{name}
+install conf/jetty.xml ${RPM_BUILD_ROOT}/etc/%{name}
+install conf/log4j.properties ${RPM_BUILD_ROOT}/etc/%{name}
+install conf/activemq-wrapper.conf ${RPM_BUILD_ROOT}/etc/%{name}
+install conf/activemq-httpd.conf ${RPM_BUILD_ROOT}/etc/httpd/conf.d
+
+# startup script
+install bin/activemq ${RPM_BUILD_ROOT}/etc/init.d
+
+# Bin and doc dirs
+install *.txt *.html ${RPM_BUILD_ROOT}%{docsdir}
+cp -r docs ${RPM_BUILD_ROOT}%{docsdir}
+
+install bin/run.jar bin/activemq-admin ${RPM_BUILD_ROOT}%{homedir}/bin
+install --directory ${RPM_BUILD_ROOT}/usr/bin
+%{__ln_s} -f %{homedir}/bin/activemq-admin ${RPM_BUILD_ROOT}/usr/bin
+
+# Runtime directory
+cp -r lib ${RPM_BUILD_ROOT}%{libdir}
+cp -r webapps/admin ${RPM_BUILD_ROOT}%{libdir}/webapps
+
+# Info provider
+install info-provider-activemq ${RPM_BUILD_ROOT}/%{libexecdir}
+
+pushd ${RPM_BUILD_ROOT}%{homedir}
+ [ -d conf ] || %{__ln_s} -f /etc/%{name} conf
+ [ -d data ] || %{__ln_s} -f %{cachedir}/data data
+ [ -d docs ] || %{__ln_s} -f %{docsdir} docs
+ [ -d lib ] || %{__ln_s} -f %{libdir}/lib lib
+ [ -d lib ] || %{__ln_s} -f %{libdir}/libexec libexec
+ [ -d log ] || %{__ln_s} -f /var/log/%{name} log
+ [ -d webapps ] || %{__ln_s} -f %{libdir}/webapps webapps
+popd
+
+
+%pre
+# Add the "activemq" user and group
+# we need a shell to be able to use su - later
+/usr/sbin/groupadd -g 92 -r activemq 2> /dev/null || :
+/usr/sbin/useradd -c "Apache Activemq" -u 92 -g activemq \
+ -s /bin/bash -r -d /usr/share/activemq activemq 2> /dev/null || :
+
+%post
+# install activemq (but don't activate)
+/sbin/chkconfig --add activemq
+
+%preun
+if [ $1 = 0 ]; then
+ [ -f /var/lock/subsys/activemq ] && /etc/init.d/activemq stop
+ [ -f /etc/init.d/activemq ] && /sbin/chkconfig --del activemq
+fi
+
+%postun
+
+%clean
+rm -rf $RPM_BUILD_ROOT
+
+
+%files
+%defattr(-,root,root)
+%attr(755,root,root) /usr/bin/activemq-admin
+%{homedir}
+%docdir %{docsdir}
+%{docsdir}
+%{libdir}
+%attr(775,activemq,activemq) %dir /var/log/%{name}
+%attr(775,activemq,activemq) %dir /var/run/%{name}
+%attr(775,root,activemq) %dir %{cachedir}/data
+%attr(755,root,root) /etc/init.d/activemq
+%config(noreplace) /etc/httpd/conf.d/activemq-httpd.conf
+%config(noreplace) /etc/%{name}/activemq.xml
+%config(noreplace) %attr(750,root,activemq) /etc/%{name}/credentials.properties
+%config(noreplace) /etc/%{name}/jetty.xml
+%config(noreplace) /etc/%{name}/activemq-wrapper.conf
+%config(noreplace) /etc/%{name}/log4j.properties
+
+%files info-provider
+%defattr(-,root,root)
+%attr(755,root,root) %{libexecdir}/info-provider-activemq
+
+%changelog
+* Sat Jan 16 2010 R.I.Pienaar <rip@devco.net> 5.3.0
+- Adjusted for ActiveMQ 5.3.0
+
+* Wed Oct 29 2008 James Casey <james.casey@cern.ch> 5.2.0-2
+- fixed defattr on subpackages
+
+* Tue Sep 02 2008 James Casey <james.casey@cern.ch> 5.2.0-1
+- Upgraded to activemq 5.2.0
+
+* Tue Sep 02 2008 James Casey <james.casey@cern.ch> 5.1.0-7
+- Added separate logging of messages whenever the logging interceptor is enabled in the config file
+- removed BrokerRegistry messages casued by REST API
+- now we don't log messages to stdout (so no duplicates in wrapper log).
+- upped the number and size of the rolling logs
+
+* Fri Aug 29 2008 James Casey <james.casey@cern.ch> 5.1.0-6
+- make ServiceData be correct LDIF
+
+* Wed Aug 27 2008 James Casey <james.casey@cern.ch> 5.1.0-5
+- changed glue path from mds-vo-name=local to =resource
+
+* Tue Aug 05 2008 James Casey <james.casey@cern.ch> 5.1.0-4
+- fixed up info-provider to give both REST and STOMP endpoints
+
+* Mon Aug 04 2008 James Casey <james.casey@cern.ch> 5.1.0-3
+- reverted out APP_NAME change to ActiveMQ from init.d since it
+ causes too many problems
+* Mon Aug 04 2008 James Casey <james.casey@cern.ch> 5.1.0-2
+- Added info-provider
+- removed mysql as a requirement
+
+* Thu Mar 20 2008 Daniel RODRIGUES <daniel.rodrigues@cern.ch> - 5.1-SNAPSHOT-1
+- Changed to version 5.1 SNAPSHOT of 18 Mar, fizing AMQ Message Store
+- small fixes to makefile
+
+* Fri Dec 14 2007 James CASEY <james.casey@cern.ch> - 5.0.0-3rc4
+- Added apache config file to forward requests to Jetty
+
+* Thu Dec 13 2007 James CASEY <james.casey@cern.ch> - 5.0.0-2rc4
+- fixed /usr/bin symlink
+- added useJmx to the default config
+
+* Thu Dec 13 2007 James CASEY <james.casey@cern.ch> - 5.0.0-RC4.1
+- Moved to RC4 of the 5.0.0 release candidates
+
+* Mon Dec 10 2007 James CASEY <james.casey@cern.ch> - 5.0-SNAPSHOT-7
+- added symlink in /usr/bin for activemq-admin
+
+* Wed Nov 26 2007 James CASEY <james.casey@cern.ch> - 5.0-SNAPSHOT-6
+- fix bug with group name setting in init.d script
+
+* Wed Nov 26 2007 James CASEY <jamesc@lxb6118.cern.ch> - 5.0-SNAPSHOT-5
+- fix typos in config file for activemq
+
+* Wed Nov 26 2007 James CASEY <jamesc@lxb6118.cern.ch> - 5.0-SNAPSHOT-4
+- add support for lib64 version of tanukiwrapper in config
+- turned off mysql persistence in the "default" config
+
+* Wed Oct 17 2007 James CASEY <jamesc@lxb6118.cern.ch> - 5.0-SNAPSHOT-2
+- more re-org to mirror how tomcat is installed.
+- support for running as activemq user
+
+* Tue Oct 16 2007 James CASEY <jamesc@lxb6118.cern.ch> - 5.0-SNAPSHOT-1
+- Initial Version
+
--- /dev/null
+3 ActiveMQ servers clustered together in a star foramt:
+
+
+ broker2 <-----> broker1 <----> broker3
+
+
+Pay attention to the names in the config file as well
+as the users. This is identical to the simple single
+broker example except with the aded amq user for the
+clustering and the connection setups in broker1
+
+Tested to work with ActiveMQ 5.5.0
--- /dev/null
+<beans
+ xmlns="http://www.springframework.org/schema/beans"
+ xmlns:amq="http://activemq.apache.org/schema/core"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
+ http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd
+ http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
+
+ <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
+ <property name="locations">
+ <value>file:${activemq.base}/conf/credentials.properties</value>
+ </property>
+ </bean>
+
+ <!--
+ For more information about what MCollective requires in this file,
+ see http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html
+ -->
+
+ <!--
+ WARNING: The elements that are direct children of <broker> MUST BE IN
+ ALPHABETICAL ORDER. This is fixed in ActiveMQ 5.6.0, but affects
+ previous versions back to 5.4.
+ https://issues.apache.org/jira/browse/AMQ-3570
+ -->
+
+ <!-- In a network of brokers, the brokerName attribute must be unique. -->
+ <broker xmlns="http://activemq.apache.org/schema/core" brokerName="broker1" useJmx="true" schedulePeriodForDestinationPurge="60000">
+ <!--
+ MCollective generally expects producer flow control to be turned off.
+ It will also generate a limitless number of single-use reply queues,
+ which should be garbage-collected after about five minutes to conserve
+ memory.
+
+ For more information, see:
+ http://activemq.apache.org/producer-flow-control.html
+ -->
+ <destinationPolicy>
+ <policyMap>
+ <policyEntries>
+ <policyEntry topic=">" producerFlowControl="false"/>
+ <policyEntry queue="*.reply.>" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" />
+ </policyEntries>
+ </policyMap>
+ </destinationPolicy>
+
+ <managementContext>
+ <managementContext createConnector="false"/>
+ </managementContext>
+
+ <!--
+ Configure network connectors for a network of brokers. The
+ MCollective ActiveMQ connector uses TWO bi-directional
+ connectors per link, because the short-lived reply queues
+ require conduitSubscriptions be set to false.
+
+ In this config, broker1 connects to both other brokers; neither
+ of the other two have a <networkConnectors> element.
+ -->
+ <networkConnectors>
+ <!-- broker1 -> broker2 -->
+ <networkConnector
+ name="broker1-broker2-topics"
+ uri="static:(tcp://broker2:61616)"
+ userName="amq"
+ password="secret"
+ duplex="true"
+ decreaseNetworkConsumerPriority="true"
+ networkTTL="2"
+ dynamicOnly="true">
+ <excludedDestinations>
+ <queue physicalName=">" />
+ </excludedDestinations>
+ </networkConnector>
+ <networkConnector
+ name="broker1-broker2-queues"
+ uri="static:(tcp://broker2:61616)"
+ userName="amq"
+ password="secret"
+ duplex="true"
+ decreaseNetworkConsumerPriority="true"
+ networkTTL="2"
+ dynamicOnly="true"
+ conduitSubscriptions="false">
+ <excludedDestinations>
+ <topic physicalName=">" />
+ </excludedDestinations>
+ </networkConnector>
+
+ <!-- broker1 -> broker3 -->
+ <networkConnector
+ name="broker1-broker3-topics"
+ uri="static:(tcp://broker3:61616)"
+ userName="amq"
+ password="secret"
+ duplex="true"
+ decreaseNetworkConsumerPriority="true"
+ networkTTL="2"
+ dynamicOnly="true">
+ <excludedDestinations>
+ <queue physicalName=">" />
+ </excludedDestinations>
+ </networkConnector>
+ <networkConnector
+ name="broker1-broker3-queues"
+ uri="static:(tcp://broker3:61616)"
+ userName="amq"
+ password="secret"
+ duplex="true"
+ decreaseNetworkConsumerPriority="true"
+ networkTTL="2"
+ dynamicOnly="true"
+ conduitSubscriptions="false">
+ <excludedDestinations>
+ <topic physicalName=">" />
+ </excludedDestinations>
+ </networkConnector>
+ </networkConnectors>
+
+ <!--
+ Configure message persistence for the broker. MCollective only
+ requires this in a network of brokers, where it's used to prevent
+ duplicate messages.
+
+ The default persistence mechanism is the KahaDB store (identified by
+ the kahaDB tag). For more information, see:
+
+ http://activemq.apache.org/persistence.html
+ -->
+ <persistenceAdapter>
+ <kahaDB directory="${activemq.base}/data/kahadb"/>
+ </persistenceAdapter>
+
+ <plugins>
+ <statisticsBrokerPlugin/>
+
+ <!--
+ This configures the users and groups used by this broker. Groups
+ are referenced below, in the write/read/admin attributes
+ of each authorizationEntry element.
+ -->
+ <simpleAuthenticationPlugin>
+ <users>
+ <authenticationUser username="amq" password="secret" groups="admins,everyone"/>
+ <authenticationUser username="mcollective" password="marionette" groups="mcollective,everyone"/>
+ <authenticationUser username="admin" password="secret" groups="mcollective,admins,everyone"/>
+ </users>
+ </simpleAuthenticationPlugin>
+
+ <!--
+ Configure which users are allowed to read and write where. Permissions
+ are organized by group; groups are configured above, in the
+ authentication plugin.
+
+ With the rules below, both servers and admin users belong to group
+ mcollective, which can both issue and respond to commands. For an
+ example that splits permissions and doesn't allow servers to issue
+ commands, see:
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#detailed-restrictions
+ -->
+ <authorizationPlugin>
+ <map>
+ <authorizationMap>
+ <authorizationEntries>
+ <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <authorizationEntry queue="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <!--
+ The advisory topics are part of ActiveMQ, and all users need access to them.
+ The "everyone" group is not special; you need to ensure every user is a member.
+ -->
+ <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
+ </authorizationEntries>
+ </authorizationMap>
+ </map>
+ </authorizationPlugin>
+ </plugins>
+
+ <!--
+ The systemUsage controls the maximum amount of space the broker will
+ use for messages. For more information, see:
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#memory-and-temp-usage-for-messages-systemusage
+ -->
+ <systemUsage>
+ <systemUsage>
+ <memoryUsage>
+ <memoryUsage limit="20 mb"/>
+ </memoryUsage>
+ <storeUsage>
+ <storeUsage limit="1 gb" name="foo"/>
+ </storeUsage>
+ <tempUsage>
+ <tempUsage limit="100 mb"/>
+ </tempUsage>
+ </systemUsage>
+ </systemUsage>
+
+ <!--
+ The transport connectors allow ActiveMQ to listen for connections over
+ a given protocol. MCollective uses Stomp, and other ActiveMQ brokers
+ use OpenWire. You'll need different URLs depending on whether you are
+ using TLS. For more information, see:
+
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#transport-connectors
+ -->
+ <transportConnectors>
+ <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
+ <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
+ </transportConnectors>
+ </broker>
+
+ <!--
+ Enable web consoles, REST and Ajax APIs and demos.
+ It also includes Camel (with its web console); see ${ACTIVEMQ_HOME}/conf/camel.xml for more info.
+
+ See ${ACTIVEMQ_HOME}/conf/jetty.xml for more details.
+ -->
+ <import resource="jetty.xml"/>
+</beans>
--- /dev/null
+<beans
+ xmlns="http://www.springframework.org/schema/beans"
+ xmlns:amq="http://activemq.apache.org/schema/core"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
+ http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd
+ http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
+
+ <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
+ <property name="locations">
+ <value>file:${activemq.base}/conf/credentials.properties</value>
+ </property>
+ </bean>
+
+ <!--
+ For more information about what MCollective requires in this file,
+ see http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html
+ -->
+
+ <!--
+ WARNING: The elements that are direct children of <broker> MUST BE IN
+ ALPHABETICAL ORDER. This is fixed in ActiveMQ 5.6.0, but affects
+ previous versions back to 5.4.
+ https://issues.apache.org/jira/browse/AMQ-3570
+ -->
+
+ <!-- In a network of brokers, the brokerName attribute must be unique. -->
+ <broker xmlns="http://activemq.apache.org/schema/core" brokerName="broker2" useJmx="true" schedulePeriodForDestinationPurge="60000">
+ <!--
+ MCollective generally expects producer flow control to be turned off.
+ It will also generate a limitless number of single-use reply queues,
+ which should be garbage-collected after about five minutes to conserve
+ memory.
+
+ For more information, see:
+ http://activemq.apache.org/producer-flow-control.html
+ -->
+ <destinationPolicy>
+ <policyMap>
+ <policyEntries>
+ <policyEntry topic=">" producerFlowControl="false"/>
+ <policyEntry queue="*.reply.>" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" />
+ </policyEntries>
+ </policyMap>
+ </destinationPolicy>
+
+ <managementContext>
+ <managementContext createConnector="false"/>
+ </managementContext>
+
+ <!--
+ Configure message persistence for the broker. MCollective only
+ requires this in a network of brokers, where it's used to prevent
+ duplicate messages.
+
+ The default persistence mechanism is the KahaDB store (identified by
+ the kahaDB tag). For more information, see:
+
+ http://activemq.apache.org/persistence.html
+ -->
+ <persistenceAdapter>
+ <kahaDB directory="${activemq.base}/data/kahadb"/>
+ </persistenceAdapter>
+
+ <plugins>
+ <statisticsBrokerPlugin/>
+
+ <!--
+ This configures the users and groups used by this broker. Groups
+ are referenced below, in the write/read/admin attributes
+ of each authorizationEntry element.
+ -->
+ <simpleAuthenticationPlugin>
+ <users>
+ <authenticationUser username="amq" password="secret" groups="admins,everyone"/>
+ <authenticationUser username="mcollective" password="marionette" groups="mcollective,everyone"/>
+ <authenticationUser username="admin" password="secret" groups="mcollective,admins,everyone"/>
+ </users>
+ </simpleAuthenticationPlugin>
+
+ <!--
+ Configure which users are allowed to read and write where. Permissions
+ are organized by group; groups are configured above, in the
+ authentication plugin.
+
+ With the rules below, both servers and admin users belong to group
+ mcollective, which can both issue and respond to commands. For an
+ example that splits permissions and doesn't allow servers to issue
+ commands, see:
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#detailed-restrictions
+ -->
+ <authorizationPlugin>
+ <map>
+ <authorizationMap>
+ <authorizationEntries>
+ <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <authorizationEntry queue="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <!--
+ The advisory topics are part of ActiveMQ, and all users need access to them.
+ The "everyone" group is not special; you need to ensure every user is a member.
+ -->
+ <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
+ </authorizationEntries>
+ </authorizationMap>
+ </map>
+ </authorizationPlugin>
+ </plugins>
+
+ <!--
+ The systemUsage controls the maximum amount of space the broker will
+ use for messages. For more information, see:
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#memory-and-temp-usage-for-messages-systemusage
+ -->
+ <systemUsage>
+ <systemUsage>
+ <memoryUsage>
+ <memoryUsage limit="20 mb"/>
+ </memoryUsage>
+ <storeUsage>
+ <storeUsage limit="1 gb" name="foo"/>
+ </storeUsage>
+ <tempUsage>
+ <tempUsage limit="100 mb"/>
+ </tempUsage>
+ </systemUsage>
+ </systemUsage>
+
+ <!--
+ The transport connectors allow ActiveMQ to listen for connections over
+ a given protocol. MCollective uses Stomp, and other ActiveMQ brokers
+ use OpenWire. You'll need different URLs depending on whether you are
+ using TLS. For more information, see:
+
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#transport-connectors
+ -->
+ <transportConnectors>
+ <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
+ <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
+ </transportConnectors>
+ </broker>
+
+ <!--
+ Enable web consoles, REST and Ajax APIs and demos.
+ It also includes Camel (with its web console); see ${ACTIVEMQ_HOME}/conf/camel.xml for more info.
+
+ See ${ACTIVEMQ_HOME}/conf/jetty.xml for more details.
+ -->
+ <import resource="jetty.xml"/>
+</beans>
--- /dev/null
+<beans
+ xmlns="http://www.springframework.org/schema/beans"
+ xmlns:amq="http://activemq.apache.org/schema/core"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
+ http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd
+ http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
+
+ <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
+ <property name="locations">
+ <value>file:${activemq.base}/conf/credentials.properties</value>
+ </property>
+ </bean>
+
+ <!--
+ For more information about what MCollective requires in this file,
+ see http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html
+ -->
+
+ <!--
+ WARNING: The elements that are direct children of <broker> MUST BE IN
+ ALPHABETICAL ORDER. This is fixed in ActiveMQ 5.6.0, but affects
+ previous versions back to 5.4.
+ https://issues.apache.org/jira/browse/AMQ-3570
+ -->
+
+ <!-- In a network of brokers, the brokerName attribute must be unique. -->
+ <broker xmlns="http://activemq.apache.org/schema/core" brokerName="broker3" useJmx="true" schedulePeriodForDestinationPurge="60000">
+ <!--
+ MCollective generally expects producer flow control to be turned off.
+ It will also generate a limitless number of single-use reply queues,
+ which should be garbage-collected after about five minutes to conserve
+ memory.
+
+ For more information, see:
+ http://activemq.apache.org/producer-flow-control.html
+ -->
+ <destinationPolicy>
+ <policyMap>
+ <policyEntries>
+ <policyEntry topic=">" producerFlowControl="false"/>
+ <policyEntry queue="*.reply.>" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" />
+ </policyEntries>
+ </policyMap>
+ </destinationPolicy>
+
+ <managementContext>
+ <managementContext createConnector="false"/>
+ </managementContext>
+
+ <!--
+ Configure message persistence for the broker. MCollective only
+ requires this in a network of brokers, where it's used to prevent
+ duplicate messages.
+
+ The default persistence mechanism is the KahaDB store (identified by
+ the kahaDB tag). For more information, see:
+
+ http://activemq.apache.org/persistence.html
+ -->
+ <persistenceAdapter>
+ <kahaDB directory="${activemq.base}/data/kahadb"/>
+ </persistenceAdapter>
+
+ <plugins>
+ <statisticsBrokerPlugin/>
+
+ <!--
+ This configures the users and groups used by this broker. Groups
+ are referenced below, in the write/read/admin attributes
+ of each authorizationEntry element.
+ -->
+ <simpleAuthenticationPlugin>
+ <users>
+ <authenticationUser username="amq" password="secret" groups="admins,everyone"/>
+ <authenticationUser username="mcollective" password="marionette" groups="mcollective,everyone"/>
+ <authenticationUser username="admin" password="secret" groups="mcollective,admins,everyone"/>
+ </users>
+ </simpleAuthenticationPlugin>
+
+ <!--
+ Configure which users are allowed to read and write where. Permissions
+ are organized by group; groups are configured above, in the
+ authentication plugin.
+
+ With the rules below, both servers and admin users belong to group
+ mcollective, which can both issue and respond to commands. For an
+ example that splits permissions and doesn't allow servers to issue
+ commands, see:
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#detailed-restrictions
+ -->
+ <authorizationPlugin>
+ <map>
+ <authorizationMap>
+ <authorizationEntries>
+ <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <authorizationEntry queue="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <!--
+ The advisory topics are part of ActiveMQ, and all users need access to them.
+ The "everyone" group is not special; you need to ensure every user is a member.
+ -->
+ <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
+ </authorizationEntries>
+ </authorizationMap>
+ </map>
+ </authorizationPlugin>
+ </plugins>
+
+ <!--
+ The systemUsage controls the maximum amount of space the broker will
+ use for messages. For more information, see:
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#memory-and-temp-usage-for-messages-systemusage
+ -->
+ <systemUsage>
+ <systemUsage>
+ <memoryUsage>
+ <memoryUsage limit="20 mb"/>
+ </memoryUsage>
+ <storeUsage>
+ <storeUsage limit="1 gb" name="foo"/>
+ </storeUsage>
+ <tempUsage>
+ <tempUsage limit="100 mb"/>
+ </tempUsage>
+ </systemUsage>
+ </systemUsage>
+
+ <!--
+ The transport connectors allow ActiveMQ to listen for connections over
+ a given protocol. MCollective uses Stomp, and other ActiveMQ brokers
+ use OpenWire. You'll need different URLs depending on whether you are
+ using TLS. For more information, see:
+
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#transport-connectors
+ -->
+ <transportConnectors>
+ <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
+ <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
+ </transportConnectors>
+ </broker>
+
+ <!--
+ Enable web consoles, REST and Ajax APIs and demos.
+ It also includes Camel (with its web console); see ${ACTIVEMQ_HOME}/conf/camel.xml for more info.
+
+ See ${ACTIVEMQ_HOME}/conf/jetty.xml for more details.
+ -->
+ <import resource="jetty.xml"/>
+</beans>
--- /dev/null
+Simple single broker setup for ActiveMQ 5.5.0.
+
+Provides 2 users, one admin and one for mcollective.
+Admin user can create all sorts of queues and topics,
+mcollective user is restricted.
--- /dev/null
+<beans
+ xmlns="http://www.springframework.org/schema/beans"
+ xmlns:amq="http://activemq.apache.org/schema/core"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
+ http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd
+ http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
+
+ <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
+ <property name="locations">
+ <value>file:${activemq.base}/conf/credentials.properties</value>
+ </property>
+ </bean>
+
+ <!--
+ For more information about what MCollective requires in this file,
+ see http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html
+ -->
+
+ <!--
+ WARNING: The elements that are direct children of <broker> MUST BE IN
+ ALPHABETICAL ORDER. This is fixed in ActiveMQ 5.6.0, but affects
+ previous versions back to 5.4.
+ https://issues.apache.org/jira/browse/AMQ-3570
+ -->
+ <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" useJmx="true" schedulePeriodForDestinationPurge="60000">
+ <!--
+ MCollective generally expects producer flow control to be turned off.
+ It will also generate a limitless number of single-use reply queues,
+ which should be garbage-collected after about five minutes to conserve
+ memory.
+
+ For more information, see:
+ http://activemq.apache.org/producer-flow-control.html
+ -->
+ <destinationPolicy>
+ <policyMap>
+ <policyEntries>
+ <policyEntry topic=">" producerFlowControl="false"/>
+ <policyEntry queue="*.reply.>" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" />
+ </policyEntries>
+ </policyMap>
+ </destinationPolicy>
+
+ <managementContext>
+ <managementContext createConnector="false"/>
+ </managementContext>
+
+ <plugins>
+ <statisticsBrokerPlugin/>
+
+ <!--
+ This configures the users and groups used by this broker. Groups
+ are referenced below, in the write/read/admin attributes
+ of each authorizationEntry element.
+ -->
+ <simpleAuthenticationPlugin>
+ <users>
+ <authenticationUser username="mcollective" password="marionette" groups="mcollective,everyone"/>
+ <authenticationUser username="admin" password="secret" groups="mcollective,admins,everyone"/>
+ </users>
+ </simpleAuthenticationPlugin>
+
+ <!--
+ Configure which users are allowed to read and write where. Permissions
+ are organized by group; groups are configured above, in the
+ authentication plugin.
+
+ With the rules below, both servers and admin users belong to group
+ mcollective, which can both issue and respond to commands. For an
+ example that splits permissions and doesn't allow servers to issue
+ commands, see:
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#detailed-restrictions
+ -->
+ <authorizationPlugin>
+ <map>
+ <authorizationMap>
+ <authorizationEntries>
+ <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <authorizationEntry queue="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <!--
+ The advisory topics are part of ActiveMQ, and all users need access to them.
+ The "everyone" group is not special; you need to ensure every user is a member.
+ -->
+ <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
+ </authorizationEntries>
+ </authorizationMap>
+ </map>
+ </authorizationPlugin>
+ </plugins>
+
+ <!--
+ The systemUsage controls the maximum amount of space the broker will
+ use for messages. For more information, see:
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#memory-and-temp-usage-for-messages-systemusage
+ -->
+ <systemUsage>
+ <systemUsage>
+ <memoryUsage>
+ <memoryUsage limit="20 mb"/>
+ </memoryUsage>
+ <storeUsage>
+ <storeUsage limit="1 gb" name="foo"/>
+ </storeUsage>
+ <tempUsage>
+ <tempUsage limit="100 mb"/>
+ </tempUsage>
+ </systemUsage>
+ </systemUsage>
+
+ <!--
+ The transport connectors allow ActiveMQ to listen for connections over
+ a given protocol. MCollective uses Stomp, and other ActiveMQ brokers
+ use OpenWire. You'll need different URLs depending on whether you are
+ using TLS. For more information, see:
+
+ http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#transport-connectors
+ -->
+ <transportConnectors>
+ <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
+ <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
+ </transportConnectors>
+ </broker>
+
+ <!--
+ Enable web consoles, REST and Ajax APIs and demos.
+ It also includes Camel (with its web console); see ${ACTIVEMQ_HOME}/conf/camel.xml for more info.
+
+ See ${ACTIVEMQ_HOME}/conf/jetty.xml for more details.
+ -->
+ <import resource="jetty.xml"/>
+</beans>
--- /dev/null
+_mco() {
+ local agents options
+
+ COMPREPLY=()
+ local cur=${COMP_WORDS[COMP_CWORD]}
+ local prev=${COMP_WORDS[COMP_CWORD-1]}
+
+ # Where are the plugins?
+ local libdir=$(sed -n 's@libdir = @@p' /etc/mcollective/client.cfg)
+
+ # All arguments by options
+ noopt=($(tr ' ' '\n' <<<${COMP_WORDS[@]} | \
+ grep -v "^$cur$" | grep -v -- '^-'))
+
+ local count_noopt=${#noopt[@]}
+ local cmd=${noopt[0]}
+ local app=${noopt[1]}
+
+ # A bug in the output of --help prevents
+ # from parsing all options, so we list the common ones here
+ local common_options="-T --target -c --config --dt --discovery-timeout \
+ -t --timeout -q --quiet -v --verbose -h --help -W --with -F \
+ --wf --with-fact -C --wc --with-class -A --wa --with-agent -I \
+ --wi --with-identity"
+
+ if [ $COMP_CWORD -eq 1 ]; then
+ apps=$($cmd completion --list-applications)
+ COMPREPLY=($(compgen -W "$apps" -- "$cur"))
+ elif [ $COMP_CWORD -gt 1 ]; then
+ options="${common_options} $($cmd $app --help | grep -o -- '-[^, ]\+')"
+
+ if [ "x${app}" = "xrpc" ]; then
+ if [[ $count_noopt -eq 2 || "x${prev}" = "x--agent" ]]; then
+ # Complete with agents
+ agents=$($cmd completion --list-agents)
+ options="$options $agents"
+ elif [[ $count_noopt -eq 3 || "x${prev}" = "x--action" ]]; then
+ # Complete with agent actions
+ rpcagent=${noopt[2]}
+ actions=$($cmd completion --list-actions \
+ --agent "$rpcagent")
+ options="$options $actions"
+ elif [ $count_noopt -gt 3 ]; then
+ # Complete with key=value
+ rpcagent=${noopt[2]}
+ rpcaction=${noopt[3]}
+ inputs=$($cmd completion --list-inputs \
+ --agent "$rpcagent" --action "$rpcaction")
+ options="$options $inputs"
+ fi
+ fi
+
+ COMPREPLY=($(compgen -W "$options" -S ' ' -- "$cur"))
+ fi
+}
+[ -n "${have:-}" ] && complete -o nospace -F _mco mco
+
--- /dev/null
+Source: mcollective
+Section: utils
+Priority: extra
+Maintainer: Riccardo Setti <giskard@debian.org>
+Build-Depends: debhelper (>= 7), dpatch, cdbs
+Standards-Version: 3.8.0
+Homepage: http://marionette-collective.org/
+
+Package: mcollective
+Architecture: all
+Depends: ruby (>= 1.8.1), mcollective-common (= ${source:Version})
+Description: build server orchestration or parallel job execution systems
+ The Marionette Collective aka. mcollective is a framework
+ to build server orchestration or parallel job execution systems.
+
+Package: mcollective-client
+Architecture: all
+Depends: ruby (>= 1.8.1), mcollective-common (= ${source:Version})
+Description: build server orchestration or parallel job execution systems
+ The Marionette Collective aka. mcollective is a framework
+ to build server orchestration or parallel job execution system
+
+Package: mcollective-common
+Replaces: mcollective (<< 2.0.0-1)
+Breaks: mcollective (<< 2.0.0-1), mcollective-client (<< 2.0.0-1)
+Architecture: all
+Depends: ruby (>= 1.8.1) , rubygems
+Description: build server orchestration or parallel job execution systems
+ The Marionette Collective aka. mcollective is a framework
+ to build server orchestration or parallel job execution systems.
+ .
+ Common files for mcollective packages.
+
+Package: mcollective-doc
+Architecture: all
+Section: doc
+Description: Documentation for mcollective
+ The Marionette Collective aka. mcollective is a framework
+ to build server orchestration or parallel job execution systems.
+ .
+ Documentation package.
+
--- /dev/null
+This package was debianized by Riccardo Setti <giskard@debian.org> on
+Mon, 04 Jan 2010 17:09:50 +0000.
+
+It was downloaded from http://code.google.com/p/mcollective
+
+Upstream Author:
+ R.I.Pienaar <rip@devco.net>
+
+Copyright:
+
+ Copyright 2009 R.I.Pienaar <rip@devco.net>
+
+License:
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+The Debian packaging is (C) 2010, Riccardo Setti <giskard@debian.org> and
+is licensed under the Apache License v2.
+
--- /dev/null
+usr/bin/mco usr/bin/
+usr/sbin/mc-* usr/sbin/
+etc/mcollective/client.cfg etc/mcollective
+usr/share/mcollective/plugins/mcollective/application usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/aggregate usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/pluginpackager usr/share/mcollective/plugins/mcollective
--- /dev/null
+usr/lib/ruby/1.8/* usr/lib/ruby/1.8/
+etc/mcollective/*.erb etc/mcollective
+usr/share/mcollective/plugins/mcollective/agent usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/audit usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/connector usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/data usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/facts usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/registration usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/security usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/validator usr/share/mcollective/plugins/mcollective
+usr/share/mcollective/plugins/mcollective/discovery usr/share/mcollective/plugins/mcollective
--- /dev/null
+usr/share/doc/mcollective/* usr/share/doc/mcollective-doc/
--- /dev/null
+#!/bin/sh
+#
+# mcollective Application Server for STOMP based agents
+#
+#
+# description: mcollective lets you build powerful Stomp compatible middleware clients in ruby without having to worry too
+# much about all the setup and management of a Stomp connection, it also provides stats, logging and so forth
+# as a bonus.
+#
+### BEGIN INIT INFO
+# Provides: mcollective
+# Required-Start: $remote_fs
+# Required-Stop: $remote_fs
+# Default-Start: 2 3 4 5
+# Default-Stop: 0 1 6
+# Short-Description: Start daemon at boot time
+# Description: Enable service provided by mcollective.
+### END INIT INFO
+
+# check permissions
+
+uid=`id -u`
+[ $uid -gt 0 ] && { echo "You need to be root to run file" ; exit 4 ; }
+
+
+# PID directory
+pidfile="/var/run/mcollectived.pid"
+
+name="mcollective"
+mcollectived=/usr/sbin/mcollectived
+daemonopts="--pid=${pidfile} --config=/etc/mcollective/server.cfg"
+
+
+# Source function library.
+. /lib/lsb/init-functions
+
+# Check that binary exists
+if ! [ -f $mcollectived ]
+then
+ echo "mcollectived binary not found"
+ exit 5
+fi
+
+# create pid file if it does not exist
+[ ! -f ${pidfile} ] && { touch ${pidfile} ; }
+
+# See how we were called.
+case "$1" in
+ start)
+ echo "Starting daemon: " $name
+ # start the program
+ start-stop-daemon -S -p ${pidfile} --oknodo -q -a ${mcollectived} -- ${daemonopts}
+ [ $? = 0 ] && { exit 0 ; } || { exit 1 ; }
+ log_success_msg "mcollective started"
+ touch $lock
+ ;;
+ stop)
+ echo "Stopping daemon: " $name
+ start-stop-daemon -K -R 5 -s "TERM" --oknodo -q -p ${pidfile}
+ [ $? = 0 ] && { exit 0 ; } || { exit 1 ; }
+ log_success_msg "mcollective stopped"
+ ;;
+ restart)
+ echo "Restarting daemon: " $name
+ $0 stop
+ sleep 2
+ $0 start
+ [ $? = 0 ] && { echo "mcollective restarted" ; exit 0 ; }
+ ;;
+ condrestart)
+ if [ -f $lock ]; then
+ $0 stop
+ # avoid race
+ sleep 2
+ $0 start
+ fi
+ ;;
+ status)
+ status_of_proc -p ${pidfile} ${mcollectived} ${name} && exit 0 || exit $?
+ ;;
+ *)
+ echo "Usage: mcollectived {start|stop|restart|condrestart|status}"
+ exit 2
+ ;;
+esac
--- /dev/null
+usr/sbin/mcollectived usr/sbin
+etc/mcollective/facts.yaml etc/mcollective
+etc/mcollective/server.cfg etc/mcollective
+etc/init.d etc/
+etc/mcollective/ssl
--- /dev/null
+pluginsdir.dpatch
--- /dev/null
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## conffile.dpatch by <giskard@debian.org
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: default config options
+
+@DPATCH@
+diff -urNad mcollective-0.4.1~/etc/client.cfg mcollective-0.4.1/etc/client.cfg
+--- mcollective-0.4.1~/etc/client.cfg 1970-01-01 00:00:00.000000000 +0000
++++ mcollective-0.4.1/etc/client.cfg 2010-01-06 14:37:24.000000000 +0000
+@@ -0,0 +1,19 @@
++libdir = /usr/share/mcollective/plugins
++logger_type = console
++loglevel = warn
++
++# Plugins
++securityprovider = psk
++plugin.psk = unset
++
++connector = activemq
++plugin.activemq.pool.size = 1
++plugin.activemq.pool.1.host = stomp1
++plugin.activemq.pool.1.port = 6163
++plugin.activemq.pool.1.user = mcollective
++plugin.activemq.pool.1.password = marionette
++
++# Facts
++factsource = yaml
++plugin.yaml = /etc/mcollective/facts.yaml
++
+diff -urNad mcollective-0.4.1~/etc/client.cfg.dist mcollective-0.4.1/etc/client.cfg.dist
+--- mcollective-0.4.1~/etc/client.cfg.dist 2010-01-02 14:54:25.000000000 +0000
++++ mcollective-0.4.1/etc/client.cfg.dist 1970-01-01 00:00:00.000000000 +0000
+@@ -1,19 +0,0 @@
+-libdir = /usr/libexec/mcollective
+-logger_type = console
+-loglevel = warn
+-
+-# Plugins
+-securityprovider = psk
+-plugin.psk = unset
+-
+-connector = stomp
+-plugin.stomp.host = stomp.your.com
+-plugin.stomp.port = 6163
+-plugin.stomp.user = mcollective
+-plugin.stomp.password = marionette
+-
+-# Facts
+-factsource = yaml
+-plugin.yaml = /etc/mcollective/facts.yaml
+-
+diff -urNad mcollective-0.4.1~/etc/facts.yaml mcollective-0.4.1/etc/facts.yaml
+--- mcollective-0.4.1~/etc/facts.yaml 1970-01-01 00:00:00.000000000 +0000
++++ mcollective-0.4.1/etc/facts.yaml 2010-01-06 14:37:27.000000000 +0000
+@@ -0,0 +1,2 @@
++---
++mcollective: 1
+diff -urNad mcollective-0.4.1~/etc/facts.yaml.dist mcollective-0.4.1/etc/facts.yaml.dist
+--- mcollective-0.4.1~/etc/facts.yaml.dist 2010-01-02 14:54:25.000000000 +0000
++++ mcollective-0.4.1/etc/facts.yaml.dist 1970-01-01 00:00:00.000000000 +0000
+@@ -1,2 +0,0 @@
+----
+-mcollective: 1
+diff -urNad mcollective-0.4.1~/etc/server.cfg mcollective-0.4.1/etc/server.cfg
+--- mcollective-0.4.1~/etc/server.cfg 1970-01-01 00:00:00.000000000 +0000
++++ mcollective-0.4.1/etc/server.cfg 2010-01-06 14:37:50.000000000 +0000
+@@ -0,0 +1,20 @@
++libdir = /usr/share/mcollective/plugins
++logfile = /var/log/mcollective.log
++loglevel = info
++daemonize = 1
++
++# Plugins
++securityprovider = psk
++plugin.psk = unset
++
++connector = activemq
++plugin.activemq.pool.size = 1
++plugin.activemq.pool.1.host = stomp1
++plugin.activemq.pool.1.port = 6163
++plugin.activemq.pool.1.user = mcollective
++plugin.activemq.pool.1.password = marionette
++
++# Facts
++factsource = yaml
++plugin.yaml = /etc/mcollective/facts.yaml
++
+diff -urNad mcollective-0.4.1~/etc/server.cfg.dist mcollective-0.4.1/etc/server.cfg.dist
+--- mcollective-0.4.1~/etc/server.cfg.dist 2010-01-02 14:54:25.000000000 +0000
++++ mcollective-0.4.1/etc/server.cfg.dist 1970-01-01 00:00:00.000000000 +0000
+@@ -1,20 +0,0 @@
+-libdir = /usr/libexec/mcollective
+-logfile = /var/log/mcollective.log
+-loglevel = info
+-daemonize = 1
+-
+-# Plugins
+-securityprovider = psk
+-plugin.psk = unset
+-
+-connector = stomp
+-plugin.stomp.host = stomp.your.com
+-plugin.stomp.port = 6163
+-plugin.stomp.user = mcollective
+-plugin.stomp.password = marionette
+-
+-# Facts
+-factsource = yaml
+-plugin.yaml = /etc/mcollective/facts.yaml
+-
--- /dev/null
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## initlsb.dpatch by <giskard@debian.org
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: Init func.
+
+@DPATCH@
+diff -urNad mcollective-0.4.1~/mcollective.init mcollective-0.4.1/mcollective.init
+--- mcollective-0.4.1~/mcollective.init 2010-01-02 14:54:25.000000000 +0000
++++ mcollective-0.4.1/mcollective.init 2010-01-07 10:52:38.000000000 +0000
+@@ -8,6 +8,15 @@
+ # much about all the setup and management of a Stomp connection, it also provides stats, logging and so forth
+ # as a bonus.
+ #
++### BEGIN INIT INFO
++# Provides: scriptname
++# Required-Start: $remote_fs $syslog
++# Required-Stop: $remote_fs $syslog
++# Default-Start: 2 3 4 5
++# Default-Stop: 0 1 6
++# Short-Description: Start daemon at boot time
++# Description: Enable service provided by daemon.
++### END INIT INFO
+
+ RUBYLIB=/usr/local/lib/site_ruby/1.8:$RUBYLIB
+ export RUBYLIB
+@@ -104,7 +113,10 @@
+ echo "mcollectived: service not started"
+ exit 1
+ fi
+- ;;
++ ;;
++ force-reload)
++ echo "not implemented"
++ ;;
+ *)
+ echo "Usage: mcollectived {start|stop|restart|condrestart|status}"
+ exit 1
--- /dev/null
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## makefile.dpatch by <giskard@debian.org
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: Add Makefile for install task
+
+@DPATCH@
+diff -urNad mcollective-0.4.1~/Makefile mcollective-0.4.1/Makefile
+--- mcollective-0.4.1~/Makefile 1970-01-01 00:00:00.000000000 +0000
++++ mcollective-0.4.1/Makefile 2010-01-06 13:43:56.000000000 +0000
+@@ -0,0 +1,38 @@
++#!/usr/bin/make -f
++
++DESTDIR=
++
++clean:
++
++install: install-bin install-lib install-conf install-plugins install-doc
++
++install-bin:
++ install -d $(DESTDIR)/usr/sbin
++ cp mc-* $(DESTDIR)/usr/sbin
++ cp mcollectived.rb $(DESTDIR)/usr/sbin/mcollectived
++
++install-lib:
++ install -d $(DESTDIR)/usr/lib/ruby/1.8/
++ cp -a lib/* $(DESTDIR)/usr/lib/ruby/1.8/
++
++install-conf:
++ install -d $(DESTDIR)/etc/mcollective/
++ install -d $(DESTDIR)/etc/init.d
++ cp etc/* $(DESTDIR)/etc/mcollective/
++ cp mcollective.init $(DESTDIR)/etc/init.d/mcollective
++
++install-plugins:
++ install -d $(DESTDIR)/usr/share/mcollective/
++ cp -a plugins $(DESTDIR)/usr/share/mcollective/
++
++install-doc:
++ install -d $(DESTDIR)/usr/share/doc/
++ cp -a doc $(DESTDIR)/usr/share/doc/mcollective
++
++uninstall:
++ rm -f $(DESTDIR)/usr/sbin/mcollectived
++ rm -rf $(DESTDIR)/usr/lib/ruby/1.8/mcollective*
++ rm -rf $(DESTDIR)/usr/share/mcollective
++ rm -rf $(DESTDIR)/etc/mcollective
++
++.PHONY: clean install uninstall
--- /dev/null
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## pluginsdir.dpatch by <gisakrd@debian.org>
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: fix plugins dir
+
+@DPATCH@
+diff -urNad mcollective-1.1.4~/etc/client.cfg.dist mcollective-1.1.4/etc/client.cfg.dist
+--- mcollective-1.1.4~/etc/client.cfg.dist 2011-04-06 14:13:08.829462165 -0700
++++ mcollective-1.1.4/etc/client.cfg.dist 2011-04-06 14:12:53.129384114 -0700
+@@ -1,6 +1,6 @@
+ main_collective = mcollective
+ collectives = mcollective
+-libdir = /usr/libexec/mcollective
++libdir = /usr/share/mcollective/plugins
+ logger_type = console
+ loglevel = warn
+
+diff -urNad mcollective-1.1.4~/etc/server.cfg.dist mcollective-1.1.4/etc/server.cfg.dist
+--- mcollective-1.1.4~/etc/server.cfg.dist 2011-04-06 14:12:30.889527230 -0700
++++ mcollective-1.1.4/etc/server.cfg.dist 2011-04-06 14:12:23.779407065 -0700
+@@ -1,6 +1,6 @@
+ main_collective = mcollective
+ collectives = mcollective
+-libdir = /usr/libexec/mcollective
++libdir = /usr/share/mcollective/plugins
+ logfile = /var/log/mcollective.log
+ loglevel = info
+ daemonize = 1
+
--- /dev/null
+#!/usr/bin/make -f
+
+DEB_MAKE_CLEAN_TARGET :=
+DEB_MAKE_INSTALL_TARGET := install DESTDIR=$(CURDIR)/debian/tmp
+
+include /usr/share/cdbs/1/rules/debhelper.mk
+include /usr/share/cdbs/1/rules/dpatch.mk
+include /usr/share/cdbs/1/class/makefile.mk
+DEB_MAKE_INVOKE = $(DEB_MAKE_ENVVARS) make -f ext/Makefile -C $(DEB_BUILDDIR)
+
+install/mcollective::
+ mv $(CURDIR)/debian/tmp/etc/mcollective/server.cfg.dist $(CURDIR)/debian/tmp/etc/mcollective/server.cfg
+ mv $(CURDIR)/debian/tmp/etc/mcollective/client.cfg.dist $(CURDIR)/debian/tmp/etc/mcollective/client.cfg
+ mv $(CURDIR)/debian/tmp/etc/mcollective/facts.yaml.dist $(CURDIR)/debian/tmp/etc/mcollective/facts.yaml
+# dh_installinit -pmcollective -o
+
+binary-fixup/mcollective::
+ chmod 640 $(CURDIR)/debian/mcollective/etc/mcollective/server.cfg
--- /dev/null
+A number of templates for the SimpleRPC DDL based help system
--- /dev/null
+<%= meta[:name].upcase %> AGENT
+<% (meta[:name].size + 7).times do %>=<% end %>
+
+<%= meta[:description] %>
+
+ Author: <%= meta[:author] %>
+ Version: <%= meta[:version] %>
+ License: <%= meta[:license] %>
+ Timeout: <%= meta[:timeout] %>
+ Home Page: <%= meta[:url] %>
+
+
+
+ACTIONS:
+========
+% actions.keys.sort.each do |action|
+ * <%= action %>
+% end
+
+% actions.keys.sort.each do |action|
+_<%= action %>_ action:
+<% (action.size + 8).times do %>-<% end %>
+<%= actions[action][:description] %>
+
+% if actions[action][:input].keys.size > 0
+ INPUT:
+% end
+% actions[action][:input].keys.sort.each do |input|
+ <%= input %>:
+ Description: <%= actions[action][:input][input][:description] %>
+ Prompt: <%= actions[action][:input][input][:prompt] %>
+ Type: <%= actions[action][:input][input][:type] %>
+% if actions[action][:input][input][:type] == :string
+ Validation: <%= actions[action][:input][input][:validation] %>
+ Length: <%= actions[action][:input][input][:maxlength] %>
+% elsif actions[action][:input][input][:type] == :list
+ Valid Values: <%= actions[action][:input][input][:list].join(", ") %>
+% end
+
+% end
+
+ OUTPUT:
+% actions[action][:output].keys.sort.each do |output|
+ <%= output %>:
+ Description: <%= actions[action][:output][output][:description] %>
+ Display As: <%= actions[action][:output][output][:display_as] %>
+
+% end
+% end
--- /dev/null
+#!/usr/bin/env ruby
+
+# Simple IRB shell for mcollective
+#
+# mc-irb nrpe
+# Determining the amount of hosts matching filter for 2 seconds .... 47
+# >> rpc :runcommand, :command => "check_disks"
+#
+# * [ ============================================================> ] 47 / 47
+#
+#
+# dev1.your.net Request Aborted
+# CRITICAL
+# Output: DISK CRITICAL - free space: / 176 MB (4% inode=86%);
+# Exit Code: 2
+# Performance Data: /=3959MB;3706;3924;0;4361 /boot=26MB;83;88;0;98 /dev/shm=0MB;217;230;0;256
+#
+# => true
+# >> mchelp
+# <shows the DDL based help for the chosen agent>
+# => true
+# >> rpc(:runcommand, :command => "check_disks") do |resp|
+# ?> puts resp[:sender] + ": " + resp[:data][:output]
+# >> end
+#
+# * [ ============================================================> ] 47 / 47
+#
+# dev1.your.net: DISK OK
+# <snip>
+# => true
+# >>
+#
+# You can access the agent variable via @agent from where you can do the usual manipulation of filters etc,
+# if you wish to switch to a different agent mid run just do newagent("some_other_agent")
+#
+# If you install the Bond gem you'll get some DDL assisted completion in the rpc method
+require 'rubygems'
+require 'irb'
+
+def consolize &block
+ yield
+
+ IRB.setup(nil)
+ irb = IRB::Irb.new
+ IRB.conf[:MAIN_CONTEXT] = irb.context
+ irb.context.evaluate("require 'irb/completion'", 0)
+
+ begin
+ require 'bond'
+ Bond.start
+
+ Bond.complete(:method => "rpc") do |e|
+ begin
+ if e.argument == 1
+ if e.arguments.last == "?"
+ puts "\n\nActions for #{@agent_name}:\n"
+
+ @agent.ddl.actions.each do |action|
+ puts "%20s - %s" % [ ":#{action}", @agent.ddl.action_interface(action)[:description] ]
+ end
+
+ print "\n" + e.line
+ end
+
+ @agent.ddl.actions
+
+ elsif e.argument > 1
+ action = eval(e.arguments[0]).to_s
+ ddl = @agent.ddl.action_interface(action)
+
+ if e.arguments.last == "?"
+ puts "\n\nArguments for #{action}:\n"
+ ddl[:input].keys.each do |input|
+ puts "%20s - %s" % [ ":#{input}", ddl[:input][input][:description] ]
+ end
+
+ print "\n" + e.line
+ end
+
+ [ddl[:input].keys, :verbose].flatten
+ end
+ rescue Exception
+ []
+ end
+ end
+ rescue Exception
+ end
+
+ trap("SIGINT") do
+ irb.signal_handle
+ end
+ catch(:IRB_EXIT) do
+ irb.eval_input
+ end
+end
+
+def mchelp
+ system("mc-rpc --agent-help #{@agent_name}|less")
+ true
+end
+
+def rpc(method_name, *args, &block)
+ unless block_given?
+ if args.size > 0
+ args = args.first
+ else
+ args = {}
+ end
+
+ if args[:verbose]
+ args.delete(:verbose)
+
+ printrpc(@agent.send(method_name, args), :verbose => true)
+ printrpcstats
+ else
+ printrpc @agent.send(method_name, args)
+ printrpcstats
+ end
+
+ else
+ @agent.send(method_name, args.first).each do |resp|
+ yield resp
+ end
+
+ printrpcstats
+ end
+
+ true
+rescue MCollective::DDLValidationError => e
+ puts "Request did not pass DDL validation: #{e}"
+end
+
+def print_filter
+ puts "Active Filter matched #{discover.size} hosts:"
+ puts "\tIdentity: #{@agent.filter['identity'].pretty_inspect}"
+ puts "\t Classes: #{@agent.filter['cf_class'].pretty_inspect}"
+ puts "\t Facts: #{@agent.filter['fact'].pretty_inspect}"
+ puts "\t Agents: #{@agent.filter['agent'].pretty_inspect}"
+
+ discover.size > 0 ? true : false
+end
+
+def newagent(agent)
+ @agent_name = agent
+
+ @options[:filter]["agent"] = []
+ @agent = rpcclient(@agent_name, :options => @options)
+
+ discover
+
+ @agent.progress = true
+
+ print_filter
+end
+
+def identity_filter(*args)
+ @agent.identity_filter(*args)
+
+ print_filter
+end
+
+def fact_filter(*args)
+ @agent.fact_filter(*args)
+
+ print_filter
+end
+
+def agent_filter(*args)
+ @agent.agent_filter(*args)
+
+ print_filter
+end
+
+def class_filter(*args)
+ @agent.class_filter(*args)
+
+ print_filter
+end
+
+def reset_filter
+ @agent.reset_filter
+
+ print_filter
+end
+
+def reset
+ @agent.reset
+
+ print_filter
+end
+
+def discover
+ @agent.discover
+end
+
+def mc?
+ puts <<EOF
+ Available Commands:
+
+ rpc - Performs an RPC request
+ reset - Resets the discovered knowledge
+ discover - Performs a new discovery showing
+ hosts that was found
+ newagent - Switches to a new agent
+ mchelp - The DDL created help for the agent
+
+ Filter Commands:
+ Filter arguments should be enclosed in "your.host.com" if they are strings
+ else use /your.host/ to match Regular expressions
+
+ identity_filter [identity] - Sets an identity filter
+ fact_filter [factname], [factvalue] - Sets a fact filter
+ class_filter [classname] - Sets a class filter
+ agent_filter [agentname] - Sets an agent filter
+ reset_filter - Resets to the filter to blank
+ print_filter - Displays current filter
+
+ Available Variables:
+
+ @agent_name - The name of the active agent
+ @agent - The active RPC client
+
+ Completions:
+
+ While doing an RPC request, press ?<tab> for a list of actions or
+ arguments, do simple :<tab> to get completion on action names and
+ arguments without description of each
+
+EOF
+ true
+end
+
+consolize do
+ require 'mcollective'
+
+
+ include MCollective::RPC
+
+ @options = rpcoptions
+
+ unless ARGV.size == 1
+ puts "Please specify an agent name on the command line"
+ exit 1
+ end
+
+ puts "The Marionette Collective Interactive Ruby Shell version #{MCollective.version}"
+
+ puts
+ newagent(ARGV[0])
+ puts
+ puts "Use mc? to get help on using this shell"
+end
--- /dev/null
+#!/usr/bin/env ruby
+
+# A very simple demonstration of writing a REST server
+# for Simple RPC clients that takes requests over HTTP
+# and returns results as JSON structures.
+
+require 'rubygems'
+require 'sinatra'
+require 'mcollective'
+require 'json'
+
+include MCollective::RPC
+
+# http://<your box>/mcollective/rpctest/echo/msg=hello%20world
+#
+# Creates a new Simple RPC client for the 'rpctest' agent, calls
+# the echo action with a message 'hello world'.
+#
+# Returns all the answers as a JSON data block
+get '/mcollective/:agent/:action/*' do
+ mc = rpcclient(params[:agent])
+ mc.discover
+
+ arguments = {}
+
+ # split up the wildcard params into key=val pairs and
+ # build the arguments hash
+ params[:splat].each do |arg|
+ arguments[$1.to_sym] = $2 if arg =~ /^(.+?)=(.+)$/
+ end
+
+ JSON.dump(mc.send(params[:action], arguments).map{|r| r.results})
+end
+
--- /dev/null
+These files are meant to be places in /usr/ports/mystuff/sysutils/ and then follow
+the OpenBSD guidelines for "custom" ports.
+
+Happy hacking
--- /dev/null
+PKG_ARCH= *
+COMMENT= The Marionette Collective
+
+DISTNAME= mcollective-2.2.1
+
+CATEGORIES= sysutils
+
+HOMEPAGE= http://puppetlabs.com/mcollective/
+MASTER_SITES= http://puppetlabs.com/downloads/mcollective/
+EXTRACT_SUFX= .tgz
+
+# GFDL
+PERMIT_PACKAGE_CDROM= Yes
+PERMIT_PACKAGE_FTP= Yes
+PERMIT_DISTFILES_CDROM= Yes
+PERMIT_DISTFILES_FTP= Yes
+
+NO_BUILD= Yes
+NO_REGRESS= Yes
+
+# makefile is in ext/
+MAKE_FILE=ext/Makefile
+
+post-patch:
+ ${SUBST_CMD} ${WRKSRC}/etc/server.cfg.dist
+ rm -f ${WRKSRC}/etc/server.cfg.dist.{orig,beforesubst}
+
+.include <bsd.port.mk>
--- /dev/null
+MD5 (mcollective-2.2.1.tgz) = 82t1Sk+HRSXR9L4x4eTaSg==
+RMD160 (mcollective-2.2.1.tgz) = PMTcw0VMqrr7rSOYApCW0q8SXok=
+SHA1 (mcollective-2.2.1.tgz) = 1yPH0CgmQa5J4VeiDEAExNbn3K4=
+SHA256 (mcollective-2.2.1.tgz) = 2KaZk2KrMXPJ7aycM+Uy+0E8hyUsONoP9HkdjpAZ4YM=
+SIZE (mcollective-2.2.1.tgz) = 1123400
--- /dev/null
+$OpenBSD$
+--- etc/server.cfg.dist.orig Thu Jun 24 15:57:17 2010
++++ etc/server.cfg.dist Thu Jun 24 15:57:25 2010
+@@ -1,5 +1,5 @@
+-libdir = /usr/libexec/mcollective
++libdir = ${PREFIX}/share/mcollective/plugins
+ logfile = /var/log/mcollective.log
+ loglevel = info
+ daemonize = 1
--- /dev/null
+diff --git ext/Makefile ext/Makefile
+index 029fda4..638d7a5 100644
+--- ext/Makefile
++++ ext/Makefile
+@@ -1,6 +1,5 @@
+ #!/usr/bin/make -f
+
+-DESTDIR=
+
+ build:
+
+@@ -9,36 +8,34 @@ clean:
+ install: install-bin install-lib install-conf install-plugins install-doc
+
+ install-bin:
+- install -d $(DESTDIR)/usr/sbin
+- install -d $(DESTDIR)/usr/bin
+- cp bin/mc-* $(DESTDIR)/usr/sbin
+- cp bin/mco $(DESTDIR)/usr/bin
+- cp bin/mcollectived $(DESTDIR)/usr/sbin/mcollectived
++ install -d $(PREFIX)/sbin
++ install -d $(PREFIX)/bin
++ cp bin/mc-* $(PREFIX)/sbin
++ cp bin/mco $(PREFIX)/bin
++ cp bin/mcollectived $(PREFIX)/sbin/mcollectived
+
+ install-lib:
+- install -d $(DESTDIR)/usr/lib/ruby/1.8/
+- cp -a lib/* $(DESTDIR)/usr/lib/ruby/1.8/
++ install -d $(PREFIX)/lib/ruby/1.8/
++ cp -R lib/* $(PREFIX)/lib/ruby/1.8/
+
+ install-conf:
+- install -d $(DESTDIR)/etc/mcollective/
+- install -d $(DESTDIR)/etc/init.d
+- cp -r etc/* $(DESTDIR)/etc/mcollective/
+- cp mcollective.init $(DESTDIR)/etc/init.d/mcollective
+- rm $(DESTDIR)/etc/mcollective/ssl/PLACEHOLDER
+- rm $(DESTDIR)/etc/mcollective/ssl/clients/PLACEHOLDER
++ install -d $(PREFIX)/share/examples/mcollective/
++ cp -R etc/* $(PREFIX)/share/examples/mcollective/
++ rm $(PREFIX)/share/examples/mcollective/ssl/PLACEHOLDER
++ rm $(PREFIX)/share/examples/mcollective/ssl/clients/PLACEHOLDER
+
+ install-plugins:
+- install -d $(DESTDIR)/usr/share/mcollective/
+- cp -a plugins $(DESTDIR)/usr/share/mcollective/
++ install -d $(PREFIX)/share/mcollective/
++ cp -R plugins $(PREFIX)/share/mcollective/
+
+ install-doc:
+- install -d $(DESTDIR)/usr/share/doc/
+- cp -a doc $(DESTDIR)/usr/share/doc/mcollective
++ install -d $(PREFIX)/share/doc/
++ cp -R doc $(PREFIX)/share/doc/mcollective
+
+ uninstall:
+- rm -f $(DESTDIR)/usr/sbin/mcollectived
+- rm -rf $(DESTDIR)/usr/lib/ruby/1.8/mcollective*
+- rm -rf $(DESTDIR)/usr/share/mcollective
+- rm -rf $(DESTDIR)/etc/mcollective
++ rm -f $(PREFIX)/sbin/mcollectived
++ rm -rf $(PREFIX)/lib/ruby/1.8/mcollective*
++ rm -rf $(PREFIX)/share/mcollective
++ rm -rf $(PREFIX)/share/examples/mcollective
+
+ .PHONY: build clean install uninstall
--- /dev/null
+The Marionette Collective aka. mcollective is a framework to
+build server orchestration or parallel job execution systems.
--- /dev/null
+If you wish to have mcollective started automatically at boot time,
+update your pkg_scripts variable, e.g. in /etc/rc.conf.local.
+See rc.d(8) for details.
--- /dev/null
+@comment $OpenBSD$
+bin/mco
+lib/ruby/
+lib/ruby/1.8/
+lib/ruby/1.8/mcollective/
+lib/ruby/1.8/mcollective.rb
+lib/ruby/1.8/mcollective/agent.rb
+lib/ruby/1.8/mcollective/agents.rb
+lib/ruby/1.8/mcollective/aggregate/
+lib/ruby/1.8/mcollective/aggregate.rb
+lib/ruby/1.8/mcollective/aggregate/base.rb
+lib/ruby/1.8/mcollective/aggregate/result/
+lib/ruby/1.8/mcollective/aggregate/result.rb
+lib/ruby/1.8/mcollective/aggregate/result/base.rb
+lib/ruby/1.8/mcollective/aggregate/result/collection_result.rb
+lib/ruby/1.8/mcollective/aggregate/result/numeric_result.rb
+lib/ruby/1.8/mcollective/application.rb
+lib/ruby/1.8/mcollective/applications.rb
+lib/ruby/1.8/mcollective/cache.rb
+lib/ruby/1.8/mcollective/client.rb
+lib/ruby/1.8/mcollective/config.rb
+lib/ruby/1.8/mcollective/connector/
+lib/ruby/1.8/mcollective/connector.rb
+lib/ruby/1.8/mcollective/connector/base.rb
+lib/ruby/1.8/mcollective/data/
+lib/ruby/1.8/mcollective/data.rb
+lib/ruby/1.8/mcollective/data/base.rb
+lib/ruby/1.8/mcollective/data/result.rb
+lib/ruby/1.8/mcollective/ddl/
+lib/ruby/1.8/mcollective/ddl.rb
+lib/ruby/1.8/mcollective/ddl/agentddl.rb
+lib/ruby/1.8/mcollective/ddl/base.rb
+lib/ruby/1.8/mcollective/ddl/dataddl.rb
+lib/ruby/1.8/mcollective/ddl/discoveryddl.rb
+lib/ruby/1.8/mcollective/ddl/validatorddl.rb
+lib/ruby/1.8/mcollective/discovery.rb
+lib/ruby/1.8/mcollective/facts/
+lib/ruby/1.8/mcollective/facts.rb
+lib/ruby/1.8/mcollective/facts/base.rb
+lib/ruby/1.8/mcollective/generators/
+lib/ruby/1.8/mcollective/generators.rb
+lib/ruby/1.8/mcollective/generators/agent_generator.rb
+lib/ruby/1.8/mcollective/generators/base.rb
+lib/ruby/1.8/mcollective/generators/data_generator.rb
+lib/ruby/1.8/mcollective/generators/templates/
+lib/ruby/1.8/mcollective/generators/templates/action_snippet.erb
+lib/ruby/1.8/mcollective/generators/templates/data_input_snippet.erb
+lib/ruby/1.8/mcollective/generators/templates/ddl.erb
+lib/ruby/1.8/mcollective/generators/templates/plugin.erb
+lib/ruby/1.8/mcollective/log.rb
+lib/ruby/1.8/mcollective/logger/
+lib/ruby/1.8/mcollective/logger.rb
+lib/ruby/1.8/mcollective/logger/base.rb
+lib/ruby/1.8/mcollective/logger/console_logger.rb
+lib/ruby/1.8/mcollective/logger/file_logger.rb
+lib/ruby/1.8/mcollective/logger/syslog_logger.rb
+lib/ruby/1.8/mcollective/matcher/
+lib/ruby/1.8/mcollective/matcher.rb
+lib/ruby/1.8/mcollective/matcher/parser.rb
+lib/ruby/1.8/mcollective/matcher/scanner.rb
+lib/ruby/1.8/mcollective/message.rb
+lib/ruby/1.8/mcollective/monkey_patches.rb
+lib/ruby/1.8/mcollective/optionparser.rb
+lib/ruby/1.8/mcollective/pluginmanager.rb
+lib/ruby/1.8/mcollective/pluginpackager/
+lib/ruby/1.8/mcollective/pluginpackager.rb
+lib/ruby/1.8/mcollective/pluginpackager/agent_definition.rb
+lib/ruby/1.8/mcollective/pluginpackager/standard_definition.rb
+lib/ruby/1.8/mcollective/registration/
+lib/ruby/1.8/mcollective/registration.rb
+lib/ruby/1.8/mcollective/registration/base.rb
+lib/ruby/1.8/mcollective/rpc/
+lib/ruby/1.8/mcollective/rpc.rb
+lib/ruby/1.8/mcollective/rpc/actionrunner.rb
+lib/ruby/1.8/mcollective/rpc/agent.rb
+lib/ruby/1.8/mcollective/rpc/audit.rb
+lib/ruby/1.8/mcollective/rpc/client.rb
+lib/ruby/1.8/mcollective/rpc/helpers.rb
+lib/ruby/1.8/mcollective/rpc/progress.rb
+lib/ruby/1.8/mcollective/rpc/reply.rb
+lib/ruby/1.8/mcollective/rpc/request.rb
+lib/ruby/1.8/mcollective/rpc/result.rb
+lib/ruby/1.8/mcollective/rpc/stats.rb
+lib/ruby/1.8/mcollective/runner.rb
+lib/ruby/1.8/mcollective/runnerstats.rb
+lib/ruby/1.8/mcollective/security/
+lib/ruby/1.8/mcollective/security.rb
+lib/ruby/1.8/mcollective/security/base.rb
+lib/ruby/1.8/mcollective/shell.rb
+lib/ruby/1.8/mcollective/ssl.rb
+lib/ruby/1.8/mcollective/unix_daemon.rb
+lib/ruby/1.8/mcollective/util.rb
+lib/ruby/1.8/mcollective/validator.rb
+lib/ruby/1.8/mcollective/vendor/
+lib/ruby/1.8/mcollective/vendor.rb
+lib/ruby/1.8/mcollective/vendor/json/
+lib/ruby/1.8/mcollective/vendor/json/.gitignore
+lib/ruby/1.8/mcollective/vendor/json/CHANGES
+lib/ruby/1.8/mcollective/vendor/json/COPYING
+lib/ruby/1.8/mcollective/vendor/json/COPYING-json-jruby
+lib/ruby/1.8/mcollective/vendor/json/GPL
+lib/ruby/1.8/mcollective/vendor/json/README
+lib/ruby/1.8/mcollective/vendor/json/README-json-jruby.markdown
+lib/ruby/1.8/mcollective/vendor/json/Rakefile
+lib/ruby/1.8/mcollective/vendor/json/TODO
+lib/ruby/1.8/mcollective/vendor/json/VERSION
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data/
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/.keep
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkComparison.log
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_fast-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_fast.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_pretty-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_pretty.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_safe-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_safe.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt.log
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_fast-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_fast.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_pretty-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_pretty.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_safe-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_safe.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure.log
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkRails#generator-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkRails#generator.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkRails.log
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkComparison.log
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkExt#parser-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkExt#parser.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkExt.log
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkPure#parser-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkPure#parser.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkPure.log
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkRails#parser-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkRails#parser.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkRails.log
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkYAML#parser-autocorrelation.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkYAML#parser.dat
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkYAML.log
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/data/.keep
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/generator2_benchmark.rb
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/generator_benchmark.rb
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/ohai.json
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/ohai.ruby
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/parser2_benchmark.rb
+lib/ruby/1.8/mcollective/vendor/json/benchmarks/parser_benchmark.rb
+lib/ruby/1.8/mcollective/vendor/json/bin/
+lib/ruby/1.8/mcollective/vendor/json/bin/edit_json.rb
+lib/ruby/1.8/mcollective/vendor/json/bin/prettify_json.rb
+lib/ruby/1.8/mcollective/vendor/json/data/
+lib/ruby/1.8/mcollective/vendor/json/data/example.json
+lib/ruby/1.8/mcollective/vendor/json/data/index.html
+lib/ruby/1.8/mcollective/vendor/json/data/prototype.js
+lib/ruby/1.8/mcollective/vendor/json/diagrams/
+lib/ruby/1.8/mcollective/vendor/json/diagrams/.keep
+lib/ruby/1.8/mcollective/vendor/json/ext/
+lib/ruby/1.8/mcollective/vendor/json/ext/json/
+lib/ruby/1.8/mcollective/vendor/json/ext/json/ext/
+lib/ruby/1.8/mcollective/vendor/json/ext/json/ext/generator/
+lib/ruby/1.8/mcollective/vendor/json/ext/json/ext/generator/extconf.rb
+lib/ruby/1.8/mcollective/vendor/json/ext/json/ext/generator/generator.c
+lib/ruby/1.8/mcollective/vendor/json/ext/json/ext/generator/generator.h
+lib/ruby/1.8/mcollective/vendor/json/ext/json/ext/parser/
+lib/ruby/1.8/mcollective/vendor/json/ext/json/ext/parser/extconf.rb
+lib/ruby/1.8/mcollective/vendor/json/ext/json/ext/parser/parser.c
+lib/ruby/1.8/mcollective/vendor/json/ext/json/ext/parser/parser.h
+lib/ruby/1.8/mcollective/vendor/json/ext/json/ext/parser/parser.rl
+lib/ruby/1.8/mcollective/vendor/json/install.rb
+lib/ruby/1.8/mcollective/vendor/json/java/
+lib/ruby/1.8/mcollective/vendor/json/java/lib/
+lib/ruby/1.8/mcollective/vendor/json/java/lib/bytelist-1.0.6.jar
+lib/ruby/1.8/mcollective/vendor/json/java/lib/jcodings.jar
+lib/ruby/1.8/mcollective/vendor/json/java/src/
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/ByteListTranscoder.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/Generator.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/GeneratorMethods.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/GeneratorService.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/GeneratorState.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/OptionsReader.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/Parser.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/Parser.rl
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/ParserService.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/RuntimeInfo.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/StringDecoder.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/StringEncoder.java
+lib/ruby/1.8/mcollective/vendor/json/java/src/json/ext/Utils.java
+lib/ruby/1.8/mcollective/vendor/json/json-java.gemspec
+lib/ruby/1.8/mcollective/vendor/json/lib/
+lib/ruby/1.8/mcollective/vendor/json/lib/json/
+lib/ruby/1.8/mcollective/vendor/json/lib/json.rb
+lib/ruby/1.8/mcollective/vendor/json/lib/json/Array.xpm
+lib/ruby/1.8/mcollective/vendor/json/lib/json/FalseClass.xpm
+lib/ruby/1.8/mcollective/vendor/json/lib/json/Hash.xpm
+lib/ruby/1.8/mcollective/vendor/json/lib/json/Key.xpm
+lib/ruby/1.8/mcollective/vendor/json/lib/json/NilClass.xpm
+lib/ruby/1.8/mcollective/vendor/json/lib/json/Numeric.xpm
+lib/ruby/1.8/mcollective/vendor/json/lib/json/String.xpm
+lib/ruby/1.8/mcollective/vendor/json/lib/json/TrueClass.xpm
+lib/ruby/1.8/mcollective/vendor/json/lib/json/add/
+lib/ruby/1.8/mcollective/vendor/json/lib/json/add/core.rb
+lib/ruby/1.8/mcollective/vendor/json/lib/json/add/rails.rb
+lib/ruby/1.8/mcollective/vendor/json/lib/json/common.rb
+lib/ruby/1.8/mcollective/vendor/json/lib/json/editor.rb
+lib/ruby/1.8/mcollective/vendor/json/lib/json/ext/
+lib/ruby/1.8/mcollective/vendor/json/lib/json/ext.rb
+lib/ruby/1.8/mcollective/vendor/json/lib/json/ext/.keep
+lib/ruby/1.8/mcollective/vendor/json/lib/json/json.xpm
+lib/ruby/1.8/mcollective/vendor/json/lib/json/pure/
+lib/ruby/1.8/mcollective/vendor/json/lib/json/pure.rb
+lib/ruby/1.8/mcollective/vendor/json/lib/json/pure/generator.rb
+lib/ruby/1.8/mcollective/vendor/json/lib/json/pure/parser.rb
+lib/ruby/1.8/mcollective/vendor/json/lib/json/version.rb
+lib/ruby/1.8/mcollective/vendor/json/tests/
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail1.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail10.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail11.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail12.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail13.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail14.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail18.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail19.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail2.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail20.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail21.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail22.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail23.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail24.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail25.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail27.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail28.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail3.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail4.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail5.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail6.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail7.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail8.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/fail9.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/pass1.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/pass15.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/pass16.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/pass17.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/pass2.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/pass26.json
+lib/ruby/1.8/mcollective/vendor/json/tests/fixtures/pass3.json
+lib/ruby/1.8/mcollective/vendor/json/tests/setup_variant.rb
+lib/ruby/1.8/mcollective/vendor/json/tests/test_json.rb
+lib/ruby/1.8/mcollective/vendor/json/tests/test_json_addition.rb
+lib/ruby/1.8/mcollective/vendor/json/tests/test_json_encoding.rb
+lib/ruby/1.8/mcollective/vendor/json/tests/test_json_fixtures.rb
+lib/ruby/1.8/mcollective/vendor/json/tests/test_json_generate.rb
+lib/ruby/1.8/mcollective/vendor/json/tests/test_json_string_matching.rb
+lib/ruby/1.8/mcollective/vendor/json/tests/test_json_unicode.rb
+lib/ruby/1.8/mcollective/vendor/json/tools/
+lib/ruby/1.8/mcollective/vendor/json/tools/fuzz.rb
+lib/ruby/1.8/mcollective/vendor/json/tools/server.rb
+lib/ruby/1.8/mcollective/vendor/load_json.rb
+lib/ruby/1.8/mcollective/vendor/load_systemu.rb
+lib/ruby/1.8/mcollective/vendor/require_vendored.rb
+lib/ruby/1.8/mcollective/vendor/systemu/
+lib/ruby/1.8/mcollective/vendor/systemu/LICENSE
+lib/ruby/1.8/mcollective/vendor/systemu/README
+lib/ruby/1.8/mcollective/vendor/systemu/README.erb
+lib/ruby/1.8/mcollective/vendor/systemu/Rakefile
+lib/ruby/1.8/mcollective/vendor/systemu/lib/
+lib/ruby/1.8/mcollective/vendor/systemu/lib/systemu.rb
+lib/ruby/1.8/mcollective/vendor/systemu/samples/
+lib/ruby/1.8/mcollective/vendor/systemu/samples/a.rb
+lib/ruby/1.8/mcollective/vendor/systemu/samples/b.rb
+lib/ruby/1.8/mcollective/vendor/systemu/samples/c.rb
+lib/ruby/1.8/mcollective/vendor/systemu/samples/d.rb
+lib/ruby/1.8/mcollective/vendor/systemu/samples/e.rb
+lib/ruby/1.8/mcollective/vendor/systemu/samples/f.rb
+lib/ruby/1.8/mcollective/vendor/systemu/systemu.gemspec
+lib/ruby/1.8/mcollective/windows_daemon.rb
+sbin/mc-call-agent
+sbin/mcollectived
+share/doc/mcollective/
+share/doc/mcollective/Array.html
+share/doc/mcollective/COPYING.html
+share/doc/mcollective/Dir.html
+share/doc/mcollective/MCollective/
+share/doc/mcollective/MCollective.html
+share/doc/mcollective/MCollective/Agent.html
+share/doc/mcollective/MCollective/Agents.html
+share/doc/mcollective/MCollective/Aggregate/
+share/doc/mcollective/MCollective/Aggregate.html
+share/doc/mcollective/MCollective/Aggregate/Base.html
+share/doc/mcollective/MCollective/Aggregate/Result/
+share/doc/mcollective/MCollective/Aggregate/Result.html
+share/doc/mcollective/MCollective/Aggregate/Result/Base.html
+share/doc/mcollective/MCollective/Aggregate/Result/CollectionResult.html
+share/doc/mcollective/MCollective/Aggregate/Result/NumericResult.html
+share/doc/mcollective/MCollective/Application.html
+share/doc/mcollective/MCollective/Applications.html
+share/doc/mcollective/MCollective/Cache.html
+share/doc/mcollective/MCollective/Client.html
+share/doc/mcollective/MCollective/Config.html
+share/doc/mcollective/MCollective/Connector/
+share/doc/mcollective/MCollective/Connector.html
+share/doc/mcollective/MCollective/Connector/Base.html
+share/doc/mcollective/MCollective/DDL/
+share/doc/mcollective/MCollective/DDL.html
+share/doc/mcollective/MCollective/DDL/AgentDDL.html
+share/doc/mcollective/MCollective/DDL/Base.html
+share/doc/mcollective/MCollective/DDL/DataDDL.html
+share/doc/mcollective/MCollective/DDL/DiscoveryDDL.html
+share/doc/mcollective/MCollective/DDL/ValidatorDDL.html
+share/doc/mcollective/MCollective/DDLValidationError.html
+share/doc/mcollective/MCollective/Data/
+share/doc/mcollective/MCollective/Data.html
+share/doc/mcollective/MCollective/Data/Base.html
+share/doc/mcollective/MCollective/Data/Result.html
+share/doc/mcollective/MCollective/Discovery.html
+share/doc/mcollective/MCollective/Facts/
+share/doc/mcollective/MCollective/Facts.html
+share/doc/mcollective/MCollective/Facts/Base.html
+share/doc/mcollective/MCollective/Generators/
+share/doc/mcollective/MCollective/Generators.html
+share/doc/mcollective/MCollective/Generators/AgentGenerator.html
+share/doc/mcollective/MCollective/Generators/Base.html
+share/doc/mcollective/MCollective/Generators/DataGenerator.html
+share/doc/mcollective/MCollective/InvalidRPCData.html
+share/doc/mcollective/MCollective/Log.html
+share/doc/mcollective/MCollective/Logger/
+share/doc/mcollective/MCollective/Logger.html
+share/doc/mcollective/MCollective/Logger/Base.html
+share/doc/mcollective/MCollective/Logger/Console_logger.html
+share/doc/mcollective/MCollective/Logger/File_logger.html
+share/doc/mcollective/MCollective/Logger/Syslog_logger.html
+share/doc/mcollective/MCollective/Matcher/
+share/doc/mcollective/MCollective/Matcher.html
+share/doc/mcollective/MCollective/Matcher/Parser.html
+share/doc/mcollective/MCollective/Matcher/Scanner.html
+share/doc/mcollective/MCollective/Message.html
+share/doc/mcollective/MCollective/MissingRPCData.html
+share/doc/mcollective/MCollective/MsgDoesNotMatchRequestID.html
+share/doc/mcollective/MCollective/MsgTTLExpired.html
+share/doc/mcollective/MCollective/NotTargettedAtUs.html
+share/doc/mcollective/MCollective/Optionparser.html
+share/doc/mcollective/MCollective/PluginManager.html
+share/doc/mcollective/MCollective/PluginPackager/
+share/doc/mcollective/MCollective/PluginPackager.html
+share/doc/mcollective/MCollective/PluginPackager/AgentDefinition.html
+share/doc/mcollective/MCollective/PluginPackager/StandardDefinition.html
+share/doc/mcollective/MCollective/RPC/
+share/doc/mcollective/MCollective/RPC.html
+share/doc/mcollective/MCollective/RPC/ActionRunner.html
+share/doc/mcollective/MCollective/RPC/Agent.html
+share/doc/mcollective/MCollective/RPC/Audit.html
+share/doc/mcollective/MCollective/RPC/Client.html
+share/doc/mcollective/MCollective/RPC/Helpers.html
+share/doc/mcollective/MCollective/RPC/Progress.html
+share/doc/mcollective/MCollective/RPC/Reply.html
+share/doc/mcollective/MCollective/RPC/Request.html
+share/doc/mcollective/MCollective/RPC/Result.html
+share/doc/mcollective/MCollective/RPC/Stats.html
+share/doc/mcollective/MCollective/RPCAborted.html
+share/doc/mcollective/MCollective/RPCError.html
+share/doc/mcollective/MCollective/Registration/
+share/doc/mcollective/MCollective/Registration.html
+share/doc/mcollective/MCollective/Registration/Base.html
+share/doc/mcollective/MCollective/Runner.html
+share/doc/mcollective/MCollective/RunnerStats.html
+share/doc/mcollective/MCollective/SSL.html
+share/doc/mcollective/MCollective/Security/
+share/doc/mcollective/MCollective/Security.html
+share/doc/mcollective/MCollective/Security/Base.html
+share/doc/mcollective/MCollective/SecurityValidationFailed.html
+share/doc/mcollective/MCollective/Shell.html
+share/doc/mcollective/MCollective/UnixDaemon.html
+share/doc/mcollective/MCollective/UnknownRPCAction.html
+share/doc/mcollective/MCollective/UnknownRPCError.html
+share/doc/mcollective/MCollective/Util.html
+share/doc/mcollective/MCollective/Validator.html
+share/doc/mcollective/MCollective/ValidatorError.html
+share/doc/mcollective/MCollective/WindowsDaemon.html
+share/doc/mcollective/Object.html
+share/doc/mcollective/README.html
+share/doc/mcollective/Rakefile.html
+share/doc/mcollective/String.html
+share/doc/mcollective/Symbol.html
+share/doc/mcollective/bin/
+share/doc/mcollective/bin/mc-call-agent.html
+share/doc/mcollective/bin/mco.html
+share/doc/mcollective/bin/mcollectived.html
+share/doc/mcollective/created.rid
+share/doc/mcollective/etc/
+share/doc/mcollective/etc/ssl/
+share/doc/mcollective/etc/ssl/PLACEHOLDER.html
+share/doc/mcollective/etc/ssl/clients/
+share/doc/mcollective/etc/ssl/clients/PLACEHOLDER.html
+share/doc/mcollective/images/
+share/doc/mcollective/images/brick.png
+share/doc/mcollective/images/brick_link.png
+share/doc/mcollective/images/bug.png
+share/doc/mcollective/images/bullet_black.png
+share/doc/mcollective/images/bullet_toggle_minus.png
+share/doc/mcollective/images/bullet_toggle_plus.png
+share/doc/mcollective/images/date.png
+share/doc/mcollective/images/find.png
+share/doc/mcollective/images/loadingAnimation.gif
+share/doc/mcollective/images/macFFBgHack.png
+share/doc/mcollective/images/package.png
+share/doc/mcollective/images/page_green.png
+share/doc/mcollective/images/page_white_text.png
+share/doc/mcollective/images/page_white_width.png
+share/doc/mcollective/images/plugin.png
+share/doc/mcollective/images/ruby.png
+share/doc/mcollective/images/tag_green.png
+share/doc/mcollective/images/wrench.png
+share/doc/mcollective/images/wrench_orange.png
+share/doc/mcollective/images/zoom.png
+share/doc/mcollective/index.html
+share/doc/mcollective/js/
+share/doc/mcollective/js/darkfish.js
+share/doc/mcollective/js/jquery.js
+share/doc/mcollective/js/quicksearch.js
+share/doc/mcollective/js/thickbox-compressed.js
+share/doc/mcollective/lib/
+share/doc/mcollective/lib/mcollective/
+share/doc/mcollective/lib/mcollective/agent_rb.html
+share/doc/mcollective/lib/mcollective/agents_rb.html
+share/doc/mcollective/lib/mcollective/aggregate/
+share/doc/mcollective/lib/mcollective/aggregate/base_rb.html
+share/doc/mcollective/lib/mcollective/aggregate/result/
+share/doc/mcollective/lib/mcollective/aggregate/result/base_rb.html
+share/doc/mcollective/lib/mcollective/aggregate/result/collection_result_rb.html
+share/doc/mcollective/lib/mcollective/aggregate/result/numeric_result_rb.html
+share/doc/mcollective/lib/mcollective/aggregate/result_rb.html
+share/doc/mcollective/lib/mcollective/aggregate_rb.html
+share/doc/mcollective/lib/mcollective/application_rb.html
+share/doc/mcollective/lib/mcollective/applications_rb.html
+share/doc/mcollective/lib/mcollective/cache_rb.html
+share/doc/mcollective/lib/mcollective/client_rb.html
+share/doc/mcollective/lib/mcollective/config_rb.html
+share/doc/mcollective/lib/mcollective/connector/
+share/doc/mcollective/lib/mcollective/connector/base_rb.html
+share/doc/mcollective/lib/mcollective/connector_rb.html
+share/doc/mcollective/lib/mcollective/data/
+share/doc/mcollective/lib/mcollective/data/base_rb.html
+share/doc/mcollective/lib/mcollective/data/result_rb.html
+share/doc/mcollective/lib/mcollective/data_rb.html
+share/doc/mcollective/lib/mcollective/ddl/
+share/doc/mcollective/lib/mcollective/ddl/agentddl_rb.html
+share/doc/mcollective/lib/mcollective/ddl/base_rb.html
+share/doc/mcollective/lib/mcollective/ddl/dataddl_rb.html
+share/doc/mcollective/lib/mcollective/ddl/discoveryddl_rb.html
+share/doc/mcollective/lib/mcollective/ddl/validatorddl_rb.html
+share/doc/mcollective/lib/mcollective/ddl_rb.html
+share/doc/mcollective/lib/mcollective/discovery_rb.html
+share/doc/mcollective/lib/mcollective/facts/
+share/doc/mcollective/lib/mcollective/facts/base_rb.html
+share/doc/mcollective/lib/mcollective/facts_rb.html
+share/doc/mcollective/lib/mcollective/generators/
+share/doc/mcollective/lib/mcollective/generators/agent_generator_rb.html
+share/doc/mcollective/lib/mcollective/generators/base_rb.html
+share/doc/mcollective/lib/mcollective/generators/data_generator_rb.html
+share/doc/mcollective/lib/mcollective/generators_rb.html
+share/doc/mcollective/lib/mcollective/log_rb.html
+share/doc/mcollective/lib/mcollective/logger/
+share/doc/mcollective/lib/mcollective/logger/base_rb.html
+share/doc/mcollective/lib/mcollective/logger/console_logger_rb.html
+share/doc/mcollective/lib/mcollective/logger/file_logger_rb.html
+share/doc/mcollective/lib/mcollective/logger/syslog_logger_rb.html
+share/doc/mcollective/lib/mcollective/logger_rb.html
+share/doc/mcollective/lib/mcollective/matcher/
+share/doc/mcollective/lib/mcollective/matcher/parser_rb.html
+share/doc/mcollective/lib/mcollective/matcher/scanner_rb.html
+share/doc/mcollective/lib/mcollective/matcher_rb.html
+share/doc/mcollective/lib/mcollective/message_rb.html
+share/doc/mcollective/lib/mcollective/monkey_patches_rb.html
+share/doc/mcollective/lib/mcollective/optionparser_rb.html
+share/doc/mcollective/lib/mcollective/pluginmanager_rb.html
+share/doc/mcollective/lib/mcollective/pluginpackager/
+share/doc/mcollective/lib/mcollective/pluginpackager/agent_definition_rb.html
+share/doc/mcollective/lib/mcollective/pluginpackager/standard_definition_rb.html
+share/doc/mcollective/lib/mcollective/pluginpackager_rb.html
+share/doc/mcollective/lib/mcollective/registration/
+share/doc/mcollective/lib/mcollective/registration/base_rb.html
+share/doc/mcollective/lib/mcollective/registration_rb.html
+share/doc/mcollective/lib/mcollective/rpc/
+share/doc/mcollective/lib/mcollective/rpc/actionrunner_rb.html
+share/doc/mcollective/lib/mcollective/rpc/agent_rb.html
+share/doc/mcollective/lib/mcollective/rpc/audit_rb.html
+share/doc/mcollective/lib/mcollective/rpc/client_rb.html
+share/doc/mcollective/lib/mcollective/rpc/helpers_rb.html
+share/doc/mcollective/lib/mcollective/rpc/progress_rb.html
+share/doc/mcollective/lib/mcollective/rpc/reply_rb.html
+share/doc/mcollective/lib/mcollective/rpc/request_rb.html
+share/doc/mcollective/lib/mcollective/rpc/result_rb.html
+share/doc/mcollective/lib/mcollective/rpc/stats_rb.html
+share/doc/mcollective/lib/mcollective/rpc_rb.html
+share/doc/mcollective/lib/mcollective/runner_rb.html
+share/doc/mcollective/lib/mcollective/runnerstats_rb.html
+share/doc/mcollective/lib/mcollective/security/
+share/doc/mcollective/lib/mcollective/security/base_rb.html
+share/doc/mcollective/lib/mcollective/security_rb.html
+share/doc/mcollective/lib/mcollective/shell_rb.html
+share/doc/mcollective/lib/mcollective/ssl_rb.html
+share/doc/mcollective/lib/mcollective/unix_daemon_rb.html
+share/doc/mcollective/lib/mcollective/util_rb.html
+share/doc/mcollective/lib/mcollective/validator_rb.html
+share/doc/mcollective/lib/mcollective/windows_daemon_rb.html
+share/doc/mcollective/lib/mcollective_rb.html
+share/doc/mcollective/rdoc.css
+share/examples/mcollective/
+@sample ${SYSCONFDIR}/mcollective/
+share/examples/mcollective/client.cfg.dist
+@sample ${SYSCONFDIR}/mcollective/client.cfg
+share/examples/mcollective/data-help.erb
+share/examples/mcollective/discovery-help.erb
+share/examples/mcollective/facts.yaml.dist
+@sample ${SYSCONFDIR}/mcollective/facts.yaml
+share/examples/mcollective/metadata-help.erb
+share/examples/mcollective/rpc-help.erb
+share/examples/mcollective/server.cfg.dist
+@sample ${SYSCONFDIR}/mcollective/server.cfg
+share/examples/mcollective/ssl/
+@sample ${SYSCONFDIR}/mcollective/ssl/
+share/examples/mcollective/ssl/clients/
+@sample ${SYSCONFDIR}/mcollective/ssl/clients/
+share/mcollective/
+share/mcollective/plugins/
+share/mcollective/plugins/mcollective/
+share/mcollective/plugins/mcollective/agent/
+share/mcollective/plugins/mcollective/agent/discovery.rb
+share/mcollective/plugins/mcollective/agent/rpcutil.ddl
+share/mcollective/plugins/mcollective/agent/rpcutil.rb
+share/mcollective/plugins/mcollective/aggregate/
+share/mcollective/plugins/mcollective/aggregate/average.rb
+share/mcollective/plugins/mcollective/aggregate/sum.rb
+share/mcollective/plugins/mcollective/aggregate/summary.rb
+share/mcollective/plugins/mcollective/application/
+share/mcollective/plugins/mcollective/application/completion.rb
+share/mcollective/plugins/mcollective/application/facts.rb
+share/mcollective/plugins/mcollective/application/find.rb
+share/mcollective/plugins/mcollective/application/help.rb
+share/mcollective/plugins/mcollective/application/inventory.rb
+share/mcollective/plugins/mcollective/application/ping.rb
+share/mcollective/plugins/mcollective/application/plugin.rb
+share/mcollective/plugins/mcollective/application/rpc.rb
+share/mcollective/plugins/mcollective/audit/
+share/mcollective/plugins/mcollective/audit/logfile.rb
+share/mcollective/plugins/mcollective/connector/
+share/mcollective/plugins/mcollective/connector/activemq.rb
+share/mcollective/plugins/mcollective/connector/rabbitmq.rb
+share/mcollective/plugins/mcollective/connector/stomp.rb
+share/mcollective/plugins/mcollective/data/
+share/mcollective/plugins/mcollective/data/agent_data.ddl
+share/mcollective/plugins/mcollective/data/agent_data.rb
+share/mcollective/plugins/mcollective/data/fstat_data.ddl
+share/mcollective/plugins/mcollective/data/fstat_data.rb
+share/mcollective/plugins/mcollective/discovery/
+share/mcollective/plugins/mcollective/discovery/flatfile.ddl
+share/mcollective/plugins/mcollective/discovery/flatfile.rb
+share/mcollective/plugins/mcollective/discovery/mc.ddl
+share/mcollective/plugins/mcollective/discovery/mc.rb
+share/mcollective/plugins/mcollective/facts/
+share/mcollective/plugins/mcollective/facts/yaml_facts.rb
+share/mcollective/plugins/mcollective/pluginpackager/
+share/mcollective/plugins/mcollective/pluginpackager/debpackage_packager.rb
+share/mcollective/plugins/mcollective/pluginpackager/ospackage_packager.rb
+share/mcollective/plugins/mcollective/pluginpackager/rpmpackage_packager.rb
+share/mcollective/plugins/mcollective/pluginpackager/templates/
+share/mcollective/plugins/mcollective/pluginpackager/templates/debian/
+share/mcollective/plugins/mcollective/pluginpackager/templates/debian/Makefile.erb
+share/mcollective/plugins/mcollective/pluginpackager/templates/debian/changelog.erb
+share/mcollective/plugins/mcollective/pluginpackager/templates/debian/compat.erb
+share/mcollective/plugins/mcollective/pluginpackager/templates/debian/control.erb
+share/mcollective/plugins/mcollective/pluginpackager/templates/debian/copyright.erb
+share/mcollective/plugins/mcollective/pluginpackager/templates/debian/rules.erb
+share/mcollective/plugins/mcollective/pluginpackager/templates/redhat/
+share/mcollective/plugins/mcollective/pluginpackager/templates/redhat/rpm_spec.erb
+share/mcollective/plugins/mcollective/registration/
+share/mcollective/plugins/mcollective/registration/agentlist.rb
+share/mcollective/plugins/mcollective/security/
+share/mcollective/plugins/mcollective/security/aes_security.rb
+share/mcollective/plugins/mcollective/security/psk.rb
+share/mcollective/plugins/mcollective/security/ssl.rb
+share/mcollective/plugins/mcollective/validator/
+share/mcollective/plugins/mcollective/validator/array_validator.ddl
+share/mcollective/plugins/mcollective/validator/array_validator.rb
+share/mcollective/plugins/mcollective/validator/ipv4address_validator.ddl
+share/mcollective/plugins/mcollective/validator/ipv4address_validator.rb
+share/mcollective/plugins/mcollective/validator/ipv6address_validator.ddl
+share/mcollective/plugins/mcollective/validator/ipv6address_validator.rb
+share/mcollective/plugins/mcollective/validator/length_validator.ddl
+share/mcollective/plugins/mcollective/validator/length_validator.rb
+share/mcollective/plugins/mcollective/validator/regex_validator.ddl
+share/mcollective/plugins/mcollective/validator/regex_validator.rb
+share/mcollective/plugins/mcollective/validator/shellsafe_validator.ddl
+share/mcollective/plugins/mcollective/validator/shellsafe_validator.rb
+share/mcollective/plugins/mcollective/validator/typecheck_validator.ddl
+share/mcollective/plugins/mcollective/validator/typecheck_validator.rb
+@rcscript ${RCDIR}/mcollectived
--- /dev/null
+#!/bin/sh
+
+daemon="${TRUEPREFIX}/sbin/mcollectived"
+
+. /etc/rc.d/rc.subr
+
+pexp=".*ruby.* ${daemon}${daemon_flags:+ ${daemon_flags}}"
+rc_reload=NO
+
+rc_cmd $1
--- /dev/null
+This script will automatically build Mac packages based on your source
+tarball. The only argument is the path to the untarred source directory.
+
+It automatically determines the MCollective version so it will only work
+with 0.4.9 and forward. It loads mcollective in ruby so it requires
+that the stomp rubygem be installed.
+
+I could just grep the mcollective.rb script but I wanted it to be future proof.
+It also requires XCode be installed to have PackageMaker available. It sets
+the server.cfg file daemonize setting to 0 since my launchd plist will handle
+that. I also have launchd set to restart mcollectived if it stops running.
+
+Let me know if you run in to trouble with it.
+
+Carl Caum - carl _at_ carlcaum.com
--- /dev/null
+#!/bin/bash
+
+MPATH=''
+BETCDIR='/etc/mcollective'
+BRUBYDIR='/Library/Ruby/Site/1.8'
+BSBINDIR='/usr/sbin'
+BBINDIR='/usr/bin'
+BLIBEXECDIR='/usr/libexec/mcollective'
+BDOCDIR='/usr/share/doc/mcollective'
+BLAUNCHDIR='/Library/LaunchDaemons'
+
+if [ -z $1 ]; then
+ echo 'Please give the path to the MCollective source directory'
+ exit 1
+else
+ MPATH=$1
+fi
+
+function msg_stomp {
+ echo 'It is recommended to install stomp on this system using ruby gems'
+ exit 2
+}
+
+function msg_xcode {
+ echo 'It is required to have the latest XCode installed'
+ exit 3
+}
+
+#Make sure we have stomp so we can load mcollective
+/usr/bin/ruby <<EOF || msg_stomp
+require 'rubygems'
+require 'stomp'
+EOF
+
+#Make sure we have PackageMaker installed
+[ -x /Developer/Applications/Utilities/PackageMaker.app/Contents/MacOS/PackageMaker ] || msg_xcode
+
+#Get the MCollective version
+export RUBYLIB=$RUBYLIB:$MPATH/lib
+mcversion=$(/usr/bin/ruby <<EOF
+require 'mcollective'
+puts MCollective.version
+EOF
+)
+
+#Make our tmp directory
+tmpbase=`basename $0`
+common_tmpdir=`mktemp -d /tmp/${tmpbase}-common_$mcversion.XXXX` || exit 1
+client_tmpdir=`mktemp -d /tmp/${tmpbase}-client_$mcversion.XXXX` || exit 1
+tmpdir=`mktemp -d /tmp/${tmpbase}_$mcversion.XXXX` || exit 1
+
+#Build the common environment
+mkdir -p "$common_tmpdir/$BRUBYDIR"
+mkdir -p "$common_tmpdir/$BLIBEXECDIR"
+mkdir -p "$common_tmpdir/$BDOCDIR"
+
+cp -r $MPATH/lib/mcollective $common_tmpdir/$BRUBYDIR/
+cp $MPATH/lib/mcollective.rb $common_tmpdir/$BRUBYDIR/
+cp -r $MPATH/plugins/mcollective $common_tmpdir/$BLIBEXECDIR/
+cp $MPATH/COPYING $common_tmpdir/$BDOCDIR/
+
+#Build the server environment
+mkdir -p "$tmpdir/$BSBINDIR"
+mkdir -p "$tmpdir/$BETCDIR"
+mkdir -p "$tmpdir/$BETCDIR/ssl/clients"
+mkdir -p "$tmpdir/$BLAUNCHDIR"
+
+cp $MPATH/mcollectived.rb $tmpdir/$BSBINDIR/mcollectived
+cp $MPATH/etc/facts.yaml.dist $tmpdir/$BETCDIR/facts.yaml
+cp $MPATH/etc/server.cfg.dist $tmpdir/$BETCDIR/server.cfg
+#This is needed for macs since launchd will handle the daemonizing
+perl -i -pe 's/daemonize = 1/daemonize = 0/' $tmpdir/$BETCDIR/server.cfg
+
+#Build the client environment
+mkdir -p "$client_tmpdir/$BETCDIR"
+mkdir -p "$client_tmpdir/$BSBINDIR"
+
+cp $MPATH/etc/client.cfg.dist $client_tmpdir/$BETCDIR/client.cfg
+cp $MPATH/etc/rpc-help.erb $client_tmpdir/$BETCDIR/
+cp $MPATH/mc-call-agent $client_tmpdir/$BSBINDIR/
+cp $MPATH/mc-controller $client_tmpdir/$BSBINDIR/
+cp $MPATH/mc-facts $client_tmpdir/$BSBINDIR/
+cp $MPATH/mc-find-hosts $client_tmpdir/$BSBINDIR/
+cp $MPATH/mc-inventory $client_tmpdir/$BSBINDIR/
+cp $MPATH/mc-ping $client_tmpdir/$BSBINDIR/
+cp $MPATH/mc-rpc $client_tmpdir/$BSBINDIR/
+
+#Build our launchd property list file
+cat - > $tmpdir/$BLAUNCHDIR/org.marionette-collective.mcollective.plist <<EOF
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+ <key>EnvironmentVariables</key>
+ <dict>
+ <key>PATH</key>
+ <string>/sbin:/usr/sbin:/bin:/usr/bin</string>
+ <key>RUBYLIB</key>
+ <string>/Library/Ruby/Site/1.8</string>
+ </dict>
+ <key>Label</key>
+ <string>org.marionette-collective.mcollective</string>
+ <key>OnDemand</key>
+ <false/>
+ <key>KeepAlive</key>
+ <true/>
+ <key>ProgramArguments</key>
+ <array>
+ <string>/usr/sbin/mcollectived</string>
+ <string>--config=/etc/mcollective/server.cfg</string>
+ </array>
+ <key>RunAtLoad</key>
+ <true/>
+ <key>ServiceDescription</key>
+ <string>MCollective Server</string>
+ <key>ServiceIPC</key>
+ <false/>
+</dict>
+</plist>
+EOF
+
+#launchd complains if the permissions aren't right
+chmod 644 $tmpdir/$BLAUNCHDIR/org.marionette-collective.mcollective.plist
+
+#Make our Packages. This requires XCode be installed
+/Developer/Applications/Utilities/PackageMaker.app/Contents/MacOS/PackageMaker -r $tmpdir --version $mcversion --title "MCollective" -l / -o MCollective_$mcversion.pkg -i org.marionette-collective.mcollective
+/Developer/Applications/Utilities/PackageMaker.app/Contents/MacOS/PackageMaker -r $common_tmpdir --version $mcversion --title "MCollective Common" -l / -o MCollective-Common_$mcversion.pkg -i org.marionette-collective.mcollective-common
+/Developer/Applications/Utilities/PackageMaker.app/Contents/MacOS/PackageMaker -r $client_tmpdir --version $mcversion --title "MCollective Client" -l / -o MCollective-Client_$mcversion.pkg -i org.marionette-collective.mcollective-client
+
+#Clean up
+rm -rf $tmpdir
+rm -rf $common_tmpdir
+rm -rf $client_tmpdir
--- /dev/null
+#!/usr/bin/perl
+
+# A simple Perl client for mcollective that just demonstrates
+# how to construct requests, send them and process results.
+#
+# This is in effect a mc-find-hosts equivelant, you can fill in
+# filters in the request and only the matching ones will reply.
+#
+# For this to work you need the SSL security plugin in MCollective
+# 1.0.0 set to operate in YAML mode.
+
+use YAML::Syck;
+use Digest::MD5 qw(md5 md5_hex md5_base64);
+use Crypt::OpenSSL::RSA;
+use MIME::Base64;
+use Net::STOMP::Client;
+use Data::Dumper;
+
+# The topics from your activemq, /topic/mcollective_dev/...
+$mcollective_prefix = "mcollective_dev";
+
+# Path to your SSL private key and what it's called so the
+# mcollectived will load yours
+$ssl_private_key = "/path/to/your.pem";
+$ssl_private_key_name = "you";
+
+# A string representing your sending host
+$mcollective_client_identity = "devel.your.com-perl";
+
+# Stomp connection parameters
+$stomp_host = "localhost";
+$stomp_port = 6163;
+$stomp_user = "your";
+$stomp_password = "secret";
+
+$YAML::Syck::ImplicitTyping = 1;
+
+$request{":msgtime"} = time();
+$request{":filter"}{"identity"} = [];
+$request{":filter"}{"fact"} = [];
+$request{":filter"}{"agent"} = [];
+$request{":filter"}{"cf_class"} = [];
+$request{":requestid"} = md5_hex(time() . $$);
+$request{":callerid"} = "cert=${ssl_private_key_name}";
+$request{":senderid"} = $mcollective_client_identity;
+$request{":body"} = Dump("ping");
+$request{":msgtarget"} = "/topic/${mcollective_prefix}.discovery.command";
+
+$key = "";
+
+open(SSL, $ssl_private_key);
+ while(<SSL>) {
+ $key = $key . $_;
+ }
+close(SSL);
+
+$rsa = Crypt::OpenSSL::RSA->new_private_key($key);
+$request{":hash"} = encode_base64($rsa->sign($request{":body"}));
+
+$mcrequest = Dump(\%request);
+
+$stomp = Net::STOMP::Client->new(host => $stomp_host, port => $stomp_port);
+$stomp->connect(login => $stomp_user, passcode => $stomp_password);
+
+$stomp->message_callback(sub {
+ my ($self, $frame) = @_;
+
+ $mc_reply = Load($frame->body);
+ $mc_body = Load($mc_reply->{":body"});
+ print $mc_reply->{":senderid"} . "> " . $mc_body . "\n";
+
+ return($self);
+ });
+
+$stomp->subscribe(destination => "/topic/${mcollective_prefix}.discovery.reply");
+$stomp->send(destination => "/topic/${mcollective_prefix}.discovery.command", body => $mcrequest);
+$stomp->wait_for_frames(callback => sub { return(0) }, timeout => 5);
+$stomp->disconnect();
+
+
--- /dev/null
+#!/bin/sh
+#
+# mcollective Application Server for STOMP based agents
+#
+# chkconfig: - 24 76
+#
+# description: mcollective lets you build powerful Stomp compatible middleware clients in ruby without having to worry too
+# much about all the setup and management of a Stomp connection, it also provides stats, logging and so forth
+# as a bonus.
+#
+### BEGIN INIT INFO
+# Provides: mcollective
+# Required-Start: $remote_fs
+# Required-Stop: $remote_fs
+# Short-Description: Start daemon at boot time
+# Description: Enable service provided by daemon.
+### END INIT INFO
+
+mcollectived="/usr/sbin/mcollectived"
+pidfile="/var/run/mcollectived.pid"
+if [ -d /var/lock/subsys ]; then
+ # RedHat/CentOS/etc who use subsys
+ lockfile="/var/lock/subsys/mcollective"
+else
+ # The rest of them
+ lockfile="/var/lock/mcollective"
+fi
+
+# Check that binary exists
+if ! [ -f $mcollectived ]; then
+ echo "mcollectived binary not found"
+ exit 5
+fi
+
+# Source function library.
+. /etc/init.d/functions
+
+if [ -f /etc/sysconfig/mcollective ]; then
+ . /etc/sysconfig/mcollective
+fi
+
+# Determine if we can use the -p option to daemon, killproc, and status.
+# RHEL < 5 can't.
+if status | grep -q -- '-p' 2>/dev/null; then
+ daemonopts="--pidfile $pidfile"
+ pidopts="-p $pidfile"
+fi
+
+start() {
+ echo -n "Starting mcollective: "
+ # Only try to start if not already started
+ if ! rh_status_q; then
+ daemon ${daemonopts} ${mcollectived} --pid=${pidfile} --config="/etc/mcollective/server.cfg"
+ fi
+ # This will be 0 if mcollective is already running
+ RETVAL=$?
+ echo
+ [ $RETVAL -eq 0 ] && touch ${lockfile}
+ return $RETVAL
+}
+
+stop() {
+ echo -n "Shutting down mcollective: "
+ # If running, try to stop it
+ if rh_status_q; then
+ killproc ${pidopts} -d 10 ${mcollectived}
+ else
+ # Non-zero status either means lockfile and pidfile need cleanup (1 and 2)
+ # or the process is already stopped (3), so we can just call true to
+ # trigger the cleanup that happens below.
+ true
+ fi
+ RETVAL=$?
+ echo
+ [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
+ return $RETVAL
+}
+
+restart() {
+ stop
+ start
+}
+
+reload_agents() {
+ echo -n "Reloading mcollective agents: "
+ killproc ${pidopts} ${mcollectived} -USR1
+ RETVAL=$?
+ echo
+ return $RETVAL
+}
+
+reload_loglevel() {
+ echo -n "Cycling mcollective logging level: "
+ killproc ${pidopts} ${mcollectived} -USR2
+ RETVAL=$?
+ echo
+ return $RETVAL
+}
+
+rh_status() {
+ status ${pidopts} ${mcollectived}
+ RETVAL=$?
+ return $RETVAL
+}
+
+rh_status_q() {
+ rh_status >/dev/null 2>&1
+}
+
+# See how we were called.
+case "$1" in
+ start)
+ start
+ ;;
+ stop)
+ stop
+ ;;
+ restart)
+ restart
+ ;;
+ condrestart)
+ rh_status_q || exit 0
+ restart
+ ;;
+ reload-agents)
+ reload_agents
+ ;;
+ reload-loglevel)
+ reload_loglevel
+ ;;
+ status)
+ rh_status
+ ;;
+ *)
+ echo "Usage: mcollectived {start|stop|restart|condrestart|reload-agents|reload-loglevel|status}"
+ RETVAL=2
+ ;;
+esac
+exit $RETVAL
--- /dev/null
+%{!?ruby_sitelib: %global ruby_sitelib %(ruby -rrbconfig -e "puts RbConfig::CONFIG['sitelibdir']")}
+%define release %{rpm_release}%{?dist}
+
+Summary: Application Server for hosting Ruby code on any capable middleware
+Name: mcollective
+Version: %{version}
+Release: %{release}
+Group: System Environment/Daemons
+License: ASL 2.0
+URL: http://puppetlabs.com/mcollective/introduction/
+Source0: http://downloads.puppetlabs.com/mcollective/%{name}-%{version}.tgz
+BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
+Requires: mcollective-common = %{version}-%{release}
+Packager: R.I.Pienaar <rip@devco.net>
+BuildArch: noarch
+
+%package common
+Summary: Common libraries for the mcollective clients and servers
+Group: System Environment/Libraries
+Requires: ruby
+Requires: ruby(abi) >= 1.8
+Requires: rubygems
+Requires: rubygem(stomp)
+
+%description common
+The Marionette Collective:
+
+Common libraries for the mcollective clients and servers
+
+%package client
+Summary: Client tools for the mcollective Application Server
+Requires: mcollective-common = %{version}-%{release}
+Group: Applications/System
+
+%description client
+The Marionette Collective:
+
+Client tools for the mcollective Application Server
+
+%description
+The Marionette Collective:
+
+Server for the mcollective Application Server
+
+%prep
+%setup -q
+
+%build
+
+%install
+rm -rf %{buildroot}
+%{__install} -d -m0755 %{buildroot}/%{ruby_sitelib}/mcollective
+%{__install} -d -m0755 %{buildroot}%{_bindir}
+%{__install} -d -m0755 %{buildroot}%{_sbindir}
+%{__install} -d -m0755 %{buildroot}%{_sysconfdir}/init.d
+%{__install} -d -m0755 %{buildroot}%{_libexecdir}/mcollective/
+%{__install} -d -m0755 %{buildroot}%{_sysconfdir}/mcollective
+%{__install} -d -m0755 %{buildroot}%{_sysconfdir}/mcollective/plugin.d
+%{__install} -d -m0755 %{buildroot}%{_sysconfdir}/mcollective/ssl
+%{__install} -d -m0755 %{buildroot}%{_sysconfdir}/mcollective/ssl/clients
+%{__install} -m0755 bin/mcollectived %{buildroot}%{_sbindir}/mcollectived
+%{__install} -m0640 etc/server.cfg.dist %{buildroot}%{_sysconfdir}/mcollective/server.cfg
+%{__install} -m0644 etc/client.cfg.dist %{buildroot}%{_sysconfdir}/mcollective/client.cfg
+%{__install} -m0444 etc/facts.yaml.dist %{buildroot}%{_sysconfdir}/mcollective/facts.yaml
+%{__install} -m0444 etc/rpc-help.erb %{buildroot}%{_sysconfdir}/mcollective/rpc-help.erb
+%{__install} -m0444 etc/data-help.erb %{buildroot}%{_sysconfdir}/mcollective/data-help.erb
+%{__install} -m0444 etc/discovery-help.erb %{buildroot}%{_sysconfdir}/mcollective/discovery-help.erb
+%{__install} -m0444 etc/metadata-help.erb %{buildroot}%{_sysconfdir}/mcollective/metadata-help.erb
+%{__install} -m0444 etc/msg-help.erb %{buildroot}%{_sysconfdir}/mcollective/msg-help.erb
+%if 0%{?suse_version}
+%{__install} -m0755 mcollective.init %{buildroot}%{_sysconfdir}/init.d/mcollective
+%else
+%{__install} -m0755 ext/redhat/mcollective.init %{buildroot}%{_sysconfdir}/init.d/mcollective
+%endif
+
+
+cp -R lib/* %{buildroot}/%{ruby_sitelib}/
+cp -R plugins/* %{buildroot}%{_libexecdir}/mcollective/
+cp bin/mc-* %{buildroot}%{_sbindir}/
+cp bin/mco %{buildroot}%{_bindir}/
+chmod 0755 %{buildroot}%{_sbindir}/*
+
+%clean
+rm -rf %{buildroot}
+
+%post
+/sbin/chkconfig --add mcollective || :
+
+%postun
+if [ "$1" -ge 1 ]; then
+ /sbin/service mcollective condrestart &>/dev/null || :
+fi
+
+%preun
+if [ "$1" = 0 ] ; then
+ /sbin/service mcollective stop > /dev/null 2>&1
+ /sbin/chkconfig --del mcollective || :
+fi
+
+%files common
+%doc COPYING
+%{ruby_sitelib}/mcollective.rb
+%{ruby_sitelib}/mcollective
+%{_libexecdir}/mcollective/mcollective
+%dir %{_sysconfdir}/mcollective
+%dir %{_sysconfdir}/mcollective/ssl
+%config %{_sysconfdir}/mcollective/*.erb
+
+%files client
+%attr(0755, root, root)%{_sbindir}/mc-call-agent
+%attr(0755, root, root)%{_bindir}/mco
+%doc COPYING
+%config(noreplace)%{_sysconfdir}/mcollective/client.cfg
+%{_libexecdir}/mcollective/mcollective/application
+%{_libexecdir}/mcollective/mcollective/pluginpackager
+
+%files
+%doc COPYING
+%{_sbindir}/mcollectived
+%{_sysconfdir}/init.d/mcollective
+%config(noreplace)%{_sysconfdir}/mcollective/server.cfg
+%config(noreplace)%{_sysconfdir}/mcollective/facts.yaml
+%dir %{_sysconfdir}/mcollective/ssl/clients
+%config(noreplace)%{_sysconfdir}/mcollective/plugin.d
+
+%changelog
+* Tue Nov 03 2009 R.I.Pienaar <rip@devco.net>
+- First release
--- /dev/null
+Building
+--------
+
+Requirements, you can get them from opencsw:
+- coreuitls (CSWcoreutils)
+- gmake (CSWgmake)
+- ggrep (CSWggrep)
+
+Just run ./build on your solaris system.
+
+Running
+-------
+
+Requirements, get them from opencsw:
+- ruby (CSWruby)
+- rubygems (CSWrubygems)
+
+Run requirements
+- rubystomp library
+ http://stomp.codehaus.org/Ruby+Client
+ Up and till version 1.0.4 it is a single file. Put in /opt/csw/lib/ruby/site_ruby/1.8/
+
+Configuration
+-------------
+
+/etc/mcollective/server.cfg
+
+Put the plugins in:
+libdir = /opt/csw/share/mcollective/plugins
+
+Credits
+-------
+Rudy Gevaert <rudy.gevaert+mcollective@ugent.be>
--- /dev/null
+#!/opt/csw/bin/gmake -f -d
+# -*- makefile -*-
+
+BUILDDIR = solaris/tmp
+PKG = solaris/pkg
+DESTDIR = ${CURDIR}/${BUILDDIR}
+PKGDIR = ${CURDIR}/${PKG}
+PKGNAME = CSWmcollective
+VERSION = $(shell cd ../.. ; RUBYLIB=./lib /opt/csw/bin/ruby18 -r mcollective -e 'puts MCollective::VERSION' )
+# If we checked out from git:
+ifeq ($(VERSION),@DEVELOPMENT_VERSION@)
+ VERSION = $(shell ggrep "PROJ_VERSION = " ../../Rakefile | cut -d' ' -f3 | sed -e 's/"//g')
+endif
+RELEASE = 1
+PKGVERSION = ${VERSION}-${RELEASE}\,REV=$(shell date +%Y.%m.%d)
+RUBY_VERSION = 1.8
+RUBY_SITE = ${DESTDIR}/opt/csw/lib/ruby/site_ruby/${RUBY_VERSION}
+
+install:
+ # install directories
+ ginstall -d $(DESTDIR)
+ ginstall -g root -d $(DESTDIR)/opt
+ ginstall -g sys -d $(DESTDIR)/var $(DESTDIR)/var/lock $(DESTDIR)/etc $(DESTDIR)/etc/opt
+ ginstall -g bin -d $(DESTDIR)/var/opt $(DESTDIR)/var/opt/csw $(DESTDIR)/var/opt/csw/svc $(DESTDIR)/var/opt/csw/svc/manifest $(DESTDIR)/var/opt/csw/svc/manifest/network
+ ginstall -g bin -d $(DESTDIR)/opt/csw/lib $(DESTDIR)/opt/csw/lib/svc $(DESTDIR)/opt/csw/lib/svc/method
+ ginstall -g bin -d $(DESTDIR)/opt/csw $(DESTDIR)/opt/csw/lib $(DESTDIR)/opt/csw/sbin $(DESTDIR)/opt/csw/bin
+ ginstall -g bin -d $(DESTDIR)/opt/csw/lib/ruby $(DESTDIR)/opt/csw/lib/ruby/site_ruby $(DESTDIR)/opt/csw/lib/ruby/site_ruby/$(RUBY_VERSION)
+ ginstall -g bin -d $(DESTDIR)/etc/opt/csw $(DESTDIR)/etc/opt/csw/mcollective
+ ginstall -g bin -d $(DESTDIR)/opt/csw/share $(DESTDIR)/opt/csw/share/mcollective
+
+ # install binaries
+ ginstall -g bin $(CURDIR)/../../bin/mc-* $(DESTDIR)/opt/csw/sbin/
+ ginstall -g bin $(CURDIR)/../../bin/mco $(DESTDIR)/opt/csw/sbin/
+ ginstall -g bin $(CURDIR)/../../bin/mcollectived $(DESTDIR)/opt/csw/sbin/mcollectived
+ # install libraries
+ gcp -a $(CURDIR)/../../lib/* $(RUBY_SITE)/
+ chgrp -R bin $(RUBY_SITE)/
+ # install example config files
+ gcp -a $(CURDIR)/../../etc/* $(DESTDIR)/etc/opt/csw/mcollective/
+ grm $(DESTDIR)/etc/opt/csw/mcollective/ssl/PLACEHOLDER
+ grm $(DESTDIR)/etc/opt/csw/mcollective/ssl/clients/PLACEHOLDER
+ chgrp -R bin $(DESTDIR)/etc/opt/csw/mcollective/
+ # install plugins
+ gcp -a $(CURDIR)/../../plugins $(DESTDIR)/opt/csw/share/mcollective/
+ # install docs
+ #ginstall -d $(DESTDIR)/opt/csw/doc $(DESTDIR)/opt/csw/doc/mcollective/
+ #gcp -a $(CURDIR)/../../doc/ $(DESTDIR)/opt/cs/doc/mcollective
+
+ ginstall -g bin $(CURDIR)/mcollective.init $(DESTDIR)/opt/csw/lib/svc/method/svc-cswmcollectived
+ ginstall -g bin $(CURDIR)/cswmcollectived.xml $(DESTDIR)/var/opt/csw/svc/manifest/network
+
+ (cat prototype.head; pkgproto $(DESTDIR)=/ ) > solaris/prototype
+ mkdir $(PKGDIR) || true
+ ginstall postinstall solaris/
+ ginstall postremove solaris/
+ ginstall preremove solaris/
+
+ ginstall pkginfo solaris/
+ (echo PKG=${PKGNAME} ) >> solaris/pkginfo
+ (echo VERSION=${PKGVERSION} ) >> solaris/pkginfo
+ (cd solaris/ ; pkgmk -o -d $(PKGDIR))
+ pkgtrans -s $(PKGDIR) $(CURDIR)/$(PKGNAME)-$(PKGVERSION)-`uname -s``uname -r`-all-CSW.pkg $(PKGNAME)
+
+clean:
+ grm -rf $(DESTDIR)
+ grm -rf $(PKGDIR)
+ grm -f solaris/prototype
+ grm -f $(PKGNAME)-$(SOLARIS_VERSION)-`uname -s``uname -r`-all-CSW.pkg
--- /dev/null
+<?xml version="1.0"?>
+<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
+
+<!-- Mcollective Manifest: Rudy Gevaert -->
+
+<service_bundle type='manifest' name='cswmcollectived:cswmcollectived:'>
+
+
+<service
+ name='network/cswmcollectived'
+ type='service'
+ version='1'>
+
+ <create_default_instance enabled='false'/>
+ <single_instance/>
+
+ <dependency name='config-file'
+ grouping='require_all'
+ restart_on='none'
+ type='path'>
+ <service_fmri value='file:///etc/opt/csw/mcollective/server.cfg'/>
+ </dependency>
+
+ <dependency name='loopback'
+ grouping='require_all'
+ restart_on='error'
+ type='service'>
+ <service_fmri value='svc:/network/loopback:default'/>
+ </dependency>
+
+ <dependency name='physical'
+ grouping='require_all'
+ restart_on='error'
+ type='service'>
+ <service_fmri value='svc:/network/physical:default'/>
+ </dependency>
+
+ <dependency name='fs-local'
+ grouping='require_all'
+ restart_on='none'
+ type='service'>
+ <service_fmri value='svc:/system/filesystem/local'/>
+ </dependency>
+
+ <exec_method
+ type='method'
+ name='start'
+ exec='/opt/csw/lib/svc/method/svc-cswmcollectived start'
+ timeout_seconds='60'>
+ <method_context>
+ <method_environment>
+ <envvar name='PATH' value='/bin:/usr/bin:/opt/csw/bin' />
+ </method_environment>
+ </method_context>
+ </exec_method>
+
+ <exec_method
+ type='method'
+ name='stop'
+ exec='/opt/csw/lib/svc/method/svc-cswmcollectived stop'
+ timeout_seconds='60' />
+
+ <exec_method
+ type='method'
+ name='refresh'
+ exec='/opt/csw/lib/svc/method/svc-cswmcollectived reload'
+ timeout_seconds='60' />
+
+ <exec_method
+ type='method'
+ name='restart'
+ exec='/opt/csw/lib/svc/method/svc-cswmcollectived restart'
+ timeout_seconds='60' />
+
+ <exec_method
+ type='method'
+ name='condrestart'
+ exec='/opt/csw/lib/svc/method/svc-cswmcollectived condrestart'
+ timeout_seconds='60' />
+
+ <exec_method
+ type='method'
+ name='status'
+ exec='/opt/csw/lib/svc/method/svc-cswmcollectived status'
+ timeout_seconds='60' />
+
+ <stability value='Unstable' />
+
+ <template>
+ <common_name>
+ <loctext xml:lang='C'>Mcollective server daemon</loctext>
+ </common_name>
+ <documentation>
+ <doc_link name='puppetlabs.com'
+ uri='http://www.puppetlabs.com/projects/mcollective' />
+ </documentation>
+ </template>
+</service>
+
+</service_bundle>
--- /dev/null
+P CWSruby ruby
+P CWSrubystomp rubystomp
+P CSWrubygems rubygems
--- /dev/null
+#!/bin/sh
+#
+# mcollective Application Server for STOMP based agents
+#
+# description: mcollective lets you build powerful Stomp compatible middleware clients in ruby without having to worry too
+# much about all the setup and management of a Stomp connection, it also provides stats, logging and so forth
+# as a bonus.
+#
+
+RUBYLIB=/opt/csw/lib/ruby/site_ruby/1.8:$RUBYLIB
+export RUBYLIB
+
+mcollectived="/opt/csw/sbin/mcollectived"
+
+lock="/var/lock/mcollective"
+
+# PID directory
+pidfile="/var/run/mcollectived.pid"
+
+# Check that binary exists
+if [ ! -f $mcollectived ]
+then
+ echo "mcollectived binary not found"
+ exit 1
+fi
+
+# See how we were called.
+case "$1" in
+ start)
+ if [ -f ${lock} ]; then
+ # we were not shut down correctly
+ if [ -s ${pidfile} ]; then
+ kill `cat ${pidfile}` >/dev/null 2>&1
+ fi
+ rm -f ${pidfile}
+
+ rm -f ${lock}
+ sleep 2
+ fi
+
+ rm -f ${pidfile}
+
+ ${mcollectived} --pid=${pidfile} --config="/etc/mcollective/server.cfg"
+ if [ $? = 0 ]; then
+ touch $lock
+ exit 0
+ else
+ exit 1
+ fi
+ ;;
+ stop)
+ if [ -s ${pidfile} ]; then
+ kill `cat ${pidfile}` >/dev/null 2>&1
+ fi
+ rm -f ${pidfile}
+
+ rm -f $lock
+ ;;
+ restart)
+ $0 stop
+ sleep 2
+ $0 start
+ ;;
+ condrestart)
+ if [ -f $lock ]; then
+ $0 stop
+ # avoid race
+ sleep 2
+ $0 start
+ fi
+ ;;
+ status)
+ if [ -f ${lock} ]; then
+ if [ -s ${pidfile} ]; then
+ if [ -e /proc/`cat ${pidfile}` ]; then
+ echo "mcollectived (`cat ${pidfile}`) is running"
+ exit 0
+ else
+ echo "mcollectived (`cat ${pidfile}`) is NOT running"
+ exit 1
+ fi
+ fi
+ else
+ echo "mcollectived: service not started"
+ exit 1
+ fi
+ ;;
+ force-reload)
+ echo "not implemented"
+ ;;
+ *)
+ echo "Usage: $0 {start|stop|restart|condrestart|status}"
+ exit 1
+ ;;
+esac
+exit 0
--- /dev/null
+NAME=mcollective- build server orchestration or parallel job execution systems
+CATEGORY=network
+DESC=The Marionette Collective aka. mcollective is a framework to build server orchestration or parallel job execution systems.
+ARCH=all
+EMAIL=rudy.gevaert@ugent.be
+VENDOR=Puppetlabs
--- /dev/null
+#! /bin/bash
+
+PKG_INSTALL_ROOT=${PKG_INSTALL_ROOT:-/}
+
+/usr/bin/test -d $PKG_INSTALL_ROOT/etc/mcollective || /usr/sbin/chroot $PKG_INSTALL_ROOT /usr/bin/ln -s /etc/opt/csw/mcollective /etc/mcollective
+/usr/sbin/chroot $PKG_INSTALL_ROOT /usr/sbin/svccfg import /var/opt/csw/svc/manifest/network/cswmcollectived.xml || /bin/true
--- /dev/null
+/usr/sbin/svcadm disable svc:network/cswmcollectived 2>/dev/null || /bin/true
+/usr/sbin/svccfg delete svc:network/cswmcollectived 2>/dev/null || /bin/true
+rm /etc/mcollective || /bin/true
--- /dev/null
+/usr/sbin/svcadm disable svc:network/cswmcollectived 2>/dev/null || /bin/true
+/usr/sbin/svccfg delete svc:network/cswmcollectived 2>/dev/null || /bin/true
--- /dev/null
+i pkginfo
+i postinstall
+i preremove
+i postremove
--- /dev/null
+#! /bin/bash
+
+IN=$1
+OUT=$2
+
+SOLARIS_VERSION=`ggrep ^VERSION solaris/pkginfo | cut -d = -f2`
+
+sed 's/VERSION="none"/VERSION="'"$SOLARIS_VERSION"'"/' $IN > $OUT
+chmod 755 $OUT
--- /dev/null
+#!/usr/bin/make -f
+
+DESTDIR=
+
+build:
+
+clean:
+
+install: install-bin install-lib install-conf install-plugins # install-doc
+
+install-bin:
+ install -d $(DESTDIR)/usr/sbin
+ install -d $(DESTDIR)/usr/bin
+ cp bin/mc-* $(DESTDIR)/usr/sbin
+ cp bin/mco $(DESTDIR)/usr/bin
+ cp bin/mcollectived $(DESTDIR)/usr/sbin/mcollectived
+ cp COPYING $(DESTDIR)/
+
+install-lib:
+ install -d $(DESTDIR)/usr/ruby/1.8/lib/ruby/site_ruby/1.8/
+ cp -rp lib/* $(DESTDIR)/usr/ruby/1.8/lib/ruby/site_ruby/1.8/
+
+install-conf:
+ install -d $(DESTDIR)/etc/mcollective/
+ install -d $(DESTDIR)/etc/init.d
+ cp -r etc/* $(DESTDIR)/etc/mcollective/
+ cp mcollective.init $(DESTDIR)/etc/init.d/mcollective
+ rm $(DESTDIR)/etc/mcollective/ssl/PLACEHOLDER
+ rm $(DESTDIR)/etc/mcollective/ssl/clients/PLACEHOLDER
+
+install-plugins:
+ install -d $(DESTDIR)/usr/share/mcollective/
+ cp -rp plugins $(DESTDIR)/usr/share/mcollective/
+
+install-doc:
+ install -d $(DESTDIR)/usr/share/doc/
+ cp -rp doc $(DESTDIR)/usr/share/doc/mcollective
+
+uninstall:
+ rm -f $(DESTDIR)/usr/sbin/mcollectived
+ rm -rf $(DESTDIR)/usr/ruby/1.8/lib/ruby/site_ruby/1.8/mcollective*
+ rm -rf $(DESTDIR)/usr/share/mcollective
+ rm -rf $(DESTDIR)/etc/mcollective
+
+.PHONY: build clean install uninstall
+
--- /dev/null
+Requirements
+------------
+
+- Ruby (pkg install pkg:/runtime/ruby-18)
+- Header files (pkg install system/header)
+- GCC to install JSON (pkg install developer/gcc-3)
+- Stomp: gem install stomp
+- Ruby-JSON: gem install json
+
+Installation
+------------
+
+Clone the github repository and install as root:
+
+ $ cd marionette-collective
+ $ make -f ext/solaris11/Makefile install
+
+This will use / as a default destination root directory.
+
+IPS package
+-----------
+
+To create an IPS package, follow the excellent guide at:
+http://www.neuhalfen.name/blog/2011/07/02/Solaris11-Packaging-IPS_simple_packages/
+
+To create a basic IPS repository (and start the associated services):
+
+ # zfs create rpool/IPS
+ # zfs set atime=off rpool/IPS
+ # zfs set mountpoint=/IPS rpool/IPS
+ # mkdir /IPS/Solaris11
+ # svcadm enable application/pkg/server
+ # svccfg -s application/pkg/server setprop pkg/inst_root=/IPS/Solaris11
+ # svccfg -s application/pkg/server setprop pkg/readonly=false
+ # pkgrepo create /IPS/Solaris11/
+ # pkgrepo set -s /IPS/Solaris11 publisher/prefix=legrand.im
+ # pkgrepo -s /IPS/Solaris11 refresh
+ # svcadm refresh application/pkg/server
+ # svcadm enable application/pkg/server
+ # pkg set-publisher -O http://localhost:80 legrand.im
+
+To create and send the package itself, from the guide above:
+
+ # mkdir ~/package
+ # cd <GITHUB>/marionette-collective
+ # cat Makefile | sed 's/DESTDIR=$/DESTDIR=~\/package/' > Makefile.package
+ # make -f ext/solaris11/Makefile.package install
+ # pkg install pkg:/file/gnu-findutils
+ # export ROOT=/
+ # export description="MCollective"
+ # export user="root"
+ # export group="root"
+ # cd ~/package
+ # cat > ../send.sh << "EOF"
+ #!/bin/sh
+ export PKGSEND="pkgsend -s http://localhost:80"
+ eval `$PKGSEND open mcollective@1.1-1`
+ $PKGSEND add license ./COPYING license=lorem_ipsum
+ $PKGSEND add set name=description value="${description}"
+ EOF
+ # gfind . -type d -not -name . -printf "\$PKGSEND add dir mode=%m owner=${user} group=${group} path=$ROOT/%h/%f \n" >> ../send.sh
+ # gfind . -type f -not -name LICENSE -printf "\$PKGSEND add file %h/%f mode=%m owner=${user} group=${group} path=$ROOT/%h/%f \n" >> ../send.sh
+ # gfind . -type l -not -name LICENSE -printf "\$PKGSEND add link path=%h/%f target=%l \n" >> ../send.sh
+ # echo '$PKGSEND close' >> ../send.sh
+ # sh -x ../send.sh
+
+The package can then be installed with:
+
+ # pkg install pkg://legrand.im/mcollective
+
+Configuration
+-------------
+
+There is no packaged configuration; you can use the following example:
+
+ # cat > /etc/mcollective/client.cfg << "EOF"
+ main_collective = mcollective
+ collectives = mcollective
+ libdir = /usr/share/mcollective/plugins
+ logfile = /dev/null
+ loglevel = info
+ # Plugins
+ securityprovider = psk
+ plugin.psk = unset
+ connector = stomp
+ plugin.stomp.host = mqserver
+ plugin.stomp.port = 6163
+ plugin.stomp.user = mcollective
+ plugin.stomp.password = changeme
+ # Facts
+ factsource = yaml
+ plugin.yaml = /etc/mcollective/facts.yaml
+ EOF
+
+License
+------
+
+http://creativecommons.org/publicdomain/zero/1.0/
+
+To the extent possible under law, Mathieu Legrand has waived all copyright and related or
+neighboring rights to this work. This work is published from: Singapore.
+
+
--- /dev/null
+#!/usr/bin/env ruby
+# == Synopsis
+#
+# stompclient: Generic client to consume and produce STOMP queues and topics, tested against
+# Apache Active MQ
+#
+# == Description
+# A simple client that can connect to an STOMP server, subscribe to topics and queues and also
+# send to topics and queues.
+#
+# == Usage
+# stompclient [OPTIONS]
+#
+# --help, -h:
+# Show Help
+#
+# --server, -s
+# The server to connect to, can also be set in STOMP_SERVER environment variable
+#
+# --port, -p
+# The port to connect to, default to 6163
+#
+# --user, -u
+# The user to connect as, can also be set in STOMP_USER environment variable
+#
+# --password, -P
+# The password to use, can also be set in STOMP_PASSWORD environment variable
+#
+# When connected to a server, use the 'help' command to see further information about
+# using the client, common commands that can be issued are:
+#
+# - subscribe /topic/foo: Subscribes to topic 'foo'
+# - /topic/foo bar: Sends 'bar' to the topic 'foo'
+# - details: Toggle the display or timestamp and topic or queue information for each message
+#
+#
+# == Changelog
+# - 20 December 2009 Include into MCollective
+# - 17 March 2009 Initial release
+#
+# R.I.Pienaar <rip@devco.net> more information at www.devco.net
+#
+# Licensed under the Apache License, Version 2.0
+
+require 'rubygems'
+require 'stomp'
+require 'readline'
+require 'thread'
+require 'getoptlong'
+
+opts = GetoptLong.new(
+ [ '--server', '-s', GetoptLong::REQUIRED_ARGUMENT],
+ [ '--port', '-p', GetoptLong::REQUIRED_ARGUMENT],
+ [ '--user', '-u', GetoptLong::REQUIRED_ARGUMENT],
+ [ '--password', '-P', GetoptLong::REQUIRED_ARGUMENT],
+ [ '--help', '-h', GetoptLong::NO_ARGUMENT]
+)
+
+@user = ENV["STOMP_USER"];
+@password = ENV["STOMP_PASSWORD"]
+@server = ENV["STOMP_SERVER"]
+@port = ENV["STOMP_PORT"] || 6163
+
+opts.each { |opt, arg|
+ case opt
+ when '--help'
+ begin
+ require 'rdoc/ri/ri_paths'
+ require 'rdoc/usage'
+ RDoc::usage
+ exit
+ rescue Exception => e
+ puts("Install RDoc::usage or view the comments in the top of the script to get detailed help") if e.to_str != "exit"
+ end
+
+ exit
+ when '--server'
+ @server = arg
+ when '--port'
+ @port = arg
+ when '--user'
+ @user = arg
+ when '--password'
+ @password = arg
+ end
+}
+
+@conn = Stomp::Connection.open(@user, @password, @server, @port, true)
+
+STDOUT.sync = true
+
+def showhelp
+ puts("List of commands:")
+ puts("\n\t- subscribe /(topic|queue)/foo subscribes to topic of queue 'foo'")
+ puts("\t- /(topic|queue|/foo bar sends msg 'bar' to topic of queue 'foo'")
+ puts("\t- quit|exit|q|^d exit")
+ puts("\t- detail show/dont show time and topic a msg was received on")
+ puts("\t- help show this help")
+end
+
+@showdetails = true
+
+Thread.new(@conn) do |amq|
+ while true
+ msg = amq.receive
+ dest = msg.headers["destination"]
+ time = Time.now.strftime('%H:%M:%S')
+
+ if @showdetails
+ msg = "\r#{time}:#{dest} > #{msg.body.chomp}\n"
+ else
+ msg = "\r#{msg.body.chomp}\n"
+ end
+
+ puts (msg)
+ end
+end
+
+loop do
+ line = Readline::readline('AMQ> ')
+ if line
+ Readline::HISTORY.push(line) if line != ""
+ else
+ exit
+ end
+
+ if (line =~ /^(\/(topic|queue)\/\S+)\s+(.+)$/)
+ puts("Sending '#{$3}' to #{$1}")
+
+ if @conn.respond_to?("publish")
+ @conn.publish($1, $3)
+ else
+ @conn.send($1, $3)
+ end
+
+ elsif (line =~ /^sub\S* (\/(topic|queue)\/\S+)$/)
+ puts("Subscribing to #{$1}")
+
+ @conn.subscribe($1)
+ elsif (line =~ /^det(ail)*$/)
+ if @showdetails
+ @showdetails = false
+ puts("No longer showing details")
+ else
+ @showdetails = true
+ puts("Showing time and topic for each msg")
+ end
+ elsif (line =~ /^(quit|exit|q)$/)
+ exit
+ elsif (line =~ /^(help|h|\?)$/)
+ showhelp
+ elsif (line =~ /^$/)
+ else
+ puts("ERROR: unrecognised input: #{line}")
+ end
+end
--- /dev/null
+snippet mcagent
+ module MCollective
+ module Agent
+ class ${1:Agentname}<RPC::Agent
+ action "${2:action name}" do
+ ${3}
+ end
+ end
+ end
+ end
--- /dev/null
+# Snippets for use with VIM and http://www.vim.org/scripts/script.php?script_id=2540
+#
+# These snippets help you write Agent DDLs. Install the VIM Snippets system
+# and copy this to your snippets directory.
+#
+# Create a file .vim/ftdetect/mcollective_ddl.vim with the following:
+#
+# au BufRead,BufNewFile *.ddl setfiletype mcollective_ddl
+#
+# Your file type should now be correctly set automatically and editing
+# DDLs should be easier.
+#
+# Please contact R.I.Pienaar <rip@devco.net> for additions and feedback,
+snippet meta
+ metadata :name => "${1:`Filename('', 'name')`}",
+ :description => "${2:description}",
+ :author => "${3:`g:snips_author`}",
+ :license => "${4:license}",
+ :version => "${5:version}",
+ :url => "${6:homepage}",
+ :timeout => ${7:run timeout}
+
+ ${8}
+snippet discovery
+ discovery do
+ capabilities ${1:capability list}
+ end
+snippet dataquery
+ dataquery :description => "${1:data query description}" do
+ ${2}
+ end
+snippet action
+ action "${1:action name}", :description => "${2:action description}" do
+ ${3}
+ end
+snippet input String
+ input :${1:input name},
+ :prompt => "${2:prompt when asking for information}",
+ :description => "${3:description of the input}",
+ :type => :string,
+ :validation => '${4:^.+$}',
+ :optional => ${5:false},
+ :maxlength => ${6:20}
+
+ ${7}
+snippet input List
+ input :${1:input name},
+ :prompt => "${2:prompt when asking for information}",
+ :description => "${3:description of the input}",
+ :type => :list,
+ :optional => ${4:false},
+ :list => [${5:list members}]
+
+ ${6}
+snippet input Numeric
+ input :${1:input name},
+ :prompt => "${2:prompt when asking for information}",
+ :description => "${3:description of the input}",
+ :type => :number,
+ :optional => ${4:false}
+
+ ${5}
+snippet input Boolean
+ input :${1:input name},
+ :prompt => "${2:prompt when asking for information}",
+ :description => "${3:description of the input}",
+ :type => :boolean,
+ :optional => ${4:false}
+
+ ${5}
+snippet output
+ output ${1:output name},
+ :description => "${2:description of this output data}",
+ :display_as => "${3:what do display}",
+ :default => ${4:nil}
+
+ ${5}
+snippet display Always
+ display :always
+
+
+snippet display Only OK results
+ display :ok
+
+
+snippet display Only failed results
+ display :failed
+
+
--- /dev/null
+These files support installing and using mcollective on MS Windows.
+
+Here are a few instructions for people who wish to do early adopter
+testing, before 2.0 is out we hope to have this packaged into a msi
+installer but your early feedback will help.
+
+Assuming you are installing mcollective into C:\marionette-collective:
+
+ * Install Ruby from http://rubyinstaller.org/, use 1.8.7
+ * Install the following gems: stomp, win32-process, win32-service,
+ sys-admin, windows-api
+ * extract the zip file or clone the git repo into C:\marionette-collective
+ * copy the files from C:\marionette-collective\ext\windows\*.* into
+ C:\marionette-collective\bin
+ * Install any plugins and their dependencies into C:\marionette-collective\plugins
+ specifically for the package and service agents you can install Puppet via gems
+ * Edit the configuration files setting:
+ * libdir = c:\marionette-collective\plugins
+ * logfile = c:\marionette-collective\mcollective.log
+ * plugin.yaml = c:\marionette-collective\etc\facts.ysml
+ * daemonize = 1
+ * change directories to c:\marionette-collective\bin and run register_service.bat
+
+At this point you would have your service registered into the windows service
+manager but set to manual start. If you start it there it should run ok.
+
+If it does not run:
+
+ * Look in the log files, set it to debug level
+ * If the log files are empty look at the command the service wrapper runs
+ and run it by hand. This will show you any early exception preventing it
+ from running. It wont succesfully start but you should see why it does
+ not get far enough to start writing logs.
+
--- /dev/null
+SET BASEDIR=%~dp0..
+SET BASEDIR=%BASEDIR:\bin\..=%
+
+SET SERVER_CONFIG=%BASEDIR%\etc\server.cfg
+SET CLIENT_CONFIG=%BASEDIR%\etc\client.cfg
+
+SET MCOLLECTIVED=%BASEDIR%\bin\mcollectived
+SET MC_STARTTYPE=manual
+REM SET MC_STARTTYPE=auto
+
+SET PATH=%BASEDIR%\bin;%PATH%
+
+SET RUBYLIB=%BASEDIR%\lib;%RUBYLIB%
+SET RUBYLIB=%RUBYLIB:\=/%
+
+SET RUBY="ruby"
--- /dev/null
+@echo off
+
+SETLOCAL
+
+call "%~dp0environment.bat" %0 %*
+
+%RUBY% -S -- mco %* --config "%CLIENT_CONFIG%"
--- /dev/null
+@echo off
+
+SETLOCAL
+
+call "%~dp0environment.bat" %0 %*
+
+%RUBY% -S -- service_manager.rb --install
--- /dev/null
+require 'optparse'
+
+opt = OptionParser.new
+
+ruby_path = ENV["RUBY"].gsub('"','')
+basedir = ENV["BASEDIR"]
+libdir = ENV["RUBYLIB"]
+mcollectived = ENV["MCOLLECTIVED"]
+configfile = ENV["SERVER_CONFIG"]
+
+unless File.exist?(ruby_path)
+ ENV["PATH"].split(File::PATH_SEPARATOR).each do |path|
+ ruby = File.join(path, "ruby.exe")
+
+ if File.exist?(ruby)
+ ruby_path = ruby
+ break
+ end
+ end
+end
+
+abort("Can't find ruby.ext in the path") unless ruby_path
+
+options = {:name => "mcollectived",
+ :display_name => "The Marionette Collective",
+ :description => "Puppet Labs server orchestration framework",
+ :command => '%s -I"%s" -- "%s" --config "%s"' % [ ruby_path, libdir, mcollectived, configfile ]}
+
+action = false
+
+opt.on("--install", "Install service") do
+ action = :install
+end
+
+opt.on("--uninstall", "Remove service") do
+ action = :uninstall
+end
+
+opt.on("--name NAME", String, "Service name (#{options[:name]})") do |n|
+ options[:name] = n
+end
+
+opt.on("--description DESCRIPTION", String, "Service description (#{options[:description]})") do |v|
+ options[:description] = v
+end
+
+opt.on("--display NAME", String, "Service display name (#{options[:display_name]})") do |n|
+ options[:display_name] = n
+end
+
+opt.on("--command COMMAND", String, "Service command (#{options[:command]})") do |c|
+ options[:command] = c
+end
+
+opt.parse!
+
+abort "Please choose an action with --install or --uninstall" unless action
+
+require 'rubygems'
+require 'win32/service'
+
+include Win32
+
+case action
+ when :install
+ if ENV["MC_STARTTYPE"] =~ /auto/i
+ start_type = Service::AUTO_START
+ else
+ start_type = Service::DEMAND_START
+ end
+
+ Service.new(
+ :service_name => options[:name],
+ :display_name => options[:display_name],
+ :description => options[:description],
+ :binary_path_name => options[:command],
+ :service_type => Service::SERVICE_WIN32_OWN_PROCESS,
+ :start_type => start_type
+ )
+
+ puts "Service %s installed" % [options[:name]]
+
+ when :uninstall
+ Service.stop(options[:name]) unless Service.status(options[:name]).current_state == 'stopped'
+
+ while Service.status(options[:name]).current_state != 'stopped'
+ puts "Waiting for service %s to stop" % [options[:name]]
+ sleep 1
+ end
+
+ Service.delete(options[:name])
+
+ puts "Service %s removed" % [options[:name]]
+end
--- /dev/null
+@echo off
+
+SETLOCAL
+
+call "%~dp0environment.bat" %0 %*
+
+%RUBY% -S -- service_manager.rb --uninstall
--- /dev/null
+#compdef mco
+
+# completion for the mcollective cli.
+#
+# for the main mco application it will complete
+# the list of available applications
+#
+# for the rpc application it will complete first
+# the list of agents, then actions and then each
+# input.
+#
+# For all other applications it will just complete
+# the common command line options, to add another
+# application simply define a function called
+# _mco_application_foo() for the foo application
+
+_mco() {
+ if (( CURRENT > 2 )); then
+ local application=${words[2]}
+
+ shift words
+
+ args=({-W,--with}'[Combined class and fact filter]' \
+ {-S,--select}'[Select filter]' \
+ {-F,--wf,--with-fact}'[Fact filter]' \
+ {-C,--wc,--with-class}'[Class filter]' \
+ {-A,--wa,--with-agent}'[Agent filter]' \
+ {-I,--wi,--with-identity}'[Identity filter]' \
+ {-T,--target}'[Target collective]' \
+ {--dm,--disc-method}'[Which discovery method to use]' \
+ {--do,--disc-option}'[Options to pass to the discovery method]' \
+ {--dt,--discovery-timeout}'[Discovery timeout]' \
+ {-t,--timeout}'[Command Timeout]' \
+ {-q,--quiet}'[Surpress verbose output]' \
+ {-c,--config}'[Path to the config file]' \
+ {-v,--verbose}'[Be verbose]' \
+ {-h,--help}'[Show complete help message]' \
+ '--nodes[List of nodes to address]' \
+ '--ttl[Time To Live for the request]' \
+ '--reply-to[Custom reply target]')
+
+ curcontext="${curcontext%:*:*}:mco-${application}"
+
+ if (( $+functions[_mco_application_$application] > 0 ));then
+ _mco_application_$application
+ fi
+
+ _arguments -s : $args
+ else
+ local -a cmdlist
+ _call_program mco-list-applications mco completion --list-applications -v | while read -A hline; do
+ cmdlist=($cmdlist "${hline}")
+ done
+
+ curcontext="${curcontext%:*:*}:mco-applications"
+
+ _describe -t mco-application 'MCollective applications' cmdlist
+ fi
+}
+
+_mco_application_rpc() {
+ local -a clist
+
+ if (( CURRENT == 3 )); then
+ _call_program mco-list-agents mco completion --list-agents -v | while read -A hline; do
+ clist=($clist "${hline}")
+ done
+
+ _describe -t mco-agents "MCollective agents" clist
+ elif (( CURRENT == 4 )); then
+ _call_program mco-list-actions mco completion --list-actions --agent=${words[2]} -v | while read -A hline; do
+ clist=($clist "${hline}")
+ done
+
+ _describe -t mco-actions "${words[2]} actions" clist
+
+ elif (( CURRENT > 4 )); then
+ _call_program mco-list-inputs mco completion --list-inputs --action=${words[3]} --agent=${words[2]} -v | while read hline; do
+ clist=($clist $hline)
+ done
+
+ _describe -t mco-inputs "${words[3]} inputs" clist -S =
+ fi
+
+ args+=(
+ {--np,--no-progress}'[Do not show the progress bar]' \
+ {--nr,--no-results}'[Do not process results, just send request]' \
+ {-1,--one}'[Send request to only one discovered node]' \
+ '--batch[Do request in batches]' \
+ '--batch-sleep[Sleep time between batches]' \
+ {--ln,--limit-nodes}'[Only send the request to a certain number of discovered nodes]' \
+ {-j,--json}'[Output result as JSON data]'
+ )
+}
--- /dev/null
+require 'rubygems'
+require 'stomp'
+require 'timeout'
+require 'digest/md5'
+require 'optparse'
+require 'singleton'
+require 'socket'
+require 'erb'
+require 'shellwords'
+require 'stringio'
+require 'rbconfig'
+require 'tempfile'
+require 'tmpdir'
+require 'mcollective/exception'
+require 'mcollective/monkey_patches'
+require "mcollective/translatable"
+require 'mcollective/cache'
+
+# == The Marionette Collective
+#
+# Framework to build and run Systems Administration agents running on a
+# publish/subscribe middleware system. The system allows you to treat your
+# network as the only true source of the state of your platform via discovery agents
+# and allow you to run agents matching discovery criteria.
+#
+# For an overview of the idea behind this and what it enables please see:
+#
+# http://www.devco.net/archives/2009/10/18/middleware_for_systems_administration.php
+module MCollective
+ autoload :Agent, "mcollective/agent"
+ autoload :Agents, "mcollective/agents"
+ autoload :Aggregate, "mcollective/aggregate"
+ autoload :Application, "mcollective/application"
+ autoload :Applications, "mcollective/applications"
+ autoload :Client, "mcollective/client"
+ autoload :Config, "mcollective/config"
+ autoload :Connector, "mcollective/connector"
+ autoload :Data, "mcollective/data"
+ autoload :DDL, "mcollective/ddl"
+ autoload :Discovery, "mcollective/discovery"
+ autoload :Facts, "mcollective/facts"
+ autoload :Logger, "mcollective/logger"
+ autoload :Log, "mcollective/log"
+ autoload :Matcher, "mcollective/matcher"
+ autoload :Message, "mcollective/message"
+ autoload :Optionparser, "mcollective/optionparser"
+ autoload :Generators, "mcollective/generators"
+ autoload :PluginManager, "mcollective/pluginmanager"
+ autoload :PluginPackager, "mcollective/pluginpackager"
+ autoload :Registration, "mcollective/registration"
+ autoload :RPC, "mcollective/rpc"
+ autoload :Runner, "mcollective/runner"
+ autoload :RunnerStats, "mcollective/runnerstats"
+ autoload :Security, "mcollective/security"
+ autoload :Shell, "mcollective/shell"
+ autoload :SSL, "mcollective/ssl"
+ autoload :Util, "mcollective/util"
+ autoload :Validator, "mcollective/validator"
+ autoload :Vendor, "mcollective/vendor"
+
+ MCollective::Vendor.load_vendored
+
+ VERSION="@DEVELOPMENT_VERSION@"
+
+ def self.version
+ VERSION
+ end
+end
--- /dev/null
+module MCollective
+ module Agent
+
+ end
+end
--- /dev/null
+module MCollective
+ # A collection of agents, loads them, reloads them and dispatches messages to them.
+ # It uses the PluginManager to store, load and manage instances of plugins.
+ class Agents
+ def initialize(agents = {})
+ @config = Config.instance
+ raise ("Configuration has not been loaded, can't load agents") unless @config.configured
+
+ @@agents = agents
+
+ loadagents
+ end
+
+ # Deletes all agents
+ def clear!
+ @@agents.each_key do |agent|
+ PluginManager.delete "#{agent}_agent"
+ Util.unsubscribe(Util.make_subscriptions(agent, :broadcast))
+ end
+
+ @@agents = {}
+ end
+
+ # Loads all agents from disk
+ def loadagents
+ Log.debug("Reloading all agents from disk")
+
+ clear!
+
+ @config.libdir.each do |libdir|
+ agentdir = "#{libdir}/mcollective/agent"
+ next unless File.directory?(agentdir)
+
+ Dir.new(agentdir).grep(/\.rb$/).each do |agent|
+ agentname = File.basename(agent, ".rb")
+ loadagent(agentname) unless PluginManager.include?("#{agentname}_agent")
+ end
+ end
+ end
+
+ # Loads a specified agent from disk if available
+ def loadagent(agentname)
+ agentfile = findagentfile(agentname)
+ return false unless agentfile
+ classname = class_for_agent(agentname)
+
+ PluginManager.delete("#{agentname}_agent")
+
+ begin
+ single_instance = ["registration", "discovery"].include?(agentname)
+
+ PluginManager.loadclass(classname)
+
+ if activate_agent?(agentname)
+ PluginManager << {:type => "#{agentname}_agent", :class => classname, :single_instance => single_instance}
+
+ # Attempt to instantiate the agent once so any validation and hooks get run
+ # this does a basic sanity check on the agent as a whole, if this fails it
+ # will be removed from the plugin list
+ PluginManager["#{agentname}_agent"]
+
+ Util.subscribe(Util.make_subscriptions(agentname, :broadcast)) unless @@agents.include?(agentname)
+
+ @@agents[agentname] = {:file => agentfile}
+ return true
+ else
+ Log.debug("Not activating agent #{agentname} due to agent policy in activate? method")
+ return false
+ end
+ rescue Exception => e
+ Log.error("Loading agent #{agentname} failed: #{e}")
+ PluginManager.delete("#{agentname}_agent")
+ return false
+ end
+ end
+
+ # Builds a class name string given a Agent name
+ def class_for_agent(agent)
+ "MCollective::Agent::#{agent.capitalize}"
+ end
+
+ # Checks if a plugin should be activated by
+ # calling #activate? on it if it responds to
+ # that method else always activate it
+ def activate_agent?(agent)
+ klass = Kernel.const_get("MCollective").const_get("Agent").const_get(agent.capitalize)
+
+ if klass.respond_to?("activate?")
+ return klass.activate?
+ else
+ Log.debug("#{klass} does not have an activate? method, activating as default")
+ return true
+ end
+ rescue Exception => e
+ Log.warn("Agent activation check for #{agent} failed: #{e.class}: #{e}")
+ return false
+ end
+
+ # searches the libdirs for agents
+ def findagentfile(agentname)
+ @config.libdir.each do |libdir|
+ agentfile = File.join([libdir, "mcollective", "agent", "#{agentname}.rb"])
+ if File.exist?(agentfile)
+ Log.debug("Found #{agentname} at #{agentfile}")
+ return agentfile
+ end
+ end
+ return false
+ end
+
+ # Determines if we have an agent with a certain name
+ def include?(agentname)
+ PluginManager.include?("#{agentname}_agent")
+ end
+
+ # Dispatches a message to an agent, accepts a block that will get run if there are
+ # any replies to process from the agent
+ def dispatch(request, connection)
+ Log.debug("Dispatching a message to agent #{request.agent}")
+
+ Thread.new do
+ begin
+ agent = PluginManager["#{request.agent}_agent"]
+
+ Timeout::timeout(agent.timeout) do
+ replies = agent.handlemsg(request.payload, connection)
+
+ # Agents can decide if they wish to reply or not,
+ # returning nil will mean nothing goes back to the
+ # requestor
+ unless replies == nil
+ yield(replies)
+ end
+ end
+ rescue Timeout::Error => e
+ Log.warn("Timeout while handling message for #{request.agent}")
+ rescue Exception => e
+ Log.error("Execution of #{request.agent} failed: #{e}")
+ Log.error(e.backtrace.join("\n\t\t"))
+ end
+ end
+ end
+
+ # Get a list of agents that we have
+ def self.agentlist
+ @@agents.keys
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Aggregate
+ autoload :Result, 'mcollective/aggregate/result'
+ autoload :Base, 'mcollective/aggregate/base'
+
+ attr_accessor :ddl, :functions, :action, :failed
+
+ def initialize(ddl)
+ @functions = []
+ @ddl = ddl
+ @action = ddl[:action]
+ @failed = []
+
+ create_functions
+ end
+
+ # Creates instances of the Aggregate functions and stores them in the function array.
+ # All aggregate call and summarize method calls operate on these function as a batch.
+ def create_functions
+ @ddl[:aggregate].each_with_index do |agg, i|
+ output = agg[:args][0]
+
+ if contains_output?(output)
+ arguments = agg[:args][1]
+ format = (arguments.delete(:format) if arguments) || nil
+ begin
+ @functions << load_function(agg[:function]).new(output, arguments, format, @action)
+ rescue Exception => e
+ Log.error("Cannot create aggregate function '#{output}'. #{e.to_s}")
+ @failed << {:name => output, :type => :startup}
+ end
+ else
+ Log.error("Cannot create aggregate function '#{output}'. '#{output}' has not been specified as a valid ddl output.")
+ @failed << {:name => output, :type => :create}
+ end
+ end
+ end
+
+ # Check if the function param is defined as an output for the action in the ddl
+ def contains_output?(output)
+ @ddl[:output].keys.include?(output)
+ end
+
+ # Call all the appropriate functions with the reply data received from RPC::Client
+ def call_functions(reply)
+ @functions.each do |function|
+ Log.debug("Calling aggregate function #{function} for result")
+ begin
+ function.process_result(reply[:data][function.output_name], reply)
+ rescue Exception => e
+ Log.error("Could not process aggregate function for '#{function.output_name}'. #{e.to_s}")
+ @failed << {:name => function.output_name, :type => :process_result}
+ @functions.delete(function)
+ end
+ end
+ end
+
+ # Finalizes the function returning a result object
+ def summarize
+ summary = @functions.map do |function|
+ begin
+ function.summarize
+ rescue Exception => e
+ Log.error("Could not summarize aggregate result for '#{function.output_name}'. #{e.to_s}")
+ @failed << {:name => function.output_name, :type => :summarize}
+ nil
+ end
+ end
+
+ summary.reject{|x| x.nil?}.sort do |x,y|
+ x.result[:output] <=> y.result[:output]
+ end
+ end
+
+ # Loads function from disk for use
+ def load_function(function_name)
+ function_name = function_name.to_s.capitalize
+
+ PluginManager.loadclass("MCollective::Aggregate::#{function_name}") unless Aggregate.const_defined?(function_name)
+ Aggregate.const_get(function_name)
+ rescue Exception
+ raise "Aggregate function file '#{function_name.downcase}.rb' cannot be loaded"
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Aggregate
+ class Base
+ attr_accessor :name, :result, :output_name, :action, :aggregate_format, :arguments
+
+ def initialize(output_name, arguments, aggregate_format, action)
+ @name = self.class.to_s
+ @output_name = output_name
+
+ # Any additional arguments passed in the ddl after the output field will
+ # be stored in the arguments array which can be used in the function
+ @arguments = arguments
+ @aggregate_format = aggregate_format
+ @action = action
+ @result = {:value => nil, :type => nil, :output => output_name}
+
+ startup_hook
+ end
+
+ ['startup_hook', 'process_result'].each do |method|
+ define_method method do
+ raise RuntimeError, "'#{method}' method of function class #{@name} has not yet been implemented"
+ end
+ end
+
+ # Stops execution of the function and returns a specific ResultObject,
+ # aggregate functions will most likely override this but this is the simplest
+ # case so we might as well default to that
+ def summarize
+ raise "Result type is not set while trying to summarize aggregate function results" unless @result[:type]
+
+ result_class(@result[:type]).new(@result, @aggregate_format, @action)
+ end
+
+ def result_class(type)
+ Result.const_get("#{type.to_s.capitalize}Result")
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Aggregate
+ module Result
+ autoload :Base, 'mcollective/aggregate/result/base'
+ autoload :NumericResult, 'mcollective/aggregate/result/numeric_result'
+ autoload :CollectionResult, 'mcollective/aggregate/result/collection_result'
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Aggregate
+ module Result
+ class Base
+ attr_accessor :result, :aggregate_format, :action
+
+ def initialize(result, aggregate_format, action)
+ raise "No aggregate_format defined in ddl or aggregate function" unless aggregate_format
+
+ @result = result
+ @aggregate_format = aggregate_format
+ @action = action
+ end
+
+ def to_s
+ raise "'to_s' method not implemented for result class '#{self.class}'"
+ end
+
+ def result_type
+ @result[:type]
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Aggregate
+ module Result
+ class CollectionResult<Base
+ def to_s
+ return "" if @result[:value].keys.include?(nil)
+
+ result = StringIO.new
+
+ @result[:value].sort{|x,y| x[1] <=> y[1]}.reverse.each do |value|
+ result.puts @aggregate_format % [value[0], value[1]]
+ end
+
+ result.string.chomp
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Aggregate
+ module Result
+ class NumericResult<Base
+ def to_s
+ return "" if @result[:value].nil?
+
+ return @aggregate_format % @result[:value]
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Application
+ include RPC
+
+ class << self
+ # Intialize a blank set of options if its the first time used
+ # else returns active options
+ def application_options
+ intialize_application_options unless @application_options
+ @application_options
+ end
+
+ # set an option in the options hash
+ def []=(option, value)
+ intialize_application_options unless @application_options
+ @application_options[option] = value
+ end
+
+ # retrieves a specific option
+ def [](option)
+ intialize_application_options unless @application_options
+ @application_options[option]
+ end
+
+ # Sets the application description, there can be only one
+ # description per application so multiple calls will just
+ # change the description
+ def description(descr)
+ self[:description] = descr
+ end
+
+ # Supplies usage information, calling multiple times will
+ # create multiple usage lines in --help output
+ def usage(usage)
+ self[:usage] << usage
+ end
+
+ def exclude_argument_sections(*sections)
+ sections = [sections].flatten
+
+ sections.each do |s|
+ raise "Unknown CLI argument section #{s}" unless ["rpc", "common", "filter"].include?(s)
+ end
+
+ intialize_application_options unless @application_options
+ self[:exclude_arg_sections] = sections
+ end
+
+ # Wrapper to create command line options
+ #
+ # - name: varaible name that will be used to access the option value
+ # - description: textual info shown in --help
+ # - arguments: a list of possible arguments that can be used
+ # to activate this option
+ # - type: a data type that ObjectParser understand of :bool or :array
+ # - required: true or false if this option has to be supplied
+ # - validate: a proc that will be called with the value used to validate
+ # the supplied value
+ #
+ # option :foo,
+ # :description => "The foo option"
+ # :arguments => ["--foo ARG"]
+ #
+ # after this the value supplied will be in configuration[:foo]
+ def option(name, arguments)
+ opt = {:name => name,
+ :description => nil,
+ :arguments => [],
+ :type => String,
+ :required => false,
+ :validate => Proc.new { true }}
+
+ arguments.each_pair{|k,v| opt[k] = v}
+
+ self[:cli_arguments] << opt
+ end
+
+ # Creates an empty set of options
+ def intialize_application_options
+ @application_options = {:description => nil,
+ :usage => [],
+ :cli_arguments => [],
+ :exclude_arg_sections => []}
+ end
+ end
+
+ # The application configuration built from CLI arguments
+ def configuration
+ @application_configuration ||= {}
+ @application_configuration
+ end
+
+ # The active options hash used for MC::Client and other configuration
+ def options
+ @options
+ end
+
+ # Calls the supplied block in an option for validation, an error raised
+ # will log to STDERR and exit the application
+ def validate_option(blk, name, value)
+ validation_result = blk.call(value)
+
+ unless validation_result == true
+ STDERR.puts "Validation of #{name} failed: #{validation_result}"
+ exit 1
+ end
+ end
+
+ # Creates a standard options hash, pass in a block to add extra headings etc
+ # see Optionparser
+ def clioptions(help)
+ oparser = Optionparser.new({:verbose => false, :progress_bar => true}, "filter", application_options[:exclude_arg_sections])
+
+ options = oparser.parse do |parser, options|
+ if block_given?
+ yield(parser, options)
+ end
+
+ RPC::Helpers.add_simplerpc_options(parser, options) unless application_options[:exclude_arg_sections].include?("rpc")
+ end
+
+ return oparser.parser.help if help
+
+ validate_cli_options
+
+ post_option_parser(configuration) if respond_to?(:post_option_parser)
+
+ return options
+ rescue Exception => e
+ application_failure(e)
+ end
+
+ # Builds an ObjectParser config, parse the CLI options and validates based
+ # on the option config
+ def application_parse_options(help=false)
+ @options ||= {:verbose => false}
+
+ @options = clioptions(help) do |parser, options|
+ parser.define_head application_description if application_description
+ parser.banner = ""
+
+ if application_usage
+ parser.separator ""
+
+ application_usage.each do |u|
+ parser.separator "Usage: #{u}"
+ end
+
+ parser.separator ""
+ end
+
+ parser.separator "Application Options" unless application_cli_arguments.empty?
+
+ parser.define_tail ""
+ parser.define_tail "The Marionette Collective #{MCollective.version}"
+
+
+ application_cli_arguments.each do |carg|
+ opts_array = []
+
+ opts_array << :on
+
+ # if a default is set from the application set it up front
+ if carg.include?(:default)
+ configuration[carg[:name]] = carg[:default]
+ end
+
+ # :arguments are multiple possible ones
+ if carg[:arguments].is_a?(Array)
+ carg[:arguments].each {|a| opts_array << a}
+ else
+ opts_array << carg[:arguments]
+ end
+
+ # type was given and its not one of our special types, just pass it onto optparse
+ opts_array << carg[:type] if carg[:type] && ![:boolean, :bool, :array].include?(carg[:type])
+
+ opts_array << carg[:description]
+
+ # Handle our special types else just rely on the optparser to handle the types
+ if [:bool, :boolean].include?(carg[:type])
+ parser.send(*opts_array) do |v|
+ validate_option(carg[:validate], carg[:name], v)
+
+ configuration[carg[:name]] = v
+ end
+
+ elsif carg[:type] == :array
+ parser.send(*opts_array) do |v|
+ validate_option(carg[:validate], carg[:name], v)
+
+ configuration[carg[:name]] = [] unless configuration.include?(carg[:name])
+ configuration[carg[:name]] << v
+ end
+
+ else
+ parser.send(*opts_array) do |v|
+ validate_option(carg[:validate], carg[:name], v)
+
+ configuration[carg[:name]] = v
+ end
+ end
+ end
+ end
+ end
+
+ def validate_cli_options
+ # Check all required parameters were set
+ validation_passed = true
+ application_cli_arguments.each do |carg|
+ # Check for required arguments
+ if carg[:required]
+ unless configuration[ carg[:name] ]
+ validation_passed = false
+ STDERR.puts "The #{carg[:name]} option is mandatory"
+ end
+ end
+ end
+
+ unless validation_passed
+ STDERR.puts "\nPlease run with --help for detailed help"
+ exit 1
+ end
+
+
+ end
+
+ # Retrieves the full hash of application options
+ def application_options
+ self.class.application_options
+ end
+
+ # Retrieve the current application description
+ def application_description
+ application_options[:description]
+ end
+
+ # Return the current usage text false if nothing is set
+ def application_usage
+ usage = application_options[:usage]
+
+ usage.empty? ? false : usage
+ end
+
+ # Returns an array of all the arguments built using
+ # calls to optin
+ def application_cli_arguments
+ application_options[:cli_arguments]
+ end
+
+ # Handles failure, if we're far enough in the initialization
+ # phase it will log backtraces if its in verbose mode only
+ def application_failure(e, err_dest=STDERR)
+ # peole can use exit() anywhere and not get nasty backtraces as a result
+ if e.is_a?(SystemExit)
+ disconnect
+ raise(e)
+ end
+
+ if e.is_a?(CodedError)
+ err_dest.puts "\nThe %s application failed to run: %s: %s\n" % [ Util.colorize(:bold, $0), Util.colorize(:bold, e.code), Util.colorize(:red, e.to_s)]
+
+ err_dest.puts
+ if options[:verbose]
+ err_dest.puts "Use the 'mco doc %s' command for details about this error" % e.code
+ else
+ err_dest.puts "Use the 'mco doc %s' command for details about this error, use -v for full error backtrace details" % e.code
+ end
+ else
+ if options[:verbose]
+ err_dest.puts "\nThe %s application failed to run: %s\n" % [ Util.colorize(:bold, $0), Util.colorize(:red, e.to_s)]
+ else
+ err_dest.puts "\nThe %s application failed to run, use -v for full error backtrace details: %s\n" % [ Util.colorize(:bold, $0), Util.colorize(:red, e.to_s)]
+ end
+ end
+
+ if options.nil? || options[:verbose]
+ e.backtrace.first << Util.colorize(:red, " <----")
+ err_dest.puts "\n%s %s" % [ Util.colorize(:red, e.to_s), Util.colorize(:bold, "(#{e.class.to_s})")]
+ e.backtrace.each{|l| err_dest.puts "\tfrom #{l}"}
+ end
+
+ disconnect
+
+ exit 1
+ end
+
+ def help
+ application_parse_options(true)
+ end
+
+ # The main logic loop, builds up the options, validate configuration and calls
+ # the main as supplied by the user. Disconnects when done and pass any exception
+ # onto the application_failure helper
+ def run
+ application_parse_options
+
+ validate_configuration(configuration) if respond_to?(:validate_configuration)
+
+ Util.setup_windows_sleeper if Util.windows?
+
+ main
+
+ disconnect
+
+ rescue Exception => e
+ application_failure(e)
+ end
+
+ def disconnect
+ MCollective::PluginManager["connector_plugin"].disconnect
+ rescue
+ end
+
+ # Fake abstract class that logs if the user tries to use an application without
+ # supplying a main override method.
+ def main
+ STDERR.puts "Applications need to supply a 'main' method"
+ exit 1
+ end
+
+ def halt_code(stats)
+ request_stats = {:discoverytime => 0,
+ :discovered => 0,
+ :okcount => 0,
+ :failcount => 0}.merge(stats.to_hash)
+
+ return 4 if request_stats[:discoverytime] == 0 && request_stats[:responses] == 0
+ return 3 if request_stats[:discovered] > 0 && request_stats[:responses] == 0
+ return 2 if request_stats[:discovered] > 0 && request_stats[:failcount] > 0
+ return 1 if request_stats[:discovered] == 0
+ return 0 if request_stats[:discoverytime] == 0 && request_stats[:discovered] == request_stats[:okcount]
+ return 0 if request_stats[:discovered] == request_stats[:okcount]
+ end
+
+ # A helper that creates a consistent exit code for applications by looking at an
+ # instance of MCollective::RPC::Stats
+ #
+ # Exit with 0 if nodes were discovered and all passed
+ # Exit with 0 if no discovery were done and > 0 responses were received, all ok
+ # Exit with 1 if no nodes were discovered
+ # Exit with 2 if nodes were discovered but some RPC requests failed
+ # Exit with 3 if nodes were discovered, but no responses received
+ # Exit with 4 if no discovery were done and no responses were received
+ def halt(stats)
+ exit(halt_code(stats))
+ end
+
+ # Wrapper around MC::RPC#rpcclient that forcably supplies our options hash
+ # if someone forgets to pass in options in an application the filters and other
+ # cli options wouldnt take effect which could have a disasterous outcome
+ def rpcclient(agent, flags = {})
+ flags[:options] = options unless flags.include?(:options)
+ flags[:exit_on_failure] = false
+
+ super
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Applications
+ def self.[](appname)
+ load_application(appname)
+ PluginManager["#{appname}_application"]
+ end
+
+ def self.run(appname)
+ load_config
+
+ begin
+ load_application(appname)
+ rescue Exception => e
+ e.backtrace.first << Util.colorize(:red, " <----")
+ STDERR.puts "Application '#{appname}' failed to load:"
+ STDERR.puts
+ STDERR.puts Util.colorize(:red, " #{e} (#{e.class})")
+ STDERR.puts
+ STDERR.puts " %s" % [e.backtrace.join("\n ")]
+ exit 1
+ end
+
+ PluginManager["#{appname}_application"].run
+ end
+
+ def self.load_application(appname)
+ return if PluginManager.include?("#{appname}_application")
+
+ load_config
+
+ PluginManager.loadclass "MCollective::Application::#{appname.capitalize}"
+ PluginManager << {:type => "#{appname}_application", :class => "MCollective::Application::#{appname.capitalize}"}
+ end
+
+ # Returns an array of applications found in the lib dirs
+ def self.list
+ load_config
+
+ PluginManager.find("application")
+ rescue SystemExit
+ exit 1
+ rescue Exception => e
+ STDERR.puts "Failed to generate application list: #{e.class}: #{e}"
+ exit 1
+ end
+
+ # Filters a string of opts out using Shellwords
+ # keeping only things related to --config and -c
+ def self.filter_extra_options(opts)
+ res = ""
+ words = Shellwords.shellwords(opts)
+ words.each_with_index do |word,idx|
+ if word == "-c"
+ return "--config=#{words[idx + 1]}"
+ elsif word == "--config"
+ return "--config=#{words[idx + 1]}"
+ elsif word =~ /\-c=/
+ return word
+ elsif word =~ /\-\-config=/
+ return word
+ end
+ end
+
+ return ""
+ end
+
+ # We need to know the config file in order to know the libdir
+ # so that we can find applications.
+ #
+ # The problem is the CLI might be stuffed with options only the
+ # app in the libdir might understand so we have a chicken and
+ # egg situation.
+ #
+ # We're parsing and filtering MCOLLECTIVE_EXTRA_OPTS removing
+ # all but config related options and parsing the options looking
+ # just for the config file.
+ #
+ # We're handling failures gracefully and finally restoring the
+ # ARG and MCOLLECTIVE_EXTRA_OPTS to the state they were before
+ # we started parsing.
+ #
+ # This is mostly a hack, when we're redoing how config works
+ # this stuff should be made less sucky
+ def self.load_config
+ return if Config.instance.configured
+
+ original_argv = ARGV.clone
+ original_extra_opts = ENV["MCOLLECTIVE_EXTRA_OPTS"].clone rescue nil
+ configfile = nil
+
+ parser = OptionParser.new
+ parser.on("--config CONFIG", "-c", "Config file") do |f|
+ configfile = f
+ end
+
+ parser.program_name = $0
+
+ parser.on("--help")
+
+ # avoid option parsers own internal version handling that sux
+ parser.on("-v", "--verbose")
+
+ if original_extra_opts
+ begin
+ # optparse will parse the whole ENV in one go and refuse
+ # to play along with the retry trick I do below so in
+ # order to handle unknown options properly I parse out
+ # only -c and --config deleting everything else and
+ # then restore the environment variable later when I
+ # am done with it
+ ENV["MCOLLECTIVE_EXTRA_OPTS"] = filter_extra_options(ENV["MCOLLECTIVE_EXTRA_OPTS"].clone)
+ parser.environment("MCOLLECTIVE_EXTRA_OPTS")
+ rescue Exception => e
+ Log.error("Failed to parse MCOLLECTIVE_EXTRA_OPTS: #{e}")
+ end
+
+ ENV["MCOLLECTIVE_EXTRA_OPTS"] = original_extra_opts.clone
+ end
+
+ begin
+ parser.parse!
+ rescue OptionParser::InvalidOption => e
+ retry
+ end
+
+ ARGV.clear
+ original_argv.each {|a| ARGV << a}
+
+ configfile = Util.config_file_for_user unless configfile
+
+ Config.instance.loadconfig(configfile)
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # A class to manage a number of named caches. Each cache can have a unique
+ # timeout for keys in it and each cache has an independent Mutex protecting
+ # access to it.
+ #
+ # This cache is setup early in the process of loading the mcollective
+ # libraries, before any threads are created etc making it suitable as a
+ # cross thread cache or just a store for Mutexes you need to use across
+ # threads like in an Agent or something.
+ #
+ # # sets up a new cache, noop if it already exist
+ # Cache.setup(:ddl, 600)
+ # => true
+ #
+ # # writes an item to the :ddl cache, this item will
+ # # be valid on the cache for 600 seconds
+ # Cache.write(:ddl, :something, "value")
+ # => "value"
+ #
+ # # reads from the cache, read failures due to non existing
+ # # data or expired data will raise an exception
+ # Cache.read(:ddl, :something)
+ # => "value"
+ #
+ # # time left till expiry, raises if nothing is found
+ # Cache.ttl(:ddl, :something)
+ # => 500
+ #
+ # # forcibly evict something from the cache
+ # Cache.invalidate!(:ddl, :something)
+ # => "value"
+ #
+ # # deletes an entire named cache and its mutexes
+ # Cache.delete!(:ddl)
+ # => true
+ #
+ # # you can also just use this cache as a global mutex store
+ # Cache.setup(:mylock)
+ #
+ # Cache.synchronize(:mylock) do
+ # do_something()
+ # end
+ #
+ module Cache
+ extend Translatable
+
+ # protects access to @cache_locks and top level @cache
+ @locks_mutex = Mutex.new
+
+ # stores a mutex per named cache
+ @cache_locks = {}
+
+ # the named caches protected by items in @cache_locks
+ @cache = {}
+
+ def self.setup(cache_name, ttl=300)
+ @locks_mutex.synchronize do
+ break if @cache_locks.include?(cache_name)
+
+ @cache_locks[cache_name] = Mutex.new
+
+ @cache_locks[cache_name].synchronize do
+ @cache[cache_name] = {:max_age => Float(ttl)}
+ end
+ end
+
+ true
+ end
+
+ def self.check_cache!(cache_name)
+ raise_code(:PLMC13, "Could not find a cache called '%{cache_name}'", :debug, :cache_name => cache_name) unless has_cache?(cache_name)
+ end
+
+ def self.has_cache?(cache_name)
+ @locks_mutex.synchronize do
+ @cache.include?(cache_name)
+ end
+ end
+
+ def self.delete!(cache_name)
+ check_cache!(cache_name)
+
+ @locks_mutex.synchronize do
+ @cache_locks.delete(cache_name)
+ @cache.delete(cache_name)
+ end
+
+ true
+ end
+
+ def self.write(cache_name, key, value)
+ check_cache!(cache_name)
+
+ @cache_locks[cache_name].synchronize do
+ @cache[cache_name][key] ||= {}
+ @cache[cache_name][key][:cache_create_time] = Time.now
+ @cache[cache_name][key][:value] = value
+ end
+
+ value
+ end
+
+ def self.read(cache_name, key)
+ check_cache!(cache_name)
+
+ unless ttl(cache_name, key) > 0
+ raise_code(:PLMC11, "Cache expired on '%{cache_name}' key '%{item}'", :debug, :cache_name => cache_name, :item => key)
+ end
+
+ log_code(:PLMC12, "Cache hit on '%{cache_name}' key '%{item}'", :debug, :cache_name => cache_name, :item => key)
+
+ @cache_locks[cache_name].synchronize do
+ @cache[cache_name][key][:value]
+ end
+ end
+
+ def self.ttl(cache_name, key)
+ check_cache!(cache_name)
+
+ @cache_locks[cache_name].synchronize do
+ unless @cache[cache_name].include?(key)
+ raise_code(:PLMC15, "No item called '%{item}' for cache '%{cache_name}'", :debug, :cache_name => cache_name, :item => key)
+ end
+
+ @cache[cache_name][:max_age] - (Time.now - @cache[cache_name][key][:cache_create_time])
+ end
+ end
+
+ def self.synchronize(cache_name)
+ check_cache!(cache_name)
+
+ raise_code(:PLMC14, "No block supplied to synchronize on cache '%{cache_name}'", :debug, :cache_name => cache_name) unless block_given?
+
+ @cache_locks[cache_name].synchronize do
+ yield
+ end
+ end
+
+ def self.invalidate!(cache_name, key)
+ check_cache!(cache_name)
+
+ @cache_locks[cache_name].synchronize do
+ return false unless @cache[cache_name].include?(key)
+
+ @cache[cache_name].delete(key)
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # Helpers for writing clients that can talk to agents, do discovery and so forth
+ class Client
+ attr_accessor :options, :stats, :discoverer
+
+ def initialize(configfile)
+ @config = Config.instance
+ @config.loadconfig(configfile) unless @config.configured
+
+ @connection = PluginManager["connector_plugin"]
+ @security = PluginManager["security_plugin"]
+
+ @security.initiated_by = :client
+ @options = nil
+ @subscriptions = {}
+
+ @discoverer = Discovery.new(self)
+ @connection.connect
+ end
+
+ # Returns the configured main collective if no
+ # specific collective is specified as options
+ def collective
+ if @options[:collective].nil?
+ @config.main_collective
+ else
+ @options[:collective]
+ end
+ end
+
+ # Disconnects cleanly from the middleware
+ def disconnect
+ Log.debug("Disconnecting from the middleware")
+ @connection.disconnect
+ end
+
+ # Sends a request and returns the generated request id, doesn't wait for
+ # responses and doesn't execute any passed in code blocks for responses
+ def sendreq(msg, agent, filter = {})
+ if msg.is_a?(Message)
+ request = msg
+ agent = request.agent
+ else
+ ttl = @options[:ttl] || @config.ttl
+ request = Message.new(msg, nil, {:agent => agent, :type => :request, :collective => collective, :filter => filter, :ttl => ttl})
+ request.reply_to = @options[:reply_to] if @options[:reply_to]
+ end
+
+ request.encode!
+
+ Log.debug("Sending request #{request.requestid} to the #{request.agent} agent with ttl #{request.ttl} in collective #{request.collective}")
+
+ subscribe(agent, :reply) unless request.reply_to
+
+ request.publish
+
+ request.requestid
+ end
+
+ def subscribe(agent, type)
+ unless @subscriptions.include?(agent)
+ subscription = Util.make_subscriptions(agent, type, collective)
+ Log.debug("Subscribing to #{type} target for agent #{agent}")
+
+ Util.subscribe(subscription)
+ @subscriptions[agent] = 1
+ end
+ end
+
+ def unsubscribe(agent, type)
+ if @subscriptions.include?(agent)
+ subscription = Util.make_subscriptions(agent, type, collective)
+ Log.debug("Unsubscribing #{type} target for #{agent}")
+
+ Util.unsubscribe(subscription)
+ @subscriptions.delete(agent)
+ end
+ end
+ # Blocking call that waits for ever for a message to arrive.
+ #
+ # If you give it a requestid this means you've previously send a request
+ # with that ID and now you just want replies that matches that id, in that
+ # case the current connection will just ignore all messages not directed at it
+ # and keep waiting for more till it finds a matching message.
+ def receive(requestid = nil)
+ reply = nil
+
+ begin
+ reply = @connection.receive
+ reply.type = :reply
+ reply.expected_msgid = requestid
+
+ reply.decode!
+
+ reply.payload[:senderid] = Digest::MD5.hexdigest(reply.payload[:senderid]) if ENV.include?("MCOLLECTIVE_ANON")
+
+ raise(MsgDoesNotMatchRequestID, "Message reqid #{requestid} does not match our reqid #{reply.requestid}") unless reply.requestid == requestid
+ rescue SecurityValidationFailed => e
+ Log.warn("Ignoring a message that did not pass security validations")
+ retry
+ rescue MsgDoesNotMatchRequestID => e
+ Log.debug("Ignoring a message for some other client")
+ retry
+ end
+
+ reply
+ end
+
+ # Performs a discovery of nodes matching the filter passed
+ # returns an array of nodes
+ #
+ # An integer limit can be supplied this will have the effect
+ # of the discovery being cancelled soon as it reached the
+ # requested limit of hosts
+ def discover(filter, timeout, limit=0)
+ discovered = @discoverer.discover(filter, timeout, limit)
+ end
+
+ # Send a request, performs the passed block for each response
+ #
+ # times = req("status", "mcollectived", options, client) {|resp|
+ # pp resp
+ # }
+ #
+ # It returns a hash of times and timeouts for discovery and total run is taken from the options
+ # hash which in turn is generally built using MCollective::Optionparser
+ def req(body, agent=nil, options=false, waitfor=0)
+ if body.is_a?(Message)
+ agent = body.agent
+ waitfor = body.discovered_hosts.size || 0
+ @options = body.options
+ end
+
+ @options = options if options
+
+ stat = {:starttime => Time.now.to_f, :discoverytime => 0, :blocktime => 0, :totaltime => 0}
+
+ timeout = @discoverer.discovery_timeout(@options[:timeout], @options[:filter])
+
+ STDOUT.sync = true
+
+ hosts_responded = 0
+ reqid = nil
+
+ begin
+ Log.debug("Publishing request to agent %s with timeout %d" % [agent, timeout])
+
+ Timeout.timeout(timeout) do
+ reqid = sendreq(body, agent, @options[:filter])
+
+ loop do
+ resp = receive(reqid)
+
+ hosts_responded += 1
+
+ yield(resp.payload)
+
+ break if (waitfor != 0 && hosts_responded >= waitfor)
+ end
+ end
+ rescue Interrupt => e
+ rescue Timeout::Error => e
+ ensure
+ unsubscribe(agent, :reply)
+ end
+
+ stat[:totaltime] = Time.now.to_f - stat[:starttime]
+ stat[:blocktime] = stat[:totaltime] - stat[:discoverytime]
+ stat[:responses] = hosts_responded
+ stat[:noresponsefrom] = []
+ stat[:requestid] = reqid
+
+ @stats = stat
+ return stat
+ end
+
+ def discovered_req(body, agent, options=false)
+ raise "Client#discovered_req has been removed, please port your agent and client to the SimpleRPC framework"
+ end
+
+ # Prints out the stats returns from req and discovered_req in a nice way
+ def display_stats(stats, options=false, caption="stomp call summary")
+ options = @options unless options
+
+ if options[:verbose]
+ puts("\n---- #{caption} ----")
+
+ if stats[:discovered]
+ puts(" Nodes: #{stats[:discovered]} / #{stats[:responses]}")
+ else
+ puts(" Nodes: #{stats[:responses]}")
+ end
+
+ printf(" Start Time: %s\n", Time.at(stats[:starttime]))
+ printf(" Discovery Time: %.2fms\n", stats[:discoverytime] * 1000)
+ printf(" Agent Time: %.2fms\n", stats[:blocktime] * 1000)
+ printf(" Total Time: %.2fms\n", stats[:totaltime] * 1000)
+
+ else
+ if stats[:discovered]
+ printf("\nFinished processing %d / %d hosts in %.2f ms\n\n", stats[:responses], stats[:discovered], stats[:blocktime] * 1000)
+ else
+ printf("\nFinished processing %d hosts in %.2f ms\n\n", stats[:responses], stats[:blocktime] * 1000)
+ end
+ end
+
+ if stats[:noresponsefrom].size > 0
+ puts("\nNo response from:\n")
+
+ stats[:noresponsefrom].each do |c|
+ puts if c % 4 == 1
+ printf("%30s", c)
+ end
+
+ puts
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # A pretty sucky config class, ripe for refactoring/improving
+ class Config
+ include Singleton
+
+ attr_accessor :mode
+
+ attr_reader :daemonize, :pluginconf, :libdir, :configured
+ attr_reader :logfile, :keeplogs, :max_log_size, :loglevel, :logfacility
+ attr_reader :identity, :daemonize, :connector, :securityprovider, :factsource
+ attr_reader :registration, :registerinterval, :classesfile
+ attr_reader :rpcauditprovider, :rpcaudit, :configdir, :rpcauthprovider
+ attr_reader :rpcauthorization, :color, :configfile, :rpchelptemplate
+ attr_reader :rpclimitmethod, :logger_type, :fact_cache_time, :collectives
+ attr_reader :main_collective, :ssl_cipher, :registration_collective
+ attr_reader :direct_addressing, :direct_addressing_threshold, :ttl, :helptemplatedir
+ attr_reader :default_discovery_method, :default_discovery_options
+
+ def initialize
+ @configured = false
+ end
+
+ def loadconfig(configfile)
+ set_config_defaults(configfile)
+
+ if File.exists?(configfile)
+ File.readlines(configfile).each do |line|
+
+ # strip blank spaces, tabs etc off the end of all lines
+ line.gsub!(/\s*$/, "")
+
+ unless line =~ /^#|^$/
+ if (line =~ /(.+?)\s*=\s*(.+)/)
+ key = $1
+ val = $2
+
+ case key
+ when "registration"
+ @registration = val.capitalize
+ when "registration_collective"
+ @registration_collective = val
+ when "registerinterval"
+ @registerinterval = val.to_i
+ when "collectives"
+ @collectives = val.split(",").map {|c| c.strip}
+ when "main_collective"
+ @main_collective = val
+ when "logfile"
+ @logfile = val
+ when "keeplogs"
+ @keeplogs = val.to_i
+ when "max_log_size"
+ @max_log_size = val.to_i
+ when "loglevel"
+ @loglevel = val
+ when "logfacility"
+ @logfacility = val
+ when "libdir"
+ paths = val.split(File::PATH_SEPARATOR)
+ paths.each do |path|
+ raise("libdir paths should be absolute paths but '%s' is relative" % path) unless Util.absolute_path?(path)
+
+ @libdir << path
+ unless $LOAD_PATH.include?(path)
+ $LOAD_PATH << path
+ end
+ end
+ when "identity"
+ @identity = val
+ when "direct_addressing"
+ @direct_addressing = false unless val =~ /^1|y/i
+ when "direct_addressing_threshold"
+ @direct_addressing_threshold = val.to_i
+ when "color"
+ val =~ /^1|y/i ? @color = true : @color = false
+ when "daemonize"
+ val =~ /^1|y/i ? @daemonize = true : @daemonize = false
+ when "securityprovider"
+ @securityprovider = val.capitalize
+ when "factsource"
+ @factsource = val.capitalize
+ when "connector"
+ @connector = val.capitalize
+ when "classesfile"
+ @classesfile = val
+ when /^plugin.(.+)$/
+ @pluginconf[$1] = val
+ when "rpcaudit"
+ val =~ /^1|y/i ? @rpcaudit = true : @rpcaudit = false
+ when "rpcauditprovider"
+ @rpcauditprovider = val.capitalize
+ when "rpcauthorization"
+ val =~ /^1|y/i ? @rpcauthorization = true : @rpcauthorization = false
+ when "rpcauthprovider"
+ @rpcauthprovider = val.capitalize
+ when "rpchelptemplate"
+ @rpchelptemplate = val
+ when "rpclimitmethod"
+ @rpclimitmethod = val.to_sym
+ when "logger_type"
+ @logger_type = val
+ when "fact_cache_time"
+ @fact_cache_time = val.to_i
+ when "ssl_cipher"
+ @ssl_cipher = val
+ when "ttl"
+ @ttl = val.to_i
+ when "helptemplatedir"
+ @helptemplatedir = val
+ when "default_discovery_options"
+ @default_discovery_options << val
+ when "default_discovery_method"
+ @default_discovery_method = val
+ else
+ raise("Unknown config parameter #{key}")
+ end
+ end
+ end
+ end
+
+ raise('The %s config file does not specify a libdir setting, cannot continue' % configfile) if @libdir.empty?
+
+ I18n.load_path = Dir[File.expand_path(File.join(File.dirname(__FILE__), "locales", "*.yml"))]
+ I18n.locale = :en
+
+ read_plugin_config_dir("#{@configdir}/plugin.d")
+
+ raise 'Identities can only match /\w\.\-/' unless @identity.match(/^[\w\.\-]+$/)
+
+ @configured = true
+
+ @libdir.each {|dir| Log.warn("Cannot find libdir: #{dir}") unless File.directory?(dir)}
+
+ if @logger_type == "syslog"
+ raise "The sylog logger is not usable on the Windows platform" if Util.windows?
+ end
+
+ PluginManager.loadclass("Mcollective::Facts::#{@factsource}_facts")
+ PluginManager.loadclass("Mcollective::Connector::#{@connector}")
+ PluginManager.loadclass("Mcollective::Security::#{@securityprovider}")
+ PluginManager.loadclass("Mcollective::Registration::#{@registration}")
+ PluginManager.loadclass("Mcollective::Audit::#{@rpcauditprovider}") if @rpcaudit
+ PluginManager << {:type => "global_stats", :class => RunnerStats.new}
+
+ Log.logmsg(:PLMC1, "The Marionette Collective version %{version} started by %{name} using config file %{config}", :info, :version => MCollective::VERSION, :name => $0, :config => configfile)
+ else
+ raise("Cannot find config file '#{configfile}'")
+ end
+ end
+
+ def set_config_defaults(configfile)
+ @stomp = Hash.new
+ @subscribe = Array.new
+ @pluginconf = Hash.new
+ @connector = "activemq"
+ @securityprovider = "Psk"
+ @factsource = "Yaml"
+ @identity = Socket.gethostname
+ @registration = "Agentlist"
+ @registerinterval = 0
+ @registration_collective = nil
+ @classesfile = "/var/lib/puppet/state/classes.txt"
+ @rpcaudit = false
+ @rpcauditprovider = ""
+ @rpcauthorization = false
+ @rpcauthprovider = ""
+ @configdir = File.dirname(configfile)
+ @color = !Util.windows?
+ @configfile = configfile
+ @logger_type = "file"
+ @keeplogs = 5
+ @max_log_size = 2097152
+ @rpclimitmethod = :first
+ @libdir = Array.new
+ @fact_cache_time = 300
+ @loglevel = "info"
+ @logfacility = "user"
+ @collectives = ["mcollective"]
+ @main_collective = @collectives.first
+ @ssl_cipher = "aes-256-cbc"
+ @direct_addressing = true
+ @direct_addressing_threshold = 10
+ @default_discovery_method = "mc"
+ @default_discovery_options = []
+ @ttl = 60
+ @mode = :client
+
+ # look in the config dir for the template so users can provide their own and windows
+ # with odd paths will just work more often, but fall back to old behavior if it does
+ # not exist
+ @rpchelptemplate = File.join(File.dirname(configfile), "rpc-help.erb")
+ @rpchelptemplate = "/etc/mcollective/rpc-help.erb" unless File.exists?(@rpchelptemplate)
+ @helptemplatedir = File.dirname(@rpchelptemplate)
+ end
+
+ def read_plugin_config_dir(dir)
+ return unless File.directory?(dir)
+
+ Dir.new(dir).each do |pluginconfigfile|
+ next unless pluginconfigfile =~ /^([\w]+).cfg$/
+
+ plugin = $1
+ File.open("#{dir}/#{pluginconfigfile}", "r").each do |line|
+ # strip blank lines
+ line.gsub!(/\s*$/, "")
+ next if line =~ /^#|^$/
+ if (line =~ /(.+?)\s*=\s*(.+)/)
+ key = $1
+ val = $2
+ @pluginconf["#{plugin}.#{key}"] = val
+ end
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # Connectors take care of transporting messages between clients and agents,
+ # the design doesn't your middleware to be very rich in features. All it
+ # really needs is the ability to send and receive messages to named queues/topics.
+ #
+ # At present there are assumptions about the naming of topics and queues that is
+ # compatible with Stomp, ie.
+ #
+ # /topic/foo.bar/baz
+ # /queue/foo.bar/baz
+ #
+ # This is the only naming format that is supported, but you could replace Stomp
+ # with something else that supports the above, see MCollective::Connector::Stomp
+ # for the default connector.
+ module Connector
+ autoload :Base, "mcollective/connector/base"
+ end
+end
--- /dev/null
+module MCollective
+ # Connector plugins handle the communications with the middleware, you can provide your own to speak
+ # to something other than Stomp, your plugins must inherit from MCollective::Connector::Base and should
+ # provide the following methods:
+ #
+ # connect - Creates a connection to the middleware, no arguments should get its parameters from the config
+ # receive - Receive data from the middleware, should act like a blocking call only returning if/when data
+ # was received. It should get data from all subscribed channels/topics. Individual messages
+ # should be returned as MCollective::Request objects with the payload provided
+ # publish - Takes a target and msg, should send the message to the supplied target topic or destination
+ # subscribe - Adds a subscription to a specific message source
+ # unsubscribe - Removes a subscription to a specific message source
+ # disconnect - Disconnects from the middleware
+ #
+ # These methods are all that's needed for a new connector protocol and should hopefully be simple
+ # enough to not have tied us to Stomp.
+ module Connector
+ class Base
+ def self.inherited(klass)
+ PluginManager << {:type => "connector_plugin", :class => klass.to_s}
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Data
+ autoload :Base, "mcollective/data/base"
+ autoload :Result, "mcollective/data/result"
+
+ def self.load_data_sources
+ PluginManager.find_and_load("data")
+
+ PluginManager.grep(/_data$/).each do |plugin|
+ begin
+ unless PluginManager[plugin].class.activate?
+ Log.debug("Disabling data plugin %s due to plugin activation policy" % plugin)
+ PluginManager.delete(plugin)
+ end
+ rescue Exception => e
+ Log.debug("Disabling data plugin %s due to exception #{e.class}: #{e}" % plugin)
+ PluginManager.delete(plugin)
+ end
+ end
+ end
+
+ def self.pluginname(plugin)
+ plugin.to_s =~ /_data$/i ? plugin.to_s.downcase : "%s_data" % plugin.to_s.downcase
+ end
+
+ def self.[](plugin)
+ PluginManager[pluginname(plugin)]
+ end
+
+ # Data.package("httpd").architecture
+ def self.method_missing(method, *args)
+ super unless PluginManager.include?(pluginname(method))
+
+ PluginManager[pluginname(method)].lookup(args.first)
+ end
+
+ def self.ddl(plugin)
+ DDL.new(pluginname(plugin), :data)
+ end
+
+ def self.ddl_validate(ddl, argument)
+ name = ddl.meta[:name]
+ query = ddl.entities[:data]
+
+ DDL.validation_fail!(:PLMC31, "No dataquery has been defined in the DDL for data plugin '%{plugin}'", :error, :plugin => name) unless query
+
+ input = query.fetch(:input, {})
+ output = query.fetch(:output, {})
+
+ DDL.validation_fail!(:PLMC32, "No output has been defined in the DDL for data plugin %{plugin}", :error, :plugin => name) if output.keys.empty?
+
+ if input[:query]
+ return true if argument.nil? && input[:query][:optional]
+
+ ddl.validate_input_argument(input, :query, argument)
+ else
+ DDL.validation_fail!(:PLMC33, "No data plugin argument was declared in the '%{plugin}' DDL but an input was supplied", :error, :plugin => name) if argument
+ return true
+ end
+ end
+
+ def self.ddl_has_output?(ddl, output)
+ ddl.entities[:data][:output].include?(output.to_sym) rescue false
+ end
+
+ # For an input where the DDL requests a boolean or some number
+ # this will convert the input to the right type where possible
+ # else just returns the origin input unedited
+ #
+ # if anything here goes wrong just return the input value
+ # this is not really the end of the world or anything since
+ # all that will happen is that DDL validation will fail and
+ # the user will get an error, no need to be too defensive here
+ def self.ddl_transform_input(ddl, input)
+ begin
+ type = ddl.entities[:data][:input][:query][:type]
+
+ case type
+ when :boolean
+ return DDL.string_to_boolean(input)
+
+ when :number, :integer, :float
+ return DDL.string_to_number(input)
+ end
+ rescue
+ end
+
+ return input
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Data
+ class Base
+ attr_reader :name, :result, :ddl, :timeout
+
+ # Register plugins that inherits base
+ def self.inherited(klass)
+ type = klass.to_s.split("::").last.downcase
+
+ PluginManager << {:type => type, :class => klass.to_s, :single_instance => false}
+ end
+
+ def initialize
+ @name = self.class.to_s.split("::").last.downcase
+ @ddl = DDL.new(@name, :data)
+ @result = Result.new(@ddl.dataquery_interface[:output])
+ @timeout = @ddl.meta[:timeout] || 1
+
+ startup_hook
+ end
+
+ def lookup(what)
+ ddl_validate(what)
+
+ Log.debug("Doing data query %s for '%s'" % [ @name, what ])
+
+ Timeout::timeout(@timeout) do
+ query_data(what)
+ end
+
+ @result
+ rescue Timeout::Error
+ # Timeout::Error is a inherited from Interrupt which seems a really
+ # strange choice, making it an equivelant of ^C and such. Catch it
+ # and raise something less critical that will not the runner to just
+ # give up the ghost
+ msg = "Data plugin %s timed out on query '%s'" % [@name, what]
+ Log.error(msg)
+ raise MsgTTLExpired, msg
+ end
+
+ def self.query(&block)
+ self.module_eval { define_method("query_data", &block) }
+ end
+
+ def ddl_validate(what)
+ Data.ddl_validate(@ddl, what)
+ end
+
+ # activate_when do
+ # file.exist?("/usr/bin/puppet")
+ # end
+ def self.activate_when(&block)
+ (class << self; self; end).instance_eval do
+ define_method("activate?", &block)
+ end
+ end
+
+ # Always be active unless a specific block is given with activate_when
+ def self.activate?
+ return true
+ end
+
+ def startup_hook;end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Data
+ class Result
+ # remove some methods that might clash with commonly
+ # used return data to improve the effectiveness of the
+ # method_missing lookup strategy
+ undef :type if method_defined?(:type)
+
+ def initialize(outputs)
+ @data = {}
+
+ outputs.keys.each do |output|
+ @data[output] = outputs[output].fetch(:default, nil)
+ end
+ end
+
+ def include?(key)
+ @data.include?(key.to_sym)
+ end
+
+ def [](key)
+ @data[key.to_sym]
+ end
+
+ def []=(key, val)
+ raise "Can only store String, Integer, Float or Boolean data but got #{val.class} for key #{key}" unless [String, Fixnum, Bignum, Float, TrueClass, FalseClass].include?(val.class)
+
+ @data[key.to_sym] = val
+ end
+
+ def keys
+ @data.keys
+ end
+
+ def method_missing(method, *args)
+ key = method.to_sym
+
+ raise NoMethodError, "undefined local variable or method `%s'" % key unless include?(key)
+
+ @data[key]
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # A set of classes that helps create data description language files
+ # for plugins. You can define meta data, actions, input and output
+ # describing the behavior of your agent or other plugins
+ #
+ # DDL files are used for input validation, constructing outputs,
+ # producing online help, informing the various display routines and
+ # so forth.
+ #
+ # A sample DDL for an agent be seen below, you'd put this in your agent
+ # dir as <agent name>.ddl
+ #
+ # metadata :name => "SimpleRPC Service Agent",
+ # :description => "Agent to manage services using the Puppet service provider",
+ # :author => "R.I.Pienaar",
+ # :license => "GPLv2",
+ # :version => "1.1",
+ # :url => "http://mcollective-plugins.googlecode.com/",
+ # :timeout => 60
+ #
+ # action "status", :description => "Gets the status of a service" do
+ # display :always
+ #
+ # input :service,
+ # :prompt => "Service Name",
+ # :description => "The service to get the status for",
+ # :type => :string,
+ # :validation => '^[a-zA-Z\-_\d]+$',
+ # :optional => true,
+ # :maxlength => 30
+ #
+ # output :status,
+ # :description => "The status of service",
+ # :display_as => "Service Status"
+ # end
+ #
+ # There are now many types of DDL and ultimately all pugins should have
+ # DDL files. The code is organized so that any plugin type will magically
+ # just work - they will be an instane of Base which has #metadata and a few
+ # common cases.
+ #
+ # For plugin types that require more specific behaviors they can just add a
+ # class here that inherits from Base and add their specific behavior.
+ #
+ # Base defines a specific behavior for input, output and metadata which we'd
+ # like to keep standard across plugin types so do not completely override the
+ # behavior of input. The methods are written that they will gladly store extra
+ # content though so you add, do not remove. See the AgentDDL class for an example
+ # where agents want a :required argument to be always set.
+ module DDL
+ autoload :Base, "mcollective/ddl/base"
+ autoload :AgentDDL, "mcollective/ddl/agentddl"
+ autoload :DataDDL, "mcollective/ddl/dataddl"
+ autoload :DiscoveryDDL, "mcollective/ddl/discoveryddl"
+
+ extend Translatable
+
+ # There used to be only one big nasty DDL class with a bunch of mashed
+ # together behaviors. It's been around for ages and we would rather not
+ # ask all the users to change their DDL.new calls to some other factory
+ # method that would have this exact same behavior.
+ #
+ # So we override the behavior of #new which is a hugely sucky thing to do
+ # but ultimately it's what would be least disrupting to code out there
+ # today. We did though change DDL to a module to make it possibly a
+ # little less suprising, possibly.
+ def self.new(*args, &blk)
+ load_and_cache(*args)
+ end
+
+ def self.load_and_cache(*args)
+ Cache.setup(:ddl, 300)
+
+ plugin = args.first
+ args.size > 1 ? type = args[1].to_s : type = "agent"
+ path = "%s/%s" % [type, plugin]
+
+ begin
+ ddl = Cache.read(:ddl, path)
+ rescue
+ begin
+ klass = DDL.const_get("%sDDL" % type.capitalize)
+ rescue NameError
+ klass = Base
+ end
+
+ ddl = Cache.write(:ddl, path, klass.new(*args))
+ end
+
+ return ddl
+ end
+
+ # As we're taking arguments on the command line we need a
+ # way to input booleans, true on the cli is a string so this
+ # method will take the ddl, find all arguments that are supposed
+ # to be boolean and if they are the strings "true"/"yes" or "false"/"no"
+ # turn them into the matching boolean
+ def self.string_to_boolean(val)
+ return true if ["true", "t", "yes", "y", "1"].include?(val.downcase)
+ return false if ["false", "f", "no", "n", "0"].include?(val.downcase)
+
+ raise_code(:PLMC17, "%{value} does not look like a boolean argument", :debug, :value => val)
+ end
+
+ # a generic string to number function, if a number looks like a float
+ # it turns it into a float else an int. This is naive but should be sufficient
+ # for numbers typed on the cli in most cases
+ def self.string_to_number(val)
+ return val.to_f if val =~ /^\d+\.\d+$/
+ return val.to_i if val =~ /^\d+$/
+
+ raise_code(:PLMC16, "%{value} does not look like a numeric value", :debug, :value => val)
+ end
+
+ # Various DDL implementations will validate and raise on error, this is a
+ # utility method to correctly setup a DDLValidationError exceptions and raise them
+ def self.validation_fail!(code, default, level, args={})
+ exception = DDLValidationError.new(code, default, level, args)
+ exception.set_backtrace caller
+
+ raise exception
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module DDL
+ # A DDL class specific to agent plugins.
+ #
+ # A full DDL can be seen below with all the possible bells and whistles present.
+ #
+ # metadata :name => "Utilities and Helpers for SimpleRPC Agents",
+ # :description => "General helpful actions that expose stats and internals to SimpleRPC clients",
+ # :author => "R.I.Pienaar <rip@devco.net>",
+ # :license => "Apache License, Version 2.0",
+ # :version => "1.0",
+ # :url => "http://marionette-collective.org/",
+ # :timeout => 10
+ #
+ # action "get_fact", :description => "Retrieve a single fact from the fact store" do
+ # display :always
+ #
+ # input :fact,
+ # :prompt => "The name of the fact",
+ # :description => "The fact to retrieve",
+ # :type => :string,
+ # :validation => '^[\w\-\.]+$',
+ # :optional => false,
+ # :maxlength => 40,
+ # :default => "fqdn"
+ #
+ # output :fact,
+ # :description => "The name of the fact being returned",
+ # :display_as => "Fact"
+ #
+ # output :value,
+ # :description => "The value of the fact",
+ # :display_as => "Value",
+ # :default => ""
+ #
+ # summarize do
+ # aggregate summary(:value)
+ # end
+ # end
+ class AgentDDL<Base
+ def initialize(plugin, plugintype=:agent, loadddl=true)
+ @process_aggregate_functions = nil
+
+ super
+ end
+
+ def input(argument, properties)
+ raise "Input needs a :optional property" unless properties.include?(:optional)
+
+ super
+ end
+
+ # Calls the summarize block defined in the ddl. Block will not be called
+ # if the ddl is getting processed on the server side. This means that
+ # aggregate plugins only have to be present on the client side.
+ #
+ # The @process_aggregate_functions variable is used by the method_missing
+ # block to determine if it should kick in, this way we very tightly control
+ # where we activate the method_missing behavior turning it into a noop
+ # otherwise to maximise the chance of providing good user feedback
+ def summarize(&block)
+ unless @config.mode == :server
+ @process_aggregate_functions = true
+ block.call
+ @process_aggregate_functions = nil
+ end
+ end
+
+ # Sets the aggregate array for the given action
+ def aggregate(function, format = {:format => nil})
+ DDL.validation_fail!(:PLMC28, "Formats supplied to aggregation functions should be a hash", :error) unless format.is_a?(Hash)
+ DDL.validation_fail!(:PLMC27, "Formats supplied to aggregation functions must have a :format key", :error) unless format.keys.include?(:format)
+ DDL.validation_fail!(:PLMC26, "Functions supplied to aggregate should be a hash", :error) unless function.is_a?(Hash)
+
+ unless (function.keys.include?(:args)) && function[:args]
+ DDL.validation_fail!(:PLMC25, "aggregate method for action '%{action}' missing a function parameter", :error, :action => entities[@current_entity][:action])
+ end
+
+ entities[@current_entity][:aggregate] ||= []
+ entities[@current_entity][:aggregate] << (format[:format].nil? ? function : function.merge(format))
+ end
+
+ # Sets the display preference to either :ok, :failed, :flatten or :always
+ # operates on action level
+ def display(pref)
+ # defaults to old behavior, complain if its supplied and invalid
+ unless [:ok, :failed, :flatten, :always].include?(pref)
+ raise "Display preference #{pref} is not valid, should be :ok, :failed, :flatten or :always"
+ end
+
+ action = @current_entity
+ @entities[action][:display] = pref
+ end
+
+ # Creates the definition for an action, you can nest input definitions inside the
+ # action to attach inputs and validation to the actions
+ #
+ # action "status", :description => "Restarts a Service" do
+ # display :always
+ #
+ # input "service",
+ # :prompt => "Service Action",
+ # :description => "The action to perform",
+ # :type => :list,
+ # :optional => true,
+ # :list => ["start", "stop", "restart", "status"]
+ #
+ # output "status",
+ # :description => "The status of the service after the action"
+ #
+ # end
+ def action(name, input, &block)
+ raise "Action needs a :description property" unless input.include?(:description)
+
+ unless @entities.include?(name)
+ @entities[name] = {}
+ @entities[name][:action] = name
+ @entities[name][:input] = {}
+ @entities[name][:output] = {}
+ @entities[name][:display] = :failed
+ @entities[name][:description] = input[:description]
+ end
+
+ # if a block is passed it might be creating input methods, call it
+ # we set @current_entity so the input block can know what its talking
+ # to, this is probably an epic hack, need to improve.
+ @current_entity = name
+ block.call if block_given?
+ @current_entity = nil
+ end
+
+ # If the method name matches a # aggregate function, we return the function
+ # with args as a hash. This will only be active if the @process_aggregate_functions
+ # is set to true which only happens in the #summarize block
+ def method_missing(name, *args, &block)
+ unless @process_aggregate_functions || is_function?(name)
+ raise NoMethodError, "undefined local variable or method `#{name}'", caller
+ end
+
+ return {:function => name, :args => args}
+ end
+
+ # Checks if a method name matches a aggregate plugin.
+ # This is used by method missing so that we dont greedily assume that
+ # every method_missing call in an agent ddl has hit a aggregate function.
+ def is_function?(method_name)
+ PluginManager.find("aggregate").include?(method_name.to_s)
+ end
+
+ # For a given action and arguments look up the DDL interface to that action
+ # and if any arguments in the DDL have a :default value assign that to any
+ # input that does not have an argument in the input arguments
+ #
+ # This is intended to only be called on clients and not on servers as the
+ # clients should never be able to publish non compliant requests and the
+ # servers should really not tamper with incoming requests since doing so
+ # might raise validation errors that were not raised on the client breaking
+ # our fail-fast approach to input validation
+ def set_default_input_arguments(action, arguments)
+ input = action_interface(action)[:input]
+
+ return unless input
+
+ input.keys.each do |key|
+ if !arguments.include?(key) && !input[key][:default].nil? && !input[key][:optional]
+ Log.debug("Setting default value for input '%s' to '%s'" % [key, input[key][:default]])
+ arguments[key] = input[key][:default]
+ end
+ end
+ end
+
+ # Helper to use the DDL to figure out if the remote call to an
+ # agent should be allowed based on action name and inputs.
+ def validate_rpc_request(action, arguments)
+ # is the action known?
+ unless actions.include?(action)
+ DDL.validation_fail!(:PLMC29, "Attempted to call action %{action} for %{plugin} but it's not declared in the DDL", :debug, :action => action, :plugin => @pluginname)
+ end
+
+ input = action_interface(action)[:input] || {}
+
+ input.keys.each do |key|
+ unless input[key][:optional]
+ unless arguments.keys.include?(key)
+ DDL.validation_fail!(:PLMC30, "Action '%{action}' needs a '%{key}' argument", :debug, :action => action, :key => key)
+ end
+ end
+
+ if arguments.keys.include?(key)
+ validate_input_argument(input, key, arguments[key])
+ end
+ end
+
+ true
+ end
+
+ # Returns the interface for a specific action
+ def action_interface(name)
+ @entities[name] || {}
+ end
+
+ # Returns an array of actions this agent support
+ def actions
+ @entities.keys
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module DDL
+ # The base class for all kinds of DDL files. DDL files when
+ # run gets parsed and builds up a hash of the basic primitive
+ # types, ideally restricted so it can be converted to JSON though
+ # today there are some Ruby Symbols in them which might be fixed
+ # laster on.
+ #
+ # The Hash being built should be stored in @entities, the format
+ # is generally not prescribed but there's a definite feel to how
+ # DDL files look so study the agent and discovery ones to see how
+ # the structure applies to very different use cases.
+ #
+ # For every plugin type you should have a single word name - that
+ # corresponds to the directory in the libdir where these plugins
+ # live. If you need anything above and beyond 'metadata' in your
+ # plugin DDL then add a PlugintypeDDL class here and add your
+ # specific behaviors to those.
+ class Base
+ include Translatable
+
+ attr_reader :meta, :entities, :pluginname, :plugintype, :usage, :requirements
+
+ def initialize(plugin, plugintype=:agent, loadddl=true)
+ @entities = {}
+ @meta = {}
+ @usage = ""
+ @config = Config.instance
+ @pluginname = plugin
+ @plugintype = plugintype.to_sym
+ @requirements = {}
+
+ loadddlfile if loadddl
+ end
+
+ # Generates help using the template based on the data
+ # created with metadata and input.
+ #
+ # If no template name is provided one will be chosen based
+ # on the plugin type. If the provided template path is
+ # not absolute then the template will be loaded relative to
+ # helptemplatedir configuration parameter
+ def help(template=nil)
+ template = template_for_plugintype unless template
+ template = File.join(@config.helptemplatedir, template) unless template.start_with?(File::SEPARATOR)
+
+ template = File.read(template)
+ meta = @meta
+ entities = @entities
+
+ unless template == "metadata-help.erb"
+ metadata_template = File.join(@config.helptemplatedir, "metadata-help.erb")
+ metadata_template = File.read(metadata_template)
+ metastring = ERB.new(metadata_template, 0, '%')
+ metastring = metastring.result(binding)
+ end
+
+ erb = ERB.new(template, 0, '%')
+ erb.result(binding)
+ end
+
+ def usage(usage_text)
+ @usage = usage_text
+ end
+
+ def template_for_plugintype
+ case @plugintype
+ when :agent
+ return "rpc-help.erb"
+ else
+ if File.exists?(File.join(@config.helptemplatedir,"#{@plugintype}-help.erb"))
+ return "#{@plugintype}-help.erb"
+ else
+ # Default help template gets loaded if plugintype-help does not exist.
+ return "metadata-help.erb"
+ end
+ end
+ end
+
+ def loadddlfile
+ if ddlfile = findddlfile
+ instance_eval(File.read(ddlfile), ddlfile, 1)
+ else
+ raise_code(:PLMC40, "Can't find DDL for %{type} plugin '%{name}'", :debug, :type => @plugintype, :name => @pluginname)
+ end
+ end
+
+ def findddlfile(ddlname=nil, ddltype=nil)
+ ddlname = @pluginname unless ddlname
+ ddltype = @plugintype unless ddltype
+
+ @config.libdir.each do |libdir|
+ ddlfile = File.join([libdir, "mcollective", ddltype.to_s, "#{ddlname}.ddl"])
+ if File.exist?(ddlfile)
+ log_code(:PLMC18, "Found %{ddlname} ddl at %{ddlfile}", :debug, :ddlname => ddlname, :ddlfile => ddlfile)
+ return ddlfile
+ end
+ end
+ return false
+ end
+
+ def validate_requirements
+ if requirement = @requirements[:mcollective]
+ if Util.mcollective_version == "@DEVELOPMENT_VERSION@"
+ log_code(:PLMC19, "DDL requirements validation being skipped in development", :warn)
+ return true
+ end
+
+ if Util.versioncmp(Util.mcollective_version, requirement) < 0
+ DDL.validation_fail!(:PLMC20, "%{type} plugin '%{name}' requires MCollective version %{requirement} or newer", :debug, :type => @plugintype.to_s.capitalize, :name => @pluginname, :requirement => requirement)
+ end
+ end
+
+ true
+ end
+
+ # validate strings, lists and booleans, we'll add more types of validators when
+ # all the use cases are clear
+ #
+ # only does validation for arguments actually given, since some might
+ # be optional. We validate the presense of the argument earlier so
+ # this is a safe assumption, just to skip them.
+ #
+ # :string can have maxlength and regex. A maxlength of 0 will bypasss checks
+ # :list has a array of valid values
+ def validate_input_argument(input, key, argument)
+ Validator.load_validators
+
+ case input[key][:type]
+ when :string
+ Validator.validate(argument, :string)
+
+ Validator.length(argument, input[key][:maxlength].to_i)
+
+ Validator.validate(argument, input[key][:validation])
+
+ when :list
+ Validator.validate(argument, input[key][:list])
+
+ else
+ Validator.validate(argument, input[key][:type])
+ end
+
+ return true
+ rescue => e
+ DDL.validation_fail!(:PLMC21, "Cannot validate input '%{input}': %{error}", :debug, :input => key, :error => e.to_s)
+ end
+
+ # Registers an input argument for a given action
+ #
+ # See the documentation for action for how to use this
+ def input(argument, properties)
+ raise_code(:PLMC22, "Cannot determine what entity input '%{entity}' belongs to", :error, :entity => @current_entity) unless @current_entity
+
+ entity = @current_entity
+
+ [:prompt, :description, :type].each do |arg|
+ raise_code(:PLMC23, "Input needs a :%{property} property", :debug, :property => arg) unless properties.include?(arg)
+ end
+
+ @entities[entity][:input][argument] = {:prompt => properties[:prompt],
+ :description => properties[:description],
+ :type => properties[:type],
+ :default => properties[:default],
+ :optional => properties[:optional]}
+
+ case properties[:type]
+ when :string
+ raise "Input type :string needs a :validation argument" unless properties.include?(:validation)
+ raise "Input type :string needs a :maxlength argument" unless properties.include?(:maxlength)
+
+ @entities[entity][:input][argument][:validation] = properties[:validation]
+ @entities[entity][:input][argument][:maxlength] = properties[:maxlength]
+
+ when :list
+ raise "Input type :list needs a :list argument" unless properties.include?(:list)
+
+ @entities[entity][:input][argument][:list] = properties[:list]
+ end
+ end
+
+ # Registers an output argument for a given action
+ #
+ # See the documentation for action for how to use this
+ def output(argument, properties)
+ raise "Cannot figure out what action input #{argument} belongs to" unless @current_entity
+ raise "Output #{argument} needs a description argument" unless properties.include?(:description)
+ raise "Output #{argument} needs a display_as argument" unless properties.include?(:display_as)
+
+ action = @current_entity
+
+ @entities[action][:output][argument] = {:description => properties[:description],
+ :display_as => properties[:display_as],
+ :default => properties[:default]}
+ end
+
+ def requires(requirement)
+ raise "Requirement should be a hash in the form :item => 'requirement'" unless requirement.is_a?(Hash)
+
+ valid_requirements = [:mcollective]
+
+ requirement.keys.each do |key|
+ unless valid_requirements.include?(key)
+ raise "Requirement %s is not a valid requirement, only %s is supported" % [key, valid_requirements.join(", ")]
+ end
+
+ @requirements[key] = requirement[key]
+ end
+
+ validate_requirements
+ end
+
+ # Registers meta data for the introspection hash
+ def metadata(meta)
+ [:name, :description, :author, :license, :version, :url, :timeout].each do |arg|
+ raise "Metadata needs a :#{arg} property" unless meta.include?(arg)
+ end
+
+ @meta = meta
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module DDL
+ # A DDL file for the data query plugins.
+ #
+ # Query plugins can today take only one input by convention in the DDL that
+ # is called :query, otherwise the input is identical to the standard input.
+ #
+ # metadata :name => "Agent",
+ # :description => "Meta data about installed MColletive Agents",
+ # :author => "R.I.Pienaar <rip@devco.net>",
+ # :license => "ASL 2.0",
+ # :version => "1.0",
+ # :url => "http://marionette-collective.org/",
+ # :timeout => 1
+ #
+ # dataquery :description => "Agent Meta Data" do
+ # input :query,
+ # :prompt => "Agent Name",
+ # :description => "Valid agent name",
+ # :type => :string,
+ # :validation => /^[\w\_]+$/,
+ # :maxlength => 20
+ #
+ # [:license, :timeout, :description, :url, :version, :author].each do |item|
+ # output item,
+ # :description => "Agent #{item}",
+ # :display_as => item.to_s.capitalize
+ # end
+ # end
+ class DataDDL<Base
+ def dataquery(input, &block)
+ raise "Data queries need a :description" unless input.include?(:description)
+ raise "Data queries can only have one definition" if @entities[:data]
+
+ @entities[:data] = {:description => input[:description],
+ :input => {},
+ :output => {}}
+
+ @current_entity = :data
+ block.call if block_given?
+ @current_entity = nil
+ end
+
+ def input(argument, properties)
+ raise "The only valid input name for a data query is 'query'" if argument != :query
+
+ super
+ end
+
+ # Returns the interface for the data query
+ def dataquery_interface
+ @entities[:data] || {}
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module DDL
+ # DDL for discovery plugins, a full example can be seen below
+ #
+ # metadata :name => "mc",
+ # :description => "MCollective Broadcast based discovery",
+ # :author => "R.I.Pienaar <rip@devco.net>",
+ # :license => "ASL 2.0",
+ # :version => "0.1",
+ # :url => "http://marionette-collective.org/",
+ # :timeout => 2
+ #
+ # discovery do
+ # capabilities [:classes, :facts, :identity, :agents, :compound]
+ # end
+ class DiscoveryDDL<Base
+ def discovery_interface
+ @entities[:discovery]
+ end
+
+ # records valid capabilities for discovery plugins
+ def capabilities(*caps)
+ caps = [caps].flatten
+
+ raise "Discovery plugin capabilities can't be empty" if caps.empty?
+
+ caps.each do |cap|
+ if [:classes, :facts, :identity, :agents, :compound].include?(cap)
+ @entities[:discovery][:capabilities] << cap
+ else
+ raise "%s is not a valid capability, valid capabilities are :classes, :facts, :identity, :agents and :compound" % cap
+ end
+ end
+ end
+
+ # Creates the definition for new discovery plugins
+ #
+ # discovery do
+ # capabilities [:classes, :facts, :identity, :agents, :compound]
+ # end
+ def discovery(&block)
+ raise "Discovery plugins can only have one definition" if @entities[:discovery]
+
+ @entities[:discovery] = {:capabilities => []}
+
+ @current_entity = :discovery
+ block.call if block_given?
+ @current_entity = nil
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module DDL
+ class ValidatorDDL<Base
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Discovery
+ def initialize(client)
+ @known_methods = find_known_methods
+ @default_method = Config.instance.default_discovery_method
+ @client = client
+ end
+
+ def find_known_methods
+ PluginManager.find("discovery")
+ end
+
+ def has_method?(method)
+ @known_methods.include?(method)
+ end
+
+ def force_direct_mode?
+ discovery_method != "mc"
+ end
+
+ def discovery_method
+ method = "mc"
+
+ if @client.options[:discovery_method]
+ method = @client.options[:discovery_method]
+ else
+ method = @default_method
+ end
+
+ raise "Unknown discovery method %s" % method unless has_method?(method)
+
+ unless method == "mc"
+ raise "Custom discovery methods require direct addressing mode" unless Config.instance.direct_addressing
+ end
+
+ return method
+ end
+
+ def discovery_class
+ method = discovery_method.capitalize
+
+ PluginManager.loadclass("MCollective::Discovery::#{method}") unless self.class.const_defined?(method)
+
+ self.class.const_get(method)
+ end
+
+ def ddl
+ @ddl ||= DDL.new(discovery_method, :discovery)
+
+ # if the discovery method got changed we might have an old DDL cached
+ # this will detect that and reread the correct DDL from disk
+ unless @ddl.meta[:name] == discovery_method
+ @ddl = DDL.new(discovery_method, :discovery)
+ end
+
+ return @ddl
+ end
+
+ # Agent filters are always present no matter what, so we cant raise an error if the capabilities
+ # suggest the discovery method cant do agents we just have to rely on the discovery plugin to not
+ # do stupid things in the presense of a agent filter
+ def check_capabilities(filter)
+ capabilities = ddl.discovery_interface[:capabilities]
+
+ unless capabilities.include?(:classes)
+ raise "Cannot use class filters while using the '%s' discovery method" % discovery_method unless filter["cf_class"].empty?
+ end
+
+ unless capabilities.include?(:facts)
+ raise "Cannot use fact filters while using the '%s' discovery method" % discovery_method unless filter["fact"].empty?
+ end
+
+ unless capabilities.include?(:identity)
+ raise "Cannot use identity filters while using the '%s' discovery method" % discovery_method unless filter["identity"].empty?
+ end
+
+ unless capabilities.include?(:compound)
+ raise "Cannot use compound filters while using the '%s' discovery method" % discovery_method unless filter["compound"].empty?
+ end
+ end
+
+ # checks if compound filters are used and then forces the 'mc' discovery plugin
+ def force_discovery_method_by_filter(filter)
+ unless discovery_method == "mc"
+ unless filter["compound"].empty?
+ Log.info "Switching to mc discovery method because compound filters are used"
+ @client.options[:discovery_method] = "mc"
+
+ return true
+ end
+ end
+
+ return false
+ end
+
+ # if a compound filter is specified and it has any function
+ # then we read the DDL for each of those plugins and sum up
+ # the timeout declared in the DDL
+ def timeout_for_compound_filter(compound_filter)
+ return 0 if compound_filter.nil? || compound_filter.empty?
+
+ timeout = 0
+
+ compound_filter.each do |filter|
+ filter.each do |statement|
+ if statement["fstatement"]
+ pluginname = Data.pluginname(statement["fstatement"]["name"])
+ ddl = DDL.new(pluginname, :data)
+ timeout += ddl.meta[:timeout]
+ end
+ end
+ end
+
+ timeout
+ end
+
+ def discovery_timeout(timeout, filter)
+ timeout = ddl.meta[:timeout] unless timeout
+
+ unless (filter["compound"] && filter["compound"].empty?)
+ timeout + timeout_for_compound_filter(filter["compound"])
+ else
+ timeout
+ end
+ end
+
+ def discover(filter, timeout, limit)
+ raise "Limit has to be an integer" unless limit.is_a?(Fixnum)
+
+ force_discovery_method_by_filter(filter)
+
+ check_capabilities(filter)
+
+ discovered = discovery_class.discover(filter, discovery_timeout(timeout, filter), limit, @client)
+
+ if limit > 0
+ return discovered[0,limit]
+ else
+ return discovered
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class CodedError<RuntimeError
+ attr_reader :code, :args, :log_level, :default
+
+ def initialize(msgid, default, level=:debug, args={})
+ @code = msgid
+ @log_level = level
+ @args = args
+ @default = default
+
+ msg = Util.t(@code, {:default => default}.merge(@args))
+
+ super(msg)
+ end
+
+ def set_backtrace(trace)
+ super
+ log(@log_level)
+ end
+
+ def log(level, log_backtrace=false)
+ Log.logexception(@code, level, self, log_backtrace)
+ end
+ end
+
+ # Exceptions for the RPC system
+ class DDLValidationError<CodedError;end
+ class ValidatorError<RuntimeError; end
+ class MsgDoesNotMatchRequestID < RuntimeError; end
+ class MsgTTLExpired<RuntimeError;end
+ class NotTargettedAtUs<RuntimeError;end
+ class RPCError<StandardError;end
+ class SecurityValidationFailed<RuntimeError;end
+
+ class InvalidRPCData<RPCError;end
+ class MissingRPCData<RPCError;end
+ class RPCAborted<RPCError;end
+ class UnknownRPCAction<RPCError;end
+ class UnknownRPCError<RPCError;end
+end
--- /dev/null
+module MCollective
+ # This is a class that gives access to the configured fact provider
+ # such as MCollectives::Facts::Facter that uses Reductive Labs facter
+ #
+ # The actual provider is pluggable and configurable using the 'factsource'
+ # configuration option.
+ #
+ # To develop a new factsource simply create a class under MCollective::Facts::
+ # and provide the following classes:
+ #
+ # self.get_fact(fact)
+ # self.has_fact?(fact)
+ #
+ # You can also just inherit from MCollective::Facts::Base and provide just the
+ #
+ # self.get_facts
+ #
+ # method that should return a hash of facts.
+ module Facts
+ autoload :Base, "mcollective/facts/base"
+
+ @@config = nil
+
+ # True if we know of a specific fact else false
+ def self.has_fact?(fact, value)
+ PluginManager["facts_plugin"].get_fact(fact) == value ? true : false
+ end
+
+ # Get the value of a fact
+ def self.get_fact(fact)
+ PluginManager["facts_plugin"].get_fact(fact)
+ end
+
+ # Get the value of a fact
+ def self.[](fact)
+ PluginManager["facts_plugin"].get_fact(fact)
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Facts
+ # A base class for fact providers, to make a new fully functional fact provider
+ # inherit from this and simply provide a self.get_facts method that returns a
+ # hash like:
+ #
+ # {"foo" => "bar",
+ # "bar" => "baz"}
+ class Base
+ def initialize
+ @facts = {}
+ @last_good_facts = {}
+ @last_facts_load = 0
+ end
+
+ # Registers new fact sources into the plugin manager
+ def self.inherited(klass)
+ PluginManager << {:type => "facts_plugin", :class => klass.to_s}
+ end
+
+ # Returns the value of a single fact
+ def get_fact(fact=nil)
+ config = Config.instance
+
+ cache_time = config.fact_cache_time || 300
+
+ Thread.exclusive do
+ begin
+ if (Time.now.to_i - @last_facts_load > cache_time.to_i ) || force_reload?
+ Log.debug("Resetting facter cache, now: #{Time.now.to_i} last-known-good: #{@last_facts_load}")
+
+ tfacts = load_facts_from_source
+
+ # Force reset to last known good state on empty facts
+ raise "Got empty facts" if tfacts.empty?
+
+ @facts.clear
+
+ tfacts.each_pair do |key,value|
+ @facts[key.to_s] = value.to_s
+ end
+
+ @last_good_facts = @facts.clone
+ @last_facts_load = Time.now.to_i
+ else
+ Log.debug("Using cached facts now: #{Time.now.to_i} last-known-good: #{@last_facts_load}")
+ end
+ rescue Exception => e
+ Log.error("Failed to load facts: #{e.class}: #{e}")
+
+ # Avoid loops where failing fact loads cause huge CPU
+ # loops, this way it only retries once every cache_time
+ # seconds
+ @last_facts_load = Time.now.to_i
+
+ # Revert to last known good state
+ @facts = @last_good_facts.clone
+ end
+ end
+
+
+ # If you do not supply a specific fact all facts will be returned
+ if fact.nil?
+ return @facts
+ else
+ @facts.include?(fact) ? @facts[fact] : nil
+ end
+ end
+
+ # Returns all facts
+ def get_facts
+ get_fact(nil)
+ end
+
+ # Returns true if we know about a specific fact, false otherwise
+ def has_fact?(fact)
+ get_fact(nil).include?(fact)
+ end
+
+ # Plugins can override this to provide forced fact invalidation
+ def force_reload?
+ false
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Generators
+ autoload :Base, "mcollective/generators/base.rb"
+ autoload :DataGenerator, "mcollective/generators/data_generator.rb"
+ autoload :AgentGenerator, "mcollective/generators/agent_generator.rb"
+ end
+end
--- /dev/null
+module MCollective
+ module Generators
+ class AgentGenerator<Base
+
+ attr_accessor :ddl, :content
+
+ def initialize(plugin_name, actions = [], name = nil, description = nil, author = nil ,
+ license = nil, version = nil, url = nil, timeout = nil)
+
+ super(name, description, author, license, version, url, timeout)
+ @plugin_name = plugin_name
+ @actions = actions || []
+ @ddl = create_ddl
+ @mod_name = "Agent"
+ @pclass = "RPC::Agent"
+ @content = create_plugin_content
+ @plugin = create_plugin_string
+ write_plugins
+ end
+
+ def create_ddl
+ action_text = ""
+ @actions.each_with_index do |action, i|
+ action_text += "action \"#{action}\", :description => \"%ACTIONDESCRIPTION%\" do\n"
+ action_text += action_help if i == 0
+ action_text += "end\n"
+ action_text += "\n" unless @actions.size == (i + 1)
+ end
+ # Use inherited method to create metadata part of the ddl
+ create_metadata_string + action_text
+ end
+
+ def create_plugin_content
+ content_text = ""
+
+ # Add actions to agent file
+ @actions.each_with_index do |action, i|
+ content_text += "%6s%s" % [" ", "action \"#{action}\" do\n"]
+ content_text += "%6s%s" % [" ", "end\n"]
+ content_text += "\n" unless @actions.size == (i + 1)
+ end
+ content_text
+ end
+
+ def action_help
+ action_snippet = File.read(File.join(File.dirname(__FILE__), "templates", "action_snippet.erb"))
+ ERB.new(action_snippet).result
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Generators
+ class Base
+ attr_accessor :meta, :plugin_name, :mod_name
+ def initialize(name, description, author, license, version, url, timeout)
+ @meta = {:name => name,
+ :description => description,
+ :author => author,
+ :license => license,
+ :version => version,
+ :url => url,
+ :timeout => timeout}
+ end
+
+ def create_metadata_string
+ ddl_template = File.read(File.join(File.dirname(__FILE__), "templates", "ddl.erb"))
+ ERB.new(ddl_template, nil, "-").result(binding)
+ end
+
+ def create_plugin_string
+ plugin_template = File.read(File.join(File.dirname(__FILE__), "templates", "plugin.erb"))
+ ERB.new(plugin_template, nil, "-").result(binding)
+ end
+
+ def write_plugins
+ begin
+ Dir.mkdir @plugin_name
+ dirname = File.join(@plugin_name, @mod_name.downcase)
+ Dir.mkdir dirname
+ puts "Created plugin directory : #{@plugin_name}"
+
+ File.open(File.join(dirname, "#{@plugin_name}.ddl"), "w"){|f| f.puts @ddl}
+ puts "Created DDL file : #{File.join(dirname, "#{@plugin_name}.ddl")}"
+
+ File.open(File.join(dirname, "#{@plugin_name}.rb"), "w"){|f| f.puts @plugin}
+ puts "Created #{@mod_name} file : #{File.join(dirname, "#{@plugin_name}.rb")}"
+ rescue Errno::EEXIST
+ raise "cannot generate '#{@plugin_name}' : plugin directory already exists."
+ rescue Exception => e
+ FileUtils.rm_rf(@plugin_name) if File.directory?(@plugin_name)
+ raise "cannot generate plugin - #{e}"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Generators
+ class DataGenerator<Base
+
+ attr_accessor :ddl, :content
+
+ def initialize(plugin_name, outputs = [], name = nil, description = nil, author = nil ,
+ license = nil, version = nil, url = nil, timeout = nil)
+
+ super(name, description, author, license, version, url, timeout)
+ @mod_name = "Data"
+ @pclass = "Base"
+ @plugin_name = plugin_name
+ @outputs = outputs
+ @ddl = create_ddl
+ @content = create_plugin_content
+ @plugin = create_plugin_string
+ write_plugins
+ end
+
+ def create_ddl
+ query_text = "dataquery :description => \"Query information\" do\n"
+ query_text += ERB.new(File.read(File.join(File.dirname(__FILE__), "templates", "data_input_snippet.erb"))).result
+
+ @outputs.each_with_index do |output,i|
+ query_text += "%2s%s" % [" ", "output :#{output},\n"]
+ query_text += "%9s%s" % [" ", ":description => \"%DESCRIPTION%\",\n"]
+ query_text += "%9s%s" % [" ", ":display_as => \"%DESCRIPTION%\"\n"]
+ query_text += "\n" unless @outputs.size == (i + 1)
+ end
+
+ query_text += "end"
+
+ # Use inherited method to create metadata part of the ddl
+ create_metadata_string + query_text
+ end
+
+ def create_plugin_content
+ content_text = "%6s%s" % [" ", "query do |what|\n"]
+
+ @outputs.each do |output|
+ content_text += "%8s%s" % [" ", "result[:#{output}] = nil\n"]
+ end
+ content_text += "%6s%s" % [" ", "end\n"]
+
+ # Add actions to agent file
+ content_text
+ end
+ end
+ end
+end
--- /dev/null
+ # Example Input
+ input :name,
+ :prompt => "%PROMPT%",
+ :description => "%DESCRIPTION%",
+ :type => %TYPE%,
+ :validation => '%VALIDATION%',
+ :optional => %OPTIONAL%,
+ :maxlength => %MAXLENGTH%
+
+ # Example output
+ output :name,
+ :description => "%DESCRIPTION%",
+ :display_as => "%DISPLAYAS%"
--- /dev/null
+ input :query,
+ :prompt => "%PROMP%",
+ :description => "%DESCRIPTION%",
+ :type => %TYPE%,
+ :validation => %VALIDATION%,
+ :maxlength => %MAXLENGTH%
+
--- /dev/null
+metadata :name => "<%= meta[:name] || "%FULLNANE%" -%>",
+ :description => "<%= meta[:description] || "%DESCRIPTION%" -%>",
+ :author => "<%= meta[:author] || "%AUTHOR%" -%>",
+ :license => "<%= meta[:license] || "%LICENSE%" -%>",
+ :version => "<%= meta[:version] || "%VERSION%" -%>",
+ :url => "<%= meta[:url] || "%URL%" -%>",
+ :timeout => <%= meta[:timeout] || "%TIMEOUT%" %>
+
--- /dev/null
+module MCollective
+ module <%= @mod_name%>
+ class <%= @plugin_name.capitalize -%><<%= @pclass%>
+<%= @content%>
+ end
+ end
+end
--- /dev/null
+en:
+ PLMC1:
+ example: "The Marionette Collective version 2.2.2 started by /usr/bin/mco using config file /etc/mcollective/client.cfg"
+ expanded: "This message gets logged each time MCollective reads it's config file. Generally this only happens once per process. It shows the version, process name and config file as a simple debugging aid"
+ pattern: "The Marionette Collective version %{version} started by %{name} using config file %{config}"
+ PLMC10:
+ example: "Failed to handle message: RuntimeError: none.rb:15:in `decodemsg': Could not decrypt message "
+ expanded: |-
+ When a message arrives from the middleware it gets decoded, security validated and then dispatched to the agent code.
+
+ There exist a number of errors that can happen here, some are handled specifically others will be logged by this "catch all" handler.
+
+ Generally there should not be many messages logged here but we include a stack trace to assist with debugging these.
+
+ The messages here do not tend to originate from your Agents unless they are syntax error related but more likely to be situations like security failures due to incorrect SSL keys and so forth
+
+ Should you come across one that is a regular accorance in your logs please open a ticket including your backtrace and we will improve the handling of that situation
+ pattern: "Failed to handle message: %{error}"
+ PLMC11:
+ example: "Cache expired on 'ddl' key 'agent/nrpe'"
+ expanded: |-
+ MCollective has an internal Cache used to speed up operations like parsing of DDL files. The cache is also usable from the agents and other plugins you write.
+
+ Each entry in the cache has an associated TTL or maximum life time, once the maximum time on an item is reached it is considered expired. Next time anything attempts to read this entry from the cache this log line will be logged.
+
+ This is part of the normal operations of MCollective and doesn't indicate any problem. We log this debug message to help you debug your own use of the cache.
+ pattern: "Cache expired on '%{cache_name}' key '%{item}'"
+ PLMC12:
+ example: "Cache hit on 'ddl' key 'agent/nrpe'"
+ expanded: |-
+ MCollective has an internal Cache used to speed up operations like parsing of DDL files. The cache is also usable from the agents and other plugins you write.
+
+ Each entry in the cache has an associated TTL or maximum life time, once the maximum time on an item is reached it is considered expired.
+
+ This log line indicates that a request for a cache entry was made, the entry had not yet expired and so the cached data is being returned.
+
+ It does not indicate a problem, it's just a debugging aid
+ pattern: "Cache hit on '%{cache_name}' key '%{item}'"
+ PLMC13:
+ example: "Could not find a cache called 'my_cache'"
+ expanded: |-
+ MCollective has an internal Cache used to speed up operations like parsing of DDL files. The cache is also usable from the agents and other plugins you write.
+
+ The cache is made up of many named caches, this error indicates that a cache has not yet been setup prior to trying to use it.
+ pattern: "Could not find a cache called '%{cache_name}'"
+ PLMC14:
+ example: "No block supplied to synchronize on cache 'my_cache'"
+ expanded: |-
+ When using the Cache to synchronize your own code across agents or other plugins you have to supply a block to synchronise.
+
+ Correct usage would be:
+
+ Cache.setup(:customer, 600)
+ Cache(:customer).synchronize do
+ # update customer record
+ end
+
+ This error is raise when the logic to update the customer record is not in a block as in the example
+ pattern: "No block supplied to synchronize on cache '%{cache_name}'"
+ PLMC15:
+ example: "No item called 'nrpe_agent' for cache 'ddl'"
+ expanded: |-
+ MCollective has an internal Cache used to speed up operations like parsing of DDL files. The cache is also usable from the agents and other plugins you write.
+
+ The cache stored items using a key, this error will be logged and raised when you try to access a item that does not yet exist in the cache.
+ pattern: "No item called '%{item}' for cache '%{cache_name}'"
+ PLMC16:
+ example: "'hello' does not look like a numeric value"
+ expanded: |-
+ When MCollective receives an argument from the command line like port=fello it consults the DDL file to determine the desired type of this parameter, it then tries to convert the input string into the correct numeric value.
+
+ This error indicates the input you provided could not be converted into the desired format.
+
+ You'll usually see this when using the 'mco rpc' command to interact with an agent. This is usually a fatal error, the request will not be sent if it does not validate against the DDL.
+ pattern: "'%{value}' does not look like a numeric value"
+ PLMC17:
+ example: "'flase' does not look like a boolean value"
+ expanded: |-
+ When MCollective receives an argument from the command line like force=true it consults the DDL file to determine the desired type of this parameter, it then tries to convert the input string into the correct boolean value.
+
+ This error indicates the input you provided could not be converted into the desired boolean format. It wil accept "true", "t", "yes", "y" and "1" as being true. It will accept "false", "f", "no", "n", "0" as being false.
+
+ You'll usually see this when using the 'mco rpc' command to interact with an agent. This is usually a fatal error, the request will not be sent if it does not validate against the DDL.
+ pattern: "'%{value}' does not look like a boolean value"
+ PLMC18:
+ example: "Found 'rpcutil' ddl at '/usr/libexec/mcollective/mcollective/agent/rpcutil.ddl'"
+ expanded: |-
+ Most plugin types have a DDL file that needs to be correctly installed for the plugin to function. There can be multiple libdirs that can provide the DDL file for a specific plugin.
+
+ This message is a message designed to help you debug issues when you think the wrong DDL file is being used.
+ pattern: "Found '%{ddlname}' ddl at '%{ddlfile}'"
+ PLMC19:
+ expanded: |-
+ Usually when MCollective run it validates all requirements are met before publishing a request or processing a request against the DDL file for the agent.
+
+ This can be difficult to satisfy in development perhaps because you are still writing your DDL files or debugging issues.
+
+ This message indicates that when MCollective detects it's running against an unreleased version of MCollective - like directly out of a Git clone - it will skip the DDL validation steps. It is logged at a warning level as this significantly changes the behaviour of the client and server.
+ pattern: "DDL requirements validation being skipped in development"
+ PLMC2:
+ expanded: "When sending the mcollectived process the USR1 signal on a Unix based machine this message will indicate that the signal got received and all agents are being reloaded from disk\n"
+ pattern: "Reloading all agents after receiving USR1 signal"
+ PLMC20:
+ example: "Agent plugin 'example' requires MCollective version 2.2.1 or newer"
+ expanded: |-
+ MCollective plugins can declare a minimum version of MCollective that they require to be functional.
+
+ MCollective validates this when it loads the plugin and this error will be logged or shown to the user when this requirement cannot be satisfied.
+ pattern: "%{type} plugin '%{name}' requires MCollective version %{requirement} or newer"
+ PLMC21:
+ example: "Cannot validate input 'service': Input string is longer than 40 character(s)"
+ expanded: |-
+ Every input you provide to a RPC request is validated against it's DDL file. This error will be shown when the supplied data does not pass validation against the DDL.
+
+ Consult the 'mco plugin doc' command to view the DDL file for your action and input.
+ pattern: "Cannot validate input '%{input}: %{error}"
+ PLMC22:
+ example: "Cannot determine what entity input 'port' belongs to"
+ expanded: |-
+ When writing a DDL you declare inputs into plugins using the input keyword. Each input has to belong to a wrapping entity like in the example below:
+
+ action "get_data", :description => "Get data from a data plugin" do
+ input :source,
+ :prompt => "Data Source",
+ :description => "The data plugin to retrieve information from",
+ :type => :string,
+ :validation => '^\w+$',
+ :optional => false,
+ :maxlength => 50
+ end
+
+ Here the input belongs to the action entity "get_data", this error indicates that an input were found without it belonging to any specific entity
+ pattern: "Cannot determine what entity input '%{input}' belongs to"
+ PLMC23:
+ example: "Input needs a :prompt property"
+ expanded: "When writing a DDL you declare inputs for all data that you pass into the plugin. Inputs have a minimally required field set and this error indicate that you did not provide some key field while describing the input."
+ pattern: "Input needs a :%{property} property"
+ PLMC24:
+ example: "Failed to load DDL for the 'rpcutil' agent, DDLs are required: RuntimeError: failed to parse DDL file"
+ expanded: |-
+ As of version 2.0.0 DDL files are required by the MCollective server. This server indicates that it either could not find the DDL for the rpcutil agent or that there was a syntax error.
+
+ Confirm that the DDL file is installed in the agent directory and that it parses correctly using the 'mco plugin doc' command.
+ pattern: "Failed to load DDL for the '%{agent}' agent, DDLs are required: %{error_class}: %{error}"
+ PLMC25:
+ example: "aggregate method for action 'rpcutil' is missing a function parameter"
+ expanded: |-
+ DDL files can declare aggregation rules for the data returned by actions as seen below:
+
+ summarize do
+ aggregate summary(:collectives)
+ end
+
+ This error indicates that you did not supply the argument - :collectives in this example.
+ pattern: "aggregate method for action '%{action}' is missing a function parameter"
+ PLMC26:
+ expanded: |-
+ Internally when MCollective parse the DDL it converts the aggregate functions into a hash with the function name and any arguments.
+
+ This error indicates that the internal translation failed, this is a critical error and would probably indicate a structure problem in your DDL or a bug in MCollective
+ pattern: "Functions supplied to aggregate should be a hash"
+ PLMC27:
+ expanded: |-
+ DDL aggregate functions can take a custom format as in the example below:
+
+ aggregate average(:total_resources, :format => "Average: %d")
+
+ This error indicate that the :format above could not be correctly parsed.
+ pattern: "Formats supplied to aggregation functions must have a :format key"
+ PLMC28:
+ expanded: |-
+ DDL aggregate functions can take a custom format as in the example below:
+
+ aggregate average(:total_resources, :format => "Average: %d")
+
+ This error indicate that the :format above was not supplied as a hash as in the above example
+ pattern: "Formats supplied to aggregation functions should be a hash"
+ PLMC29:
+ example: "Attempted to call action 'yum_clean' for 'package' but it's not declared in the DDL"
+ expanded: |-
+ Every agent has a DDL file that describes valid actions that they can perform
+
+ This error indicates you attempted to perform an action that does not exist. Review the plugin documentation using 'mco plugin doc' for correct usage
+ pattern: "Attempted to call action '%{action}' for '%{plugin}' but it's not declared in the DDL"
+ PLMC3:
+ expanded: |-
+ When sending the mcollectived process the USR2 signal on a Unix based machine this message indicates that the signal for received and the logging level is being adjusted to the next higher level or back down to debug if it was already at it's highest level.
+
+ This message will be followed by another message similar to the one below:
+
+ Logging level is now WARN
+ pattern: "Cycling logging level due to USR2 signal"
+ PLMC30:
+ example: "Action 'get_fact' needs a 'fact' argument"
+ expanded: "Actions can declare that they expect a required input, this error indicates that you did not supply the required input"
+ pattern: "Action '%{action}' needs a '%{key}' argument"
+ PLMC31:
+ example: "No dataquery has been defined in the DDL for data plugin 'package'"
+ expanded: |-
+ Each data plugin requires a DDL that has a 'dataquery' block in it.
+
+ dataquery :description => "Agent Meta Data" do
+ .
+ .
+ end
+
+ This error indicates that the DDL file for a specific data plugin did not contain this block.
+ pattern: "No dataquery has been defined in the DDL for data plugin '%{plugin}'"
+ PLMC32:
+ expanded: "Data plugins must return data. The DDL files for a data plugin must declare at least one output parameter else you will see this error."
+ pattern: "No output has been defined in the DDL for data plugin '%{plugin}'"
+ PLMC33:
+ example: "No data plugin argument was declared in the 'puppet' DDL but an input was supplied"
+ expanded: |-
+ Data plugins can take an optional input argument. This error indicate that you supplied an input argument but the plugin does not actually expect any input.
+
+ Review the documentation for the data plugin using 'mco plugin doc'
+ pattern: "No data plugin argument was declared in the '%{plugin}' DDL but an input was supplied"
+ PLMC34:
+ example: "setting meta data in agents have been deprecated, DDL files are now being used for this information. Please update the 'package' agent"
+ expanded: |-
+ In the past each MCollective agent had a metadata section containing information like the timeout.
+
+ As of 2.2.0 the agents will now consult the DDL files that ship with each agent for this purpose so the metadata in agents are not used at all.
+
+ In order to remove confusing duplication setting metadata in agents have been deprecated and from version 2.4.0 will not be supported at all.
+
+ Please update your agent by removing the metadata section from it and make sure the DDL file has the correct information instead.
+ pattern: "setting meta data in agents have been deprecated, DDL files are now being used for this information. Please update the '%{agent}' agent"
+ PLMC35:
+ expanded: |-
+ The MCollective client can ask that the agent just performs the action and never respond. Like when supplying the --no-results option to the 'mco rpc' application.
+
+ This log line indicates that the request was received and interpreted as such and no reply will be sent. This does not indicate a problem generally it's just there to assist with debugging of problems.
+ pattern: "Client did not request a response, surpressing reply"
+ PLMC36:
+ example: "Unknown action 'png' for agent 'rpcutil'"
+ expanded: |-
+ Agents are made up of a number of named actions. When the MCollective Server receives a request it double checks if the agent in question actually implements the logic for a specific action.
+
+ When it cannot find the implementation this error will be raised. This is an unusual situation since at this point on both the client and the server the DDL will already have been used to validate the request and the DDL would have indicated that the action is valid.
+
+ Check your agent code and make sure you have code like:
+
+ action "the_action" do
+ .
+ .
+ end
+ pattern: "Unknown action '%{action}' for agent '%{agent}'"
+ PLMC37:
+ example: "Starting default activation checks for the 'rpcutil' agent"
+ expanded: |-
+ Each time the MCollective daemon starts it loads each agent from disk. It then tries to determine if the agent should start on this node by using the activate_when method or per-agent configuration.
+
+ This is a debug statement that shows you it is about to start interacting with the logic in the agent to determine if it should be made available or not.
+ pattern: "Starting default activation checks for the '%{agent}' agent"
+ PLMC38:
+ example: "Found plugin configuration 'exim.activate_agent' with value 'y'"
+ expanded: |-
+ The administrator can choose that a certain agent that is deployed on this machine should not be made available to the network.
+
+ To do this you would add a configuration value like this example to the mcollective server.cfg:
+
+ plugin.exim.activate_agent = n
+
+ If this value is "1", "y" or "true" the agent will be activated else it will be disabled.
+ pattern: "Found plugin configuration '%{agent}.activate_agent' with value '%{should_activate}'"
+ PLMC39:
+ example: "Audit failed with an error, processing the request will continue."
+ expanded: |-
+ Every MCollective request can be audited. In the event where the auditing fails the processing continues.
+
+ At present the failure handling is not configurable, in future the administrator might elect to not run unaudited requests via configuration.
+ PLMC4:
+ example: "Failed to start registration plugin: ArgumentError: meta.rb:6:in `gsub': wrong number of arguments (0 for 2)"
+ expanded: |
+ Registration plugins are loaded into the MCollective daemon at startup and ran on a regular interval.
+
+ This message indicate that on first start this plugin failed to run, it will show the exception class, line and exception message to assist with debugging
+ pattern: "Failed to start registration plugin: %{error}"
+ PLMC40:
+ example: "Can't find DDL for agent plugin 'puppet'"
+ expanded: |-
+ MCollective plugins come with DDL files that describe their behaviours, versions, capabilities and requirements.
+
+ This error indicate that a DDL file for the plugin mentioned could not be found - it could be that you have a typing error in your command line or an installation error.
+ pattern: "Can't find DDL for %{type} plugin '%{name}'"
+ PLMC5:
+ expanded: |-
+ In previous versions of MCollective a application called 'controller' were included that had the ability to request agent reloads and other commands that would control the runner.
+
+ Unfortunately this method of controlling the daemon was never considered stable or reliable and has since been deprecate for removal during the 2.3.x development series
+ pattern: "Received a control message, possibly via 'mco controller' but this has been deprecated"
+ PLMC6:
+ expanded: |-
+ When a specific MCollective daemon receives a message from a network it validates the filter before processing the message. The filter is the list of classes, facts or other conditions that are associated with any message.
+
+ In the case where the filter does not match the specific host this line gets logged.
+
+ It's often the case for broadcast messages that all MCollective nodes will receive a message but only a subset of nodes are targetted using the filters, in that situation the nodes that received the request but should not respond to it will see this log line.
+
+ It does not indicate anything to be concerned about but if you find debugging a problem and expect a node to have responded when it shouldn't this log line will give you a hint that some condition specified in the filter did not match the local host
+ pattern: "Message does not pass filters, ignoring"
+ PLMC7:
+ example: "Exiting after signal: SignalException: runner.rb:6:in `run': Interrupt"
+ expanded: |-
+ When the MCollective daemon gets a signal from the Operating System that it does not specifically handle it will log this line before exiting.
+
+ You would see this whenever the daemon is stopped by init script or when sending it a kill signal, it will then proceed to disconnect from the middleware and exit its main loop
+ pattern: "Exiting after signal: %{error}"
+ PLMC8:
+ example: "Handling message for agent 'rpcutil' on collective 'eu_collective' with requestid 'a8a78d0ff555c931f045b6f448129846'"
+ expanded: |-
+ After receiving a message from the middleware, decoding it, validating it's security credentials and doing other checks on it the MCollective daemon will pass it off to the actual agent code for processing.
+
+ Prior to doing so it will log this line indicating the agent name and sub-collective and other information that will assist in correlating the message sent from the client with those in the server logs being processed.
+ pattern: "Handling message for agent '%{agent}' on collective '%{collective} with requestid '%{requestid}'"
+ PLMC9:
+ example: "Expired Message: message 8b4fe522f0d0541dabe83ec10b7fa446 from cert=client@node created at 1358840888 is 653 seconds old, TTL is 60"
+ expanded: |-
+ Requests sent from clients to servers all have a creation time and a maximum validity time called a TTL.
+
+ This message indicates that a message was received from the network but that it was determined to be too based on the TTL settings.
+
+ Usually this happens because your clocks are not in sync - something that can be fixed by rolling out a tool like ntp across your server estate.
+
+ It might also happen during very slow network conditions or when the TTL is set too low for your general network latency.
+ pattern: "Expired Message: %{error}"
--- /dev/null
+module MCollective
+ # A simple class that allows logging at various levels.
+ class Log
+ class << self
+ @logger = nil
+
+ VALID_LEVELS = [:error, :fatal, :debug, :warn, :info]
+
+ # Obtain the class name of the currently configured logger
+ def logger
+ @logger.class
+ end
+
+ # Logs at info level
+ def info(msg)
+ log(:info, msg)
+ end
+
+ # Logs at warn level
+ def warn(msg)
+ log(:warn, msg)
+ end
+
+ # Logs at debug level
+ def debug(msg)
+ log(:debug, msg)
+ end
+
+ # Logs at fatal level
+ def fatal(msg)
+ log(:fatal, msg)
+ end
+
+ # Logs at error level
+ def error(msg)
+ log(:error, msg)
+ end
+
+ # handle old code that relied on this class being a singleton
+ def instance
+ self
+ end
+
+ # increments the active log level
+ def cycle_level
+ @logger.cycle_level if @configured
+ end
+
+ def config_and_check_level(level)
+ configure unless @configured
+ check_level(level)
+ @logger.should_log?(level)
+ end
+
+ def check_level(level)
+ raise "Unknown log level" unless valid_level?(level)
+ end
+
+ def valid_level?(level)
+ VALID_LEVELS.include?(level)
+ end
+
+ def message_for(msgid, args={})
+ "%s: %s" % [msgid, Util.t(msgid, args)]
+ end
+
+ def logexception(msgid, level, e, backtrace=false, args={})
+ return false unless config_and_check_level(level)
+
+ origin = File.basename(e.backtrace[1])
+
+ if e.is_a?(CodedError)
+ msg = "%s: %s" % [e.code, e.to_s]
+ else
+ error_string = "%s: %s" % [e.class, e.to_s]
+ msg = message_for(msgid, args.merge(:error => error_string))
+ end
+
+ log(level, msg, origin)
+
+ if backtrace
+ e.backtrace.each do |line|
+ log(level, "%s: %s" % [msgid, line], origin)
+ end
+ end
+ end
+
+ # Logs a message at a certain level, the message must be
+ # a token that will be looked up from the i18n localization
+ # database
+ #
+ # Messages can interprolate strings from the args hash, a
+ # message with "foo %{bar}" in the localization database
+ # will use args[:bar] for the value there, the interprolation
+ # is handled by the i18n library itself
+ def logmsg(msgid, default, level, args={})
+ return false unless config_and_check_level(level)
+
+ msg = message_for(msgid, {:default => default}.merge(args))
+
+ log(level, msg)
+ end
+
+ # logs a message at a certain level
+ def log(level, msg, origin=nil)
+ return unless config_and_check_level(level)
+
+ origin = from unless origin
+
+ if @logger
+ @logger.log(level, origin, msg)
+ else
+ t = Time.new.strftime("%H:%M:%S")
+
+ STDERR.puts "#{t}: #{level}: #{origin}: #{msg}"
+ end
+ end
+
+ # sets the logger class to use
+ def set_logger(logger)
+ @logger = logger
+ end
+
+ # configures the logger class, if the config has not yet been loaded
+ # we default to the console logging class and do not set @configured
+ # so that future calls to the log method will keep attempting to configure
+ # the logger till we eventually get a logging preference from the config
+ # module
+ def configure(logger=nil)
+ unless logger
+ logger_type = "console"
+
+ config = Config.instance
+
+ if config.configured
+ logger_type = config.logger_type
+ @configured = true
+ end
+
+ require "mcollective/logger/%s_logger" % logger_type.downcase
+
+ logger_class = MCollective::Logger.const_get("%s_logger" % logger_type.capitalize)
+
+ set_logger(logger_class.new)
+ else
+ set_logger(logger)
+ @configured = true
+ end
+
+
+ @logger.start
+ rescue Exception => e
+ @configured = false
+ STDERR.puts "Could not start logger: #{e.class} #{e}"
+ end
+
+ def unconfigure
+ @configured = false
+ set_logger(nil)
+ end
+
+ # figures out the filename that called us
+ def from
+ from = File.basename(caller[2])
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Logger
+ autoload :Base, "mcollective/logger/base"
+ end
+end
--- /dev/null
+module MCollective
+ module Logger
+ # A base class for logging providers.
+ #
+ # Logging providers should provide the following:
+ #
+ # * start - all you need to do to setup your logging
+ # * set_logging_level - set your logging to :info, :warn, etc
+ # * valid_levels - a hash of maps from :info to your internal level name
+ # * log - what needs to be done to log a specific message
+ class Base
+ attr_reader :active_level
+
+ def initialize
+ @known_levels = [:debug, :info, :warn, :error, :fatal]
+
+ # Sanity check the class that impliments the logging
+ @known_levels.each do |lvl|
+ raise "Logger class did not specify a map for #{lvl}" unless valid_levels.include?(lvl)
+ end
+ end
+
+ def should_log?(level)
+ @known_levels.index(level) >= @known_levels.index(@active_level)
+ end
+
+ # Figures out the next level and sets it
+ def cycle_level
+ lvl = get_next_level
+ set_level(lvl)
+
+ log(lvl, "", "Logging level is now #{lvl.to_s.upcase}")
+ end
+
+ # Sets a new level and record it in @active_level
+ def set_level(level)
+ set_logging_level(level)
+ @active_level = level.to_sym
+ end
+
+ private
+ def map_level(level)
+ raise "Logger class do not know how to handle #{level} messages" unless valid_levels.include?(level.to_sym)
+
+ valid_levels[level.to_sym]
+ end
+
+ # Gets the next level in the list, cycles down to the firt once it reaches the end
+ def get_next_level
+ # if all else fails, always go to debug mode
+ nextlvl = :debug
+
+ if @known_levels.index(@active_level) == (@known_levels.size - 1)
+ nextlvl = @known_levels.first
+ else
+ idx = @known_levels.index(@active_level) + 1
+ nextlvl = @known_levels[idx]
+ end
+
+ nextlvl
+ end
+
+ # Abstract methods to ensure the logging implimentations supply what they should
+ def valid_levels
+ raise "The logging class did not supply a valid_levels method"
+ end
+
+ def log(level, from, msg)
+ raise "The logging class did not supply a log method"
+ end
+
+ def start
+ raise "The logging class did not supply a start method"
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Logger
+ # Implements a syslog based logger using the standard ruby syslog class
+ class Console_logger<Base
+ def start
+ set_level(:info)
+
+ config = Config.instance
+ set_level(config.loglevel.to_sym) if config.configured
+ end
+
+ def set_logging_level(level)
+ # nothing to do here, we ignore high levels when we log
+ end
+
+ def valid_levels
+ {:info => :info,
+ :warn => :warning,
+ :debug => :debug,
+ :fatal => :crit,
+ :error => :err}
+ end
+
+ def log(level, from, msg, normal_output=STDERR, last_resort_output=STDERR)
+ time = Time.new.strftime("%Y/%m/%d %H:%M:%S")
+
+ normal_output.puts("%s %s: %s %s" % [colorize(level, level), time, from, msg])
+ rescue
+ # if this fails we probably cant show the user output at all,
+ # STDERR it as last resort
+ last_resort_output.puts("#{level}: #{msg}")
+ end
+
+ # Set some colors for various logging levels, will honor the
+ # color configuration option and return nothing if its configured
+ # not to
+ def color(level)
+ colorize = Config.instance.color
+
+ colors = {:error => Util.color(:red),
+ :fatal => Util.color(:red),
+ :warn => Util.color(:yellow),
+ :info => Util.color(:green),
+ :reset => Util.color(:reset)}
+
+ if colorize
+ return colors[level] || ""
+ else
+ return ""
+ end
+ end
+
+ # Helper to return a string in specific color
+ def colorize(level, msg)
+ "%s%s%s" % [ color(level), msg, color(:reset) ]
+ end
+ end
+ end
+end
--- /dev/null
+require 'logger'
+
+module MCollective
+ module Logger
+ # Impliments a file based logger using the standard ruby logger class
+ #
+ # To configure you should set:
+ #
+ # - config.logfile
+ # - config.keeplogs defaults to 2097152
+ # - config.max_log_size defaults to 5
+ class File_logger<Base
+ def start
+ config = Config.instance
+
+ @logger = ::Logger.new(config.logfile, config.keeplogs, config.max_log_size)
+ @logger.formatter = ::Logger::Formatter.new
+
+ set_level(config.loglevel.to_sym)
+ end
+
+ def set_logging_level(level)
+ @logger.level = map_level(level)
+ rescue Exception => e
+ @logger.level = ::Logger::DEBUG
+ log(:error, "", "Could not set logging to #{level} using debug instead: #{e.class} #{e}")
+ end
+
+ def valid_levels
+ {:info => ::Logger::INFO,
+ :warn => ::Logger::WARN,
+ :debug => ::Logger::DEBUG,
+ :fatal => ::Logger::FATAL,
+ :error => ::Logger::ERROR}
+ end
+
+ def log(level, from, msg)
+ @logger.add(map_level(level)) { "#{from} #{msg}" }
+ rescue
+ # if this fails we probably cant show the user output at all,
+ # STDERR it as last resort
+ STDERR.puts("#{level}: #{msg}")
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Logger
+ # Implements a syslog based logger using the standard ruby syslog class
+ class Syslog_logger<Base
+ require 'syslog'
+
+ include Syslog::Constants
+
+ def start
+ config = Config.instance
+
+ facility = syslog_facility(config.logfacility)
+ level = config.loglevel.to_sym
+
+ Syslog.close if Syslog.opened?
+ Syslog.open(File.basename($0), 3, facility)
+
+ set_level(level)
+ end
+
+ def syslog_facility(facility)
+ begin
+ Syslog.const_get("LOG_#{facility.upcase}")
+ rescue NameError => e
+ STDERR.puts "Invalid syslog facility #{facility} supplied, reverting to USER"
+ Syslog::LOG_USER
+ end
+ end
+
+ def set_logging_level(level)
+ # noop
+ end
+
+ def valid_levels
+ {:info => :info,
+ :warn => :warning,
+ :debug => :debug,
+ :fatal => :crit,
+ :error => :err}
+ end
+
+ def log(level, from, msg)
+ Syslog.send(map_level(level), "#{from} #{msg}")
+ rescue
+ # if this fails we probably cant show the user output at all,
+ # STDERR it as last resort
+ STDERR.puts("#{level}: #{msg}")
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # A parser and scanner that creates a stack machine for a simple
+ # fact and class matching language used on the CLI to facilitate
+ # a rich discovery language
+ #
+ # Language EBNF
+ #
+ # compound = ["("] expression [")"] {["("] expression [")"]}
+ # expression = [!|not]statement ["and"|"or"] [!|not] statement
+ # char = A-Z | a-z | < | > | => | =< | _ | - |* | / { A-Z | a-z | < | > | => | =< | _ | - | * | / | }
+ # int = 0|1|2|3|4|5|6|7|8|9{|0|1|2|3|4|5|6|7|8|9|0}
+ module Matcher
+ autoload :Parser, "mcollective/matcher/parser"
+ autoload :Scanner, "mcollective/matcher/scanner"
+
+ # Helper creates a hash from a function call string
+ def self.create_function_hash(function_call)
+ func_hash = {}
+ f = ""
+ func_parts = function_call.split(/(!=|>=|<=|<|>|=)/)
+ func_hash["r_compare"] = func_parts.pop
+ func_hash["operator"] = func_parts.pop
+ func = func_parts.join
+
+ # Deal with dots in function parameters and functions without dot values
+ if func.match(/^.+\(.*\)$/)
+ f = func
+ else
+ func_parts = func.split(".")
+ func_hash["value"] = func_parts.pop
+ f = func_parts.join(".")
+ end
+
+ # Deal with regular expression matches
+ if func_hash["r_compare"] =~ /^\/.*\/$/
+ func_hash["operator"] = "=~" if func_hash["operator"] == "="
+ func_hash["operator"] = "!=~" if func_hash["operator"] == "!="
+ func_hash["r_compare"] = Regexp.new(func_hash["r_compare"].gsub(/^\/|\/$/, ""))
+ # Convert = operators to == so they can be propperly evaluated
+ elsif func_hash["operator"] == "="
+ func_hash["operator"] = "=="
+ end
+
+ # Grab function name and parameters from left compare string
+ func_hash["name"], func_hash["params"] = f.split("(")
+ if func_hash["params"] == ")"
+ func_hash["params"] = nil
+ else
+
+ # Walk the function parameters from the front and from the
+ # back removing the first and last instances of single of
+ # double qoutes. We do this to handle the case where params
+ # contain escaped qoutes.
+ func_hash["params"] = func_hash["params"].gsub(")", "")
+ func_quotes = func_hash["params"].split(/('|")/)
+
+ func_quotes.each_with_index do |item, i|
+ if item.match(/'|"/)
+ func_quotes.delete_at(i)
+ break
+ end
+ end
+
+ func_quotes.reverse.each_with_index do |item,i|
+ if item.match(/'|"/)
+ func_quotes.delete_at(func_quotes.size - i - 1)
+ break
+ end
+ end
+
+ func_hash["params"] = func_quotes.join
+ end
+
+ func_hash
+ end
+
+ # Returns the result of an executed function
+ def self.execute_function(function_hash)
+ # In the case where a data plugin isn't present there are two ways we can handle
+ # the raised exception. The function result can either be false or the entire
+ # expression can fail.
+ #
+ # In the case where we return the result as false it opens us op to unexpected
+ # negation behavior.
+ #
+ # !foo('bar').name = bar
+ #
+ # In this case the user would expect discovery to match on all machines where
+ # the name value of the foo function does not equal bar. If a non existent function
+ # returns false then it is posible to match machines where the name value of the
+ # foo function is bar.
+ #
+ # Instead we raise a DDLValidationError to prevent this unexpected behavior from
+ # happening.
+
+ result = Data.send(function_hash["name"], function_hash["params"])
+
+ if function_hash["value"]
+ eval_result = result.send(function_hash["value"])
+ return eval_result
+ else
+ return result
+ end
+ rescue NoMethodError
+ Log.debug("cannot execute discovery function '#{function_hash["name"]}'. data plugin not found")
+ raise DDLValidationError
+ end
+
+ # Evaluates a compound statement
+ def self.eval_compound_statement(expression)
+ if expression.values.first =~ /^\//
+ return Util.has_cf_class?(expression.values.first)
+ elsif expression.values.first =~ />=|<=|=|<|>/
+ optype = expression.values.first.match(/>=|<=|=|<|>/)
+ name, value = expression.values.first.split(optype[0])
+ unless value.split("")[0] == "/"
+ optype[0] == "=" ? optype = "==" : optype = optype[0]
+ else
+ optype = "=~"
+ end
+
+ return Util.has_fact?(name,value, optype).to_s
+ else
+ return Util.has_cf_class?(expression.values.first)
+ end
+ end
+
+ # Returns the result of an evaluated compound statement that
+ # includes a function
+ def self.eval_compound_fstatement(function_hash)
+ l_compare = execute_function(function_hash)
+
+ # Prevent unwanted discovery by limiting comparison operators
+ # on Strings and Booleans
+ if((l_compare.is_a?(String) || l_compare.is_a?(TrueClass) || l_compare.is_a?(FalseClass)) && function_hash["operator"].match(/<|>/))
+ Log.debug "Cannot do > and < comparison on Booleans and Strings '#{l_compare} #{function_hash["operator"]} #{function_hash["r_compare"]}'"
+ return false
+ end
+
+ # Prevent backticks in function parameters
+ if function_hash["params"] =~ /`/
+ Log.debug("Cannot use backticks in function parameters")
+ return false
+ end
+
+ # Escape strings for evaluation
+ function_hash["r_compare"] = "\"#{function_hash["r_compare"]}\"" if(l_compare.is_a?(String) && !(function_hash["operator"] =~ /=~|!=~/))
+
+ # Do a regex comparison if right compare string is a regex
+ if function_hash["operator"] =~ /(=~|!=~)/
+ # Fail if left compare value isn't a string
+ unless l_compare.is_a?(String)
+ Log.debug("Cannot do a regex check on a non string value.")
+ return false
+ else
+ compare_result = l_compare.match(function_hash["r_compare"])
+ # Flip return value for != operator
+ if function_hash["operator"] == "!=~"
+ !((compare_result.nil?) ? false : true)
+ else
+ (compare_result.nil?) ? false : true
+ end
+ end
+ # Otherwise evaluate the logical comparison
+ else
+ l_compare = "\"#{l_compare}\"" if l_compare.is_a?(String)
+ result = eval("#{l_compare} #{function_hash["operator"]} #{function_hash["r_compare"]}")
+ (result.nil?) ? false : result
+ end
+ end
+
+ # Creates a callstack to be evaluated from a compound evaluation string
+ def self.create_compound_callstack(call_string)
+ callstack = Matcher::Parser.new(call_string).execution_stack
+ callstack.each_with_index do |statement, i|
+ if statement.keys.first == "fstatement"
+ callstack[i]["fstatement"] = create_function_hash(statement.values.first)
+ end
+ end
+ callstack
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Matcher
+ class Parser
+ attr_reader :scanner, :execution_stack
+
+ def initialize(args)
+ @scanner = Scanner.new(args)
+ @execution_stack = []
+ @parse_errors = []
+ @token_errors = []
+ @paren_errors = []
+ parse
+ exit_with_token_errors if @token_errors.size > 0
+ exit_with_parse_errors if @parse_errors.size > 0
+ exit_with_paren_errors if @paren_errors.size > 0
+ end
+
+ # Exit and highlight any malformed tokens
+ def exit_with_token_errors
+ @token_errors.each do |error_range|
+ (error_range[0]..error_range[1]).each do |i|
+ @scanner.arguments[i] = Util.colorize(:red, @scanner.arguments[i])
+ end
+ end
+ raise "Malformed token(s) found while parsing -S input #{@scanner.arguments.join}"
+ end
+
+ def exit_with_parse_errors
+ @parse_errors.each do |error_range|
+ (error_range[0]..error_range[1]).each do |i|
+ @scanner.arguments[i] = Util.colorize(:red, @scanner.arguments[i])
+ end
+ end
+ raise "Parse errors found while parsing -S input #{ @scanner.arguments.join}"
+ end
+
+ def exit_with_paren_errors
+ @paren_errors.each do |i|
+ @scanner.arguments[i] = Util.colorize(:red, @scanner.arguments[i])
+ end
+ raise "Missing parenthesis found while parsing -S input #{@scanner.arguments.join}"
+ end
+
+ # Parse the input string, one token at a time a contruct the call stack
+ def parse
+ pre_index = @scanner.token_index
+ p_token,p_token_value = nil
+ c_token,c_token_value = @scanner.get_token
+ parenth = 0
+
+ while (c_token != nil)
+ @scanner.token_index += 1
+ n_token, n_token_value = @scanner.get_token
+
+ unless n_token == " "
+ case c_token
+ when "bad_token"
+ @token_errors << c_token_value
+
+ when "and"
+ unless (n_token =~ /not|fstatement|statement|\(/) || (scanner.token_index == scanner.arguments.size) && !(n_token == nil)
+ @parse_errors << [pre_index, scanner.token_index]
+ end
+
+ if p_token == nil
+ @parse_errors << [pre_index - c_token.size, scanner.token_index]
+ elsif (p_token == "and" || p_token == "or")
+ @parse_errors << [pre_index - 1 - p_token.size, pre_index - 1]
+ end
+
+ when "or"
+ unless (n_token =~ /not|fstatement|statement|\(/) || (scanner.token_index == scanner.arguments.size) && !(n_token == nil)
+ @parse_errors << [pre_index, scanner.token_index]
+ end
+
+ if p_token == nil
+ @parse_errors << [pre_index - c_token.size, scanner.token_index]
+ elsif (p_token == "and" || p_token == "or")
+ @parse_errors << [pre_index - 1 - p_token.size, pre_index - 1]
+ end
+
+ when "not"
+ unless n_token =~ /fstatement|statement|\(|not/ && !(n_token == nil)
+ @parse_errors << [pre_index, scanner.token_index]
+ end
+
+ when "statement","fstatement"
+ unless n_token =~ /and|or|\)/
+ unless scanner.token_index == scanner.arguments.size
+ @parse_errors << [pre_index, scanner.token_index]
+ end
+ end
+
+ when ")"
+ unless (n_token =~ /|and|or|not|\(/)
+ unless(scanner.token_index == scanner.arguments.size)
+ @parse_errors << [pre_index, scanner.token_index]
+ end
+ end
+ unless @paren_errors.empty?
+ @paren_errors.pop
+ else
+ @paren_errors.push((n_token.nil?) ? scanner.token_index - 1: scanner.token_index - n_token_value.size)
+ end
+
+ when "("
+ unless n_token =~ /fstatement|statement|not|\(/
+ @parse_errors << [pre_index, scanner.token_index]
+ end
+ @paren_errors.push((n_token.nil?) ? scanner.token_index - 1: scanner.token_index - n_token_value.size)
+
+ else
+ @parse_errors << [pre_index, scanner.token_index]
+ end
+
+ unless n_token == " " ||c_token == "bad_token"
+ @execution_stack << {c_token => c_token_value}
+ end
+
+ p_token, p_token_value = c_token, c_token_value
+ c_token, c_token_value = n_token, n_token_value
+ end
+ pre_index = @scanner.token_index
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Matcher
+ class Scanner
+ attr_accessor :arguments, :token_index
+
+ def initialize(arguments)
+ @token_index = 0
+ @arguments = arguments.split("")
+ @seperation_counter = 0
+ @white_spaces = 0
+ end
+
+ # Scans the input string and identifies single language tokens
+ def get_token
+ if @token_index >= @arguments.size
+ return nil
+ end
+
+ case @arguments[@token_index]
+ when "("
+ return "(", "("
+
+ when ")"
+ return ")", ")"
+
+ when "n"
+ if (@arguments[@token_index + 1] == "o") && (@arguments[@token_index + 2] == "t") && ((@arguments[@token_index + 3] == " ") || (@arguments[@token_index + 3] == "("))
+ @token_index += 2
+ return "not", "not"
+ else
+ gen_statement
+ end
+
+ when "!"
+ return "not", "not"
+
+ when "a"
+ if (@arguments[@token_index + 1] == "n") && (@arguments[@token_index + 2] == "d") && ((@arguments[@token_index + 3] == " ") || (@arguments[@token_index + 3] == "("))
+ @token_index += 2
+ return "and", "and"
+ else
+ gen_statement
+ end
+
+ when "o"
+ if (@arguments[@token_index + 1] == "r") && ((@arguments[@token_index + 2] == " ") || (@arguments[@token_index + 2] == "("))
+ @token_index += 1
+ return "or", "or"
+ else
+ gen_statement
+ end
+
+ when " "
+ return " ", " "
+
+ else
+ gen_statement
+ end
+ end
+
+ private
+ # Helper generates a statement token
+ def gen_statement
+ func = false
+ current_token_value = ""
+ j = @token_index
+
+ begin
+ if (@arguments[j] == "/")
+ begin
+ current_token_value << @arguments[j]
+ j += 1
+ end until (j >= @arguments.size) || (@arguments[j] =~ /\s/)
+ elsif (@arguments[j] =~ /=|<|>/)
+ while !(@arguments[j] =~ /=|<|>/)
+ current_token_value << @arguments[j]
+ j += 1
+ end
+
+ current_token_value << @arguments[j]
+ j += 1
+
+ if @arguments[j] == "/"
+ begin
+ current_token_value << @arguments[j]
+ j += 1
+ if @arguments[j] == "/"
+ current_token_value << "/"
+ break
+ end
+ end until (j >= @arguments.size) || (@arguments[j] =~ /\//)
+ else
+ while (j < @arguments.size) && ((@arguments[j] != " ") && (@arguments[j] != ")"))
+ current_token_value << @arguments[j]
+ j += 1
+ end
+ end
+ else
+ begin
+ # Identify and tokenize regular expressions by ignoring everything between /'s
+ if @arguments[j] == '/'
+ current_token_value << '/'
+ j+=1
+ while(j < @arguments.size && @arguments[j] != '/')
+ current_token_value << @arguments[j]
+ j += 1
+ end
+ current_token_value << @arguments[j] if @arguments[j]
+ break
+ end
+ if @arguments[j+1] == "("
+ func = true
+ be_greedy = true
+ end
+ current_token_value << @arguments[j]
+ if be_greedy
+ while !(j+1 >= @arguments.size) && @arguments[j] != ')'
+ j += 1
+ current_token_value << @arguments[j]
+ end
+ j += 1
+ be_greedy = false
+ else
+ j += 1
+ end
+ if(@arguments[j] == ' ')
+ break if(is_klass?(j) && !(@arguments[j-1] =~ /=|<|>/))
+ end
+ if( (@arguments[j] == ' ') && (@seperation_counter < 2) && !(current_token_value.match(/^.+(=|<|>).+$/)) )
+ if((index = lookahead(j)))
+ j = index
+ end
+ end
+ end until (j >= @arguments.size) || (@arguments[j] =~ /\s|\)/)
+ @seperation_counter = 0
+ end
+ rescue Exception => e
+ raise "An exception was raised while trying to tokenize '#{current_token_value} - #{e}'"
+ end
+
+ @token_index += current_token_value.size + @white_spaces - 1
+ @white_spaces = 0
+
+ # bar(
+ if current_token_value.match(/.+?\($/)
+ return "bad_token", [@token_index - current_token_value.size + 1, @token_index]
+ # /foo/=bar
+ elsif current_token_value.match(/^\/.+?\/(<|>|=).+/)
+ return "bad_token", [@token_index - current_token_value.size + 1, @token_index]
+ elsif current_token_value.match(/^.+?\/(<|>|=).+/)
+ return "bad_token", [@token_index - current_token_value.size + 1, @token_index]
+ else
+ if func
+ if current_token_value.match(/^.+?\((\s*(')[^']*(')\s*(,\s*(')[^']*('))*)?\)(\.[a-zA-Z0-9_]+)?((!=|<=|>=|=|>|<).+)?$/) ||
+ current_token_value.match(/^.+?\((\s*(")[^"]*(")\s*(,\s*(")[^"]*("))*)?\)(\.[a-zA-Z0-9_]+)?((!=|<=|>=|=|>|<).+)?$/)
+ return "fstatement", current_token_value
+ else
+ return "bad_token", [@token_index - current_token_value.size + 1, @token_index]
+ end
+ else
+ slash_err = false
+ current_token_value.split('').each do |c|
+ if c == '/'
+ slash_err = !slash_err
+ end
+ end
+ return "bad_token", [@token_index - current_token_value.size + 1, @token_index] if slash_err
+ return "statement", current_token_value
+ end
+ end
+ end
+
+ # Deal with special puppet class statement
+ def is_klass?(j)
+ while(j < @arguments.size && @arguments[j] == ' ')
+ j += 1
+ end
+
+ if @arguments[j] =~ /=|<|>/
+ return false
+ else
+ return true
+ end
+ end
+
+ # Eat spaces while looking for the next comparison symbol
+ def lookahead(index)
+ index += 1
+ while(index <= @arguments.size)
+ @white_spaces += 1
+ unless(@arguments[index] =~ /\s/)
+ @seperation_counter +=1
+ return index
+ end
+ index += 1
+ end
+ return nil
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # container for a message, its headers, agent, collective and other meta data
+ class Message
+ attr_reader :message, :request, :validated, :msgtime, :payload, :type, :expected_msgid, :reply_to
+ attr_accessor :headers, :agent, :collective, :filter
+ attr_accessor :requestid, :discovered_hosts, :options, :ttl
+
+ VALIDTYPES = [:message, :request, :direct_request, :reply]
+
+ # payload - the message body without headers etc, just the text
+ # message - the original message received from the middleware
+ # options[:base64] - if the body base64 encoded?
+ # options[:agent] - the agent the message is for/from
+ # options[:collective] - the collective its for/from
+ # options[:headers] - the message headers
+ # options[:type] - an indicator about the type of message, :message, :request, :direct_request or :reply
+ # options[:request] - if this is a reply this should old the message we are replying to
+ # options[:filter] - for requests, the filter to encode into the message
+ # options[:options] - the normal client options hash
+ # options[:ttl] - the maximum amount of seconds this message can be valid for
+ # options[:expected_msgid] - in the case of replies this is the msgid it is expecting in the replies
+ # options[:requestid] - specific request id to use else one will be generated
+ def initialize(payload, message, options = {})
+ options = {:base64 => false,
+ :agent => nil,
+ :headers => {},
+ :type => :message,
+ :request => nil,
+ :filter => Util.empty_filter,
+ :options => {},
+ :ttl => 60,
+ :expected_msgid => nil,
+ :requestid => nil,
+ :collective => nil}.merge(options)
+
+ @payload = payload
+ @message = message
+ @requestid = options[:requestid]
+ @discovered_hosts = nil
+ @reply_to = nil
+
+ @type = options[:type]
+ @headers = options[:headers]
+ @base64 = options[:base64]
+ @filter = options[:filter]
+ @expected_msgid = options[:expected_msgid]
+ @options = options[:options]
+
+ @ttl = @options[:ttl] || Config.instance.ttl
+ @msgtime = 0
+
+ @validated = false
+
+ if options[:request]
+ @request = options[:request]
+ @agent = request.agent
+ @collective = request.collective
+ @type = :reply
+ else
+ @agent = options[:agent]
+ @collective = options[:collective]
+ end
+
+ base64_decode!
+ end
+
+ # Sets the message type to one of the known types. In the case of :direct_request
+ # the list of hosts to communicate with should have been set with #discovered_hosts
+ # else an exception will be raised. This is for extra security, we never accidentally
+ # want to send a direct request without a list of hosts or something weird like that
+ # as it might result in a filterless broadcast being sent.
+ #
+ # Additionally you simply cannot set :direct_request if direct_addressing was not enabled
+ # this is to force a workflow that doesnt not yield in a mistake when someone might assume
+ # direct_addressing is enabled when its not.
+ def type=(type)
+ raise "Unknown message type #{type}" unless VALIDTYPES.include?(type)
+
+ if type == :direct_request
+ raise "Direct requests is not enabled using the direct_addressing config option" unless Config.instance.direct_addressing
+
+ unless @discovered_hosts && !@discovered_hosts.empty?
+ raise "Can only set type to :direct_request if discovered_hosts have been set"
+ end
+
+ # clear out the filter, custom discovery sources might interpret the filters
+ # different than the remote mcollectived and in directed mode really the only
+ # filter that matters is the agent filter
+ @filter = Util.empty_filter
+ @filter["agent"] << @agent
+ end
+
+ @type = type
+ end
+
+ # Sets a custom reply-to target for requests. The connector plugin should inspect this
+ # when constructing requests and set this header ensuring replies will go to the custom target
+ # otherwise the connector should just do what it usually does
+ def reply_to=(target)
+ raise "Custom reply targets can only be set on requests" unless [:request, :direct_request].include?(@type)
+
+ @reply_to = target
+ end
+
+ # in the case of reply messages we are expecting replies to a previously
+ # created message. This stores a hint to that previously sent message id
+ # and can be used by other classes like the security plugins as a means
+ # of optimizing their behavior like by ignoring messages not directed
+ # at us.
+ def expected_msgid=(msgid)
+ raise "Can only store the expected msgid for reply messages" unless @type == :reply
+ @expected_msgid = msgid
+ end
+
+ def base64_decode!
+ return unless @base64
+
+ @payload = SSL.base64_decode(@payload)
+ @base64 = false
+ end
+
+ def base64_encode!
+ return if @base64
+
+ @payload = SSL.base64_encode(@payload)
+ @base64 = true
+ end
+
+ def base64?
+ @base64
+ end
+
+ def encode!
+ case type
+ when :reply
+ raise "Cannot encode a reply message if no request has been associated with it" unless request
+ raise 'callerid in original request is not valid, surpressing reply to potentially forged request' unless PluginManager["security_plugin"].valid_callerid?(request.payload[:callerid])
+
+ @requestid = request.payload[:requestid]
+ @payload = PluginManager["security_plugin"].encodereply(agent, payload, requestid, request.payload[:callerid])
+ when :request, :direct_request
+ validate_compount_filter(@filter["compound"]) unless @filter["compound"].empty?
+
+ @requestid = create_reqid unless @requestid
+ @payload = PluginManager["security_plugin"].encoderequest(Config.instance.identity, payload, requestid, filter, agent, collective, ttl)
+ else
+ raise "Cannot encode #{type} messages"
+ end
+ end
+
+ def validate_compount_filter(compound_filter)
+ compound_filter.each do |filter|
+ filter.each do |statement|
+ if statement["fstatement"]
+ functionname = statement["fstatement"]["name"]
+ pluginname = Data.pluginname(functionname)
+ value = statement["fstatement"]["value"]
+
+ begin
+ ddl = DDL.new(pluginname, :data)
+ rescue
+ raise DDLValidationError, "Could not find DDL for data plugin #{pluginname}, cannot use #{functionname}() in discovery"
+ end
+
+ # parses numbers and booleans entered as strings into proper
+ # types of data so that DDL validation will pass
+ statement["fstatement"]["params"] = Data.ddl_transform_input(ddl, statement["fstatement"]["params"])
+
+ Data.ddl_validate(ddl, statement["fstatement"]["params"])
+
+ unless value && Data.ddl_has_output?(ddl, value)
+ raise DDLValidationError, "#{functionname}() does not return a #{value} value"
+ end
+ end
+ end
+ end
+ end
+
+ def decode!
+ raise "Cannot decode message type #{type}" unless [:request, :reply].include?(type)
+
+ @payload = PluginManager["security_plugin"].decodemsg(self)
+
+ if type == :request
+ raise 'callerid in request is not valid, surpressing reply to potentially forged request' unless PluginManager["security_plugin"].valid_callerid?(payload[:callerid])
+ end
+
+ [:collective, :agent, :filter, :requestid, :ttl, :msgtime].each do |prop|
+ instance_variable_set("@#{prop}", payload[prop]) if payload.include?(prop)
+ end
+ end
+
+ # Perform validation against the message by checking filters and ttl
+ def validate
+ raise "Can only validate request messages" unless type == :request
+
+ msg_age = Time.now.utc.to_i - msgtime
+
+ if msg_age > ttl
+ cid = ""
+ cid += payload[:callerid] + "@" if payload.include?(:callerid)
+ cid += payload[:senderid]
+
+ if msg_age > ttl
+ PluginManager["global_stats"].ttlexpired
+
+ raise(MsgTTLExpired, "message #{requestid} from #{cid} created at #{msgtime} is #{msg_age} seconds old, TTL is #{ttl}")
+ end
+ end
+
+ raise(NotTargettedAtUs, "Received message is not targetted to us") unless PluginManager["security_plugin"].validate_filter?(payload[:filter])
+
+ @validated = true
+ end
+
+ # publish a reply message by creating a target name and sending it
+ def publish
+ Timeout.timeout(2) do
+ # If we've been specificaly told about hosts that were discovered
+ # use that information to do P2P calls if appropriate else just
+ # send it as is.
+ if @discovered_hosts && Config.instance.direct_addressing
+ if @discovered_hosts.size <= Config.instance.direct_addressing_threshold
+ self.type = :direct_request
+ Log.debug("Handling #{requestid} as a direct request")
+ end
+
+ PluginManager["connector_plugin"].publish(self)
+ else
+ PluginManager["connector_plugin"].publish(self)
+ end
+ end
+ end
+
+ def create_reqid
+ # we gsub out the -s so that the format of the id does not
+ # change from previous versions, these should just be more
+ # unique than previous ones
+ SSL.uuid.gsub("-", "")
+ end
+ end
+end
--- /dev/null
+# start_with? was introduced in 1.8.7, we need to support
+# 1.8.5 and 1.8.6
+class String
+ def start_with?(str)
+ return self[0 .. (str.length-1)] == str
+ end unless method_defined?("start_with?")
+end
+
+# Make arrays of Symbols sortable
+class Symbol
+ include Comparable
+
+ def <=>(other)
+ self.to_s <=> other.to_s
+ end unless method_defined?("<=>")
+end
+
+# This provides an alias for RbConfig to Config for versions of Ruby older then
+# # version 1.8.5. This allows us to use RbConfig in place of the older Config in
+# # our code and still be compatible with at least Ruby 1.8.1.
+# require 'rbconfig'
+unless defined? ::RbConfig
+ ::RbConfig = ::Config
+end
+
+# a method # that walks an array in groups, pass a block to
+# call the block on each sub array
+class Array
+ def in_groups_of(chunk_size, padded_with=nil, &block)
+ arr = self.clone
+
+ # how many to add
+ padding = chunk_size - (arr.size % chunk_size)
+
+ # pad at the end
+ arr.concat([padded_with] * padding) unless padding == chunk_size
+
+ # how many chunks we'll make
+ count = arr.size / chunk_size
+
+ # make that many arrays
+ result = []
+ count.times {|s| result << arr[s * chunk_size, chunk_size]}
+
+ if block_given?
+ result.each_with_index do |a, i|
+ case block.arity
+ when 1
+ yield(a)
+ when 2
+ yield(a, (i == result.size - 1))
+ else
+ raise "Expected 1 or 2 arguments, got #{block.arity}"
+ end
+ end
+ else
+ result
+ end
+ end unless method_defined?(:in_groups_of)
+end
+
+class String
+ def bytes(&block)
+ # This should not be necessary, really ...
+ require 'enumerator'
+ return to_enum(:each_byte) unless block_given?
+ each_byte(&block)
+ end unless method_defined?(:bytes)
+end
+
+class Dir
+ def self.mktmpdir(prefix_suffix=nil, tmpdir=nil)
+ case prefix_suffix
+ when nil
+ prefix = "d"
+ suffix = ""
+ when String
+ prefix = prefix_suffix
+ suffix = ""
+ when Array
+ prefix = prefix_suffix[0]
+ suffix = prefix_suffix[1]
+ else
+ raise ArgumentError, "unexpected prefix_suffix: #{prefix_suffix.inspect}"
+ end
+ tmpdir ||= Dir.tmpdir
+ t = Time.now.strftime("%Y%m%d")
+ n = nil
+ begin
+ path = "#{tmpdir}/#{prefix}#{t}-#{$$}-#{rand(0x100000000).to_s(36)}"
+ path << "-#{n}" if n
+ path << suffix
+ Dir.mkdir(path, 0700)
+ rescue Errno::EEXIST
+ n ||= 0
+ n += 1
+ retry
+ end
+
+ if block_given?
+ begin
+ yield path
+ ensure
+ FileUtils.remove_entry_secure path
+ end
+ else
+ path
+ end
+ end unless method_defined?(:mktmpdir)
+
+ def self.tmpdir
+ tmp = '.'
+ for dir in [ENV['TMPDIR'], ENV['TMP'], ENV['TEMP'], '/tmp']
+ if dir and stat = File.stat(dir) and stat.directory? and stat.writable?
+ tmp = dir
+ break
+ end rescue nil
+ end
+ File.expand_path(tmp)
+ end unless method_defined?(:tmpdir)
+end
--- /dev/null
+module MCollective
+ # A simple helper to build cli tools that supports a uniform command line
+ # layout.
+ class Optionparser
+ attr_reader :parser
+
+ # Creates a new instance of the parser, you can supply defaults and include named groups of options.
+ #
+ # Starts a parser that defaults to verbose and that includs the filter options:
+ #
+ # oparser = MCollective::Optionparser.new({:verbose => true}, "filter")
+ #
+ # Stats a parser in non verbose mode that does support discovery
+ #
+ # oparser = MCollective::Optionparser.new()
+ #
+ # Starts a parser in verbose mode that does not show the common options:
+ #
+ # oparser = MCollective::Optionparser.new({:verbose => true}, "filter", "common")
+ def initialize(defaults = {}, include_sections = nil, exclude_sections = nil)
+ @parser = ::OptionParser.new
+
+ @include = [include_sections].flatten
+ @exclude = [exclude_sections].flatten
+
+ @options = Util.default_options
+
+ @options.merge!(defaults)
+ end
+
+ # Parse the options returning the options, you can pass a block that adds additional options
+ # to the Optionparser.
+ #
+ # The sample below starts a parser that also prompts for --arguments in addition to the defaults.
+ # It also sets the description and shows a usage message specific to this app.
+ #
+ # options = oparser.parse{|parser, options|
+ # parser.define_head "Control the mcollective controller daemon"
+ # parser.banner = "Usage: sh-mcollective [options] command"
+ #
+ # parser.on('--arg', '--argument ARGUMENT', 'Argument to pass to agent') do |v|
+ # options[:argument] = v
+ # end
+ # }
+ #
+ # Users can set default options that get parsed in using the MCOLLECTIVE_EXTRA_OPTS environemnt
+ # variable
+ def parse(&block)
+ yield(@parser, @options) if block_given?
+
+ add_required_options
+
+ add_common_options unless @exclude.include?("common")
+
+ @include.each do |i|
+ next if @exclude.include?(i)
+
+ options_name = "add_#{i}_options"
+ send(options_name) if respond_to?(options_name)
+ end
+
+ @parser.environment("MCOLLECTIVE_EXTRA_OPTS")
+
+ @parser.parse!
+
+ @options[:collective] = Config.instance.main_collective unless @options[:collective]
+
+ @options
+ end
+
+ # These options will be added if you pass 'filter' into the include list of the
+ # constructor.
+ def add_filter_options
+ @parser.separator ""
+ @parser.separator "Host Filters"
+
+ @parser.on('-W', '--with FILTER', 'Combined classes and facts filter') do |f|
+ f.split(" ").each do |filter|
+ begin
+ fact_parsed = parse_fact(filter)
+ @options[:filter]["fact"] << fact_parsed
+ rescue
+ @options[:filter]["cf_class"] << filter
+ end
+ end
+ end
+
+ @parser.on('-S', '--select FILTER', 'Compound filter combining facts and classes') do |f|
+ @options[:filter]["compound"] << Matcher.create_compound_callstack(f)
+ end
+
+ @parser.on('-F', '--wf', '--with-fact fact=val', 'Match hosts with a certain fact') do |f|
+ fact_parsed = parse_fact(f)
+
+ @options[:filter]["fact"] << fact_parsed if fact_parsed
+ end
+
+ @parser.on('-C', '--wc', '--with-class CLASS', 'Match hosts with a certain config management class') do |f|
+ @options[:filter]["cf_class"] << f
+ end
+
+ @parser.on('-A', '--wa', '--with-agent AGENT', 'Match hosts with a certain agent') do |a|
+ @options[:filter]["agent"] << a
+ end
+
+ @parser.on('-I', '--wi', '--with-identity IDENT', 'Match hosts with a certain configured identity') do |a|
+ @options[:filter]["identity"] << a
+ end
+ end
+
+ # These options should always be present
+ def add_required_options
+ @parser.on('-c', '--config FILE', 'Load configuratuion from file rather than default') do |f|
+ @options[:config] = f
+ end
+
+ @parser.on('-v', '--verbose', 'Be verbose') do |v|
+ @options[:verbose] = v
+ end
+
+ @parser.on('-h', '--help', 'Display this screen') do
+ puts @parser
+ exit! 1
+ end
+ end
+
+ # These options will be added to most cli tools
+ def add_common_options
+ @parser.separator ""
+ @parser.separator "Common Options"
+
+ @parser.on('-T', '--target COLLECTIVE', 'Target messages to a specific sub collective') do |f|
+ @options[:collective] = f
+ end
+
+ @parser.on('--dt', '--discovery-timeout SECONDS', Integer, 'Timeout for doing discovery') do |t|
+ @options[:disctimeout] = t
+ end
+
+ @parser.on('-t', '--timeout SECONDS', Integer, 'Timeout for calling remote agents') do |t|
+ @options[:timeout] = t
+ end
+
+ @parser.on('-q', '--quiet', 'Do not be verbose') do |v|
+ @options[:verbose] = false
+ end
+
+ @parser.on('--ttl TTL', 'Set the message validity period') do |v|
+ @options[:ttl] = v.to_i
+ end
+
+ @parser.on('--reply-to TARGET', 'Set a custom target for replies') do |v|
+ @options[:reply_to] = v
+ end
+
+ @parser.on('--dm', '--disc-method METHOD', 'Which discovery method to use') do |v|
+ raise "Discovery method is already set by a competing option" if @options[:discovery_method] && @options[:discovery_method] != v
+ @options[:discovery_method] = v
+ end
+
+ @parser.on('--do', '--disc-option OPTION', 'Options to pass to the discovery method') do |a|
+ @options[:discovery_options] << a
+ end
+
+ @parser.on("--nodes FILE", "List of nodes to address") do |v|
+ raise "Cannot mix --disc-method, --disc-option and --nodes" if @options[:discovery_method] || @options[:discovery_options].size > 0
+ raise "Cannot read the discovery file #{v}" unless File.readable?(v)
+
+ @options[:discovery_method] = "flatfile"
+ @options[:discovery_options] = v
+ end
+ end
+
+ private
+ # Parse a fact filter string like foo=bar into the tuple hash thats needed
+ def parse_fact(fact)
+ Util.parse_fact_string(fact)
+ end
+
+ end
+end
--- /dev/null
+module MCollective
+ # A simple plugin manager, it stores one plugin each of a specific type
+ # the idea is that we can only have one security provider, one connector etc.
+ module PluginManager
+ @plugins = {}
+
+ # Adds a plugin to the list of plugins, we expect a hash like:
+ #
+ # {:type => "base",
+ # :class => foo.new}
+ #
+ # or like:
+ # {:type => "base",
+ # :class => "Foo::Bar"}
+ #
+ # In the event that we already have a class with the given type
+ # an exception will be raised.
+ #
+ # If the :class passed is a String then we will delay instantiation
+ # till the first time someone asks for the plugin, this is because most likely
+ # the registration gets done by inherited() hooks, at which point the plugin class is not final.
+ #
+ # If we were to do a .new here the Class initialize method would get called and not
+ # the plugins, we there for only initialize the classes when they get requested via []
+ #
+ # By default all plugin instances are cached and returned later so there's
+ # always a single instance. You can pass :single_instance => false when
+ # calling this to instruct it to always return a new instance when a copy
+ # is requested. This only works with sending a String for :class.
+ def self.<<(plugin)
+ plugin[:single_instance] = true unless plugin.include?(:single_instance)
+
+ type = plugin[:type]
+ klass = plugin[:class]
+ single = plugin[:single_instance]
+
+ raise("Plugin #{type} already loaded") if @plugins.include?(type)
+
+
+ # If we get a string then store 'nil' as the instance, signalling that we'll
+ # create the class later on demand.
+ if klass.is_a?(String)
+ @plugins[type] = {:loadtime => Time.now, :class => klass, :instance => nil, :single => single}
+ Log.debug("Registering plugin #{type} with class #{klass} single_instance: #{single}")
+ else
+ @plugins[type] = {:loadtime => Time.now, :class => klass.class, :instance => klass, :single => true}
+ Log.debug("Registering plugin #{type} with class #{klass.class} single_instance: true")
+ end
+ end
+
+ # Removes a plugim the list
+ def self.delete(plugin)
+ @plugins.delete(plugin) if @plugins.include?(plugin)
+ end
+
+ # Finds out if we have a plugin with the given name
+ def self.include?(plugin)
+ @plugins.include?(plugin)
+ end
+
+ # Provides a list of plugins we know about
+ def self.pluginlist
+ @plugins.keys.sort
+ end
+
+ # deletes all registered plugins
+ def self.clear
+ @plugins.clear
+ end
+
+ # Gets a plugin by type
+ def self.[](plugin)
+ raise("No plugin #{plugin} defined") unless @plugins.include?(plugin)
+
+ klass = @plugins[plugin][:class]
+
+ if @plugins[plugin][:single]
+ # Create an instance of the class if one hasn't been done before
+ if @plugins[plugin][:instance] == nil
+ Log.debug("Returning new plugin #{plugin} with class #{klass}")
+ @plugins[plugin][:instance] = create_instance(klass)
+ else
+ Log.debug("Returning cached plugin #{plugin} with class #{klass}")
+ end
+
+ @plugins[plugin][:instance]
+ else
+ Log.debug("Returning new plugin #{plugin} with class #{klass}")
+ create_instance(klass)
+ end
+ end
+
+ # use eval to create an instance of a class
+ def self.create_instance(klass)
+ begin
+ eval("#{klass}.new")
+ rescue Exception => e
+ raise("Could not create instance of plugin #{klass}: #{e}")
+ end
+ end
+
+ # Finds plugins in all configured libdirs
+ #
+ # find("agent")
+ #
+ # will return an array of just agent names, for example:
+ #
+ # ["puppetd", "package"]
+ #
+ # Can also be used to find files of other extensions:
+ #
+ # find("agent", "ddl")
+ #
+ # Will return the same list but only of files with extension .ddl
+ # in the agent subdirectory
+ def self.find(type, extension="rb")
+ extension = ".#{extension}" unless extension.match(/^\./)
+
+ plugins = []
+
+ Config.instance.libdir.each do |libdir|
+ plugdir = File.join([libdir, "mcollective", type.to_s])
+ next unless File.directory?(plugdir)
+
+ Dir.new(plugdir).grep(/#{extension}$/).map do |plugin|
+ plugins << File.basename(plugin, extension)
+ end
+ end
+
+ plugins.sort.uniq
+ end
+
+ # Finds and loads from disk all plugins from all libdirs that match
+ # certain criteria.
+ #
+ # find_and_load("pluginpackager")
+ #
+ # Will find all .rb files in the libdir/mcollective/pluginpackager/
+ # directory in all libdirs and load them from disk.
+ #
+ # You can influence what plugins get loaded using a block notation:
+ #
+ # find_and_load("pluginpackager") do |plugin|
+ # plugin.match(/puppet/)
+ # end
+ #
+ # This will load only plugins matching /puppet/
+ def self.find_and_load(type, extension="rb")
+ extension = ".#{extension}" unless extension.match(/^\./)
+
+ klasses = find(type, extension).map do |plugin|
+ if block_given?
+ next unless yield(plugin)
+ end
+
+ "%s::%s::%s" % [ "MCollective", type.capitalize, plugin.capitalize ]
+ end.compact
+
+ klasses.sort.uniq.each {|klass| loadclass(klass, true)}
+ end
+
+ # Loads a class from file by doing some simple search/replace
+ # on class names and then doing a require.
+ def self.loadclass(klass, squash_failures=false)
+ fname = klass.gsub("::", "/").downcase + ".rb"
+
+ Log.debug("Loading #{klass} from #{fname}")
+
+ load fname
+ rescue Exception => e
+ Log.error("Failed to load #{klass}: #{e}")
+ raise unless squash_failures
+ end
+
+ # Grep's over the plugin list and returns the list found
+ def self.grep(regex)
+ @plugins.keys.grep(regex)
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module PluginPackager
+ # Plugin definition classes
+ autoload :AgentDefinition, "mcollective/pluginpackager/agent_definition"
+ autoload :StandardDefinition, "mcollective/pluginpackager/standard_definition"
+
+ # Package implementation plugins
+ def self.load_packagers
+ PluginManager.find_and_load("pluginpackager")
+ end
+
+ def self.[](klass)
+ const_get("#{klass}")
+ end
+
+ # Fetch and return metadata from plugin DDL
+ def self.get_metadata(path, type)
+ ddl = DDL.new("package", type.to_sym, false)
+
+ begin
+ ddl_file = File.read(Dir.glob(File.join(path, type, "*.ddl")).first)
+ rescue Exception
+ raise "failed to load ddl file in plugin directory : #{File.join(path, type)}"
+ end
+ ddl.instance_eval ddl_file
+
+ return ddl.meta, ddl.requirements[:mcollective]
+ end
+
+ # Checks if a directory is present and not empty
+ def self.check_dir_present(path)
+ (File.directory?(path) && !Dir.glob(File.join(path, "*")).empty?)
+ end
+
+ # Quietly calls a block if verbose parameter is false
+ def self.do_quietly?(verbose, &block)
+ unless verbose
+ old_stdout = $stdout.clone
+ $stdout.reopen(File.new("/dev/null", "w"))
+ begin
+ block.call
+ rescue Exception => e
+ $stdout.reopen old_stdout
+ raise e
+ ensure
+ $stdout.reopen old_stdout
+ end
+ else
+ block.call
+ end
+ end
+
+ # Checks if a build tool is present on the system
+ def self.build_tool?(build_tool)
+ ENV["PATH"].split(File::PATH_SEPARATOR).each do |path|
+ builder = File.join(path, build_tool)
+ if File.exists?(builder)
+ return true
+ end
+ end
+ false
+ end
+
+ def self.safe_system(*args)
+ raise RuntimeError, "Failed: #{args.join(' ')}" unless system *args
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module PluginPackager
+ # MCollective Agent Plugin package
+ class AgentDefinition
+ attr_accessor :path, :packagedata, :metadata, :target_path, :vendor, :iteration, :preinstall
+ attr_accessor :plugintype, :dependencies, :postinstall, :mcname, :mcversion
+
+ def initialize(path, name, vendor, preinstall, postinstall, iteration, dependencies, mcdependency, plugintype)
+ @plugintype = plugintype
+ @path = path
+ @packagedata = {}
+ @iteration = iteration || 1
+ @preinstall = preinstall
+ @postinstall = postinstall
+ @vendor = vendor || "Puppet Labs"
+ @dependencies = dependencies || []
+ @target_path = File.expand_path(@path)
+ @metadata, mcversion = PluginPackager.get_metadata(@path, "agent")
+ @mcname = mcdependency[:mcname] || "mcollective"
+ @mcversion = mcdependency[:mcversion] || mcversion
+ @dependencies << {:name => "#{@mcname}-common", :version => @mcversion}
+
+ @metadata[:name] = (name || @metadata[:name]).downcase.gsub(/\s+|_/, "-")
+ identify_packages
+ end
+
+ # Identify present packages and populate packagedata hash.
+ def identify_packages
+ common_package = common
+ @packagedata[:common] = common_package if common_package
+ agent_package = agent
+ @packagedata[:agent] = agent_package if agent_package
+ client_package = client
+ @packagedata[:client] = client_package if client_package
+ end
+
+ # Obtain Agent package files and dependencies.
+ def agent
+ agent = {:files => [],
+ :dependencies => @dependencies.clone,
+ :description => "Agent plugin for #{@metadata[:name]}"}
+
+ agentdir = File.join(@path, "agent")
+
+ if PluginPackager.check_dir_present agentdir
+ ddls = Dir.glob(File.join(agentdir, "*.ddl"))
+ agent[:files] = (Dir.glob(File.join(agentdir, "*")) - ddls)
+ else
+ return nil
+ end
+ agent[:plugindependency] = {:name => "#{@mcname}-#{@metadata[:name]}-common", :version => @metadata[:version], :iteration => @iteration}
+ agent
+ end
+
+ # Obtain client package files and dependencies.
+ def client
+ client = {:files => [],
+ :dependencies => @dependencies.clone,
+ :description => "Client plugin for #{@metadata[:name]}"}
+
+ clientdir = File.join(@path, "application")
+ aggregatedir = File.join(@path, "aggregate")
+
+ client[:files] += Dir.glob(File.join(clientdir, "*")) if PluginPackager.check_dir_present clientdir
+ client[:files] += Dir.glob(File.join(aggregatedir, "*")) if PluginPackager.check_dir_present aggregatedir
+ client[:plugindependency] = {:name => "#{@mcname}-#{@metadata[:name]}-common", :version => @metadata[:version], :iteration => @iteration}
+ client[:files].empty? ? nil : client
+ end
+
+ # Obtain common package files and dependencies.
+ def common
+ common = {:files =>[],
+ :dependencies => @dependencies.clone,
+ :description => "Common libraries for #{@metadata[:name]}"}
+
+ datadir = File.join(@path, "data", "**")
+ utildir = File.join(@path, "util", "**", "**")
+ ddldir = File.join(@path, "agent", "*.ddl")
+ validatordir = File.join(@path, "validator", "**")
+
+ [datadir, utildir, validatordir, ddldir].each do |directory|
+ common[:files] += Dir.glob(directory)
+ end
+
+ # We fail if there is no ddl file present
+ if common[:files].grep(/^.*\.ddl$/).empty?
+ raise "cannot create package - No ddl file found in #{File.join(@path, "agent")}"
+ end
+
+ common[:files].empty? ? nil : common
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module PluginPackager
+ class StandardDefinition
+ attr_accessor :path, :packagedata, :metadata, :target_path, :vendor, :iteration
+ attr_accessor :plugintype, :preinstall, :postinstall, :dependencies, :mcname, :mcversion
+
+ def initialize(path, name, vendor, preinstall, postinstall, iteration, dependencies, mcdependency, plugintype)
+ @plugintype = plugintype
+ @path = path
+ @packagedata = {}
+ @iteration = iteration || 1
+ @preinstall = preinstall
+ @postinstall = postinstall
+ @vendor = vendor || "Puppet Labs"
+ @dependencies = dependencies || []
+ @target_path = File.expand_path(@path)
+ @metadata, mcversion = PluginPackager.get_metadata(@path, @plugintype)
+
+ @mcname = mcdependency[:mcname] || "mcollective"
+ @mcversion = mcdependency[:mcversion] || mcversion
+ @dependencies << {:name => "#{mcname}-common", :version => @mcversion}
+ @metadata[:name] = (name || @metadata[:name]).downcase.gsub(/\s+|_/, "-")
+ identify_packages
+ end
+
+ # Identify present packages and populate the packagedata hash
+ def identify_packages
+ common_package = common
+ @packagedata[:common] = common_package if common_package
+ plugin_package = plugin
+ @packagedata[@plugintype] = plugin_package if plugin_package
+ end
+
+ # Obtain standard plugin files and dependencies
+ def plugin
+ plugindata = {:files => [],
+ :dependencies => @dependencies.clone,
+ :description => "#{@name} #{@plugintype} plugin for the Marionette Collective."}
+
+ plugindir = File.join(@path, @plugintype.to_s)
+ if PluginPackager.check_dir_present plugindir
+ plugindata[:files] = Dir.glob(File.join(plugindir, "*"))
+ else
+ return nil
+ end
+
+ plugindata[:plugindependency] = {:name => "#{@mcname}-#{@metadata[:name]}-common",
+ :version => @metadata[:version],
+ :iteration => @iteration} if @packagedata[:common]
+ plugindata
+ end
+
+ # Obtain list of common files
+ def common
+ common = {:files => [],
+ :dependencies => @dependencies.clone,
+ :description => "Common libraries for #{@name} connector plugin"}
+
+ commondir = File.join(@path, "util")
+ if PluginPackager.check_dir_present commondir
+ common[:files] = Dir.glob(File.join(commondir, "*"))
+ return common
+ else
+ return nil
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # Registration is implimented using a module structure and installations can
+ # configure which module they want to use.
+ #
+ # We provide a simple one that just sends back the list of current known agents
+ # in MCollective::Registration::Agentlist, you can create your own:
+ #
+ # Create a module in plugins/mcollective/registration/<yourplugin>.rb
+ #
+ # You can inherit from MCollective::Registration::Base in which case you just need
+ # to supply a _body_ method, whatever this method returns will be send to the
+ # middleware connection for an agent called _registration_
+ module Registration
+ autoload :Base, "mcollective/registration/base"
+ end
+end
--- /dev/null
+module MCollective
+ module Registration
+ # This is a base class that other registration plugins can use
+ # to handle regular announcements to the mcollective
+ #
+ # The configuration file determines how often registration messages
+ # gets sent using the _registerinterval_ option, the plugin runs in the
+ # background in a thread.
+ class Base
+ # Register plugins that inherits base
+ def self.inherited(klass)
+ PluginManager << {:type => "registration_plugin", :class => klass.to_s}
+ end
+
+ # Creates a background thread that periodically send a registration notice.
+ #
+ # The actual registration notices comes from the 'body' method of the registration
+ # plugins.
+ def run(connection)
+ return false if interval == 0
+
+ Thread.new do
+ loop do
+ begin
+ publish(body)
+
+ sleep interval
+ rescue Exception => e
+ Log.error("Sending registration message failed: #{e}")
+ sleep interval
+ end
+ end
+ end
+ end
+
+ def config
+ Config.instance
+ end
+
+ def msg_filter
+ filter = Util.empty_filter
+ filter["agent"] << "registration"
+ filter
+ end
+
+ def target_collective
+ main_collective = config.main_collective
+
+ collective = config.registration_collective || main_collective
+
+ unless config.collectives.include?(collective)
+ Log.warn("Sending registration to #{main_collective}: #{collective} is not a valid collective")
+ collective = main_collective
+ end
+
+ return collective
+ end
+
+ def interval
+ config.registerinterval
+ end
+
+ def publish(message)
+ unless message
+ Log.debug("Skipping registration due to nil body")
+ else
+ req = Message.new(message, nil, {:type => :request, :agent => "registration", :collective => target_collective, :filter => msg_filter})
+ req.encode!
+
+ Log.debug("Sending registration #{req.requestid} to collective #{req.collective}")
+
+ req.publish
+ end
+ end
+ end
+ end
+end
--- /dev/null
+require 'pp'
+
+module MCollective
+ # Toolset to create a standard interface of client and agent using
+ # an RPC metaphor, standard compliant agents will make it easier
+ # to create generic clients like web interfaces etc
+ module RPC
+ autoload :ActionRunner, "mcollective/rpc/actionrunner"
+ autoload :Agent, "mcollective/rpc/agent"
+ autoload :Audit, "mcollective/rpc/audit"
+ autoload :Client, "mcollective/rpc/client"
+ autoload :Helpers, "mcollective/rpc/helpers"
+ autoload :Progress, "mcollective/rpc/progress"
+ autoload :Reply, "mcollective/rpc/reply"
+ autoload :Request, "mcollective/rpc/request"
+ autoload :Result, "mcollective/rpc/result"
+ autoload :Stats, "mcollective/rpc/stats"
+
+ # Creates a standard options hash, pass in a block to add extra headings etc
+ # see Optionparser
+ def rpcoptions
+ oparser = MCollective::Optionparser.new({:verbose => false, :progress_bar => true}, "filter")
+
+ options = oparser.parse do |parser, options|
+ if block_given?
+ yield(parser, options)
+ end
+
+ Helpers.add_simplerpc_options(parser, options)
+ end
+
+ return options
+ end
+
+ # Wrapper to create clients, supposed to be used as
+ # a mixin:
+ #
+ # include MCollective::RPC
+ #
+ # exim = rpcclient("exim")
+ # printrpc exim.mailq
+ #
+ # or
+ #
+ # rpcclient("exim") do |exim|
+ # printrpc exim.mailq
+ # end
+ #
+ # It will take a few flags:
+ # :configfile => "etc/client.cfg"
+ # :options => options
+ # :exit_on_failure => true
+ #
+ # Options would be a build up options hash from the Optionparser
+ # you can use the rpcoptions helper to create this
+ #
+ # :exit_on_failure is true by default, and causes the application to
+ # exit if there is a failure constructing the RPC client. Set this flag
+ # to false to cause an Exception to be raised instead.
+ def rpcclient(agent, flags = {})
+ configfile = flags[:configfile] || "/etc/mcollective/client.cfg"
+ options = flags[:options] || nil
+
+ if flags.key?(:exit_on_failure)
+ exit_on_failure = flags[:exit_on_failure]
+ else
+ # We exit on failure by default for CLI-friendliness
+ exit_on_failure = true
+ end
+
+ begin
+ if options
+ rpc = Client.new(agent, :configfile => options[:config], :options => options)
+ @options = rpc.options
+ else
+ rpc = Client.new(agent, :configfile => configfile)
+ @options = rpc.options
+ end
+ rescue Exception => e
+ if exit_on_failure
+ puts("Could not create RPC client: #{e}")
+ exit!
+ else
+ raise e
+ end
+ end
+
+ if block_given?
+ yield(rpc)
+ else
+ return rpc
+ end
+ end
+
+ # means for other classes to drop stats into this module
+ # its a bit hacky but needed so that the mixin methods like
+ # printrpcstats can easily get access to it without
+ # users having to pass it around in params.
+ def self.stats(stats)
+ @@stats = stats
+ end
+
+ # means for other classes to drop discovered hosts into this module
+ # its a bit hacky but needed so that the mixin methods like
+ # printrpcstats can easily get access to it without
+ # users having to pass it around in params.
+ def self.discovered(discovered)
+ @@discovered = discovered
+ end
+
+ # Prints stats, requires stats to be saved from elsewhere
+ # using the MCollective::RPC.stats method.
+ #
+ # If you've passed -v on the command line a detailed stat block
+ # will be printed, else just a one liner.
+ #
+ # You can pass flags into it:
+ #
+ # printrpcstats :caption => "Foo", :summarize => true
+ #
+ # This will use "Foo" as the caption to the stats in verbose
+ # mode and print out any aggregate summary information if present
+ def printrpcstats(flags={})
+ return unless @options[:output_format] == :console
+
+ flags = {:summarize => false, :caption => "rpc stats"}.merge(flags)
+
+ verbose = @options[:verbose] rescue verbose = false
+
+ begin
+ stats = @@stats
+ rescue
+ puts("no stats to display")
+ return
+ end
+
+ puts stats.report(flags[:caption], flags[:summarize], verbose)
+ end
+
+ # Prints the result of an RPC call.
+ #
+ # In the default quiet mode - no flattening or verbose - only results
+ # that produce an error will be printed
+ #
+ # To get details of each result run with the -v command line option.
+ def printrpc(result, flags = {})
+ verbose = @options[:verbose] rescue verbose = false
+ verbose = flags[:verbose] || verbose
+ flatten = flags[:flatten] || false
+ format = @options[:output_format]
+ forced_mode = @options[:force_display_mode] || false
+
+ result_text = Helpers.rpcresults(result, {:verbose => verbose, :flatten => flatten, :format => format, :force_display_mode => forced_mode})
+
+ if result.is_a?(Array) && format == :console
+ puts "\n%s\n" % [ result_text ]
+ else
+ # when we get just one result to print dont pad them all with
+ # blank spaces etc, just print the individual result with no
+ # padding
+ puts result_text unless result_text == ""
+ end
+ end
+
+ # Wrapper for MCollective::Util.empty_filter? to make clients less fugly
+ # to write - ticket #18
+ def empty_filter?(options)
+ if options.include?(:filter)
+ Util.empty_filter?(options[:filter])
+ else
+ Util.empty_filter?(options)
+ end
+ end
+
+ def self.const_missing(const_name)
+ super unless const_name == :DDL
+
+ Log.warn("MCollective::RPC::DDL is deprecatd, please use MCollective::DDL instead")
+ MCollective::DDL
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # A helper used by RPC::Agent#implemented_by to delegate an action to
+ # an external script. At present only JSON based serialization is
+ # supported in future ones based on key=val pairs etc will be added
+ #
+ # It serializes the request object into an input file and creates an
+ # empty output file. It then calls the external command reading the
+ # output file at the end.
+ #
+ # any STDERR gets logged at error level and any STDOUT gets logged at
+ # info level.
+ #
+ # It will interpret the exit code from the application the same way
+ # RPC::Reply#fail! and #fail handles their codes creating a consistent
+ # interface, the message part of the fail message will come from STDERR
+ #
+ # Generally externals should just exit with code 1 on failure and print to
+ # STDERR, this is exactly what Perl die() does and translates perfectly
+ # to our model
+ #
+ # It uses the MCollective::Shell wrapper to call the external application
+ class ActionRunner
+ attr_reader :command, :agent, :action, :format, :stdout, :stderr, :request
+
+ def initialize(command, request, format=:json)
+ @agent = request.agent
+ @action = request.action
+ @format = format
+ @request = request
+ @command = path_to_command(command)
+ @stdout = ""
+ @stderr = ""
+ end
+
+ def run
+ unless canrun?(command)
+ Log.warn("Cannot run #{to_s}")
+ raise RPCAborted, "Cannot execute #{to_s}"
+ end
+
+ Log.debug("Running #{to_s}")
+
+ request_file = saverequest(request)
+ reply_file = tempfile("reply")
+ reply_file.close
+
+ runner = shell(command, request_file.path, reply_file.path)
+
+ runner.runcommand
+
+ Log.debug("#{command} exited with #{runner.status.exitstatus}")
+
+ stderr.each_line {|l| Log.error("#{to_s}: #{l.chomp}")} unless stderr.empty?
+ stdout.each_line {|l| Log.info("#{to_s}: #{l.chomp}")} unless stdout.empty?
+
+ {:exitstatus => runner.status.exitstatus,
+ :stdout => runner.stdout,
+ :stderr => runner.stderr,
+ :data => load_results(reply_file.path)}
+ ensure
+ request_file.close! if request_file.respond_to?("close!")
+ reply_file.close! if reply_file.respond_to?("close")
+ end
+
+ def shell(command, infile, outfile)
+ env = {"MCOLLECTIVE_REQUEST_FILE" => infile,
+ "MCOLLECTIVE_REPLY_FILE" => outfile}
+
+ Shell.new("#{command} #{infile} #{outfile}", :cwd => Dir.tmpdir, :stdout => stdout, :stderr => stderr, :environment => env)
+ end
+
+ def load_results(file)
+ Log.debug("Attempting to load results in #{format} format from #{file}")
+
+ data = {}
+
+ if respond_to?("load_#{format}_results")
+ tempdata = send("load_#{format}_results", file)
+
+ tempdata.each_pair do |k,v|
+ data[k.to_sym] = v
+ end
+ end
+
+ data
+ rescue Exception => e
+ {}
+ end
+
+ def load_json_results(file)
+ return {} unless File.readable?(file)
+
+ JSON.load(File.read(file)) || {}
+ rescue JSON::ParserError
+ {}
+ end
+
+ def saverequest(req)
+ Log.debug("Attempting to save request in #{format} format")
+
+ if respond_to?("save_#{format}_request")
+ data = send("save_#{format}_request", req)
+
+ request_file = tempfile("request")
+ request_file.puts data
+ request_file.close
+ end
+
+ request_file
+ end
+
+ def save_json_request(req)
+ req.to_json
+ end
+
+ def canrun?(command)
+ File.executable?(command)
+ end
+
+ def to_s
+ "%s#%s command: %s" % [ agent, action, command ]
+ end
+
+ def tempfile(prefix)
+ Tempfile.new("mcollective_#{prefix}", Dir.tmpdir)
+ end
+
+ def path_to_command(command)
+ unless command[0,1] == File::SEPARATOR
+ Config.instance.libdir.each do |libdir|
+ command_file = File.join(libdir, "agent", agent, command)
+
+ return command_file if File.exist?(command_file)
+ end
+ end
+
+ return command
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # A wrapper around the traditional agent, it takes care of a lot of the tedious setup
+ # you would do for each agent allowing you to just create methods following a naming
+ # standard leaving the heavy lifting up to this clas.
+ #
+ # See http://marionette-collective.org/simplerpc/agents.html
+ #
+ # It only really makes sense to use this with a Simple RPC client on the other end, basic
+ # usage would be:
+ #
+ # module MCollective
+ # module Agent
+ # class Helloworld<RPC::Agent
+ # action "hello" do
+ # reply[:msg] = "Hello #{request[:name]}"
+ # end
+ #
+ # action "foo" do
+ # implemented_by "/some/script.sh"
+ # end
+ # end
+ # end
+ # end
+ #
+ # If you wish to implement the logic for an action using an external script use the
+ # implemented_by method that will cause your script to be run with 2 arguments.
+ #
+ # The first argument is a file containing JSON with the request and the 2nd argument
+ # is where the script should save its output as a JSON hash.
+ #
+ # We also currently have the validation code in here, this will be moved to plugins soon.
+ class Agent
+ include Translatable
+ extend Translatable
+
+ attr_accessor :reply, :request, :agent_name
+ attr_reader :logger, :config, :timeout, :ddl, :meta
+
+ def initialize
+ @agent_name = self.class.to_s.split("::").last.downcase
+
+ load_ddl
+
+ @logger = Log.instance
+ @config = Config.instance
+
+ # if we have a global authorization provider enable it
+ # plugins can still override it per plugin
+ self.class.authorized_by(@config.rpcauthprovider) if @config.rpcauthorization
+
+ startup_hook
+ end
+
+ def load_ddl
+ @ddl = DDL.new(@agent_name, :agent)
+ @meta = @ddl.meta
+ @timeout = @meta[:timeout] || 10
+
+ rescue Exception => e
+ DDL.validation_fail!(:PLMC24, "Failed to load DDL for the '%{agent}' agent, DDLs are required: %{error_class}: %{error}", :error, :agent => @agent_name, :error_class => e.class, :error => e.to_s)
+ end
+
+ def handlemsg(msg, connection)
+ @request = RPC::Request.new(msg, @ddl)
+ @reply = RPC::Reply.new(@request.action, @ddl)
+
+ begin
+ # Incoming requests need to be validated against the DDL thus reusing
+ # all the work users put into creating DDLs and creating a consistant
+ # quality of input validation everywhere with the a simple once off
+ # investment of writing a DDL
+ @request.validate!
+
+ # Calls the authorization plugin if any is defined
+ # if this raises an exception we wil just skip processing this
+ # message
+ authorization_hook(@request) if respond_to?("authorization_hook")
+
+ # Audits the request, currently continues processing the message
+ # we should make this a configurable so that an audit failure means
+ # a message wont be processed by this node depending on config
+ audit_request(@request, connection)
+
+ before_processing_hook(msg, connection)
+
+ if respond_to?("#{@request.action}_action")
+ send("#{@request.action}_action")
+ else
+ log_code(:PLMC36, "Unknown action '%{action}' for agent '%{agent}'", :warn, :action => @request.action, :agent => @request.agent)
+ raise UnknownRPCAction, "Unknown action '#{@request.action}' for agent '#{@request.agent}'"
+ end
+ rescue RPCAborted => e
+ @reply.fail e.to_s, 1
+
+ rescue UnknownRPCAction => e
+ @reply.fail e.to_s, 2
+
+ rescue MissingRPCData => e
+ @reply.fail e.to_s, 3
+
+ rescue InvalidRPCData, DDLValidationError => e
+ @reply.fail e.to_s, 4
+
+ rescue UnknownRPCError => e
+ Log.error("%s#%s failed: %s: %s" % [@agent_name, @request.action, e.class, e.to_s])
+ Log.error(e.backtrace.join("\n\t"))
+ @reply.fail e.to_s, 5
+
+ rescue Exception => e
+ Log.error("%s#%s failed: %s: %s" % [@agent_name, @request.action, e.class, e.to_s])
+ Log.error(e.backtrace.join("\n\t"))
+ @reply.fail e.to_s, 5
+
+ end
+
+ after_processing_hook
+
+ if @request.should_respond?
+ return @reply.to_hash
+ else
+ log_code(:PLMC35, "Client did not request a response, surpressing reply", :debug)
+ return nil
+ end
+ end
+
+ # By default RPC Agents support a toggle in the configuration that
+ # can enable and disable them based on the agent name
+ #
+ # Example an agent called Foo can have:
+ #
+ # plugin.foo.activate_agent = false
+ #
+ # and this will prevent the agent from loading on this particular
+ # machine.
+ #
+ # Agents can use the activate_when helper to override this for example:
+ #
+ # activate_when do
+ # File.exist?("/usr/bin/puppet")
+ # end
+ def self.activate?
+ agent_name = self.to_s.split("::").last.downcase
+
+ log_code(:PLMC37, "Starting default activation checks for the '%{agent}' agent", :debug, :agent => agent_name)
+
+ should_activate = Config.instance.pluginconf["#{agent_name}.activate_agent"]
+
+ if should_activate
+ log_code(:PLMC38, "Found plugin configuration '%{agent}.activate_agent' with value '%{should_activate}'", :debug, :agent => agent_name, :should_activate => should_activate)
+
+ unless should_activate =~ /^1|y|true$/
+ return false
+ end
+ end
+
+ return true
+ end
+
+ # Returns an array of actions this agent support
+ def self.actions
+ public_instance_methods.sort.grep(/_action$/).map do |method|
+ $1 if method =~ /(.+)_action$/
+ end
+ end
+
+ private
+ # Runs a command via the MC::Shell wrapper, options are as per MC::Shell
+ #
+ # The simplest use is:
+ #
+ # out = ""
+ # err = ""
+ # status = run("echo 1", :stdout => out, :stderr => err)
+ #
+ # reply[:out] = out
+ # reply[:error] = err
+ # reply[:exitstatus] = status
+ #
+ # This can be simplified as:
+ #
+ # reply[:exitstatus] = run("echo 1", :stdout => :out, :stderr => :error)
+ #
+ # You can set a command specific environment and cwd:
+ #
+ # run("echo 1", :cwd => "/tmp", :environment => {"FOO" => "BAR"})
+ #
+ # This will run 'echo 1' from /tmp with FOO=BAR in addition to a setting forcing
+ # LC_ALL = C. To prevent LC_ALL from being set either set it specifically or:
+ #
+ # run("echo 1", :cwd => "/tmp", :environment => nil)
+ #
+ # Exceptions here will be handled by the usual agent exception handler or any
+ # specific one you create, if you dont it will just fall through and be sent
+ # to the client.
+ #
+ # If the shell handler fails to return a Process::Status instance for exit
+ # status this method will return -1 as the exit status
+ def run(command, options={})
+ shellopts = {}
+
+ # force stderr and stdout to be strings as the library
+ # will append data to them if given using the << method.
+ #
+ # if the data pased to :stderr or :stdin is a Symbol
+ # add that into the reply hash with that Symbol
+ [:stderr, :stdout].each do |k|
+ if options.include?(k)
+ if options[k].is_a?(Symbol)
+ reply[ options[k] ] = ""
+ shellopts[k] = reply[ options[k] ]
+ else
+ if options[k].respond_to?("<<")
+ shellopts[k] = options[k]
+ else
+ reply.fail! "#{k} should support << while calling run(#{command})"
+ end
+ end
+ end
+ end
+
+ [:stdin, :cwd, :environment].each do |k|
+ if options.include?(k)
+ shellopts[k] = options[k]
+ end
+ end
+
+ shell = Shell.new(command, shellopts)
+
+ shell.runcommand
+
+ if options[:chomp]
+ shellopts[:stdout].chomp! if shellopts[:stdout].is_a?(String)
+ shellopts[:stderr].chomp! if shellopts[:stderr].is_a?(String)
+ end
+
+ shell.status.exitstatus rescue -1
+ end
+
+ # Registers meta data for the introspection hash
+ def self.metadata(data)
+ agent = File.basename(caller.first).split(":").first
+
+ log_code(:PLMC34, "setting meta data in agents have been deprecated, DDL files are now being used for this information. Please update the '%{agent}' agent", :warn, :agent => agent)
+ end
+
+ # Creates the needed activate? class in a manner similar to the other
+ # helpers like action, authorized_by etc
+ #
+ # activate_when do
+ # File.exist?("/usr/bin/puppet")
+ # end
+ def self.activate_when(&block)
+ (class << self; self; end).instance_eval do
+ define_method("activate?", &block)
+ end
+ end
+
+ # Creates a new action with the block passed and sets some defaults
+ #
+ # action "status" do
+ # # logic here to restart service
+ # end
+ def self.action(name, &block)
+ raise "Need to pass a body for the action" unless block_given?
+
+ self.module_eval { define_method("#{name}_action", &block) }
+ end
+
+ # Helper that creates a method on the class that will call your authorization
+ # plugin. If your plugin raises an exception that will abort the request
+ def self.authorized_by(plugin)
+ plugin = plugin.to_s.capitalize
+
+ # turns foo_bar into FooBar
+ plugin = plugin.to_s.split("_").map {|v| v.capitalize}.join
+ pluginname = "MCollective::Util::#{plugin}"
+
+ PluginManager.loadclass(pluginname) unless MCollective::Util.constants.include?(plugin)
+
+ class_eval("
+ def authorization_hook(request)
+ #{pluginname}.authorize(request)
+ end
+ ")
+ end
+
+ # Validates a data member, if validation is a regex then it will try to match it
+ # else it supports testing object types only:
+ #
+ # validate :msg, String
+ # validate :msg, /^[\w\s]+$/
+ #
+ # There are also some special helper validators:
+ #
+ # validate :command, :shellsafe
+ # validate :command, :ipv6address
+ # validate :command, :ipv4address
+ # validate :command, :boolean
+ # validate :command, ["start", "stop"]
+ #
+ # It will raise appropriate exceptions that the RPC system understand
+ def validate(key, validation)
+ raise MissingRPCData, "please supply a #{key} argument" unless @request.include?(key)
+
+ Validator.validate(@request[key], validation)
+ rescue ValidatorError => e
+ raise InvalidRPCData, "Input %s did not pass validation: %s" % [ key, e.message ]
+ end
+
+ # convenience wrapper around Util#shellescape
+ def shellescape(str)
+ Util.shellescape(str)
+ end
+
+ # handles external actions
+ def implemented_by(command, type=:json)
+ runner = ActionRunner.new(command, request, type)
+
+ res = runner.run
+
+ reply.fail! "Did not receive data from #{command}" unless res.include?(:data)
+ reply.fail! "Reply data from #{command} is not a Hash" unless res[:data].is_a?(Hash)
+
+ reply.data.merge!(res[:data])
+
+ if res[:exitstatus] > 0
+ reply.fail "Failed to run #{command}: #{res[:stderr]}", res[:exitstatus]
+ end
+ rescue Exception => e
+ Log.warn("Unhandled #{e.class} exception during #{request.agent}##{request.action}: #{e}")
+ reply.fail! "Unexpected failure calling #{command}: #{e.class}: #{e}"
+ end
+
+ # Called at the end of the RPC::Agent standard initialize method
+ # use this to adjust meta parameters, timeouts and any setup you
+ # need to do.
+ #
+ # This will not be called right when the daemon starts up, we use
+ # lazy loading and initialization so it will only be called the first
+ # time a request for this agent arrives.
+ def startup_hook
+ end
+
+ # Called just after a message was received from the middleware before
+ # it gets passed to the handlers. @request and @reply will already be
+ # set, the msg passed is the message as received from the normal
+ # mcollective runner and the connection is the actual connector.
+ def before_processing_hook(msg, connection)
+ end
+
+ # Called at the end of processing just before the response gets sent
+ # to the middleware.
+ #
+ # This gets run outside of the main exception handling block of the agent
+ # so you should handle any exceptions you could raise yourself. The reason
+ # it is outside of the block is so you'll have access to even status codes
+ # set by the exception handlers. If you do raise an exception it will just
+ # be passed onto the runner and processing will fail.
+ def after_processing_hook
+ end
+
+ # Gets called right after a request was received and calls audit plugins
+ #
+ # Agents can disable auditing by just overriding this method with a noop one
+ # this might be useful for agents that gets a lot of requests or simply if you
+ # do not care for the auditing in a specific agent.
+ def audit_request(msg, connection)
+ PluginManager["rpcaudit_plugin"].audit_request(msg, connection) if @config.rpcaudit
+ rescue Exception => e
+ logexception(:PLMC39, "Audit failed with an error, processing the request will continue.", :warn, e)
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # Auditing of requests is done only for SimpleRPC requests, you provide
+ # a plugin in the MCollective::Audit::* namespace which the SimpleRPC
+ # framework calls for each message
+ #
+ # We provide a simple one that logs to a logfile in the class
+ # MCollective::Audit::Logfile you can create your own:
+ #
+ # Create a class in plugins/mcollective/audit/<yourplugin>.rb
+ #
+ # You must inherit from MCollective::RPC::Audit which will take
+ # care of registering you with the plugin system.
+ #
+ # Your plugin must provide audit_request(request, connection)
+ # the request parameter will be an instance of MCollective::RPC::Request
+ #
+ # To enable auditing you should set:
+ #
+ # rpcaudit = 1
+ # rpcauditprovider = Logfile
+ #
+ # in the config file this will enable logging using the
+ # MCollective::Audit::Logile class
+ #
+ # The Audit class acts as a base for audit plugins and takes care of registering them
+ # with the plugin manager
+ class Audit
+ def self.inherited(klass)
+ PluginManager << {:type => "rpcaudit_plugin", :class => klass.to_s}
+ end
+
+ def audit_request(request, connection)
+ @log.error("audit_request is not implimented in #{this.class}")
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # The main component of the Simple RPC client system, this wraps around MCollective::Client
+ # and just brings in a lot of convention and standard approached.
+ class Client
+ attr_accessor :timeout, :verbose, :filter, :config, :progress, :ttl, :reply_to
+ attr_reader :client, :stats, :ddl, :agent, :limit_targets, :limit_method, :output_format, :batch_size, :batch_sleep_time, :batch_mode
+ attr_reader :discovery_options, :discovery_method, :limit_seed
+
+ @@initial_options = nil
+
+ # Creates a stub for a remote agent, you can pass in an options array in the flags
+ # which will then be used else it will just create a default options array with
+ # filtering enabled based on the standard command line use.
+ #
+ # rpc = RPC::Client.new("rpctest", :configfile => "client.cfg", :options => options)
+ #
+ # You typically would not call this directly you'd use MCollective::RPC#rpcclient instead
+ # which is a wrapper around this that can be used as a Mixin
+ def initialize(agent, flags = {})
+ if flags.include?(:options)
+ initial_options = flags[:options]
+
+ elsif @@initial_options
+ initial_options = Marshal.load(@@initial_options)
+
+ else
+ oparser = MCollective::Optionparser.new({:verbose => false, :progress_bar => true, :mcollective_limit_targets => false, :batch_size => nil, :batch_sleep_time => 1}, "filter")
+
+ initial_options = oparser.parse do |parser, opts|
+ if block_given?
+ yield(parser, opts)
+ end
+
+ Helpers.add_simplerpc_options(parser, opts)
+ end
+
+ @@initial_options = Marshal.dump(initial_options)
+ end
+
+ @initial_options = initial_options
+
+ @config = initial_options[:config]
+ @client = MCollective::Client.new(@config)
+ @client.options = initial_options
+
+ @stats = Stats.new
+ @agent = agent
+ @timeout = initial_options[:timeout] || 5
+ @verbose = initial_options[:verbose]
+ @filter = initial_options[:filter] || Util.empty_filter
+ @discovered_agents = nil
+ @progress = initial_options[:progress_bar]
+ @limit_targets = initial_options[:mcollective_limit_targets]
+ @limit_method = Config.instance.rpclimitmethod
+ @limit_seed = initial_options[:limit_seed] || nil
+ @output_format = initial_options[:output_format] || :console
+ @force_direct_request = false
+ @reply_to = initial_options[:reply_to]
+ @discovery_method = initial_options[:discovery_method] || Config.instance.default_discovery_method
+ @discovery_options = initial_options[:discovery_options] || []
+ @force_display_mode = initial_options[:force_display_mode] || false
+
+ @batch_size = Integer(initial_options[:batch_size] || 0)
+ @batch_sleep_time = Float(initial_options[:batch_sleep_time] || 1)
+ @batch_mode = @batch_size > 0
+
+ agent_filter agent
+
+ @discovery_timeout = @initial_options.fetch(:disctimeout, nil)
+
+ @collective = @client.collective
+ @ttl = initial_options[:ttl] || Config.instance.ttl
+
+ # if we can find a DDL for the service override
+ # the timeout of the client so we always magically
+ # wait appropriate amounts of time.
+ #
+ # We add the discovery timeout to the ddl supplied
+ # timeout as the discovery timeout tends to be tuned
+ # for local network conditions and fact source speed
+ # which would other wise not be accounted for and
+ # some results might get missed.
+ #
+ # We do this only if the timeout is the default 5
+ # seconds, so that users cli overrides will still
+ # get applied
+ #
+ # DDLs are required, failure to find a DDL is fatal
+ @ddl = DDL.new(agent)
+ @stats.ddl = @ddl
+ @timeout = @ddl.meta[:timeout] + discovery_timeout if @timeout == 5
+
+ # allows stderr and stdout to be overridden for testing
+ # but also for web apps that might not want a bunch of stuff
+ # generated to actual file handles
+ if initial_options[:stderr]
+ @stderr = initial_options[:stderr]
+ else
+ @stderr = STDERR
+ @stderr.sync = true
+ end
+
+ if initial_options[:stdout]
+ @stdout = initial_options[:stdout]
+ else
+ @stdout = STDOUT
+ @stdout.sync = true
+ end
+ end
+
+ # Disconnects cleanly from the middleware
+ def disconnect
+ @client.disconnect
+ end
+
+ # Returns help for an agent if a DDL was found
+ def help(template)
+ @ddl.help(template)
+ end
+
+ # Creates a suitable request hash for the SimpleRPC agent.
+ #
+ # You'd use this if you ever wanted to take care of sending
+ # requests on your own - perhaps via Client#sendreq if you
+ # didn't care for responses.
+ #
+ # In that case you can just do:
+ #
+ # msg = your_rpc.new_request("some_action", :foo => :bar)
+ # filter = your_rpc.filter
+ #
+ # your_rpc.client.sendreq(msg, msg[:agent], filter)
+ #
+ # This will send a SimpleRPC request to the action some_action
+ # with arguments :foo = :bar, it will return immediately and
+ # you will have no indication at all if the request was receieved or not
+ #
+ # Clearly the use of this technique should be limited and done only
+ # if your code requires such a thing
+ def new_request(action, data)
+ callerid = PluginManager["security_plugin"].callerid
+
+ raise 'callerid received from security plugin is not valid' unless PluginManager["security_plugin"].valid_callerid?(callerid)
+
+ {:agent => @agent,
+ :action => action,
+ :caller => callerid,
+ :data => data}
+ end
+
+ # For the provided arguments and action the input arguments get
+ # modified by supplying any defaults provided in the DDL for arguments
+ # that were not supplied in the request
+ #
+ # We then pass the modified arguments to the DDL for validation
+ def validate_request(action, args)
+ raise "No DDL found for agent %s cannot validate inputs" % @agent unless @ddl
+
+ @ddl.set_default_input_arguments(action, args)
+ @ddl.validate_rpc_request(action, args)
+ end
+
+ # Magic handler to invoke remote methods
+ #
+ # Once the stub is created using the constructor or the RPC#rpcclient helper you can
+ # call remote actions easily:
+ #
+ # ret = rpc.echo(:msg => "hello world")
+ #
+ # This will call the 'echo' action of the 'rpctest' agent and return the result as an array,
+ # the array will be a simplified result set from the usual full MCollective::Client#req with
+ # additional error codes and error text:
+ #
+ # {
+ # :sender => "remote.box.com",
+ # :statuscode => 0,
+ # :statusmsg => "OK",
+ # :data => "hello world"
+ # }
+ #
+ # If :statuscode is 0 then everything went find, if it's 1 then you supplied the correct arguments etc
+ # but the request could not be completed, you'll find a human parsable reason in :statusmsg then.
+ #
+ # Codes 2 to 5 maps directly to UnknownRPCAction, MissingRPCData, InvalidRPCData and UnknownRPCError
+ # see below for a description of those, in each case :statusmsg would be the reason for failure.
+ #
+ # To get access to the full result of the MCollective::Client#req calls you can pass in a block:
+ #
+ # rpc.echo(:msg => "hello world") do |resp|
+ # pp resp
+ # end
+ #
+ # In this case resp will the result from MCollective::Client#req. Instead of returning simple
+ # text and codes as above you'll also need to handle the following exceptions:
+ #
+ # UnknownRPCAction - There is no matching action on the agent
+ # MissingRPCData - You did not supply all the needed parameters for the action
+ # InvalidRPCData - The data you did supply did not pass validation
+ # UnknownRPCError - Some other error prevented the agent from running
+ #
+ # During calls a progress indicator will be shown of how many results we've received against
+ # how many nodes were discovered, you can disable this by setting progress to false:
+ #
+ # rpc.progress = false
+ #
+ # This supports a 2nd mode where it will send the SimpleRPC request and never handle the
+ # responses. It's a bit like UDP, it sends the request with the filter attached and you
+ # only get back the requestid, you have no indication about results.
+ #
+ # You can invoke this using:
+ #
+ # puts rpc.echo(:process_results => false)
+ #
+ # This will output just the request id.
+ #
+ # Batched processing is supported:
+ #
+ # printrpc rpc.ping(:batch_size => 5)
+ #
+ # This will do everything exactly as normal but communicate to only 5
+ # agents at a time
+ def method_missing(method_name, *args, &block)
+ # set args to an empty hash if nothings given
+ args = args[0]
+ args = {} if args.nil?
+
+ action = method_name.to_s
+
+ @stats.reset
+
+ validate_request(action, args)
+
+ # if a global batch size is set just use that else set it
+ # in the case that it was passed as an argument
+ batch_mode = args.include?(:batch_size) || @batch_mode
+ batch_size = args.delete(:batch_size) || @batch_size
+ batch_sleep_time = args.delete(:batch_sleep_time) || @batch_sleep_time
+
+ # if we were given a batch_size argument thats 0 and batch_mode was
+ # determined to be on via global options etc this will allow a batch_size
+ # of 0 to disable or batch_mode for this call only
+ batch_mode = (batch_mode && Integer(batch_size) > 0)
+
+ # Handle single target requests by doing discovery and picking
+ # a random node. Then do a custom request specifying a filter
+ # that will only match the one node.
+ if @limit_targets
+ target_nodes = pick_nodes_from_discovered(@limit_targets)
+ Log.debug("Picked #{target_nodes.join(',')} as limited target(s)")
+
+ custom_request(action, args, target_nodes, {"identity" => /^(#{target_nodes.join('|')})$/}, &block)
+ elsif batch_mode
+ call_agent_batched(action, args, options, batch_size, batch_sleep_time, &block)
+ else
+ call_agent(action, args, options, :auto, &block)
+ end
+ end
+
+ # Constructs custom requests with custom filters and discovery data
+ # the idea is that this would be used in web applications where you
+ # might be using a cached copy of data provided by a registration agent
+ # to figure out on your own what nodes will be responding and what your
+ # filter would be.
+ #
+ # This will help you essentially short circuit the traditional cycle of:
+ #
+ # mc discover / call / wait for discovered nodes
+ #
+ # by doing discovery however you like, contructing a filter and a list of
+ # nodes you expect responses from.
+ #
+ # Other than that it will work exactly like a normal call, blocks will behave
+ # the same way, stats will be handled the same way etcetc
+ #
+ # If you just wanted to contact one machine for example with a client that
+ # already has other filter options setup you can do:
+ #
+ # puppet.custom_request("runonce", {}, ["your.box.com"], {:identity => "your.box.com"})
+ #
+ # This will do runonce action on just 'your.box.com', no discovery will be
+ # done and after receiving just one response it will stop waiting for responses
+ #
+ # If direct_addressing is enabled in the config file you can provide an empty
+ # hash as a filter, this will force that request to be a directly addressed
+ # request which technically does not need filters. If you try to use this
+ # mode with direct addressing disabled an exception will be raise
+ def custom_request(action, args, expected_agents, filter = {}, &block)
+ validate_request(action, args)
+
+ if filter == {} && !Config.instance.direct_addressing
+ raise "Attempted to do a filterless custom_request without direct_addressing enabled, preventing unexpected call to all nodes"
+ end
+
+ @stats.reset
+
+ custom_filter = Util.empty_filter
+ custom_options = options.clone
+
+ # merge the supplied filter with the standard empty one
+ # we could just use the merge method but I want to be sure
+ # we dont merge in stuff that isnt actually valid
+ ["identity", "fact", "agent", "cf_class", "compound"].each do |ftype|
+ if filter.include?(ftype)
+ custom_filter[ftype] = [filter[ftype], custom_filter[ftype]].flatten
+ end
+ end
+
+ # ensure that all filters at least restrict the call to the agent we're a proxy for
+ custom_filter["agent"] << @agent unless custom_filter["agent"].include?(@agent)
+ custom_options[:filter] = custom_filter
+
+ # Fake out the stats discovery would have put there
+ @stats.discovered_agents([expected_agents].flatten)
+
+ # Handle fire and forget requests
+ #
+ # If a specific reply-to was set then from the client perspective this should
+ # be a fire and forget request too since no response will ever reach us - it
+ # will go to the reply-to destination
+ if args[:process_results] == false || @reply_to
+ return fire_and_forget_request(action, args, custom_filter)
+ end
+
+ # Now do a call pretty much exactly like in method_missing except with our own
+ # options and discovery magic
+ if block_given?
+ call_agent(action, args, custom_options, [expected_agents].flatten) do |r|
+ block.call(r)
+ end
+ else
+ call_agent(action, args, custom_options, [expected_agents].flatten)
+ end
+ end
+
+ def discovery_timeout
+ return @discovery_timeout if @discovery_timeout
+ return @client.discoverer.ddl.meta[:timeout]
+ end
+
+ def discovery_timeout=(timeout)
+ @discovery_timeout = Float(timeout)
+
+ # we calculate the overall timeout from the DDL of the agent and
+ # the supplied discovery timeout unless someone specifically
+ # specifies a timeout to the constructor
+ #
+ # But if we also then specifically set a discovery_timeout on the
+ # agent that has to override the supplied timeout so we then
+ # calculate a correct timeout based on DDL timeout and the
+ # supplied discovery timeout
+ @timeout = @ddl.meta[:timeout] + discovery_timeout
+ end
+
+ # Sets the discovery method. If we change the method there are a
+ # number of steps to take:
+ #
+ # - set the new method
+ # - if discovery options were provided, re-set those to initially
+ # provided ones else clear them as they might now apply to a
+ # different provider
+ # - update the client options so it knows there is a new discovery
+ # method in force
+ # - reset discovery data forcing a discover on the next request
+ #
+ # The remaining item is the discovery timeout, we leave that as is
+ # since that is the user supplied timeout either via initial options
+ # or via specifically setting it on the client.
+ def discovery_method=(method)
+ @discovery_method = method
+
+ if @initial_options[:discovery_options]
+ @discovery_options = @initial_options[:discovery_options]
+ else
+ @discovery_options.clear
+ end
+
+ @client.options = options
+
+ reset
+ end
+
+ def discovery_options=(options)
+ @discovery_options = [options].flatten
+ reset
+ end
+
+ # Sets the class filter
+ def class_filter(klass)
+ @filter["cf_class"] << klass
+ @filter["cf_class"].compact!
+ reset
+ end
+
+ # Sets the fact filter
+ def fact_filter(fact, value=nil, operator="=")
+ return if fact.nil?
+ return if fact == false
+
+ if value.nil?
+ parsed = Util.parse_fact_string(fact)
+ @filter["fact"] << parsed unless parsed == false
+ else
+ parsed = Util.parse_fact_string("#{fact}#{operator}#{value}")
+ @filter["fact"] << parsed unless parsed == false
+ end
+
+ @filter["fact"].compact!
+ reset
+ end
+
+ # Sets the agent filter
+ def agent_filter(agent)
+ @filter["agent"] << agent
+ @filter["agent"].compact!
+ reset
+ end
+
+ # Sets the identity filter
+ def identity_filter(identity)
+ @filter["identity"] << identity
+ @filter["identity"].compact!
+ reset
+ end
+
+ # Set a compound filter
+ def compound_filter(filter)
+ @filter["compound"] << Matcher.create_compound_callstack(filter)
+ reset
+ end
+
+ # Resets various internal parts of the class, most importantly it clears
+ # out the cached discovery
+ def reset
+ @discovered_agents = nil
+ end
+
+ # Reet the filter to an empty one
+ def reset_filter
+ @filter = Util.empty_filter
+ agent_filter @agent
+ end
+
+ # Does discovery based on the filters set, if a discovery was
+ # previously done return that else do a new discovery.
+ #
+ # Alternatively if identity filters are given and none of them are
+ # regular expressions then just use the provided data as discovered
+ # data, avoiding discovery
+ #
+ # Discovery can be forced if direct_addressing is enabled by passing
+ # in an array of nodes with :nodes or JSON data like those produced
+ # by mcollective RPC JSON output using :json
+ #
+ # Will show a message indicating its doing discovery if running
+ # verbose or if the :verbose flag is passed in.
+ #
+ # Use reset to force a new discovery
+ def discover(flags={})
+ flags.keys.each do |key|
+ raise "Unknown option #{key} passed to discover" unless [:verbose, :hosts, :nodes, :json].include?(key)
+ end
+
+ flags.include?(:verbose) ? verbose = flags[:verbose] : verbose = @verbose
+
+ verbose = false unless @output_format == :console
+
+ # flags[:nodes] and flags[:hosts] are the same thing, we should never have
+ # allowed :hosts as that was inconsistent with the established terminology
+ flags[:nodes] = flags.delete(:hosts) if flags.include?(:hosts)
+
+ reset if flags[:nodes] || flags[:json]
+
+ unless @discovered_agents
+ # if either hosts or JSON is supplied try to figure out discovery data from there
+ # if direct_addressing is not enabled this is a critical error as the user might
+ # not have supplied filters so raise an exception
+ if flags[:nodes] || flags[:json]
+ raise "Can only supply discovery data if direct_addressing is enabled" unless Config.instance.direct_addressing
+
+ hosts = []
+
+ if flags[:nodes]
+ hosts = Helpers.extract_hosts_from_array(flags[:nodes])
+ elsif flags[:json]
+ hosts = Helpers.extract_hosts_from_json(flags[:json])
+ end
+
+ raise "Could not find any hosts in discovery data provided" if hosts.empty?
+
+ @discovered_agents = hosts
+ @force_direct_request = true
+
+ # if an identity filter is supplied and it is all strings no regex we can use that
+ # as discovery data, technically the identity filter is then redundant if we are
+ # in direct addressing mode and we could empty it out but this use case should
+ # only really be for a few -I's on the CLI
+ #
+ # For safety we leave the filter in place for now, that way we can support this
+ # enhancement also in broadcast mode.
+ #
+ # This is only needed for the 'mc' discovery method, other methods might change
+ # the concept of identity to mean something else so we should pass the full
+ # identity filter to them
+ elsif options[:filter]["identity"].size > 0 && @discovery_method == "mc"
+ regex_filters = options[:filter]["identity"].select{|i| i.match("^\/")}.size
+
+ if regex_filters == 0
+ @discovered_agents = options[:filter]["identity"].clone
+ @force_direct_request = true if Config.instance.direct_addressing
+ end
+ end
+ end
+
+ # All else fails we do it the hard way using a traditional broadcast
+ unless @discovered_agents
+ @stats.time_discovery :start
+
+ @client.options = options
+
+ # if compound filters are used the only real option is to use the mc
+ # discovery plugin since its the only capable of using data queries etc
+ # and we do not want to degrade that experience just to allow compounds
+ # on other discovery plugins the UX would be too bad raising complex sets
+ # of errors etc.
+ @client.discoverer.force_discovery_method_by_filter(options[:filter])
+
+ if verbose
+ actual_timeout = @client.discoverer.discovery_timeout(discovery_timeout, options[:filter])
+
+ if actual_timeout > 0
+ @stderr.print("Discovering hosts using the %s method for %d second(s) .... " % [@client.discoverer.discovery_method, actual_timeout])
+ else
+ @stderr.print("Discovering hosts using the %s method .... " % [@client.discoverer.discovery_method])
+ end
+ end
+
+ # if the requested limit is a pure number and not a percent
+ # and if we're configured to use the first found hosts as the
+ # limit method then pass in the limit thus minimizing the amount
+ # of work we do in the discover phase and speeding it up significantly
+ if @limit_method == :first and @limit_targets.is_a?(Fixnum)
+ @discovered_agents = @client.discover(@filter, discovery_timeout, @limit_targets)
+ else
+ @discovered_agents = @client.discover(@filter, discovery_timeout)
+ end
+
+ @stderr.puts(@discovered_agents.size) if verbose
+
+ @force_direct_request = @client.discoverer.force_direct_mode?
+
+ @stats.time_discovery :end
+ end
+
+ @stats.discovered_agents(@discovered_agents)
+ RPC.discovered(@discovered_agents)
+
+ @discovered_agents
+ end
+
+ # Provides a normal options hash like you would get from
+ # Optionparser
+ def options
+ {:disctimeout => discovery_timeout,
+ :timeout => @timeout,
+ :verbose => @verbose,
+ :filter => @filter,
+ :collective => @collective,
+ :output_format => @output_format,
+ :ttl => @ttl,
+ :discovery_method => @discovery_method,
+ :discovery_options => @discovery_options,
+ :force_display_mode => @force_display_mode,
+ :config => @config}
+ end
+
+ # Sets the collective we are communicating with
+ def collective=(c)
+ raise "Unknown collective #{c}" unless Config.instance.collectives.include?(c)
+
+ @collective = c
+ @client.options = options
+ reset
+ end
+
+ # Sets and sanity checks the limit_targets variable
+ # used to restrict how many nodes we'll target
+ def limit_targets=(limit)
+ if limit.is_a?(String)
+ raise "Invalid limit specified: #{limit} valid limits are /^\d+%*$/" unless limit =~ /^\d+%*$/
+
+ begin
+ @limit_targets = Integer(limit)
+ rescue
+ @limit_targets = limit
+ end
+ else
+ @limit_targets = Integer(limit)
+ end
+ end
+
+ # Sets and sanity check the limit_method variable
+ # used to determine how to limit targets if limit_targets is set
+ def limit_method=(method)
+ method = method.to_sym unless method.is_a?(Symbol)
+
+ raise "Unknown limit method #{method} must be :random or :first" unless [:random, :first].include?(method)
+
+ @limit_method = method
+ end
+
+ # Sets the batch size, if the size is set to 0 that will disable batch mode
+ def batch_size=(limit)
+ raise "Can only set batch size if direct addressing is supported" unless Config.instance.direct_addressing
+
+ @batch_size = Integer(limit)
+ @batch_mode = @batch_size > 0
+ end
+
+ def batch_sleep_time=(time)
+ raise "Can only set batch sleep time if direct addressing is supported" unless Config.instance.direct_addressing
+
+ @batch_sleep_time = Float(time)
+ end
+
+ # Pick a number of nodes from the discovered nodes
+ #
+ # The count should be a string that can be either
+ # just a number or a percentage like 10%
+ #
+ # It will select nodes from the discovered list based
+ # on the rpclimitmethod configuration option which can
+ # be either :first or anything else
+ #
+ # - :first would be a simple way to do a distance based
+ # selection
+ # - anything else will just pick one at random
+ # - if random chosen, and batch-seed set, then set srand
+ # for the generator, and reset afterwards
+ def pick_nodes_from_discovered(count)
+ if count =~ /%$/
+ pct = Integer((discover.size * (count.to_f / 100)))
+ pct == 0 ? count = 1 : count = pct
+ else
+ count = Integer(count)
+ end
+
+ return discover if discover.size <= count
+
+ result = []
+
+ if @limit_method == :first
+ return discover[0, count]
+ else
+ # we delete from the discovered list because we want
+ # to be sure there is no chance that the same node will
+ # be randomly picked twice. So we have to clone the
+ # discovered list else this method will only ever work
+ # once per discovery cycle and not actually return the
+ # right nodes.
+ haystack = discover.clone
+
+ if @limit_seed
+ haystack.sort!
+ srand(@limit_seed)
+ end
+
+ count.times do
+ rnd = rand(haystack.size)
+ result << haystack.delete_at(rnd)
+ end
+
+ # Reset random number generator to fresh seed
+ # As our seed from options is most likely short
+ srand if @limit_seed
+ end
+
+ [result].flatten
+ end
+
+ def load_aggregate_functions(action, ddl)
+ return nil unless ddl
+ return nil unless ddl.action_interface(action).keys.include?(:aggregate)
+
+ return Aggregate.new(ddl.action_interface(action))
+
+ rescue => e
+ Log.error("Failed to load aggregate functions, calculating summaries disabled: %s: %s (%s)" % [e.backtrace.first, e.to_s, e.class])
+ return nil
+ end
+
+ def aggregate_reply(reply, aggregate)
+ return nil unless aggregate
+
+ aggregate.call_functions(reply)
+ return aggregate
+ rescue Exception => e
+ Log.error("Failed to calculate aggregate summaries for reply from %s, calculating summaries disabled: %s: %s (%s)" % [reply[:senderid], e.backtrace.first, e.to_s, e.class])
+ return nil
+ end
+
+ def rpc_result_from_reply(agent, action, reply)
+ Result.new(agent, action, {:sender => reply[:senderid], :statuscode => reply[:body][:statuscode],
+ :statusmsg => reply[:body][:statusmsg], :data => reply[:body][:data]})
+ end
+
+ # for requests that do not care for results just
+ # return the request id and don't do any of the
+ # response processing.
+ #
+ # We send the :process_results flag with to the
+ # nodes so they can make decisions based on that.
+ #
+ # Should only be called via method_missing
+ def fire_and_forget_request(action, args, filter=nil)
+ validate_request(action, args)
+
+ req = new_request(action.to_s, args)
+
+ filter = options[:filter] unless filter
+
+ message = Message.new(req, nil, {:agent => @agent, :type => :request, :collective => @collective, :filter => filter, :options => options})
+ message.reply_to = @reply_to if @reply_to
+
+ return @client.sendreq(message, nil)
+ end
+
+ # Calls an agent in a way very similar to call_agent but it supports batching
+ # the queries to the network.
+ #
+ # The result sets, stats, block handling etc is all exactly like you would expect
+ # from normal call_agent.
+ #
+ # This is used by method_missing and works only with direct addressing mode
+ def call_agent_batched(action, args, opts, batch_size, sleep_time, &block)
+ raise "Batched requests requires direct addressing" unless Config.instance.direct_addressing
+ raise "Cannot bypass result processing for batched requests" if args[:process_results] == false
+
+ batch_size = Integer(batch_size)
+ sleep_time = Float(sleep_time)
+
+ Log.debug("Calling #{agent}##{action} in batches of #{batch_size} with sleep time of #{sleep_time}")
+
+ @force_direct_request = true
+
+ discovered = discover
+ results = []
+ respcount = 0
+
+ if discovered.size > 0
+ req = new_request(action.to_s, args)
+
+ aggregate = load_aggregate_functions(action, @ddl)
+
+ if @progress && !block_given?
+ twirl = Progress.new
+ @stdout.puts
+ @stdout.print twirl.twirl(respcount, discovered.size)
+ end
+
+ @stats.requestid = nil
+
+ discovered.in_groups_of(batch_size) do |hosts, last_batch|
+ message = Message.new(req, nil, {:agent => @agent, :type => :direct_request, :collective => @collective, :filter => opts[:filter], :options => opts})
+
+ # first time round we let the Message object create a request id
+ # we then re-use it for future requests to keep auditing sane etc
+ @stats.requestid = message.create_reqid unless @stats.requestid
+ message.requestid = @stats.requestid
+
+ message.discovered_hosts = hosts.clone.compact
+
+ @client.req(message) do |resp|
+ respcount += 1
+
+ if block_given?
+ aggregate = process_results_with_block(action, resp, block, aggregate)
+ else
+ @stdout.print twirl.twirl(respcount, discovered.size) if @progress
+
+ result, aggregate = process_results_without_block(resp, action, aggregate)
+
+ results << result
+ end
+ end
+
+ @stats.noresponsefrom.concat @client.stats[:noresponsefrom]
+ @stats.responses += @client.stats[:responses]
+ @stats.blocktime += @client.stats[:blocktime] + sleep_time
+ @stats.totaltime += @client.stats[:totaltime]
+ @stats.discoverytime += @client.stats[:discoverytime]
+
+ sleep sleep_time unless last_batch
+ end
+
+ @stats.aggregate_summary = aggregate.summarize if aggregate
+ @stats.aggregate_failures = aggregate.failed if aggregate
+ else
+ @stderr.print("\nNo request sent, we did not discover any nodes.")
+ end
+
+ @stats.finish_request
+
+ RPC.stats(@stats)
+
+ @stdout.print("\n") if @progress
+
+ if block_given?
+ return stats
+ else
+ return [results].flatten
+ end
+ end
+
+ # Handles traditional calls to the remote agents with full stats
+ # blocks, non blocks and everything else supported.
+ #
+ # Other methods of calling the nodes can reuse this code by
+ # for example specifying custom options and discovery data
+ def call_agent(action, args, opts, disc=:auto, &block)
+ # Handle fire and forget requests and make sure
+ # the :process_results value is set appropriately
+ #
+ # specific reply-to requests should be treated like
+ # fire and forget since the client will never get
+ # the responses
+ if args[:process_results] == false || @reply_to
+ return fire_and_forget_request(action, args)
+ else
+ args[:process_results] = true
+ end
+
+ # Do discovery when no specific discovery array is given
+ #
+ # If an array is given set the force_direct_request hint that
+ # will tell the message object to be a direct request one
+ if disc == :auto
+ discovered = discover
+ else
+ @force_direct_request = true if Config.instance.direct_addressing
+ discovered = disc
+ end
+
+ req = new_request(action.to_s, args)
+
+ message = Message.new(req, nil, {:agent => @agent, :type => :request, :collective => @collective, :filter => opts[:filter], :options => opts})
+ message.discovered_hosts = discovered.clone
+
+ results = []
+ respcount = 0
+
+ if discovered.size > 0
+ message.type = :direct_request if @force_direct_request
+
+ if @progress && !block_given?
+ twirl = Progress.new
+ @stdout.puts
+ @stdout.print twirl.twirl(respcount, discovered.size)
+ end
+
+ aggregate = load_aggregate_functions(action, @ddl)
+
+ @client.req(message) do |resp|
+ respcount += 1
+
+ if block_given?
+ aggregate = process_results_with_block(action, resp, block, aggregate)
+ else
+ @stdout.print twirl.twirl(respcount, discovered.size) if @progress
+
+ result, aggregate = process_results_without_block(resp, action, aggregate)
+
+ results << result
+ end
+ end
+
+ @stats.aggregate_summary = aggregate.summarize if aggregate
+ @stats.aggregate_failures = aggregate.failed if aggregate
+ @stats.client_stats = @client.stats
+ else
+ @stderr.print("\nNo request sent, we did not discover any nodes.")
+ end
+
+ @stats.finish_request
+
+ RPC.stats(@stats)
+
+ @stdout.print("\n\n") if @progress
+
+ if block_given?
+ return stats
+ else
+ return [results].flatten
+ end
+ end
+
+ # Handles result sets that has no block associated, sets fails and ok
+ # in the stats object and return a hash of the response to send to the
+ # caller
+ def process_results_without_block(resp, action, aggregate)
+ @stats.node_responded(resp[:senderid])
+
+ result = rpc_result_from_reply(@agent, action, resp)
+ aggregate = aggregate_reply(result, aggregate) if aggregate
+
+ if resp[:body][:statuscode] == 0 || resp[:body][:statuscode] == 1
+ @stats.ok if resp[:body][:statuscode] == 0
+ @stats.fail if resp[:body][:statuscode] == 1
+ else
+ @stats.fail
+ end
+
+ [result, aggregate]
+ end
+
+ # process client requests by calling a block on each result
+ # in this mode we do not do anything fancy with the result
+ # objects and we raise exceptions if there are problems with
+ # the data
+ def process_results_with_block(action, resp, block, aggregate)
+ @stats.node_responded(resp[:senderid])
+
+ result = rpc_result_from_reply(@agent, action, resp)
+ aggregate = aggregate_reply(result, aggregate) if aggregate
+
+ if resp[:body][:statuscode] == 0 || resp[:body][:statuscode] == 1
+ @stats.ok if resp[:body][:statuscode] == 0
+ @stats.fail if resp[:body][:statuscode] == 1
+ @stats.time_block_execution :start
+
+ case block.arity
+ when 1
+ block.call(resp)
+ when 2
+ block.call(resp, result)
+ end
+
+ @stats.time_block_execution :end
+ else
+ @stats.fail
+
+ case resp[:body][:statuscode]
+ when 2
+ raise UnknownRPCAction, resp[:body][:statusmsg]
+ when 3
+ raise MissingRPCData, resp[:body][:statusmsg]
+ when 4
+ raise InvalidRPCData, resp[:body][:statusmsg]
+ when 5
+ raise UnknownRPCError, resp[:body][:statusmsg]
+ end
+ end
+
+ return aggregate
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # Various utilities for the RPC system
+ class Helpers
+ # Parse JSON output as produced by printrpc and extract
+ # the "sender" of each rpc response
+ #
+ # The simplist valid JSON based data would be:
+ #
+ # [
+ # {"sender" => "example.com"},
+ # {"sender" => "another.com"}
+ # ]
+ def self.extract_hosts_from_json(json)
+ hosts = JSON.parse(json)
+
+ raise "JSON hosts list is not an array" unless hosts.is_a?(Array)
+
+ hosts.map do |host|
+ raise "JSON host list is not an array of Hashes" unless host.is_a?(Hash)
+ raise "JSON host list does not have senders in it" unless host.include?("sender")
+
+ host["sender"]
+ end.uniq
+ end
+
+ # Given an array of something, make sure each is a string
+ # chomp off any new lines and return just the array of hosts
+ def self.extract_hosts_from_array(hosts)
+ [hosts].flatten.map do |host|
+ raise "#{host} should be a string" unless host.is_a?(String)
+ host.chomp
+ end
+ end
+
+ # Returns a blob of text representing the results in a standard way
+ #
+ # It tries hard to do sane things so you often
+ # should not need to write your own display functions
+ #
+ # If the agent you are getting results for has a DDL
+ # it will use the hints in there to do the right thing specifically
+ # it will look at the values of display in the DDL to choose
+ # when to show results
+ #
+ # If you do not have a DDL you can pass these flags:
+ #
+ # printrpc exim.mailq, :flatten => true
+ # printrpc exim.mailq, :verbose => true
+ #
+ # If you've asked it to flatten the result it will not print sender
+ # hostnames, it will just print the result as if it's one huge result,
+ # handy for things like showing a combined mailq.
+ def self.rpcresults(result, flags = {})
+ flags = {:verbose => false, :flatten => false, :format => :console, :force_display_mode => false}.merge(flags)
+
+ result_text = ""
+ ddl = nil
+
+ # if running in verbose mode, just use the old style print
+ # no need for all the DDL helpers obfuscating the result
+ if flags[:format] == :json
+ if STDOUT.tty?
+ result_text = JSON.pretty_generate(result)
+ else
+ result_text = result.to_json
+ end
+ else
+ if flags[:verbose]
+ result_text = old_rpcresults(result, flags)
+ else
+ [result].flatten.each do |r|
+ begin
+ ddl ||= DDL.new(r.agent).action_interface(r.action.to_s)
+
+ sender = r[:sender]
+ status = r[:statuscode]
+ message = r[:statusmsg]
+ result = r[:data]
+
+ if flags[:force_display_mode]
+ display = flags[:force_display_mode]
+ else
+ display = ddl[:display]
+ end
+
+ # appand the results only according to what the DDL says
+ case display
+ when :ok
+ if status == 0
+ result_text << text_for_result(sender, status, message, result, ddl)
+ end
+
+ when :failed
+ if status > 0
+ result_text << text_for_result(sender, status, message, result, ddl)
+ end
+
+ when :always
+ result_text << text_for_result(sender, status, message, result, ddl)
+
+ when :flatten
+ result_text << text_for_flattened_result(status, result)
+
+ end
+ rescue Exception => e
+ # no DDL so just do the old style print unchanged for
+ # backward compat
+ result_text = old_rpcresults(result, flags)
+ end
+ end
+ end
+ end
+
+ result_text
+ end
+
+ # Return text representing a result
+ def self.text_for_result(sender, status, msg, result, ddl)
+ statusses = ["",
+ Util.colorize(:red, "Request Aborted"),
+ Util.colorize(:yellow, "Unknown Action"),
+ Util.colorize(:yellow, "Missing Request Data"),
+ Util.colorize(:yellow, "Invalid Request Data"),
+ Util.colorize(:red, "Unknown Request Status")]
+
+ result_text = "%-40s %s\n" % [sender, statusses[status]]
+ result_text << " %s\n" % [Util.colorize(:yellow, msg)] unless msg == "OK"
+
+ # only print good data, ignore data that results from failure
+ if status == 0
+ if result.is_a?(Hash)
+ # figure out the lengths of the display as strings, we'll use
+ # it later to correctly justify the output
+ lengths = result.keys.map do |k|
+ begin
+ ddl[:output][k][:display_as].size
+ rescue
+ k.to_s.size
+ end
+ end
+
+ result.keys.sort_by{|k| k}.each do |k|
+ # get all the output fields nicely lined up with a
+ # 3 space front padding
+ begin
+ display_as = ddl[:output][k][:display_as]
+ rescue
+ display_as = k.to_s
+ end
+
+ display_length = display_as.size
+ padding = lengths.max - display_length + 3
+ result_text << " " * padding
+
+ result_text << "#{display_as}:"
+
+ if [String, Numeric].include?(result[k].class)
+ lines = result[k].to_s.split("\n")
+
+ if lines.empty?
+ result_text << "\n"
+ else
+ lines.each_with_index do |line, i|
+ i == 0 ? padtxt = " " : padtxt = " " * (padding + display_length + 2)
+
+ result_text << "#{padtxt}#{line}\n"
+ end
+ end
+ else
+ padding = " " * (lengths.max + 5)
+ result_text << " " << result[k].pretty_inspect.split("\n").join("\n" << padding) << "\n"
+ end
+ end
+ elsif status == 1
+ # for status 1 we dont want to show half baked
+ # data by default since the DDL will supply all the defaults
+ # it just doesnt look right
+ else
+ result_text << "\n\t" + result.pretty_inspect.split("\n").join("\n\t")
+ end
+ end
+
+ result_text << "\n"
+ result_text
+ end
+
+ # Returns text representing a flattened result of only good data
+ def self.text_for_flattened_result(status, result)
+ result_text = ""
+
+ if status <= 1
+ unless result.is_a?(String)
+ result_text << result.pretty_inspect
+ else
+ result_text << result
+ end
+ end
+ end
+
+ # Backward compatible display block for results without a DDL
+ def self.old_rpcresults(result, flags = {})
+ result_text = ""
+
+ if flags[:flatten]
+ result.each do |r|
+ if r[:statuscode] <= 1
+ data = r[:data]
+
+ unless data.is_a?(String)
+ result_text << data.pretty_inspect
+ else
+ result_text << data
+ end
+ else
+ result_text << r.pretty_inspect
+ end
+ end
+
+ result_text << ""
+ else
+ [result].flatten.each do |r|
+
+ if flags[:verbose]
+ result_text << "%-40s: %s\n" % [r[:sender], r[:statusmsg]]
+
+ if r[:statuscode] <= 1
+ r[:data].pretty_inspect.split("\n").each {|m| result_text += " #{m}"}
+ result_text << "\n\n"
+ elsif r[:statuscode] == 2
+ # dont print anything, no useful data to display
+ # past what was already shown
+ elsif r[:statuscode] == 3
+ # dont print anything, no useful data to display
+ # past what was already shown
+ elsif r[:statuscode] == 4
+ # dont print anything, no useful data to display
+ # past what was already shown
+ else
+ result_text << " #{r[:statusmsg]}"
+ end
+ else
+ unless r[:statuscode] == 0
+ result_text << "%-40s %s\n" % [r[:sender], Util.colorize(:red, r[:statusmsg])]
+ end
+ end
+ end
+ end
+
+ result_text << ""
+ end
+
+ # Add SimpleRPC common options
+ def self.add_simplerpc_options(parser, options)
+ parser.separator ""
+ parser.separator "RPC Options"
+
+ # add SimpleRPC specific options to all clients that use our library
+ parser.on('--np', '--no-progress', 'Do not show the progress bar') do |v|
+ options[:progress_bar] = false
+ end
+
+ parser.on('--one', '-1', 'Send request to only one discovered nodes') do |v|
+ options[:mcollective_limit_targets] = 1
+ end
+
+ parser.on('--batch SIZE', Integer, 'Do requests in batches') do |v|
+ options[:batch_size] = v
+ end
+
+ parser.on('--batch-sleep SECONDS', Float, 'Sleep time between batches') do |v|
+ options[:batch_sleep_time] = v
+ end
+
+ parser.on('--limit-seed NUMBER', Integer, 'Seed value for deterministic random batching') do |v|
+ options[:limit_seed] = v
+ end
+
+ parser.on('--limit-nodes COUNT', '--ln', '--limit', 'Send request to only a subset of nodes, can be a percentage') do |v|
+ raise "Invalid limit specified: #{v} valid limits are /^\d+%*$/" unless v =~ /^\d+%*$/
+
+ if v =~ /^\d+$/
+ options[:mcollective_limit_targets] = v.to_i
+ else
+ options[:mcollective_limit_targets] = v
+ end
+ end
+
+ parser.on('--json', '-j', 'Produce JSON output') do |v|
+ options[:progress_bar] = false
+ options[:output_format] = :json
+ end
+
+ parser.on('--display MODE', 'Influence how results are displayed. One of ok, all or failed') do |v|
+ if v == "all"
+ options[:force_display_mode] = :always
+ else
+ options[:force_display_mode] = v.intern
+ end
+
+ raise "--display has to be one of 'ok', 'all' or 'failed'" unless [:ok, :failed, :always].include?(options[:force_display_mode])
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # Class that shows a progress bar, currently only supports a twirling
+ # progress bar.
+ #
+ # You can specify a size for the progress bar if you want if you dont
+ # it will use the helper functions to figure out terminal dimensions
+ # and draw an appropriately sized bar
+ #
+ # p = Progress.new
+ # 100.times {|i| print p.twirl(i+1, 100) + "\r"};puts
+ #
+ # * [ ==================================================> ] 100 / 100
+ class Progress
+ def initialize(size=nil)
+ @twirl = ['|', '/', '-', "\\", '|', '/', '-', "\\"]
+ @twirldex = 0
+
+ if size
+ @size = size
+ else
+ cols = Util.terminal_dimensions[0] - 22
+
+ # Defaults back to old behavior if it
+ # couldn't figure out the size or if
+ # its more than 60 wide
+ if cols <= 0
+ @size = 0
+ elsif cols > 60
+ @size = 60
+ else
+ @size = cols
+ end
+ end
+ end
+
+ def twirl(current, total)
+ # if the size is negative there is just not enough
+ # space on the terminal, return a simpler version
+ return "\r#{current} / #{total}" if @size == 0
+
+ if current == total
+ txt = "\r %s [ " % Util.colorize(:green, "*")
+ else
+ txt = "\r %s [ " % Util.colorize(:red, @twirl[@twirldex])
+ end
+
+ dashes = ((current.to_f / total) * @size).round
+
+ dashes.times { txt << "=" }
+ txt << ">"
+
+ (@size - dashes).times { txt << " " }
+
+ txt << " ] #{current} / #{total}"
+
+ @twirldex == 7 ? @twirldex = 0 : @twirldex += 1
+
+ return txt
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # Simple class to manage compliant replies to MCollective::RPC
+ class Reply
+ attr_accessor :statuscode, :statusmsg, :data
+
+ def initialize(action, ddl)
+ @data = {}
+ @statuscode = 0
+ @statusmsg = "OK"
+ @ddl = ddl
+ @action = action
+
+ begin
+ initialize_data
+ rescue Exception => e
+ Log.warn("Could not pre-populate reply data from the DDL: %s: %s" % [e.class, e.to_s ])
+ end
+ end
+
+ def initialize_data
+ unless @ddl.actions.include?(@action)
+ raise "No action '%s' defined for agent '%s' in the DDL" % [@action, @ddl.pluginname]
+ end
+
+ interface = @ddl.action_interface(@action)
+
+ interface[:output].keys.each do |output|
+ @data[output] = interface[:output][output][:default]
+ end
+ end
+
+ # Helper to fill in statusmsg and code on failure
+ def fail(msg, code=1)
+ @statusmsg = msg
+ @statuscode = code
+ end
+
+ # Helper that fills in statusmsg and code but also raises an appropriate error
+ def fail!(msg, code=1)
+ @statusmsg = msg
+ @statuscode = code
+
+ case code
+ when 1
+ raise RPCAborted, msg
+
+ when 2
+ raise UnknownRPCAction, msg
+
+ when 3
+ raise MissingRPCData, msg
+
+ when 4
+ raise InvalidRPCData, msg
+
+ else
+ raise UnknownRPCError, msg
+ end
+ end
+
+ # Write to the data hash
+ def []=(key, val)
+ @data[key] = val
+ end
+
+ # Read from the data hash
+ def [](key)
+ @data[key]
+ end
+
+ def fetch(key, default)
+ @data.fetch(key, default)
+ end
+
+ # Returns a compliant Hash of the reply that should be sent
+ # over the middleware
+ def to_hash
+ return {:statuscode => @statuscode,
+ :statusmsg => @statusmsg,
+ :data => @data}
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # Simple class to manage compliant requests for MCollective::RPC agents
+ class Request
+ attr_accessor :time, :action, :data, :sender, :agent, :uniqid, :caller, :ddl
+
+ def initialize(msg, ddl)
+ @time = msg[:msgtime]
+ @action = msg[:body][:action]
+ @data = msg[:body][:data]
+ @sender = msg[:senderid]
+ @agent = msg[:body][:agent]
+ @uniqid = msg[:requestid]
+ @caller = msg[:callerid] || "unknown"
+ @ddl = ddl
+ end
+
+ # If data is a hash, quick helper to get access to it's include? method
+ # else returns false
+ def include?(key)
+ return false unless @data.is_a?(Hash)
+ return @data.include?(key)
+ end
+
+ # If no :process_results is specified always respond else respond
+ # based on the supplied property
+ def should_respond?
+ return @data[:process_results] if @data.include?(:process_results)
+
+ return true
+ end
+
+ # If data is a hash, gives easy access to its members, else returns nil
+ def [](key)
+ return nil unless @data.is_a?(Hash)
+ return @data[key]
+ end
+
+ def fetch(key, default)
+ return nil unless @data.is_a?(Hash)
+ return @data.fetch(key, default)
+ end
+
+ def to_hash
+ return {:agent => @agent,
+ :action => @action,
+ :data => @data}
+ end
+
+ # Validate the request against the DDL
+ def validate!
+ @ddl.validate_rpc_request(@action, @data)
+ end
+
+ def to_json
+ to_hash.merge!({:sender => @sender,
+ :callerid => @callerid,
+ :uniqid => @uniqid}).to_json
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # Simple class to manage compliant results from MCollective::RPC agents
+ #
+ # Currently it just fakes Hash behaviour to the result to remain backward
+ # compatible but it also knows which agent and action produced it so you
+ # can associate results to a DDL
+ class Result
+ attr_reader :agent, :action, :results
+
+ include Enumerable
+
+ def initialize(agent, action, result={})
+ @agent = agent
+ @action = action
+ @results = result
+ end
+
+ def [](idx)
+ @results[idx]
+ end
+
+ def []=(idx, item)
+ @results[idx] = item
+ end
+
+ def fetch(key, default)
+ @results.fetch(key, default)
+ end
+
+ def each
+ @results.each_pair {|k,v| yield(k,v) }
+ end
+
+ def to_json(*a)
+ {:agent => @agent,
+ :action => @action,
+ :sender => @results[:sender],
+ :statuscode => @results[:statuscode],
+ :statusmsg => @results[:statusmsg],
+ :data => @results[:data]}.to_json(*a)
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # Class to wrap all the stats and to keep track of some timings
+ class Stats
+ attr_accessor :noresponsefrom, :starttime, :discoverytime, :blocktime, :responses, :totaltime
+ attr_accessor :discovered, :discovered_nodes, :okcount, :failcount, :noresponsefrom, :responsesfrom
+ attr_accessor :requestid, :aggregate_summary, :ddl, :aggregate_failures
+
+ def initialize
+ reset
+ end
+
+ # Resets stats, if discovery time is set we keep it as it was
+ def reset
+ @noresponsefrom = []
+ @responsesfrom = []
+ @responses = 0
+ @starttime = Time.now.to_f
+ @discoverytime = 0 unless @discoverytime
+ @blocktime = 0
+ @totaltime = 0
+ @discovered = 0
+ @discovered_nodes = []
+ @okcount = 0
+ @failcount = 0
+ @noresponsefrom = []
+ @requestid = nil
+ @aggregate_summary = []
+ @aggregate_failures = []
+ end
+
+ # returns a hash of our stats
+ def to_hash
+ {:noresponsefrom => @noresponsefrom,
+ :starttime => @starttime,
+ :discoverytime => @discoverytime,
+ :blocktime => @blocktime,
+ :responses => @responses,
+ :totaltime => @totaltime,
+ :discovered => @discovered,
+ :discovered_nodes => @discovered_nodes,
+ :noresponsefrom => @noresponsefrom,
+ :okcount => @okcount,
+ :requestid => @requestid,
+ :failcount => @failcount,
+ :aggregate_summary => @aggregate_summary,
+ :aggregate_failures => @aggregate_failures}
+ end
+
+ # Fake hash access to keep things backward compatible
+ def [](key)
+ to_hash[key]
+ rescue
+ nil
+ end
+
+ # increment the count of ok hosts
+ def ok
+ @okcount += 1
+ rescue
+ @okcount = 1
+ end
+
+ # increment the count of failed hosts
+ def fail
+ @failcount += 1
+ rescue
+ @failcount = 1
+ end
+
+ # Re-initializes the object with stats from the basic client
+ def client_stats=(stats)
+ @noresponsefrom = stats[:noresponsefrom]
+ @responses = stats[:responses]
+ @starttime = stats[:starttime]
+ @blocktime = stats[:blocktime]
+ @totaltime = stats[:totaltime]
+ @requestid = stats[:requestid]
+ @discoverytime = stats[:discoverytime] if @discoverytime == 0
+ end
+
+ # Utility to time discovery from :start to :end
+ def time_discovery(action)
+ if action == :start
+ @discovery_start = Time.now.to_f
+ elsif action == :end
+ @discoverytime = Time.now.to_f - @discovery_start
+ else
+ raise("Uknown discovery action #{action}")
+ end
+ rescue
+ @discoverytime = 0
+ end
+
+ # helper to time block execution time
+ def time_block_execution(action)
+ if action == :start
+ @block_start = Time.now.to_f
+ elsif action == :end
+ @blocktime += Time.now.to_f - @block_start
+ else
+ raise("Uknown block action #{action}")
+ end
+ rescue
+ @blocktime = 0
+ end
+
+ # Update discovered and discovered_nodes based on
+ # discovery results
+ def discovered_agents(agents)
+ @discovered_nodes = agents
+ @discovered = agents.size
+ end
+
+ # Helper to calculate total time etc
+ def finish_request
+ @totaltime = @blocktime + @discoverytime
+
+ # figures out who we had no responses from
+ dhosts = @discovered_nodes.clone
+ @responsesfrom.each {|r| dhosts.delete(r)}
+ @noresponsefrom = dhosts
+ rescue
+ @totaltime = 0
+ @noresponsefrom = []
+ end
+
+ # Helper to keep track of who we received responses from
+ def node_responded(node)
+ @responsesfrom << node
+ rescue
+ @responsesfrom = [node]
+ end
+
+ def text_for_aggregates
+ result = StringIO.new
+
+ @aggregate_summary.each do |aggregate|
+ output_item = aggregate.result[:output]
+
+ begin
+ action_interface = @ddl.action_interface(aggregate.action)
+ display_as = action_interface[:output][output_item][:display_as]
+ rescue
+ display_as = output_item
+ end
+
+ if aggregate.is_a?(Aggregate::Result::Base)
+ aggregate_report = aggregate.to_s
+ else
+ next
+ end
+
+ result.puts Util.colorize(:bold, "Summary of %s:" % display_as)
+ result.puts
+ unless aggregate_report == ""
+ result.puts aggregate.to_s.split("\n").map{|x| " " + x}.join("\n")
+ else
+ result.puts Util.colorize(:yellow, " No aggregate summary could be computed")
+ end
+ result.puts
+ end
+
+ @aggregate_failures.each do |failed|
+ case(failed[:type])
+ when :startup
+ message = "exception raised while processing startup hook"
+ when :create
+ message = "unspecified output '#{failed[:name]}' for the action"
+ when :process_result
+ message = "exception raised while processing result data"
+ when :summarize
+ message = "exception raised while summarizing"
+ end
+
+ result.puts Util.colorize(:bold, "Summary of %s:" % failed[:name])
+ result.puts
+ result.puts Util.colorize(:yellow, " Could not compute summary - %s" % message)
+ result.puts
+ end
+
+ result.string
+ end
+
+ # Returns a blob of text representing the request status based on the
+ # stats contained in this class
+ def report(caption = "rpc stats", summarize = true, verbose = false)
+ result_text = []
+
+ if verbose
+ if @aggregate_summary.size > 0 && summarize
+ result_text << text_for_aggregates
+ else
+ result_text << ""
+ end
+
+ result_text << Util.colorize(:yellow, "---- #{caption} ----")
+
+ if @discovered
+ @responses < @discovered ? color = :red : color = :reset
+ result_text << " Nodes: %s / %s" % [ Util.colorize(color, @discovered), Util.colorize(color, @responses) ]
+ else
+ result_text << " Nodes: #{@responses}"
+ end
+
+ @failcount < 0 ? color = :red : color = :reset
+
+ result_text << " Pass / Fail: %s / %s" % [Util.colorize(color, @okcount), Util.colorize(color, @failcount) ]
+ result_text << " Start Time: %s" % [Time.at(@starttime)]
+ result_text << " Discovery Time: %.2fms" % [@discoverytime * 1000]
+ result_text << " Agent Time: %.2fms" % [@blocktime * 1000]
+ result_text << " Total Time: %.2fms" % [@totaltime * 1000]
+ else
+ if @discovered
+ @responses < @discovered ? color = :red : color = :green
+
+ if @aggregate_summary.size + @aggregate_failures.size > 0 && summarize
+ result_text << text_for_aggregates
+ else
+ result_text << ""
+ end
+
+ result_text << "Finished processing %s / %s hosts in %.2f ms" % [Util.colorize(color, @responses), Util.colorize(color, @discovered), @blocktime * 1000]
+ else
+ result_text << "Finished processing %s hosts in %.2f ms" % [Util.colorize(:bold, @responses), @blocktime * 1000]
+ end
+ end
+
+ if no_response_report != ""
+ result_text << "" << no_response_report
+ end
+
+ result_text.join("\n")
+ end
+
+ # Returns a blob of text indicating what nodes did not respond
+ def no_response_report
+ result_text = StringIO.new
+
+ if @noresponsefrom.size > 0
+ result_text.puts
+ result_text.puts Util.colorize(:red, "No response from:")
+ result_text.puts
+
+ @noresponsefrom.sort.in_groups_of(3) do |c|
+ result_text.puts " %-30s%-30s%-30s" % c
+ end
+
+ result_text.puts
+ end
+
+ result_text.string
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # The main runner for the daemon, supports running in the foreground
+ # and the background, keeps detailed stats and provides hooks to access
+ # all this information
+ class Runner
+ include Translatable
+
+ def initialize(configfile)
+ @config = Config.instance
+ @config.loadconfig(configfile) unless @config.configured
+ @config.mode = :server
+
+ @stats = PluginManager["global_stats"]
+
+ @security = PluginManager["security_plugin"]
+ @security.initiated_by = :node
+
+ @connection = PluginManager["connector_plugin"]
+ @connection.connect
+
+ @agents = Agents.new
+
+ unless Util.windows?
+ Signal.trap("USR1") do
+ log_code(:PLMC2, "Reloading all agents after receiving USR1 signal", :info)
+ @agents.loadagents
+ end
+
+ Signal.trap("USR2") do
+ log_code(:PLMC3, "Cycling logging level due to USR2 signal", :info)
+
+ Log.cycle_level
+ end
+ else
+ Util.setup_windows_sleeper
+ end
+ end
+
+ # Starts the main loop, before calling this you should initialize the MCollective::Config singleton.
+ def run
+ Data.load_data_sources
+
+ Util.subscribe(Util.make_subscriptions("mcollective", :broadcast))
+ Util.subscribe(Util.make_subscriptions("mcollective", :directed)) if @config.direct_addressing
+
+ # Start the registration plugin if interval isn't 0
+ begin
+ PluginManager["registration_plugin"].run(@connection) unless @config.registerinterval == 0
+ rescue Exception => e
+ logexception(:PLMC4, "Failed to start registration plugin: %{error}", :error, e)
+ end
+
+ loop do
+ begin
+ request = receive
+
+ unless request.agent == "mcollective"
+ agentmsg(request)
+ else
+ log_code(:PLMC5, "Received a control message, possibly via 'mco controller' but this has been deprecated", :error)
+ end
+ rescue SignalException => e
+ logexception(:PLMC7, "Exiting after signal: %{error}", :warn, e)
+ @connection.disconnect
+ raise
+
+ rescue MsgTTLExpired => e
+ logexception(:PLMC9, "Expired Message: %{error}", :warn, e)
+
+ rescue NotTargettedAtUs => e
+ log_code(:PLMC6, "Message does not pass filters, ignoring", :debug)
+
+ rescue Exception => e
+ logexception(:PLMC10, "Failed to handle message: %{error}", :warn, e, true)
+ end
+ end
+ end
+
+ private
+ # Deals with messages directed to agents
+ def agentmsg(request)
+ log_code(:PLMC8, "Handling message for agent '%{agent}' on collective '%{collective} with requestid '%{requestid}'", :debug, :agent => request.agent, :collective => request.collective, :requestid => request.requestid)
+
+ @agents.dispatch(request, @connection) do |reply_message|
+ reply(reply_message, request) if reply_message
+ end
+ end
+
+ # Deals with messages sent to our control topic
+ def controlmsg(request)
+ Log.debug("Handling message for mcollectived controller")
+
+ begin
+ case request.payload[:body]
+ when /^stats$/
+ reply(@stats.to_hash, request)
+
+ when /^reload_agent (.+)$/
+ reply("reloaded #{$1} agent", request) if @agents.loadagent($1)
+
+ when /^reload_agents$/
+
+ reply("reloaded all agents", request) if @agents.loadagents
+
+ else
+ Log.error("Received an unknown message to the controller")
+
+ end
+ rescue Exception => e
+ Log.error("Failed to handle control message: #{e}")
+ end
+ end
+
+ # Receive a message from the connection handler
+ def receive
+ request = @connection.receive
+ request.type = :request
+
+ @stats.received
+
+ request.decode!
+ request.validate
+
+ request
+ end
+
+ # Sends a reply to a specific target topic
+ def reply(msg, request)
+ msg = Message.new(msg, nil, :request => request)
+ msg.encode!
+ msg.publish
+
+ @stats.sent
+ end
+ end
+end
+
--- /dev/null
+module MCollective
+ # Class to store stats about the mcollectived, it should live in the PluginManager
+ # so that agents etc can get hold of it and return the stats to callers
+ class RunnerStats
+ def initialize
+ @starttime = Time.now.to_i
+ @validated = 0
+ @unvalidated = 0
+ @filtered = 0
+ @passed = 0
+ @total = 0
+ @replies = 0
+ @ttlexpired = 0
+
+ @mutex = Mutex.new
+ end
+
+ # Records a message that failed TTL checks
+ def ttlexpired
+ Log.debug("Incrementing ttl expired stat")
+ @ttlexpired += 1
+ end
+
+ # Records a message that passed the filters
+ def passed
+ Log.debug("Incrementing passed stat")
+ @passed += 1
+ end
+
+ # Records a message that didnt pass the filters
+ def filtered
+ Log.debug("Incrementing filtered stat")
+ @filtered += 1
+ end
+
+ # Records a message that validated ok
+ def validated
+ Log.debug("Incrementing validated stat")
+ @validated += 1
+ end
+
+ def unvalidated
+ Log.debug("Incrementing unvalidated stat")
+ @unvalidated += 1
+ end
+
+ # Records receipt of a message
+ def received
+ Log.debug("Incrementing total stat")
+ @total += 1
+ end
+
+ # Records sending a message
+ def sent
+ @mutex.synchronize do
+ Log.debug("Incrementing replies stat")
+ @replies += 1
+ end
+ end
+
+ # Returns a hash with all stats
+ def to_hash
+ stats = {:validated => @validated,
+ :unvalidated => @unvalidated,
+ :passed => @passed,
+ :filtered => @filtered,
+ :starttime => @starttime,
+ :total => @total,
+ :ttlexpired => @ttlexpired,
+ :replies => @replies}
+
+ reply = {:stats => stats,
+ :threads => [],
+ :pid => Process.pid,
+ :times => {} }
+
+ ::Process.times.each_pair{|k,v|
+ k = k.to_sym
+ reply[:times][k] = v
+ }
+
+ Thread.list.each do |t|
+ reply[:threads] << "#{t.inspect}"
+ end
+
+ reply[:agents] = Agents.agentlist
+ reply
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # Security is implimented using a module structure and installations
+ # can configure which module they want to use.
+ #
+ # Security modules deal with various aspects of authentication and authorization:
+ #
+ # - Determines if a filter excludes this host from dealing with a request
+ # - Serialization and Deserialization of messages
+ # - Validation of messages against keys, certificates or whatever the class choose to impliment
+ # - Encoding and Decoding of messages
+ #
+ # To impliment a new security class using SSL for example you would inherit from the base
+ # class and only impliment:
+ #
+ # - decodemsg
+ # - encodereply
+ # - encoderequest
+ # - validrequest?
+ #
+ # Each of these methods should increment various stats counters, see the default MCollective::Security::Psk module for examples of this
+ #
+ # Filtering can be extended by providing a new validate_filter? method.
+ module Security
+ autoload :Base, "mcollective/security/base"
+ end
+end
--- /dev/null
+module MCollective
+ module Security
+ # This is a base class the other security modules should inherit from
+ # it handles statistics and validation of messages that should in most
+ # cases apply to all security models.
+ #
+ # To create your own security plugin you should provide a plugin that inherits
+ # from this and provides the following methods:
+ #
+ # decodemsg - Decodes a message that was received from the middleware
+ # encodereply - Encodes a reply message to a previous request message
+ # encoderequest - Encodes a new request message
+ # validrequest? - Validates a request received from the middleware
+ #
+ # Optionally if you are identifying users by some other means like certificate name
+ # you can provide your own callerid method that can provide the rest of the system
+ # with an id, and you would see this id being usable in SimpleRPC authorization methods
+ #
+ # The @initiated_by variable will be set to either :client or :node depending on
+ # who is using this plugin. This is to help security providers that operate in an
+ # asymetric mode like public/private key based systems.
+ #
+ # Specifics of each of these are a bit fluid and the interfaces for this is not
+ # set in stone yet, specifically the encode methods will be provided with a helper
+ # that takes care of encoding the core requirements. The best place to see how security
+ # works is by looking at the provided MCollective::Security::PSK plugin.
+ class Base
+ attr_reader :stats
+ attr_accessor :initiated_by
+
+ # Register plugins that inherits base
+ def self.inherited(klass)
+ PluginManager << {:type => "security_plugin", :class => klass.to_s}
+ end
+
+ # Initializes configuration and logging as well as prepare a zero'd hash of stats
+ # various security methods and filter validators should increment stats, see MCollective::Security::Psk for a sample
+ def initialize
+ @config = Config.instance
+ @log = Log
+ @stats = PluginManager["global_stats"]
+ end
+
+ # Takes a Hash with a filter in it and validates it against host information.
+ #
+ # At present this supports filter matches against the following criteria:
+ #
+ # - puppet_class|cf_class - Presence of a configuration management class in
+ # the file configured with classesfile
+ # - agent - Presence of a MCollective agent with a supplied name
+ # - fact - The value of a fact avout this system
+ # - identity - the configured identity of the system
+ #
+ # TODO: Support REGEX and/or multiple filter keys to be AND'd
+ def validate_filter?(filter)
+ failed = 0
+ passed = 0
+
+ passed = 1 if Util.empty_filter?(filter)
+
+ filter.keys.each do |key|
+ case key
+ when /puppet_class|cf_class/
+ filter[key].each do |f|
+ Log.debug("Checking for class #{f}")
+ if Util.has_cf_class?(f) then
+ Log.debug("Passing based on configuration management class #{f}")
+ passed += 1
+ else
+ Log.debug("Failing based on configuration management class #{f}")
+ failed += 1
+ end
+ end
+
+ when "compound"
+ filter[key].each do |compound|
+ result = false
+ truth_values = []
+
+ begin
+ compound.each do |expression|
+ case expression.keys.first
+ when "statement"
+ truth_values << Matcher.eval_compound_statement(expression).to_s
+ when "fstatement"
+ truth_values << Matcher.eval_compound_fstatement(expression.values.first)
+ when "and"
+ truth_values << "&&"
+ when "or"
+ truth_values << "||"
+ when "("
+ truth_values << "("
+ when ")"
+ truth_values << ")"
+ when "not"
+ truth_values << "!"
+ end
+ end
+
+ result = eval(truth_values.join(" "))
+ rescue DDLValidationError
+ result = false
+ end
+
+ if result
+ Log.debug("Passing based on class and fact composition")
+ passed +=1
+ else
+ Log.debug("Failing based on class and fact composition")
+ failed +=1
+ end
+ end
+
+ when "agent"
+ filter[key].each do |f|
+ if Util.has_agent?(f) || f == "mcollective"
+ Log.debug("Passing based on agent #{f}")
+ passed += 1
+ else
+ Log.debug("Failing based on agent #{f}")
+ failed += 1
+ end
+ end
+
+ when "fact"
+ filter[key].each do |f|
+ if Util.has_fact?(f[:fact], f[:value], f[:operator])
+ Log.debug("Passing based on fact #{f[:fact]} #{f[:operator]} #{f[:value]}")
+ passed += 1
+ else
+ Log.debug("Failing based on fact #{f[:fact]} #{f[:operator]} #{f[:value]}")
+ failed += 1
+ end
+ end
+
+ when "identity"
+ unless filter[key].empty?
+ # Identity filters should not be 'and' but 'or' as each node can only have one identity
+ matched = filter[key].select{|f| Util.has_identity?(f)}.size
+
+ if matched == 1
+ Log.debug("Passing based on identity")
+ passed += 1
+ else
+ Log.debug("Failed based on identity")
+ failed += 1
+ end
+ end
+ end
+ end
+
+ if failed == 0 && passed > 0
+ Log.debug("Message passed the filter checks")
+
+ @stats.passed
+
+ return true
+ else
+ Log.debug("Message failed the filter checks")
+
+ @stats.filtered
+
+ return false
+ end
+ end
+
+ def create_reply(reqid, agent, body)
+ Log.debug("Encoded a message for request #{reqid}")
+
+ {:senderid => @config.identity,
+ :requestid => reqid,
+ :senderagent => agent,
+ :msgtime => Time.now.utc.to_i,
+ :body => body}
+ end
+
+ def create_request(reqid, filter, msg, initiated_by, target_agent, target_collective, ttl=60)
+ Log.debug("Encoding a request for agent '#{target_agent}' in collective #{target_collective} with request id #{reqid}")
+
+ {:body => msg,
+ :senderid => @config.identity,
+ :requestid => reqid,
+ :filter => filter,
+ :collective => target_collective,
+ :agent => target_agent,
+ :callerid => callerid,
+ :ttl => ttl,
+ :msgtime => Time.now.utc.to_i}
+ end
+
+ # Give a MC::Message instance and a message id this will figure out if you the incoming
+ # message id matches the one the Message object is expecting and raise if its not
+ #
+ # Mostly used by security plugins to figure out if they should do the hard work of decrypting
+ # etc messages that would only later on be ignored
+ def should_process_msg?(msg, msgid)
+ if msg.expected_msgid
+ unless msg.expected_msgid == msgid
+ msgtext = "Got a message with id %s but was expecting %s, ignoring message" % [msgid, msg.expected_msgid]
+ Log.debug msgtext
+ raise MsgDoesNotMatchRequestID, msgtext
+ end
+ end
+
+ true
+ end
+
+ # Validates a callerid. We do not want to allow things like \ and / in
+ # callerids since other plugins make assumptions that these are safe strings.
+ #
+ # callerids are generally in the form uid=123 or cert=foo etc so we do that
+ # here but security plugins could override this for some complex uses
+ def valid_callerid?(id)
+ !!id.match(/^[\w]+=[\w\.\-]+$/)
+ end
+
+ # Returns a unique id for the caller, by default we just use the unix
+ # user id, security plugins can provide their own means of doing ids.
+ def callerid
+ "uid=#{Process.uid}"
+ end
+
+ # Security providers should provide this, see MCollective::Security::Psk
+ def validrequest?(req)
+ Log.error("validrequest? is not implemented in #{self.class}")
+ end
+
+ # Security providers should provide this, see MCollective::Security::Psk
+ def encoderequest(sender, msg, filter={})
+ Log.error("encoderequest is not implemented in #{self.class}")
+ end
+
+ # Security providers should provide this, see MCollective::Security::Psk
+ def encodereply(sender, msg, requestcallerid=nil)
+ Log.error("encodereply is not implemented in #{self.class}")
+ end
+
+ # Security providers should provide this, see MCollective::Security::Psk
+ def decodemsg(msg)
+ Log.error("decodemsg is not implemented in #{self.class}")
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # Wrapper around systemu that handles executing of system commands
+ # in a way that makes stdout, stderr and status available. Supports
+ # timeouts and sets a default sane environment.
+ #
+ # s = Shell.new("date", opts)
+ # s.runcommand
+ # puts s.stdout
+ # puts s.stderr
+ # puts s.status.exitstatus
+ #
+ # Options hash can have:
+ #
+ # cwd - the working directory the command will be run from
+ # stdin - a string that will be sent to stdin of the program
+ # stdout - a variable that will receive stdout, must support <<
+ # stderr - a variable that will receive stdin, must support <<
+ # environment - the shell environment, defaults to include LC_ALL=C
+ # set to nil to clear the environment even of LC_ALL
+ #
+ class Shell
+ attr_reader :environment, :command, :status, :stdout, :stderr, :stdin, :cwd
+
+ def initialize(command, options={})
+ @environment = {"LC_ALL" => "C"}
+ @command = command
+ @status = nil
+ @stdout = ""
+ @stderr = ""
+ @stdin = nil
+ @cwd = Dir.tmpdir
+
+ options.each do |opt, val|
+ case opt.to_s
+ when "stdout"
+ raise "stdout should support <<" unless val.respond_to?("<<")
+ @stdout = val
+
+ when "stderr"
+ raise "stderr should support <<" unless val.respond_to?("<<")
+ @stderr = val
+
+ when "stdin"
+ raise "stdin should be a String" unless val.is_a?(String)
+ @stdin = val
+
+ when "cwd"
+ raise "Directory #{val} does not exist" unless File.directory?(val)
+ @cwd = val
+
+ when "environment"
+ if val.nil?
+ @environment = {}
+ else
+ @environment.merge!(val.dup)
+ end
+ end
+ end
+ end
+
+ # Actually does the systemu call passing in the correct environment, stdout and stderr
+ def runcommand
+ opts = {"env" => @environment,
+ "stdout" => @stdout,
+ "stderr" => @stderr,
+ "cwd" => @cwd}
+
+ opts["stdin"] = @stdin if @stdin
+
+ # Running waitpid on the cid here will start a thread
+ # with the waitpid in it, this way even if the thread
+ # that started this process gets killed due to agent
+ # timeout or such there will still be a waitpid waiting
+ # for the child to exit and not leave zombies.
+ @status = systemu(@command, opts) do |cid|
+ begin
+ sleep 1
+ Process::waitpid(cid)
+ rescue SystemExit
+ rescue Errno::ECHILD
+ rescue Exception => e
+ Log.info("Unexpected exception received while waiting for child process: #{e.class}: #{e}")
+ end
+ end
+ end
+ end
+end
--- /dev/null
+require 'openssl'
+require 'base64'
+require 'digest/sha1'
+
+module MCollective
+ # A class that assists in encrypting and decrypting data using a
+ # combination of RSA and AES
+ #
+ # Data will be AES encrypted for speed, the Key used in # the AES
+ # stage will be encrypted using RSA
+ #
+ # ssl = SSL.new(public_key, private_key, passphrase)
+ #
+ # data = File.read("largefile.dat")
+ #
+ # crypted_data = ssl.encrypt_with_private(data)
+ #
+ # pp crypted_data
+ #
+ # This will result in a hash of data like:
+ #
+ # crypted = {:key => "crd4NHvG....=",
+ # :data => "XWXlqN+i...=="}
+ #
+ # The key and data will all be base 64 encoded already by default
+ # you can pass a 2nd parameter as false to encrypt_with_private and
+ # counterparts that will prevent the base 64 encoding
+ #
+ # You can pass the data hash into ssl.decrypt_with_public which
+ # should return your original data
+ #
+ # There are matching methods for using a public key to encrypt
+ # data to be decrypted using a private key
+ class SSL
+ attr_reader :public_key_file, :private_key_file, :ssl_cipher
+
+ def initialize(pubkey=nil, privkey=nil, passphrase=nil, cipher=nil)
+ @public_key_file = pubkey
+ @private_key_file = privkey
+
+ @public_key = read_key(:public, pubkey)
+ @private_key = read_key(:private, privkey, passphrase)
+
+ @ssl_cipher = "aes-256-cbc"
+ @ssl_cipher = Config.instance.ssl_cipher if Config.instance.ssl_cipher
+ @ssl_cipher = cipher if cipher
+
+ raise "The supplied cipher '#{@ssl_cipher}' is not supported" unless OpenSSL::Cipher.ciphers.include?(@ssl_cipher)
+ end
+
+ # Encrypts supplied data using AES and then encrypts using RSA
+ # the key and IV
+ #
+ # Return a hash with everything optionally base 64 encoded
+ def encrypt_with_public(plain_text, base64=true)
+ crypted = aes_encrypt(plain_text)
+
+ if base64
+ key = base64_encode(rsa_encrypt_with_public(crypted[:key]))
+ data = base64_encode(crypted[:data])
+ else
+ key = rsa_encrypt_with_public(crypted[:key])
+ data = crypted[:data]
+ end
+
+ {:key => key, :data => data}
+ end
+
+ # Encrypts supplied data using AES and then encrypts using RSA
+ # the key and IV
+ #
+ # Return a hash with everything optionally base 64 encoded
+ def encrypt_with_private(plain_text, base64=true)
+ crypted = aes_encrypt(plain_text)
+
+ if base64
+ key = base64_encode(rsa_encrypt_with_private(crypted[:key]))
+ data = base64_encode(crypted[:data])
+ else
+ key = rsa_encrypt_with_private(crypted[:key])
+ data = crypted[:data]
+ end
+
+ {:key => key, :data => data}
+ end
+
+ # Decrypts data, expects a hash as create with crypt_with_public
+ def decrypt_with_private(crypted, base64=true)
+ raise "Crypted data should include a key" unless crypted.include?(:key)
+ raise "Crypted data should include data" unless crypted.include?(:data)
+
+ if base64
+ key = rsa_decrypt_with_private(base64_decode(crypted[:key]))
+ aes_decrypt(key, base64_decode(crypted[:data]))
+ else
+ key = rsa_decrypt_with_private(crypted[:key])
+ aes_decrypt(key, crypted[:data])
+ end
+ end
+
+ # Decrypts data, expects a hash as create with crypt_with_private
+ def decrypt_with_public(crypted, base64=true)
+ raise "Crypted data should include a key" unless crypted.include?(:key)
+ raise "Crypted data should include data" unless crypted.include?(:data)
+
+ if base64
+ key = rsa_decrypt_with_public(base64_decode(crypted[:key]))
+ aes_decrypt(key, base64_decode(crypted[:data]))
+ else
+ key = rsa_decrypt_with_public(crypted[:key])
+ aes_decrypt(key, crypted[:data])
+ end
+ end
+
+ # Use the public key to RSA encrypt data
+ def rsa_encrypt_with_public(plain_string)
+ raise "No public key set" unless @public_key
+
+ @public_key.public_encrypt(plain_string)
+ end
+
+ # Use the private key to RSA decrypt data
+ def rsa_decrypt_with_private(crypt_string)
+ raise "No private key set" unless @private_key
+
+ @private_key.private_decrypt(crypt_string)
+ end
+
+ # Use the private key to RSA encrypt data
+ def rsa_encrypt_with_private(plain_string)
+ raise "No private key set" unless @private_key
+
+ @private_key.private_encrypt(plain_string)
+ end
+
+ # Use the public key to RSA decrypt data
+ def rsa_decrypt_with_public(crypt_string)
+ raise "No public key set" unless @public_key
+
+ @public_key.public_decrypt(crypt_string)
+ end
+
+ # encrypts a string, returns a hash of key, iv and data
+ def aes_encrypt(plain_string)
+ cipher = OpenSSL::Cipher::Cipher.new(ssl_cipher)
+ cipher.encrypt
+
+ key = cipher.random_key
+
+ cipher.key = key
+ cipher.pkcs5_keyivgen(key)
+ encrypted_data = cipher.update(plain_string) + cipher.final
+
+ {:key => key, :data => encrypted_data}
+ end
+
+ # decrypts a string given key, iv and data
+ def aes_decrypt(key, crypt_string)
+ cipher = OpenSSL::Cipher::Cipher.new(ssl_cipher)
+
+ cipher.decrypt
+ cipher.key = key
+ cipher.pkcs5_keyivgen(key)
+ decrypted_data = cipher.update(crypt_string) + cipher.final
+ end
+
+ # Signs a string using the private key
+ def sign(string, base64=false)
+ sig = @private_key.sign(OpenSSL::Digest::SHA1.new, string)
+
+ base64 ? base64_encode(sig) : sig
+ end
+
+ # Using the public key verifies that a string was signed using the private key
+ def verify_signature(signature, string, base64=false)
+ signature = base64_decode(signature) if base64
+
+ @public_key.verify(OpenSSL::Digest::SHA1.new, signature, string)
+ end
+
+ # base 64 encode a string
+ def base64_encode(string)
+ SSL.base64_encode(string)
+ end
+
+ def self.base64_encode(string)
+ Base64.encode64(string)
+ end
+
+ # base 64 decode a string
+ def base64_decode(string)
+ SSL.base64_decode(string)
+ end
+
+ def self.base64_decode(string)
+ Base64.decode64(string)
+ end
+
+ def md5(string)
+ SSL.md5(string)
+ end
+
+ def self.md5(string)
+ Digest::MD5.hexdigest(string)
+ end
+
+ # Creates a RFC 4122 version 5 UUID. If string is supplied it will produce repeatable
+ # UUIDs for that string else a random 128bit string will be used from OpenSSL::BN
+ #
+ # Code used with permission from:
+ # https://github.com/kwilczynski/puppet-functions/blob/master/lib/puppet/parser/functions/uuid.rb
+ #
+ def self.uuid(string=nil)
+ string ||= OpenSSL::Random.random_bytes(16).unpack('H*').shift
+
+ uuid_name_space_dns = "\x6b\xa7\xb8\x10\x9d\xad\x11\xd1\x80\xb4\x00\xc0\x4f\xd4\x30\xc8"
+
+ sha1 = Digest::SHA1.new
+ sha1.update(uuid_name_space_dns)
+ sha1.update(string)
+
+ # first 16 bytes..
+ bytes = sha1.digest[0, 16].bytes.to_a
+
+ # version 5 adjustments
+ bytes[6] &= 0x0f
+ bytes[6] |= 0x50
+
+ # variant is DCE 1.1
+ bytes[8] &= 0x3f
+ bytes[8] |= 0x80
+
+ bytes = [4, 2, 2, 2, 6].collect do |i|
+ bytes.slice!(0, i).pack('C*').unpack('H*')
+ end
+
+ bytes.join('-')
+ end
+
+ # Reads either a :public or :private key from disk, uses an
+ # optional passphrase to read the private key
+ def read_key(type, key=nil, passphrase=nil)
+ return key if key.nil?
+
+ raise "Could not find key #{key}" unless File.exist?(key)
+ raise "#{type} key file '#{key}' is empty" if File.zero?(key)
+
+ if type == :public
+ begin
+ key = OpenSSL::PKey::RSA.new(File.read(key))
+ rescue OpenSSL::PKey::RSAError
+ key = OpenSSL::X509::Certificate.new(File.read(key)).public_key
+ end
+
+ # Ruby < 1.9.3 had a bug where it does not correctly clear the
+ # queue of errors while reading a key. It tries various ways
+ # to read the key and each failing attempt pushes an error onto
+ # the queue. With pubkeys only the 3rd attempt pass leaving 2
+ # stale errors on the error queue.
+ #
+ # In 1.9.3 they fixed this by simply discarding the errors after
+ # every attempt. So we simulate this fix here for older rubies
+ # as without it we get SSL_read errors from the Stomp+TLS sessions
+ #
+ # We do this only on 1.8 relying on 1.9.3 to do the right thing
+ # and we do not support 1.9 less than 1.9.3
+ #
+ # See http://bugs.ruby-lang.org/issues/4550
+ OpenSSL.errors if Util.ruby_version =~ /^1.8/
+
+ return key
+ elsif type == :private
+ return OpenSSL::PKey::RSA.new(File.read(key), passphrase)
+ else
+ raise "Can only load :public or :private keys"
+ end
+ end
+
+ end
+end
--- /dev/null
+module MCollective
+ module Translatable
+ def t(msgid, default, args={})
+ Util.t(msgid, {:default => default}.merge(args))
+ end
+
+ def log_code(msgid, default, level, args={})
+ msg = "%s: %s" % [msgid, Util.t(msgid, {:default => default}.merge(args))]
+
+ Log.log(level, msg, File.basename(caller[1]))
+ end
+
+ def raise_code(msgid, default, level, args={})
+ exception = CodedError.new(msgid, default, level, args)
+ exception.set_backtrace caller
+
+ raise exception
+ end
+
+ def logexception(msgid, default, level, e, backtrace=false)
+ Log.logexception(msgid, level, e, backtrace)
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class UnixDaemon
+ # Daemonize the current process
+ def self.daemonize
+ fork do
+ Process.setsid
+ exit if fork
+ Dir.chdir('/tmp')
+ STDIN.reopen('/dev/null')
+ STDOUT.reopen('/dev/null', 'a')
+ STDERR.reopen('/dev/null', 'a')
+
+ yield
+ end
+ end
+
+ def self.daemonize_runner(pid=nil)
+ raise "The Unix Daemonizer can not be used on the Windows Platform" if Util.windows?
+
+ UnixDaemon.daemonize do
+ if pid
+ begin
+ File.open(pid, 'w') {|f| f.write(Process.pid) }
+ rescue Exception => e
+ end
+ end
+
+ begin
+ runner = Runner.new(nil)
+ runner.run
+ ensure
+ File.unlink(pid) if pid && File.exist?(pid)
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # Some basic utility helper methods useful to clients, agents, runner etc.
+ module Util
+ # Finds out if this MCollective has an agent by the name passed
+ #
+ # If the passed name starts with a / it's assumed to be regex
+ # and will use regex to match
+ def self.has_agent?(agent)
+ agent = Regexp.new(agent.gsub("\/", "")) if agent.match("^/")
+
+ if agent.is_a?(Regexp)
+ if Agents.agentlist.grep(agent).size > 0
+ return true
+ else
+ return false
+ end
+ else
+ return Agents.agentlist.include?(agent)
+ end
+
+ false
+ end
+
+ # On windows ^c can't interrupt the VM if its blocking on
+ # IO, so this sets up a dummy thread that sleeps and this
+ # will have the end result of being interruptable at least
+ # once a second. This is a common pattern found in Rails etc
+ def self.setup_windows_sleeper
+ Thread.new { loop { sleep 1 } } if Util.windows?
+ end
+
+ # Checks if this node has a configuration management class by parsing the
+ # a text file with just a list of classes, recipes, roles etc. This is
+ # ala the classes.txt from puppet.
+ #
+ # If the passed name starts with a / it's assumed to be regex
+ # and will use regex to match
+ def self.has_cf_class?(klass)
+ klass = Regexp.new(klass.gsub("\/", "")) if klass.match("^/")
+ cfile = Config.instance.classesfile
+
+ Log.debug("Looking for configuration management classes in #{cfile}")
+
+ begin
+ File.readlines(cfile).each do |k|
+ if klass.is_a?(Regexp)
+ return true if k.chomp.match(klass)
+ else
+ return true if k.chomp == klass
+ end
+ end
+ rescue Exception => e
+ Log.warn("Parsing classes file '#{cfile}' failed: #{e.class}: #{e}")
+ end
+
+ false
+ end
+
+ # Gets the value of a specific fact, mostly just a duplicate of MCollective::Facts.get_fact
+ # but it kind of goes with the other classes here
+ def self.get_fact(fact)
+ Facts.get_fact(fact)
+ end
+
+ # Compares fact == value,
+ #
+ # If the passed value starts with a / it's assumed to be regex
+ # and will use regex to match
+ def self.has_fact?(fact, value, operator)
+
+ Log.debug("Comparing #{fact} #{operator} #{value}")
+ Log.debug("where :fact = '#{fact}', :operator = '#{operator}', :value = '#{value}'")
+
+ fact = Facts[fact]
+ return false if fact.nil?
+
+ fact = fact.clone
+
+ if operator == '=~'
+ # to maintain backward compat we send the value
+ # as /.../ which is what 1.0.x needed. this strips
+ # off the /'s wich is what we need here
+ if value =~ /^\/(.+)\/$/
+ value = $1
+ end
+
+ return true if fact.match(Regexp.new(value))
+
+ elsif operator == "=="
+ return true if fact == value
+
+ elsif ['<=', '>=', '<', '>', '!='].include?(operator)
+ # Yuk - need to type cast, but to_i and to_f are overzealous
+ if value =~ /^[0-9]+$/ && fact =~ /^[0-9]+$/
+ fact = Integer(fact)
+ value = Integer(value)
+ elsif value =~ /^[0-9]+.[0-9]+$/ && fact =~ /^[0-9]+.[0-9]+$/
+ fact = Float(fact)
+ value = Float(value)
+ end
+
+ return true if eval("fact #{operator} value")
+ end
+
+ false
+ end
+
+ # Checks if the configured identity matches the one supplied
+ #
+ # If the passed name starts with a / it's assumed to be regex
+ # and will use regex to match
+ def self.has_identity?(identity)
+ identity = Regexp.new(identity.gsub("\/", "")) if identity.match("^/")
+
+ if identity.is_a?(Regexp)
+ return Config.instance.identity.match(identity)
+ else
+ return true if Config.instance.identity == identity
+ end
+
+ false
+ end
+
+ # Checks if the passed in filter is an empty one
+ def self.empty_filter?(filter)
+ filter == empty_filter || filter == {}
+ end
+
+ # Creates an empty filter
+ def self.empty_filter
+ {"fact" => [],
+ "cf_class" => [],
+ "agent" => [],
+ "identity" => [],
+ "compound" => []}
+ end
+
+ # Picks a config file defaults to ~/.mcollective
+ # else /etc/mcollective/client.cfg
+ def self.config_file_for_user
+ # expand_path is pretty lame, it relies on HOME environment
+ # which isnt't always there so just handling all exceptions
+ # here as cant find reverting to default
+ begin
+ config = File.expand_path("~/.mcollective")
+
+ unless File.readable?(config) && File.file?(config)
+ config = "/etc/mcollective/client.cfg"
+ end
+ rescue Exception => e
+ config = "/etc/mcollective/client.cfg"
+ end
+
+ return config
+ end
+
+ # Creates a standard options hash
+ def self.default_options
+ {:verbose => false,
+ :disctimeout => nil,
+ :timeout => 5,
+ :config => config_file_for_user,
+ :collective => nil,
+ :discovery_method => nil,
+ :discovery_options => Config.instance.default_discovery_options,
+ :filter => empty_filter}
+ end
+
+ def self.make_subscriptions(agent, type, collective=nil)
+ config = Config.instance
+
+ raise("Unknown target type #{type}") unless [:broadcast, :directed, :reply].include?(type)
+
+ if collective.nil?
+ config.collectives.map do |c|
+ {:agent => agent, :type => type, :collective => c}
+ end
+ else
+ raise("Unknown collective '#{collective}' known collectives are '#{config.collectives.join ', '}'") unless config.collectives.include?(collective)
+
+ [{:agent => agent, :type => type, :collective => collective}]
+ end
+ end
+
+ # Helper to subscribe to a topic on multiple collectives or just one
+ def self.subscribe(targets)
+ connection = PluginManager["connector_plugin"]
+
+ targets = [targets].flatten
+
+ targets.each do |target|
+ connection.subscribe(target[:agent], target[:type], target[:collective])
+ end
+ end
+
+ # Helper to unsubscribe to a topic on multiple collectives or just one
+ def self.unsubscribe(targets)
+ connection = PluginManager["connector_plugin"]
+
+ targets = [targets].flatten
+
+ targets.each do |target|
+ connection.unsubscribe(target[:agent], target[:type], target[:collective])
+ end
+ end
+
+ # Wrapper around PluginManager.loadclass
+ def self.loadclass(klass)
+ PluginManager.loadclass(klass)
+ end
+
+ # Parse a fact filter string like foo=bar into the tuple hash thats needed
+ def self.parse_fact_string(fact)
+ if fact =~ /^([^ ]+?)[ ]*=>[ ]*(.+)/
+ return {:fact => $1, :value => $2, :operator => '>=' }
+ elsif fact =~ /^([^ ]+?)[ ]*=<[ ]*(.+)/
+ return {:fact => $1, :value => $2, :operator => '<=' }
+ elsif fact =~ /^([^ ]+?)[ ]*(<=|>=|<|>|!=|==|=~)[ ]*(.+)/
+ return {:fact => $1, :value => $3, :operator => $2 }
+ elsif fact =~ /^(.+?)[ ]*=[ ]*\/(.+)\/$/
+ return {:fact => $1, :value => "/#{$2}/", :operator => '=~' }
+ elsif fact =~ /^([^= ]+?)[ ]*=[ ]*(.+)/
+ return {:fact => $1, :value => $2, :operator => '==' }
+ else
+ raise "Could not parse fact #{fact} it does not appear to be in a valid format"
+ end
+ end
+
+ # Escapes a string so it's safe to use in system() or backticks
+ #
+ # Taken from Shellwords#shellescape since it's only in a few ruby versions
+ def self.shellescape(str)
+ return "''" if str.empty?
+
+ str = str.dup
+
+ # Process as a single byte sequence because not all shell
+ # implementations are multibyte aware.
+ str.gsub!(/([^A-Za-z0-9_\-.,:\/@\n])/n, "\\\\\\1")
+
+ # A LF cannot be escaped with a backslash because a backslash + LF
+ # combo is regarded as line continuation and simply ignored.
+ str.gsub!(/\n/, "'\n'")
+
+ return str
+ end
+
+ def self.windows?
+ !!(RbConfig::CONFIG['host_os'] =~ /mswin|win32|dos|mingw|cygwin/i)
+ end
+
+ # Return color codes, if the config color= option is false
+ # just return a empty string
+ def self.color(code)
+ colorize = Config.instance.color
+
+ colors = {:red => "\e[31m",
+ :green => "\e[32m",
+ :yellow => "\e[33m",
+ :cyan => "\e[36m",
+ :bold => "\e[1m",
+ :reset => "\e[0m"}
+
+ if colorize
+ return colors[code] || ""
+ else
+ return ""
+ end
+ end
+
+ # Helper to return a string in specific color
+ def self.colorize(code, msg)
+ "%s%s%s" % [ color(code), msg, color(:reset) ]
+ end
+
+ # Returns the current ruby version as per RUBY_VERSION, mostly
+ # doing this here to aid testing
+ def self.ruby_version
+ RUBY_VERSION
+ end
+
+ def self.mcollective_version
+ MCollective::VERSION
+ end
+
+ # Returns an aligned_string of text relative to the size of the terminal
+ # window. If a line in the string exceeds the width of the terminal window
+ # the line will be chopped off at the whitespace chacter closest to the
+ # end of the line and prepended to the next line, keeping all indentation.
+ #
+ # The terminal size is detected by default, but custom line widths can
+ # passed. All strings will also be left aligned with 5 whitespace characters
+ # by default.
+ def self.align_text(text, console_cols = nil, preamble = 5)
+ unless console_cols
+ console_cols = terminal_dimensions[0]
+
+ # if unknown size we default to the typical unix default
+ console_cols = 80 if console_cols == 0
+ end
+
+ console_cols -= preamble
+
+ # Return unaligned text if console window is too small
+ return text if console_cols <= 0
+
+ # If console is 0 this implies unknown so we assume the common
+ # minimal unix configuration of 80 characters
+ console_cols = 80 if console_cols <= 0
+
+ text = text.split("\n")
+ piece = ''
+ whitespace = 0
+
+ text.each_with_index do |line, i|
+ whitespace = 0
+
+ while whitespace < line.length && line[whitespace].chr == ' '
+ whitespace += 1
+ end
+
+ # If the current line is empty, indent it so that a snippet
+ # from the previous line is aligned correctly.
+ if line == ""
+ line = (" " * whitespace)
+ end
+
+ # If text was snipped from the previous line, prepend it to the
+ # current line after any current indentation.
+ if piece != ''
+ # Reset whitespaces to 0 if there are more whitespaces than there are
+ # console columns
+ whitespace = 0 if whitespace >= console_cols
+
+ # If the current line is empty and being prepended to, create a new
+ # empty line in the text so that formatting is preserved.
+ if text[i + 1] && line == (" " * whitespace)
+ text.insert(i + 1, "")
+ end
+
+ # Add the snipped text to the current line
+ line.insert(whitespace, "#{piece} ")
+ end
+
+ piece = ''
+
+ # Compare the line length to the allowed line length.
+ # If it exceeds it, snip the offending text from the line
+ # and store it so that it can be prepended to the next line.
+ if line.length > (console_cols + preamble)
+ reverse = console_cols
+
+ while line[reverse].chr != ' '
+ reverse -= 1
+ end
+
+ piece = line.slice!(reverse, (line.length - 1)).lstrip
+ end
+
+ # If a snippet exists when all the columns in the text have been
+ # updated, create a new line and append the snippet to it, using
+ # the same left alignment as the last line in the text.
+ if piece != '' && text[i+1].nil?
+ text[i+1] = "#{' ' * (whitespace)}#{piece}"
+ piece = ''
+ end
+
+ # Add the preamble to the line and add it to the text
+ line = ((' ' * preamble) + line)
+ text[i] = line
+ end
+
+ text.join("\n")
+ end
+
+ # Figures out the columns and lines of the current tty
+ #
+ # Returns [0, 0] if it can't figure it out or if you're
+ # not running on a tty
+ def self.terminal_dimensions(stdout = STDOUT, environment = ENV)
+ return [0, 0] unless stdout.tty?
+
+ return [80, 40] if Util.windows?
+
+ if environment["COLUMNS"] && environment["LINES"]
+ return [environment["COLUMNS"].to_i, environment["LINES"].to_i]
+
+ elsif environment["TERM"] && command_in_path?("tput")
+ return [`tput cols`.to_i, `tput lines`.to_i]
+
+ elsif command_in_path?('stty')
+ return `stty size`.scan(/\d+/).map {|s| s.to_i }
+ else
+ return [0, 0]
+ end
+ rescue
+ [0, 0]
+ end
+
+ # Checks in PATH returns true if the command is found
+ def self.command_in_path?(command)
+ found = ENV["PATH"].split(File::PATH_SEPARATOR).map do |p|
+ File.exist?(File.join(p, command))
+ end
+
+ found.include?(true)
+ end
+
+ # compare two software versions as commonly found in
+ # package versions.
+ #
+ # returns 0 if a == b
+ # returns -1 if a < b
+ # returns 1 if a > b
+ #
+ # Code originally from Puppet but refactored to a more
+ # ruby style that fits in better with this code base
+ def self.versioncmp(version_a, version_b)
+ vre = /[-.]|\d+|[^-.\d]+/
+ ax = version_a.scan(vre)
+ bx = version_b.scan(vre)
+
+ until ax.empty? || bx.empty?
+ a = ax.shift
+ b = bx.shift
+
+ next if a == b
+ next if a == '-' && b == '-'
+ return -1 if a == '-'
+ return 1 if b == '-'
+ next if a == '.' && b == '.'
+ return -1 if a == '.'
+ return 1 if b == '.'
+
+ if a =~ /^[^0]\d+$/ && b =~ /^[^0]\d+$/
+ return Integer(a) <=> Integer(b)
+ else
+ return a.upcase <=> b.upcase
+ end
+ end
+
+ version_a <=> version_b
+ end
+
+ # we should really use Pathname#absolute? but it's not in all the
+ # ruby versions we support and it comes down to roughly this
+ def self.absolute_path?(path, separator=File::SEPARATOR, alt_separator=File::ALT_SEPARATOR)
+ if alt_separator
+ path_matcher = /^[#{Regexp.quote alt_separator}#{Regexp.quote separator}]/
+ else
+ path_matcher = /^#{Regexp.quote separator}/
+ end
+
+ !!path.match(path_matcher)
+ end
+
+ # Looks up and interprolate the hash values into a i18n string
+ def self.t(msgid, args={})
+ if msgid.is_a?(Symbol)
+ I18n.t("%s.pattern" % msgid, args)
+ else
+ I18n.t(msgid, args)
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Validator
+ @last_load = nil
+
+ # Loads the validator plugins. Validators will only be loaded every 5 minutes
+ def self.load_validators
+ if load_validators?
+ @last_load = Time.now.to_i
+ PluginManager.find_and_load("validator")
+ end
+ end
+
+ # Returns and instance of the Plugin class from which objects can be created.
+ # Valid plugin names are
+ # :valplugin
+ # "valplugin"
+ # "ValpluginValidator"
+ def self.[](klass)
+ if klass.is_a?(Symbol)
+ klass = validator_class(klass)
+ elsif !(klass.match(/.*Validator$/))
+ klass = validator_class(klass)
+ end
+
+ const_get(klass)
+ end
+
+ # Allows validation plugins to be called like module methods : Validator.validate()
+ def self.method_missing(method, *args, &block)
+ if has_validator?(method)
+ validator = Validator[method].validate(*args)
+ else
+ raise ValidatorError, "Unknown validator: '#{method}'."
+ end
+ end
+
+ def self.has_validator?(validator)
+ const_defined?(validator_class(validator))
+ end
+
+ def self.validator_class(validator)
+ "#{validator.to_s.capitalize}Validator"
+ end
+
+ def self.load_validators?
+ return true if @last_load.nil?
+
+ (@last_load - Time.now.to_i) > 300
+ end
+
+ # Generic validate method that will call the correct validator
+ # plugin based on the type of the validation parameter
+ def self.validate(validator, validation)
+ Validator.load_validators
+
+ begin
+ if [:integer, :boolean, :float, :number, :string].include?(validation)
+ Validator.typecheck(validator, validation)
+
+ else
+ case validation
+ when Regexp,String
+ Validator.regex(validator, validation)
+
+ when Symbol
+ Validator.send(validation, validator)
+
+ when Array
+ Validator.array(validator, validation)
+
+ when Class
+ Validator.typecheck(validator, validation)
+ end
+ end
+ rescue => e
+ raise ValidatorError, e.to_s
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ # Simple module to manage vendored code.
+ #
+ # To vendor a library simply download its whole git repo or untar
+ # into vendor/libraryname and create a load_libraryname.rb file
+ # to add its libdir into the $:.
+ #
+ # Once you have that file, add a require line in vendor/require_vendored.rb
+ # which will run after all the load_* files.
+ #
+ # The intention is to not change vendored libraries and to eventually
+ # make adding them in optional so that distros can simply adjust their
+ # packaging to exclude this directory and the various load_xxx.rb scripts
+ # if they wish to install these gems as native packages.
+ class Vendor
+ class << self
+ def vendor_dir
+ File.join([File.dirname(File.expand_path(__FILE__)), "vendor"])
+ end
+
+ def load_entry(entry)
+ Log.debug("Loading vendored #{$1}")
+ load "#{vendor_dir}/#{entry}"
+ end
+
+ def require_libs
+ require 'mcollective/vendor/require_vendored'
+ end
+
+ def load_vendored
+ Dir.entries(vendor_dir).each do |entry|
+ if entry.match(/load_(\w+?)\.rb$/)
+ load_entry entry
+ end
+ end
+
+ require_libs
+ end
+ end
+ end
+end
--- /dev/null
+.DS_Store
+test/rails/fixtures
+nbproject/
+vendor/**/*
+*.swp
+pkg
+.bundle
+.rvmrc
--- /dev/null
+rvm:
+ - 1.8.7
+ - 1.9.2
+ - 1.9.3
+ - ree
+ - rbx
+ - jruby
--- /dev/null
+h1. Changelog
+
+h2. 0.5.0
+
+* "Extract Backend::ActiveRecord to a separate gem":https://github.com/svenfuchs/i18n/commit/197dacebad356b910d69fa69a719c2ad10cf49e6 (see "i18n-active_record":https://github.com/svenfuchs/i18n-active_record)
+* "Improve exception handling":https://github.com/svenfuchs/i18n/commit/2913ff9a7544f223f60e7d7b32c2a0e1af89812b (deprectates I18n.default_exception_handler)
+* "Change MissingTranslationData message to 'translation missing: foo.bar'":https://github.com/svenfuchs/i18n/commit/68fdfe47952325411afe5942e971ce10b2bdf900
+* "Expose MissingTranslationsData#keys method":https://github.com/svenfuchs/i18n/commit/3a37a389ecaac9670355b334e23e775549ee9822
+* "Improve Cascade#lookup (add default options)":https://github.com/svenfuchs/i18n/commit/0b9a1f2058a2be9543106cc19d08071c359511e1
+* "Finally remove deprecated interpolation syntax":https://github.com/svenfuchs/i18n/commit/2d43846d2b2a2e596f30fa58ea1c9ddb2243bb64
+
+h2. 0.4.2 (2010-10-26)
+
+* "Improve UTF8 handling":http://github.com/svenfuchs/i18n/commit/e8d5820a3b08eeca28de1a2b9c8a6ad2b9e6476c
+* "Expose I18n::VERSION":http://github.com/svenfuchs/i18n/commit/b832037bac94c7144f45f3ff5e3b4e4089781726
+* "Better deprecation output":http://github.com/svenfuchs/i18n/commit/2bee924464b8a9c33d3d7852eb1c8423aa38cc25
+
+h2. 0.4.1 (2010-06-05)
+
+* "Fix interpolation failure on Ruby 1.9":http://github.com/svenfuchs/i18n/commit/8d45bedb11c4136c00e853d104b00a8e67ec4894
+
+h2. 0.4.0 (2010-05-27)
+
+* "The localization proc also receives the object as option":http://github.com/svenfuchs/i18n/commit/4a8cd9fa660daaa3078e24c5851353ca377d9213
+
+h2. 0.4.0.beta1 (2010-05-03)
+
+* "Renamed Fast backend to Memoize backend":http://github.com/svenfuchs/i18n/commit/f7f7dc12c00a19d3876223771e14f8671ff313cd
+
+* "Deprecate {{}} as interpolation syntax":http://github.com/svenfuchs/i18n/commit/8894ee521ef5788c415b625a6daf522af4c416e0
+
+* "Allow nil translation to be stored again":http://github.com/svenfuchs/i18n/commit/f2074f1e82d10c2e9a801c8cc2f2a0c7c30703ba
+
+h2. 0.4.0.beta (2010-04-30)
+
+* "Added a KeyValue backend":http://github.com/svenfuchs/i18n/commit/28ca5f53ade7f545f8c0804e93564d4686b416a4
+
+* "Added transliteration support":http://github.com/svenfuchs/i18n/commit/928fdb4794959e779e90f360eb01ba043672d8d5
+
+* "Create Flatten backend module to aid handling flatten translations":http://github.com/svenfuchs/i18n/commit/2ec9d6998aa8facd7b15a3ef47a96cf2471cd8a1
+
+* "Decouple the external separator (used when storing translations) from the internal separator in Fast and ActiveRecord backends":http://github.com/svenfuchs/i18n/commit/274cb4daa0ca5e3b2bd23b45eb7f9fc58f75a79d
+
+h2. 0.3.7 (2010-04-17)
+
+* "Speed up I18n.normalize_keys by caching reused normalizations and producing less garbage":http://github.com/svenfuchs/i18n/commit/819dac0fea9c29e6545801aa107e63e355728cd4
+
+h2. 0.3.6 (2010-03-23)
+
+* "Move gettext po parser to lib":http://github.com/svenfuchs/i18n/commit/b2f038663b55727ac2327e6f07a46ba5d69d600c
+
+* "Move I18n configuration to I18n.config":http://github.com/svenfuchs/i18n/commit/4a7baea86663ead8c681008c3e80a622f0546b07
+
+h2. 0.3.5 (2010-02-26)
+
+* "Delegate I18n.normalize_translation_keys to I18n.normalize_keys and deprecate
+the former":http://github.com/svenfuchs/i18n/commit/7284b04d5f5dd9679cb68875515cdd0cdfc96fef
+
+h2. 0.3.4 (2010-02-25)
+
+* "Rename I18n.normalize_translation_keys to I18n.normalize_keys and finally make it public":http://github.com/svenfuchs/i18n/commit/20b05fe5802df6c90fb70a4e3760b2b851b791b3
+
+* "Added CLDR supoprt":http://github.com/svenfuchs/i18n/commit/860eadf671a231e7f5dffb1bb27fa318ff7a8786
+
+h2. 0.3.3 (2009-12-29)
+
+* "Use lib/i18n/version":http://github.com/svenfuchs/i18n/commit/ff426c8e7a2438b814cb303adadec292dacb752e
+
+* "Added a benchmark suite":http://github.com/svenfuchs/i18n/commit/f9b5b9b113097724638bdab96862ffa404e67e70
+
+* "Ensure links can be handled recursively":http://github.com/svenfuchs/i18n/commit/2c50bd209f3fc24fe9dfa694c81be64340f09b7d
+
+* "Make sure we can lookup false values as translation data":http://github.com/svenfuchs/i18n/commit/561c82ba4b8921d03bfdf56cb2d0c2f287629001
+
+* "Added Fast backend module":http://github.com/svenfuchs/i18n/commit/bd2f09f0a251ca793b0e8ecc7e32177a2f091c23
+
+* "Added InterpolationCompiler backend module":http://github.com/svenfuchs/i18n/commit/91810887d1abfb28996a9183bc9004678290d28b
+
+h2. 0.3.2 (2009-12-12)
+
+* "Added Cascade backend":http://github.com/svenfuchs/i18n/commit/8009aef293e9ef8564c9005090d8380feabcaf6f
+
+h2. 0.3.1 (2009-12-11)
+
+* "Add PoParser to gemspec":http://github.com/svenfuchs/i18n/commit/d6b2763f39c932f66adb039b96882a472f883c51
+* "Enable custom separators for ActiveRecord backend":http://github.com/svenfuchs/i18n/commit/9341d3fcfc951cc31807ba672d2b5d90909ef3e5
+* "Pass interpolation values to interpolation procs":http://github.com/svenfuchs/i18n/commit/39c2ed8fbad645671cd5520ce7ad0aeefe2b0cca
+* "Fix that ngettext supports keys with dots":http://github.com/svenfuchs/i18n/commit/7362a43c34364d500de8899cfcca6bf1a5e6d1c8
+
+h2. 0.3.0 (2009-11-30)
+
+* "Gettext backend and helpers":http://github.com/svenfuchs/i18n/commit/35a1740d2f10b808548af352006950da4017e374
+* "Metadata module":http://github.com/svenfuchs/i18n/commit/2677208555179b36fcbe958c0e8bc642cf5bc020
+* "Basic ActiveRecord backend":http://github.com/svenfuchs/i18n/commit/786632d0b42de423ecf0969622efc87f1691e2a2
+* "Set encoding to UTF8 for all files":http://github.com/svenfuchs/i18n/commit/9be3d4a311b5bf583eec5d39986176cc40c112f2
+* "Chain backend":http://github.com/svenfuchs/i18n/commit/08259ffb88b3005403648d77bc1cbca0b92f3cf5
+* "Backend/cache implementation":http://github.com/svenfuchs/i18n/commit/e7bf15351cd2e27f5972eb40e65a5dd6f4a0feed
+* "Pluralization module":http://github.com/svenfuchs/i18n/commit/9ca4c9ed52d4706566a6abeb2d78722dcc5d4764
+* "add and adapt Globalize2 fallback implementation":http://github.com/svenfuchs/i18n/commit/1b37a303b27d6222b17162804b06323e5628768f
+* "move Simple backend implementation to a Base backend class and extend Simple from Base.":http://github.com/svenfuchs/i18n/commit/32ddc80a04e6aa247f6d6613bde7f78c73396cb4
+
+h2. 0.2.0 (2009-07-12)
+
+* "Allow using Ruby 1.9 syntax for string interpolation (API addition)":http://github.com/svenfuchs/i18n/commit/c6e0b06d512f2af57199a843a1d8a40241b32861
+* "Allow configuring the default scope separator, allow to pass a custom scope separator(API addition)":http://github.com/svenfuchs/i18n/commit/5b75bfbc348061adc11e3790187a187275bfd471 (e.g. I18n.t(:'foo|bar', :separator => '|')
+* "Pass :format option to #translate for #localize more useful lambda support":http://github.com/svenfuchs/i18n/commit/e277711b3c844fe7589b8d3f9af0f7d1b969a273
+* "Refactor Simple backend #resolve to #default and #resolve for more consistency. Now allows to pass lambdas as defaults and re-resolve Symbols":http://github.com/svenfuchs/i18n/commit/8c4ce3d923ce5fa73e973fe28217e18165549aba
+* "Add lambda support to #translate (API addition)":http://github.com/svenfuchs/i18n/commit/c90e62d8f7d3d5b78f34cfe328d871b58884f115
+* "Add lambda support to #localize (API addition)":http://github.com/svenfuchs/i18n/commit/9d390afcf33f3f469bb95e6888147152f6cc7442
+
+h2. 0.1.3 (2009-02-27)
+
+* "Remove unnecessary string encoding handling in the i18n simple backend which made the backend break on Ruby 1.9":http://github.com/svenfuchs/i18n/commit/4c3a970783861a94f2e89f46714fb3434e4f4f8d
+
+h2. 0.1.2 (2009-01-09)
+
+* "added #available_locales (returns an array of locales for which translations are available)":http://github.com/svenfuchs/i18n/commit/411f8fe7c8f3f89e9b6b921fa62ed66cb92f3af4
+* "flatten load_path before using it so that a nested array of paths won't throw up":http://github.com/svenfuchs/i18n/commit/d473a068a2b90aba98135deb225d6eb6d8104d70
+
+h2. 0.1.1 (2008-11-20)
+
+* "Use :'en' as a default locale (in favor of :'en-US')":http://github.com/svenfuchs/i18n/commit/c4b10b246aecf7da78cb2568dd0d2ab7e6b8a230
+* "Add #reload! to Simple backend":http://github.com/svenfuchs/i18n/commit/36dd2bd9973b9e1559728749a9daafa44693e964
+
+h2. 0.1.0 (2008-10-25)
+
+* "Fix Simple backend to distinguish false from nil values":http://github.com/svenfuchs/i18n/commit/39d9a47da14b5f3ba126af48923af8c30e135166
+* "Add #load_path to public api, add initialize to simple backend and remove #load_translations from public api":http://github.com/svenfuchs/i18n/commit/c4c5649e6bc8f020f1aaf5a5470bde048e22c82d
+* "Speed up Backend::Simple#interpolate":http://github.com/svenfuchs/i18n/commit/9e1ac6bf8833304e036323ec9932b9f33c468a35
+* "Remove #populate and #store_translations from public API":http://github.com/svenfuchs/i18n/commit/f4e514a80be7feb509f66824ee311905e2940900
+* "Use :other instead of :many as a plural key":http://github.com/svenfuchs/i18n/commit/0f8f20a2552bf6a2aa758d8fdd62a7154e4a1bf6
+* "Use a class instead of a module for Simple backend":http://github.com/svenfuchs/i18n/commit/08f051aa61320c17debde24a83268bc74e33b995
+* "Make Simple backend #interpolate deal with non-ASCII string encodings":http://github.com/svenfuchs/i18n/commit/d84a3f3f55543c084d5dc5d1fed613b8df148789
+* "Fix default arrays of non-existant keys returning the default array":http://github.com/svenfuchs/i18n/commit/6c04ca86c87f97dc78f07c2a4023644e5ba8b839
+
+h2. Initial implementation (June/July 2008)
+
+Initial implementation by "Sven Fuchs":http://www.workingwithrails.com/person/9963-sven-fuchs based on previous discussion/consensus of the rails-i18n team (alphabetical order) and many others:
+
+* "Matt Aimonetti":http://railsontherun.com
+* "Sven Fuchs":http://www.workingwithrails.com/person/9963-sven-fuchs
+* "Joshua Harvey":http://www.workingwithrails.com/person/759-joshua-harvey
+* "Saimon Moore":http://saimonmoore.net
+* "Stephan Soller":http://www.arkanis-development.de
+
+h2. More information
+
+* "Homepage":http://rails-i18n.org
+* "Wiki":http://rails-i18n.org/wiki
+* "Mailinglist":http://groups.google.com/group/rails-i18n
+* "About the project/history":http://www.artweb-design.de/2008/7/18/finally-ruby-on-rails-gets-internationalized
+* "Initial API Intro":http://www.artweb-design.de/2008/7/18/the-ruby-on-rails-i18n-core-api
--- /dev/null
+Copyright (c) 2008 The Ruby I18n team
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
\ No newline at end of file
--- /dev/null
+h1. Ruby I18n
+
+!https://secure.travis-ci.org/svenfuchs/i18n.png?branch=master(Build Status)!:http://travis-ci.org/svenfuchs/i18n
+
+Ruby Internationalization and localization solution.
+
+Features:
+
+* translation and localization
+* interpolation of values to translations (Ruby 1.9 compatible syntax)
+* pluralization (CLDR compatible)
+* customizable transliteration to ASCII
+* flexible defaults
+* bulk lookup
+* lambdas as translation data
+* custom key/scope separator
+* custom exception handlers
+* extensible architecture with a swappable backend
+
+Pluggable features:
+
+* Cache
+* Pluralization: lambda pluralizers stored as translation data
+* Locale fallbacks, RFC4647 compliant (optionally: RFC4646 locale validation)
+* Gettext support
+* Translation metadata
+
+Alternative backends:
+
+* Chain
+* ActiveRecord (optionally: ActiveRecord::Missing and ActiveRecord::StoreProcs)
+* KeyValue (uses active_support/json and cannot store procs)
+
+For more information and lots of resources see: "http://ruby-i18n.org/wiki":http://ruby-i18n.org/wiki
+
+h2. Installation
+
+gem install i18n
+
+h4. Rails version warning
+
+On Rails < 2.3.6 the method I18n.localize will fail with MissingInterpolationArgument (issue "20":http://github.com/svenfuchs/i18n/issues/issue/20). Upgrade to Rails 2.3.6 or higher (2.3.8 preferably) is recommended.
+
+h3. Installation on Rails < 2.3.5 (deprecated)
+
+Up to version 2.3.4 Rails will not accept i18n gems > 0.1.3. There is an unpacked
+gem inside of active_support/lib/vendor which gets loaded unless gem 'i18n', '~> 0.1.3'.
+This requirement is relaxed in "6da03653":http://github.com/rails/rails/commit/6da03653
+
+The new i18n gem can be loaded from vendor/plugins like this:
+
+<pre>
+ def reload_i18n!
+ raise "Move to i18n version 0.2.0 or greater" if Rails.version > "2.3.4"
+
+ $:.grep(/i18n/).each { |path| $:.delete(path) }
+ I18n::Backend.send :remove_const, "Simple"
+ $: << Rails.root.join('vendor', 'plugins', 'i18n', 'lib').to_s
+ end
+</pre>
+
+Then you can `reload_i18n!` inside an i18n initializer.
+
+h2. Tests
+
+You can run tests both with
+
+* `rake test` or just `rake`
+* run any test file directly, e.g. `ruby -Ilib -Itest test/api/simple_test.rb`
+* run all tests with `ruby -Ilib -Itest test/all.rb`
+
+You can run all tests against all Gemfiles with
+
+* `ruby test/run_all.rb`
+
+The structure of the test suite is a bit unusual as it uses modules to reuse
+particular tests in different test cases.
+
+The reason for this is that we need to enforce the I18n API across various
+combinations of extensions. E.g. the Simple backend alone needs to support
+the same API as any combination of feature and/or optimization modules included
+to the Simple backend. We test this by reusing the same API defition (implemented
+as test methods) in test cases with different setups.
+
+You can find the test cases that enforce the API in test/api. And you can find
+the API definition test methods in test/api/tests.
+
+All other test cases (e.g. as defined in test/backend, test/core\_ext) etc.
+follow the usual test setup and should be easy to grok.
+
+h2. Authors
+
+* "Sven Fuchs":http://www.artweb-design.de
+* "Joshua Harvey":http://www.workingwithrails.com/person/759-joshua-harvey
+* "Stephan Soller":http://www.arkanis-development.de
+* "Saimon Moore":http://saimonmoore.net
+* "Matt Aimonetti":http://railsontherun.com
+
+h2. Contributors
+
+http://github.com/svenfuchs/i18n/contributors
+
+h2. License
+
+MIT License. See the included MIT-LICENSE file.
--- /dev/null
+require 'rake/testtask'
+
+task :default => [:test]
+
+Rake::TestTask.new(:test) do |t|
+ t.libs << 'lib'
+ t.libs << 'test'
+ t.pattern = "#{File.dirname(__FILE__)}/test/all.rb"
+ t.verbose = true
+ t.warning = true
+end
+Rake::Task['test'].comment = "Run all i18n tests"
+
+# require "rake/gempackagetask"
+# require "rake/clean"
+# CLEAN << "pkg" << "doc" << "coverage" << ".yardoc"
+#
+# Rake::GemPackageTask.new(eval(File.read("i18n.gemspec"))) { |pkg| }
+#
+# begin
+# require "yard"
+# YARD::Rake::YardocTask.new do |t|
+# t.options = ["--output-dir=doc"]
+# t.options << "--files" << ["CHANGELOG.textile", "contributors.txt", "MIT-LICENSE"].join(",")
+# end
+# rescue LoadError
+# end
--- /dev/null
+require 'i18n/version'
+require 'i18n/exceptions'
+require 'i18n/interpolate/ruby'
+
+module I18n
+ autoload :Backend, 'i18n/backend'
+ autoload :Config, 'i18n/config'
+ autoload :Gettext, 'i18n/gettext'
+ autoload :Locale, 'i18n/locale'
+ autoload :Tests, 'i18n/tests'
+
+ RESERVED_KEYS = [:scope, :default, :separator, :resolve, :object, :fallback, :format, :cascade, :throw, :raise, :rescue_format]
+ RESERVED_KEYS_PATTERN = /%\{(#{RESERVED_KEYS.join("|")})\}/
+
+ extend Module.new {
+ # Gets I18n configuration object.
+ def config
+ Thread.current[:i18n_config] ||= I18n::Config.new
+ end
+
+ # Sets I18n configuration object.
+ def config=(value)
+ Thread.current[:i18n_config] = value
+ end
+
+ # Write methods which delegates to the configuration object
+ %w(locale backend default_locale available_locales default_separator
+ exception_handler load_path).each do |method|
+ module_eval <<-DELEGATORS, __FILE__, __LINE__ + 1
+ def #{method}
+ config.#{method}
+ end
+
+ def #{method}=(value)
+ config.#{method} = (value)
+ end
+ DELEGATORS
+ end
+
+ # Tells the backend to reload translations. Used in situations like the
+ # Rails development environment. Backends can implement whatever strategy
+ # is useful.
+ def reload!
+ config.backend.reload!
+ end
+
+ # Translates, pluralizes and interpolates a given key using a given locale,
+ # scope, and default, as well as interpolation values.
+ #
+ # *LOOKUP*
+ #
+ # Translation data is organized as a nested hash using the upper-level keys
+ # as namespaces. <em>E.g.</em>, ActionView ships with the translation:
+ # <tt>:date => {:formats => {:short => "%b %d"}}</tt>.
+ #
+ # Translations can be looked up at any level of this hash using the key argument
+ # and the scope option. <em>E.g.</em>, in this example <tt>I18n.t :date</tt>
+ # returns the whole translations hash <tt>{:formats => {:short => "%b %d"}}</tt>.
+ #
+ # Key can be either a single key or a dot-separated key (both Strings and Symbols
+ # work). <em>E.g.</em>, the short format can be looked up using both:
+ # I18n.t 'date.formats.short'
+ # I18n.t :'date.formats.short'
+ #
+ # Scope can be either a single key, a dot-separated key or an array of keys
+ # or dot-separated keys. Keys and scopes can be combined freely. So these
+ # examples will all look up the same short date format:
+ # I18n.t 'date.formats.short'
+ # I18n.t 'formats.short', :scope => 'date'
+ # I18n.t 'short', :scope => 'date.formats'
+ # I18n.t 'short', :scope => %w(date formats)
+ #
+ # *INTERPOLATION*
+ #
+ # Translations can contain interpolation variables which will be replaced by
+ # values passed to #translate as part of the options hash, with the keys matching
+ # the interpolation variable names.
+ #
+ # <em>E.g.</em>, with a translation <tt>:foo => "foo %{bar}"</tt> the option
+ # value for the key +bar+ will be interpolated into the translation:
+ # I18n.t :foo, :bar => 'baz' # => 'foo baz'
+ #
+ # *PLURALIZATION*
+ #
+ # Translation data can contain pluralized translations. Pluralized translations
+ # are arrays of singluar/plural versions of translations like <tt>['Foo', 'Foos']</tt>.
+ #
+ # Note that <tt>I18n::Backend::Simple</tt> only supports an algorithm for English
+ # pluralization rules. Other algorithms can be supported by custom backends.
+ #
+ # This returns the singular version of a pluralized translation:
+ # I18n.t :foo, :count => 1 # => 'Foo'
+ #
+ # These both return the plural version of a pluralized translation:
+ # I18n.t :foo, :count => 0 # => 'Foos'
+ # I18n.t :foo, :count => 2 # => 'Foos'
+ #
+ # The <tt>:count</tt> option can be used both for pluralization and interpolation.
+ # <em>E.g.</em>, with the translation
+ # <tt>:foo => ['%{count} foo', '%{count} foos']</tt>, count will
+ # be interpolated to the pluralized translation:
+ # I18n.t :foo, :count => 1 # => '1 foo'
+ #
+ # *DEFAULTS*
+ #
+ # This returns the translation for <tt>:foo</tt> or <tt>default</tt> if no translation was found:
+ # I18n.t :foo, :default => 'default'
+ #
+ # This returns the translation for <tt>:foo</tt> or the translation for <tt>:bar</tt> if no
+ # translation for <tt>:foo</tt> was found:
+ # I18n.t :foo, :default => :bar
+ #
+ # Returns the translation for <tt>:foo</tt> or the translation for <tt>:bar</tt>
+ # or <tt>default</tt> if no translations for <tt>:foo</tt> and <tt>:bar</tt> were found.
+ # I18n.t :foo, :default => [:bar, 'default']
+ #
+ # *BULK LOOKUP*
+ #
+ # This returns an array with the translations for <tt>:foo</tt> and <tt>:bar</tt>.
+ # I18n.t [:foo, :bar]
+ #
+ # Can be used with dot-separated nested keys:
+ # I18n.t [:'baz.foo', :'baz.bar']
+ #
+ # Which is the same as using a scope option:
+ # I18n.t [:foo, :bar], :scope => :baz
+ #
+ # *LAMBDAS*
+ #
+ # Both translations and defaults can be given as Ruby lambdas. Lambdas will be
+ # called and passed the key and options.
+ #
+ # E.g. assuming the key <tt>:salutation</tt> resolves to:
+ # lambda { |key, options| options[:gender] == 'm' ? "Mr. %{options[:name]}" : "Mrs. %{options[:name]}" }
+ #
+ # Then <tt>I18n.t(:salutation, :gender => 'w', :name => 'Smith') will result in "Mrs. Smith".
+ #
+ # It is recommended to use/implement lambdas in an "idempotent" way. E.g. when
+ # a cache layer is put in front of I18n.translate it will generate a cache key
+ # from the argument values passed to #translate. Therefor your lambdas should
+ # always return the same translations/values per unique combination of argument
+ # values.
+ def translate(*args)
+ options = args.last.is_a?(Hash) ? args.pop : {}
+ key = args.shift
+ backend = config.backend
+ locale = options.delete(:locale) || config.locale
+ handling = options.delete(:throw) && :throw || options.delete(:raise) && :raise # TODO deprecate :raise
+
+ raise I18n::ArgumentError if key.is_a?(String) && key.empty?
+
+ result = catch(:exception) do
+ if key.is_a?(Array)
+ key.map { |k| backend.translate(locale, k, options) }
+ else
+ backend.translate(locale, key, options)
+ end
+ end
+ result.is_a?(MissingTranslation) ? handle_exception(handling, result, locale, key, options) : result
+ end
+ alias :t :translate
+
+ # Wrapper for <tt>translate</tt> that adds <tt>:raise => true</tt>. With
+ # this option, if no translation is found, it will raise <tt>I18n::MissingTranslationData</tt>
+ def translate!(key, options={})
+ translate(key, options.merge(:raise => true))
+ end
+ alias :t! :translate!
+
+ # Transliterates UTF-8 characters to ASCII. By default this method will
+ # transliterate only Latin strings to an ASCII approximation:
+ #
+ # I18n.transliterate("Ærøskøbing")
+ # # => "AEroskobing"
+ #
+ # I18n.transliterate("日本語")
+ # # => "???"
+ #
+ # It's also possible to add support for per-locale transliterations. I18n
+ # expects transliteration rules to be stored at
+ # <tt>i18n.transliterate.rule</tt>.
+ #
+ # Transliteration rules can either be a Hash or a Proc. Procs must accept a
+ # single string argument. Hash rules inherit the default transliteration
+ # rules, while Procs do not.
+ #
+ # *Examples*
+ #
+ # Setting a Hash in <locale>.yml:
+ #
+ # i18n:
+ # transliterate:
+ # rule:
+ # ü: "ue"
+ # ö: "oe"
+ #
+ # Setting a Hash using Ruby:
+ #
+ # store_translations(:de, :i18n => {
+ # :transliterate => {
+ # :rule => {
+ # "ü" => "ue",
+ # "ö" => "oe"
+ # }
+ # }
+ # )
+ #
+ # Setting a Proc:
+ #
+ # translit = lambda {|string| MyTransliterator.transliterate(string) }
+ # store_translations(:xx, :i18n => {:transliterate => {:rule => translit})
+ #
+ # Transliterating strings:
+ #
+ # I18n.locale = :en
+ # I18n.transliterate("Jürgen") # => "Jurgen"
+ # I18n.locale = :de
+ # I18n.transliterate("Jürgen") # => "Juergen"
+ # I18n.transliterate("Jürgen", :locale => :en) # => "Jurgen"
+ # I18n.transliterate("Jürgen", :locale => :de) # => "Juergen"
+ def transliterate(*args)
+ options = args.pop if args.last.is_a?(Hash)
+ key = args.shift
+ locale = options && options.delete(:locale) || config.locale
+ handling = options && (options.delete(:throw) && :throw || options.delete(:raise) && :raise)
+ replacement = options && options.delete(:replacement)
+ config.backend.transliterate(locale, key, replacement)
+ rescue I18n::ArgumentError => exception
+ handle_exception(handling, exception, locale, key, options || {})
+ end
+
+ # Localizes certain objects, such as dates and numbers to local formatting.
+ def localize(object, options = {})
+ locale = options.delete(:locale) || config.locale
+ format = options.delete(:format) || :default
+ config.backend.localize(locale, object, format, options)
+ end
+ alias :l :localize
+
+ # Executes block with given I18n.locale set.
+ def with_locale(tmp_locale = nil)
+ if tmp_locale
+ current_locale = self.locale
+ self.locale = tmp_locale
+ end
+ yield
+ ensure
+ self.locale = current_locale if tmp_locale
+ end
+
+ # Merges the given locale, key and scope into a single array of keys.
+ # Splits keys that contain dots into multiple keys. Makes sure all
+ # keys are Symbols.
+ def normalize_keys(locale, key, scope, separator = nil)
+ separator ||= I18n.default_separator
+
+ keys = []
+ keys.concat normalize_key(locale, separator)
+ keys.concat normalize_key(scope, separator)
+ keys.concat normalize_key(key, separator)
+ keys
+ end
+
+ # making these private until Ruby 1.9.2 can send to protected methods again
+ # see http://redmine.ruby-lang.org/repositories/revision/ruby-19?rev=24280
+ private
+
+ # Any exceptions thrown in translate will be sent to the @@exception_handler
+ # which can be a Symbol, a Proc or any other Object unless they're forced to
+ # be raised or thrown (MissingTranslation).
+ #
+ # If exception_handler is a Symbol then it will simply be sent to I18n as
+ # a method call. A Proc will simply be called. In any other case the
+ # method #call will be called on the exception_handler object.
+ #
+ # Examples:
+ #
+ # I18n.exception_handler = :default_exception_handler # this is the default
+ # I18n.default_exception_handler(exception, locale, key, options) # will be called like this
+ #
+ # I18n.exception_handler = lambda { |*args| ... } # a lambda
+ # I18n.exception_handler.call(exception, locale, key, options) # will be called like this
+ #
+ # I18n.exception_handler = I18nExceptionHandler.new # an object
+ # I18n.exception_handler.call(exception, locale, key, options) # will be called like this
+ def handle_exception(handling, exception, locale, key, options)
+ case handling
+ when :raise
+ raise(exception.respond_to?(:to_exception) ? exception.to_exception : exception)
+ when :throw
+ throw :exception, exception
+ else
+ case handler = options[:exception_handler] || config.exception_handler
+ when Symbol
+ send(handler, exception, locale, key, options)
+ else
+ handler.call(exception, locale, key, options)
+ end
+ end
+ end
+
+ def normalize_key(key, separator)
+ normalized_key_cache[separator][key] ||=
+ case key
+ when Array
+ key.map { |k| normalize_key(k, separator) }.flatten
+ else
+ keys = key.to_s.split(separator)
+ keys.delete('')
+ keys.map! { |k| k.to_sym }
+ keys
+ end
+ end
+
+ def normalized_key_cache
+ @normalized_key_cache ||= Hash.new { |h,k| h[k] = {} }
+ end
+
+ # DEPRECATED. Use I18n.normalize_keys instead.
+ def normalize_translation_keys(locale, key, scope, separator = nil)
+ puts "I18n.normalize_translation_keys is deprecated. Please use the class I18n.normalize_keys instead."
+ normalize_keys(locale, key, scope, separator)
+ end
+
+ # DEPRECATED. Please use the I18n::ExceptionHandler class instead.
+ def default_exception_handler(exception, locale, key, options)
+ puts "I18n.default_exception_handler is deprecated. Please use the class I18n::ExceptionHandler instead " +
+ "(an instance of which is set to I18n.exception_handler by default)."
+ exception.is_a?(MissingTranslation) ? exception.message : raise(exception)
+ end
+ }
+end
--- /dev/null
+module I18n
+ module Backend
+ autoload :Base, 'i18n/backend/base'
+ autoload :InterpolationCompiler, 'i18n/backend/interpolation_compiler'
+ autoload :Cache, 'i18n/backend/cache'
+ autoload :Cascade, 'i18n/backend/cascade'
+ autoload :Chain, 'i18n/backend/chain'
+ autoload :Fallbacks, 'i18n/backend/fallbacks'
+ autoload :Flatten, 'i18n/backend/flatten'
+ autoload :Gettext, 'i18n/backend/gettext'
+ autoload :KeyValue, 'i18n/backend/key_value'
+ autoload :Memoize, 'i18n/backend/memoize'
+ autoload :Metadata, 'i18n/backend/metadata'
+ autoload :Pluralization, 'i18n/backend/pluralization'
+ autoload :Simple, 'i18n/backend/simple'
+ autoload :Transliterator, 'i18n/backend/transliterator'
+ end
+end
--- /dev/null
+require 'yaml'
+require 'i18n/core_ext/hash'
+require 'i18n/core_ext/kernel/surpress_warnings'
+
+module I18n
+ module Backend
+ module Base
+ include I18n::Backend::Transliterator
+
+ # Accepts a list of paths to translation files. Loads translations from
+ # plain Ruby (*.rb) or YAML files (*.yml). See #load_rb and #load_yml
+ # for details.
+ def load_translations(*filenames)
+ filenames = I18n.load_path if filenames.empty?
+ filenames.flatten.each { |filename| load_file(filename) }
+ end
+
+ # This method receives a locale, a data hash and options for storing translations.
+ # Should be implemented
+ def store_translations(locale, data, options = {})
+ raise NotImplementedError
+ end
+
+ def translate(locale, key, options = {})
+ raise InvalidLocale.new(locale) unless locale
+ entry = key && lookup(locale, key, options[:scope], options)
+
+ if options.empty?
+ entry = resolve(locale, key, entry, options)
+ else
+ count, default = options.values_at(:count, :default)
+ values = options.except(*RESERVED_KEYS)
+ entry = entry.nil? && default ?
+ default(locale, key, default, options) : resolve(locale, key, entry, options)
+ end
+
+ throw(:exception, I18n::MissingTranslation.new(locale, key, options)) if entry.nil?
+ entry = entry.dup if entry.is_a?(String)
+
+ entry = pluralize(locale, entry, count) if count
+ entry = interpolate(locale, entry, values) if values
+ entry
+ end
+
+ # Acts the same as +strftime+, but uses a localized version of the
+ # format string. Takes a key from the date/time formats translations as
+ # a format argument (<em>e.g.</em>, <tt>:short</tt> in <tt>:'date.formats'</tt>).
+ def localize(locale, object, format = :default, options = {})
+ raise ArgumentError, "Object must be a Date, DateTime or Time object. #{object.inspect} given." unless object.respond_to?(:strftime)
+
+ if Symbol === format
+ key = format
+ type = object.respond_to?(:sec) ? 'time' : 'date'
+ options = options.merge(:raise => true, :object => object, :locale => locale)
+ format = I18n.t(:"#{type}.formats.#{key}", options)
+ end
+
+ # format = resolve(locale, object, format, options)
+ format = format.to_s.gsub(/%[aAbBp]/) do |match|
+ case match
+ when '%a' then I18n.t(:"date.abbr_day_names", :locale => locale, :format => format)[object.wday]
+ when '%A' then I18n.t(:"date.day_names", :locale => locale, :format => format)[object.wday]
+ when '%b' then I18n.t(:"date.abbr_month_names", :locale => locale, :format => format)[object.mon]
+ when '%B' then I18n.t(:"date.month_names", :locale => locale, :format => format)[object.mon]
+ when '%p' then I18n.t(:"time.#{object.hour < 12 ? :am : :pm}", :locale => locale, :format => format) if object.respond_to? :hour
+ end
+ end
+
+ object.strftime(format)
+ end
+
+ # Returns an array of locales for which translations are available
+ # ignoring the reserved translation meta data key :i18n.
+ def available_locales
+ raise NotImplementedError
+ end
+
+ def reload!
+ @skip_syntax_deprecation = false
+ end
+
+ protected
+
+ # The method which actually looks up for the translation in the store.
+ def lookup(locale, key, scope = [], options = {})
+ raise NotImplementedError
+ end
+
+ # Evaluates defaults.
+ # If given subject is an Array, it walks the array and returns the
+ # first translation that can be resolved. Otherwise it tries to resolve
+ # the translation directly.
+ def default(locale, object, subject, options = {})
+ options = options.dup.reject { |key, value| key == :default }
+ case subject
+ when Array
+ subject.each do |item|
+ result = resolve(locale, object, item, options) and return result
+ end and nil
+ else
+ resolve(locale, object, subject, options)
+ end
+ end
+
+ # Resolves a translation.
+ # If the given subject is a Symbol, it will be translated with the
+ # given options. If it is a Proc then it will be evaluated. All other
+ # subjects will be returned directly.
+ def resolve(locale, object, subject, options = {})
+ return subject if options[:resolve] == false
+ result = catch(:exception) do
+ case subject
+ when Symbol
+ I18n.translate(subject, options.merge(:locale => locale, :throw => true))
+ when Proc
+ date_or_time = options.delete(:object) || object
+ resolve(locale, object, subject.call(date_or_time, options))
+ else
+ subject
+ end
+ end
+ result unless result.is_a?(MissingTranslation)
+ end
+
+ # Picks a translation from an array according to English pluralization
+ # rules. It will pick the first translation if count is not equal to 1
+ # and the second translation if it is equal to 1. Other backends can
+ # implement more flexible or complex pluralization rules.
+ def pluralize(locale, entry, count)
+ return entry unless entry.is_a?(Hash) && count
+
+ key = :zero if count == 0 && entry.has_key?(:zero)
+ key ||= count == 1 ? :one : :other
+ raise InvalidPluralizationData.new(entry, count) unless entry.has_key?(key)
+ entry[key]
+ end
+
+ # Interpolates values into a given string.
+ #
+ # interpolate "file %{file} opened by %%{user}", :file => 'test.txt', :user => 'Mr. X'
+ # # => "file test.txt opened by %{user}"
+ def interpolate(locale, string, values = {})
+ if string.is_a?(::String) && !values.empty?
+ I18n.interpolate(string, values)
+ else
+ string
+ end
+ end
+
+ # Loads a single translations file by delegating to #load_rb or
+ # #load_yml depending on the file extension and directly merges the
+ # data to the existing translations. Raises I18n::UnknownFileType
+ # for all other file extensions.
+ def load_file(filename)
+ type = File.extname(filename).tr('.', '').downcase
+ raise UnknownFileType.new(type, filename) unless respond_to?(:"load_#{type}", true)
+ data = send(:"load_#{type}", filename)
+ raise InvalidLocaleData.new(filename) unless data.is_a?(Hash)
+ data.each { |locale, d| store_translations(locale, d || {}) }
+ end
+
+ # Loads a plain Ruby translations file. eval'ing the file must yield
+ # a Hash containing translation data with locales as toplevel keys.
+ def load_rb(filename)
+ eval(IO.read(filename), binding, filename)
+ end
+
+ # Loads a YAML translations file. The data must have locales as
+ # toplevel keys.
+ def load_yml(filename)
+ begin
+ YAML.load_file(filename)
+ rescue TypeError
+ nil
+ rescue SyntaxError
+ nil
+ end
+ end
+ end
+ end
+end
--- /dev/null
+# This module allows you to easily cache all responses from the backend - thus
+# speeding up the I18n aspects of your application quite a bit.
+#
+# To enable caching you can simply include the Cache module to the Simple
+# backend - or whatever other backend you are using:
+#
+# I18n::Backend::Simple.send(:include, I18n::Backend::Cache)
+#
+# You will also need to set a cache store implementation that you want to use:
+#
+# I18n.cache_store = ActiveSupport::Cache.lookup_store(:memory_store)
+#
+# You can use any cache implementation you want that provides the same API as
+# ActiveSupport::Cache (only the methods #fetch and #write are being used).
+#
+# The cache_key implementation assumes that you only pass values to
+# I18n.translate that return a valid key from #hash (see
+# http://www.ruby-doc.org/core/classes/Object.html#M000337).
+#
+# If you use a lambda as a default value in your translation like this:
+#
+# I18n.t(:"date.order", :default => lambda {[:month, :day, :year]})
+#
+# Then you will always have a cache miss, because each time this method
+# is called the lambda will have a different hash value. If you know
+# the result of the lambda is a constant as in the example above, then
+# to cache this you can make the lambda a constant, like this:
+#
+# DEFAULT_DATE_ORDER = lambda {[:month, :day, :year]}
+# ...
+# I18n.t(:"date.order", :default => DEFAULT_DATE_ORDER)
+#
+# If the lambda may result in different values for each call then consider
+# also using the Memoize backend.
+#
+module I18n
+ class << self
+ @@cache_store = nil
+ @@cache_namespace = nil
+
+ def cache_store
+ @@cache_store
+ end
+
+ def cache_store=(store)
+ @@cache_store = store
+ end
+
+ def cache_namespace
+ @@cache_namespace
+ end
+
+ def cache_namespace=(namespace)
+ @@cache_namespace = namespace
+ end
+
+ def perform_caching?
+ !cache_store.nil?
+ end
+ end
+
+ module Backend
+ # TODO Should the cache be cleared if new translations are stored?
+ module Cache
+ def translate(locale, key, options = {})
+ I18n.perform_caching? ? fetch(cache_key(locale, key, options)) { super } : super
+ end
+
+ protected
+
+ def fetch(cache_key, &block)
+ result = _fetch(cache_key, &block)
+ throw(:exception, result) if result.is_a?(MissingTranslation)
+ result = result.dup if result.frozen? rescue result
+ result
+ end
+
+ def _fetch(cache_key, &block)
+ result = I18n.cache_store.read(cache_key) and return result
+ result = catch(:exception, &block)
+ I18n.cache_store.write(cache_key, result) unless result.is_a?(Proc)
+ result
+ end
+
+ def cache_key(locale, key, options)
+ # This assumes that only simple, native Ruby values are passed to I18n.translate.
+ "i18n/#{I18n.cache_namespace}/#{locale}/#{key.hash}/#{USE_INSPECT_HASH ? options.inspect.hash : options.hash}"
+ end
+
+ private
+ # In Ruby < 1.9 the following is true: { :foo => 1, :bar => 2 }.hash == { :foo => 2, :bar => 1 }.hash
+ # Therefore we must use the hash of the inspect string instead to avoid cache key colisions.
+ USE_INSPECT_HASH = RUBY_VERSION <= "1.9"
+ end
+ end
+end
--- /dev/null
+# The Cascade module adds the ability to do cascading lookups to backends that
+# are compatible to the Simple backend.
+#
+# By cascading lookups we mean that for any key that can not be found the
+# Cascade module strips one segment off the scope part of the key and then
+# tries to look up the key in that scope.
+#
+# E.g. when a lookup for the key :"foo.bar.baz" does not yield a result then
+# the segment :bar will be stripped off the scope part :"foo.bar" and the new
+# scope :foo will be used to look up the key :baz. If that does not succeed
+# then the remaining scope segment :foo will be omitted, too, and again the
+# key :baz will be looked up (now with no scope).
+#
+# To enable a cascading lookup one passes the :cascade option:
+#
+# I18n.t(:'foo.bar.baz', :cascade => true)
+#
+# This will return the first translation found for :"foo.bar.baz", :"foo.baz"
+# or :baz in this order.
+#
+# The cascading lookup takes precedence over resolving any given defaults.
+# I.e. defaults will kick in after the cascading lookups haven't succeeded.
+#
+# This behavior is useful for libraries like ActiveRecord validations where
+# the library wants to give users a bunch of more or less fine-grained options
+# of scopes for a particular key.
+#
+# Thanks to Clemens Kofler for the initial idea and implementation! See
+# http://github.com/clemens/i18n-cascading-backend
+
+module I18n
+ module Backend
+ module Cascade
+ def lookup(locale, key, scope = [], options = {})
+ return super unless cascade = options[:cascade]
+
+ cascade = { :step => 1 } unless cascade.is_a?(Hash)
+ step = cascade[:step] || 1
+ offset = cascade[:offset] || 1
+ separator = options[:separator] || I18n.default_separator
+ skip_root = cascade.has_key?(:skip_root) ? cascade[:skip_root] : true
+
+ scope = I18n.normalize_keys(nil, key, scope, separator)
+ key = (scope.slice!(-offset, offset) || []).join(separator)
+
+ begin
+ result = super
+ return result unless result.nil?
+ scope = scope.dup
+ end while (!scope.empty? || !skip_root) && scope.slice!(-step, step)
+ end
+ end
+ end
+end
--- /dev/null
+module I18n
+ module Backend
+ # Backend that chains multiple other backends and checks each of them when
+ # a translation needs to be looked up. This is useful when you want to use
+ # standard translations with a Simple backend but store custom application
+ # translations in a database or other backends.
+ #
+ # To use the Chain backend instantiate it and set it to the I18n module.
+ # You can add chained backends through the initializer or backends
+ # accessor:
+ #
+ # # preserves the existing Simple backend set to I18n.backend
+ # I18n.backend = I18n::Backend::Chain.new(I18n::Backend::ActiveRecord.new, I18n.backend)
+ #
+ # The implementation assumes that all backends added to the Chain implement
+ # a lookup method with the same API as Simple backend does.
+ class Chain
+ module Implementation
+ include Base
+
+ attr_accessor :backends
+
+ def initialize(*backends)
+ self.backends = backends
+ end
+
+ def reload!
+ backends.each { |backend| backend.reload! }
+ end
+
+ def store_translations(locale, data, options = {})
+ backends.first.store_translations(locale, data, options)
+ end
+
+ def available_locales
+ backends.map { |backend| backend.available_locales }.flatten.uniq
+ end
+
+ def translate(locale, key, default_options = {})
+ namespace = nil
+ options = default_options.except(:default)
+
+ backends.each do |backend|
+ catch(:exception) do
+ options = default_options if backend == backends.last
+ translation = backend.translate(locale, key, options)
+ if namespace_lookup?(translation, options)
+ namespace ||= {}
+ namespace.merge!(translation)
+ elsif !translation.nil?
+ return translation
+ end
+ end
+ end
+
+ return namespace if namespace
+ throw(:exception, I18n::MissingTranslation.new(locale, key, options))
+ end
+
+ def localize(locale, object, format = :default, options = {})
+ backends.each do |backend|
+ catch(:exception) do
+ result = backend.localize(locale, object, format, options) and return result
+ end
+ end
+ throw(:exception, I18n::MissingTranslation.new(locale, format, options))
+ end
+
+ protected
+ def namespace_lookup?(result, options)
+ result.is_a?(Hash) && !options.has_key?(:count)
+ end
+ end
+
+ include Implementation
+ end
+ end
+end
--- /dev/null
+# I18n locale fallbacks are useful when you want your application to use
+# translations from other locales when translations for the current locale are
+# missing. E.g. you might want to use :en translations when translations in
+# your applications main locale :de are missing.
+#
+# To enable locale fallbacks you can simply include the Fallbacks module to
+# the Simple backend - or whatever other backend you are using:
+#
+# I18n::Backend::Simple.include(I18n::Backend::Fallbacks)
+module I18n
+ @@fallbacks = nil
+
+ class << self
+ # Returns the current fallbacks implementation. Defaults to +I18n::Locale::Fallbacks+.
+ def fallbacks
+ @@fallbacks ||= I18n::Locale::Fallbacks.new
+ end
+
+ # Sets the current fallbacks implementation. Use this to set a different fallbacks implementation.
+ def fallbacks=(fallbacks)
+ @@fallbacks = fallbacks
+ end
+ end
+
+ module Backend
+ module Fallbacks
+ # Overwrites the Base backend translate method so that it will try each
+ # locale given by I18n.fallbacks for the given locale. E.g. for the
+ # locale :"de-DE" it might try the locales :"de-DE", :de and :en
+ # (depends on the fallbacks implementation) until it finds a result with
+ # the given options. If it does not find any result for any of the
+ # locales it will then throw MissingTranslation as usual.
+ #
+ # The default option takes precedence over fallback locales only when
+ # it's a Symbol. When the default contains a String, Proc or Hash
+ # it is evaluated last after all the fallback locales have been tried.
+ def translate(locale, key, options = {})
+ return super if options[:fallback]
+ default = extract_non_symbol_default!(options) if options[:default]
+
+ options[:fallback] = true
+ I18n.fallbacks[locale].each do |fallback|
+ catch(:exception) do
+ result = super(fallback, key, options)
+ return result unless result.nil?
+ end
+ end
+ options.delete(:fallback)
+
+ return super(locale, nil, options.merge(:default => default)) if default
+ throw(:exception, I18n::MissingTranslation.new(locale, key, options))
+ end
+
+ def extract_non_symbol_default!(options)
+ defaults = [options[:default]].flatten
+ first_non_symbol_default = defaults.detect{|default| !default.is_a?(Symbol)}
+ if first_non_symbol_default
+ options[:default] = defaults[0, defaults.index(first_non_symbol_default)]
+ end
+ return first_non_symbol_default
+ end
+
+ end
+ end
+end
--- /dev/null
+module I18n
+ module Backend
+ # This module contains several helpers to assist flattening translations.
+ # You may want to flatten translations for:
+ #
+ # 1) speed up lookups, as in the Memoize backend;
+ # 2) In case you want to store translations in a data store, as in ActiveRecord backend;
+ #
+ # You can check both backends above for some examples.
+ # This module also keeps all links in a hash so they can be properly resolved when flattened.
+ module Flatten
+ SEPARATOR_ESCAPE_CHAR = "\001"
+ FLATTEN_SEPARATOR = "."
+
+ # normalize_keys the flatten way. This method is significantly faster
+ # and creates way less objects than the one at I18n.normalize_keys.
+ # It also handles escaping the translation keys.
+ def self.normalize_flat_keys(locale, key, scope, separator)
+ keys = [scope, key].flatten.compact
+ separator ||= I18n.default_separator
+
+ if separator != FLATTEN_SEPARATOR
+ keys.map! do |k|
+ k.to_s.tr("#{FLATTEN_SEPARATOR}#{separator}",
+ "#{SEPARATOR_ESCAPE_CHAR}#{FLATTEN_SEPARATOR}")
+ end
+ end
+
+ keys.join(".")
+ end
+
+ # Receives a string and escape the default separator.
+ def self.escape_default_separator(key) #:nodoc:
+ key.to_s.tr(FLATTEN_SEPARATOR, SEPARATOR_ESCAPE_CHAR)
+ end
+
+ # Shortcut to I18n::Backend::Flatten.normalize_flat_keys
+ # and then resolve_links.
+ def normalize_flat_keys(locale, key, scope, separator)
+ key = I18n::Backend::Flatten.normalize_flat_keys(locale, key, scope, separator)
+ resolve_link(locale, key)
+ end
+
+ # Store flattened links.
+ def links
+ @links ||= Hash.new { |h,k| h[k] = {} }
+ end
+
+ # Flatten keys for nested Hashes by chaining up keys:
+ #
+ # >> { "a" => { "b" => { "c" => "d", "e" => "f" }, "g" => "h" }, "i" => "j"}.wind
+ # => { "a.b.c" => "d", "a.b.e" => "f", "a.g" => "h", "i" => "j" }
+ #
+ def flatten_keys(hash, escape, prev_key=nil, &block)
+ hash.each_pair do |key, value|
+ key = escape_default_separator(key) if escape
+ curr_key = [prev_key, key].compact.join(FLATTEN_SEPARATOR).to_sym
+ yield curr_key, value
+ flatten_keys(value, escape, curr_key, &block) if value.is_a?(Hash)
+ end
+ end
+
+ # Receives a hash of translations (where the key is a locale and
+ # the value is another hash) and return a hash with all
+ # translations flattened.
+ #
+ # Nested hashes are included in the flattened hash just if subtree
+ # is true and Symbols are automatically stored as links.
+ def flatten_translations(locale, data, escape, subtree)
+ hash = {}
+ flatten_keys(data, escape) do |key, value|
+ if value.is_a?(Hash)
+ hash[key] = value if subtree
+ else
+ store_link(locale, key, value) if value.is_a?(Symbol)
+ hash[key] = value
+ end
+ end
+ hash
+ end
+
+ protected
+
+ def store_link(locale, key, link)
+ links[locale.to_sym][key.to_s] = link.to_s
+ end
+
+ def resolve_link(locale, key)
+ key, locale = key.to_s, locale.to_sym
+ links = self.links[locale]
+
+ if links.key?(key)
+ links[key]
+ elsif link = find_link(locale, key)
+ store_link(locale, key, key.gsub(*link))
+ else
+ key
+ end
+ end
+
+ def find_link(locale, key) #:nodoc:
+ links[locale].each do |from, to|
+ return [from, to] if key[0, from.length] == from
+ end && nil
+ end
+
+ def escape_default_separator(key) #:nodoc:
+ I18n::Backend::Flatten.escape_default_separator(key)
+ end
+
+ end
+ end
+end
\ No newline at end of file
--- /dev/null
+require 'i18n/gettext'
+require 'i18n/gettext/po_parser'
+
+# Experimental support for using Gettext po files to store translations.
+#
+# To use this you can simply include the module to the Simple backend - or
+# whatever other backend you are using.
+#
+# I18n::Backend::Simple.include(I18n::Backend::Gettext)
+#
+# Now you should be able to include your Gettext translation (*.po) files to
+# the I18n.load_path so they're loaded to the backend and you can use them as
+# usual:
+#
+# I18n.load_path += Dir["path/to/locales/*.po"]
+#
+# Following the Gettext convention this implementation expects that your
+# translation files are named by their locales. E.g. the file en.po would
+# contain the translations for the English locale.
+module I18n
+ module Backend
+ module Gettext
+ class PoData < Hash
+ def set_comment(msgid_or_sym, comment)
+ # ignore
+ end
+ end
+
+ protected
+ def load_po(filename)
+ locale = ::File.basename(filename, '.po').to_sym
+ data = normalize(locale, parse(filename))
+ { locale => data }
+ end
+
+ def parse(filename)
+ GetText::PoParser.new.parse(::File.read(filename), PoData.new)
+ end
+
+ def normalize(locale, data)
+ data.inject({}) do |result, (key, value)|
+ unless key.nil? || key.empty?
+ key = key.gsub(I18n::Gettext::CONTEXT_SEPARATOR, '|')
+ key, value = normalize_pluralization(locale, key, value) if key.index("\000")
+
+ parts = key.split('|').reverse
+ normalized = parts.inject({}) do |_normalized, part|
+ { part => _normalized.empty? ? value : _normalized }
+ end
+
+ result.deep_merge!(normalized)
+ end
+ result
+ end
+ end
+
+ def normalize_pluralization(locale, key, value)
+ # FIXME po_parser includes \000 chars that can not be turned into Symbols
+ key = key.gsub("\000", I18n::Gettext::PLURAL_SEPARATOR).split(I18n::Gettext::PLURAL_SEPARATOR).first
+
+ keys = I18n::Gettext.plural_keys(locale)
+ values = value.split("\000")
+ raise "invalid number of plurals: #{values.size}, keys: #{keys.inspect} on #{locale} locale for msgid #{key.inspect} with values #{values.inspect}" if values.size != keys.size
+
+ result = {}
+ values.each_with_index { |_value, ix| result[keys[ix]] = _value }
+ [key, result]
+ end
+
+ end
+ end
+end
--- /dev/null
+# The InterpolationCompiler module contains optimizations that can tremendously
+# speed up the interpolation process on the Simple backend.
+#
+# It works by defining a pre-compiled method on stored translation Strings that
+# already bring all the knowledge about contained interpolation variables etc.
+# so that the actual recurring interpolation will be very fast.
+#
+# To enable pre-compiled interpolations you can simply include the
+# InterpolationCompiler module to the Simple backend:
+#
+# I18n::Backend::Simple.include(I18n::Backend::InterpolationCompiler)
+#
+# Note that InterpolationCompiler does not yield meaningful results and consequently
+# should not be used with Ruby 1.9 (YARV) but improves performance everywhere else
+# (jRuby, Rubinius and 1.8.7).
+module I18n
+ module Backend
+ module InterpolationCompiler
+ module Compiler
+ extend self
+
+ TOKENIZER = /(%%\{[^\}]+\}|%\{[^\}]+\})/
+ INTERPOLATION_SYNTAX_PATTERN = /(%)?(%\{([^\}]+)\})/
+
+ def compile_if_an_interpolation(string)
+ if interpolated_str?(string)
+ string.instance_eval <<-RUBY_EVAL, __FILE__, __LINE__
+ def i18n_interpolate(v = {})
+ "#{compiled_interpolation_body(string)}"
+ end
+ RUBY_EVAL
+ end
+
+ string
+ end
+
+ def interpolated_str?(str)
+ str.kind_of?(::String) && str =~ INTERPOLATION_SYNTAX_PATTERN
+ end
+
+ protected
+ # tokenize("foo %{bar} baz %%{buz}") # => ["foo ", "%{bar}", " baz ", "%%{buz}"]
+ def tokenize(str)
+ str.split(TOKENIZER)
+ end
+
+ def compiled_interpolation_body(str)
+ tokenize(str).map do |token|
+ (matchdata = token.match(INTERPOLATION_SYNTAX_PATTERN)) ? handle_interpolation_token(token, matchdata) : escape_plain_str(token)
+ end.join
+ end
+
+ def handle_interpolation_token(interpolation, matchdata)
+ escaped, pattern, key = matchdata.values_at(1, 2, 3)
+ escaped ? pattern : compile_interpolation_token(key.to_sym)
+ end
+
+ def compile_interpolation_token(key)
+ "\#{#{interpolate_or_raise_missing(key)}}"
+ end
+
+ def interpolate_or_raise_missing(key)
+ escaped_key = escape_key_sym(key)
+ RESERVED_KEYS.include?(key) ? reserved_key(escaped_key) : interpolate_key(escaped_key)
+ end
+
+ def interpolate_key(key)
+ [direct_key(key), nil_key(key), missing_key(key)].join('||')
+ end
+
+ def direct_key(key)
+ "((t = v[#{key}]) && t.respond_to?(:call) ? t.call : t)"
+ end
+
+ def nil_key(key)
+ "(v.has_key?(#{key}) && '')"
+ end
+
+ def missing_key(key)
+ "raise(MissingInterpolationArgument.new(#{key}, self))"
+ end
+
+ def reserved_key(key)
+ "raise(ReservedInterpolationKey.new(#{key}, self))"
+ end
+
+ def escape_plain_str(str)
+ str.gsub(/"|\\|#/) {|x| "\\#{x}"}
+ end
+
+ def escape_key_sym(key)
+ # rely on Ruby to do all the hard work :)
+ key.to_sym.inspect
+ end
+ end
+
+ def interpolate(locale, string, values)
+ if string.respond_to?(:i18n_interpolate)
+ string.i18n_interpolate(values)
+ elsif values
+ super
+ else
+ string
+ end
+ end
+
+ def store_translations(locale, data, options = {})
+ compile_all_strings_in(data)
+ super
+ end
+
+ protected
+ def compile_all_strings_in(data)
+ data.each_value do |value|
+ Compiler.compile_if_an_interpolation(value)
+ compile_all_strings_in(value) if value.kind_of?(Hash)
+ end
+ end
+ end
+ end
+end
--- /dev/null
+require 'i18n/backend/base'
+require 'active_support/json'
+require 'active_support/ordered_hash' # active_support/json/encoding uses ActiveSupport::OrderedHash but does not require it
+
+module I18n
+ module Backend
+ # This is a basic backend for key value stores. It receives on
+ # initialization the store, which should respond to three methods:
+ #
+ # * store#[](key) - Used to get a value
+ # * store#[]=(key, value) - Used to set a value
+ # * store#keys - Used to get all keys
+ #
+ # Since these stores only supports string, all values are converted
+ # to JSON before being stored, allowing it to also store booleans,
+ # hashes and arrays. However, this store does not support Procs.
+ #
+ # As the ActiveRecord backend, Symbols are just supported when loading
+ # translations from the filesystem or through explicit store translations.
+ #
+ # Also, avoid calling I18n.available_locales since it's a somehow
+ # expensive operation in most stores.
+ #
+ # == Example
+ #
+ # To setup I18n to use TokyoCabinet in memory is quite straightforward:
+ #
+ # require 'rufus/tokyo/cabinet' # gem install rufus-tokyo
+ # I18n.backend = I18n::Backend::KeyValue.new(Rufus::Tokyo::Cabinet.new('*'))
+ #
+ # == Performance
+ #
+ # You may make this backend even faster by including the Memoize module.
+ # However, notice that you should properly clear the cache if you change
+ # values directly in the key-store.
+ #
+ # == Subtrees
+ #
+ # In most backends, you are allowed to retrieve part of a translation tree:
+ #
+ # I18n.backend.store_translations :en, :foo => { :bar => :baz }
+ # I18n.t "foo" #=> { :bar => :baz }
+ #
+ # This backend supports this feature by default, but it slows down the storage
+ # of new data considerably and makes hard to delete entries. That said, you are
+ # allowed to disable the storage of subtrees on initialization:
+ #
+ # I18n::Backend::KeyValue.new(@store, false)
+ #
+ # This is useful if you are using a KeyValue backend chained to a Simple backend.
+ class KeyValue
+ module Implementation
+ attr_accessor :store
+
+ include Base, Flatten
+
+ def initialize(store, subtrees=true)
+ @store, @subtrees = store, subtrees
+ end
+
+ def store_translations(locale, data, options = {})
+ escape = options.fetch(:escape, true)
+ flatten_translations(locale, data, escape, @subtrees).each do |key, value|
+ key = "#{locale}.#{key}"
+
+ case value
+ when Hash
+ if @subtrees && (old_value = @store[key])
+ old_value = ActiveSupport::JSON.decode(old_value)
+ value = old_value.deep_symbolize_keys.deep_merge!(value) if old_value.is_a?(Hash)
+ end
+ when Proc
+ raise "Key-value stores cannot handle procs"
+ end
+
+ @store[key] = ActiveSupport::JSON.encode([value]) unless value.is_a?(Symbol)
+ end
+ end
+
+ def available_locales
+ locales = @store.keys.map { |k| k =~ /\./; $` }
+ locales.uniq!
+ locales.compact!
+ locales.map! { |k| k.to_sym }
+ locales
+ end
+
+ protected
+
+ def lookup(locale, key, scope = [], options = {})
+ key = normalize_flat_keys(locale, key, scope, options[:separator])
+ value = @store["#{locale}.#{key}"]
+ value = ActiveSupport::JSON.decode(value)[0] if value
+ value.is_a?(Hash) ? value.deep_symbolize_keys : value
+ end
+ end
+
+ include Implementation
+ end
+ end
+end
--- /dev/null
+# Memoize module simply memoizes the values returned by lookup using
+# a flat hash and can tremendously speed up the lookup process in a backend.
+#
+# To enable it you can simply include the Memoize module to your backend:
+#
+# I18n::Backend::Simple.include(I18n::Backend::Memoize)
+#
+# Notice that it's the responsibility of the backend to define whenever the
+# cache should be cleaned.
+module I18n
+ module Backend
+ module Memoize
+ def available_locales
+ @memoized_locales ||= super
+ end
+
+ def store_translations(locale, data, options = {})
+ reset_memoizations!(locale)
+ super
+ end
+
+ def reload!
+ reset_memoizations!
+ super
+ end
+
+ protected
+
+ def lookup(locale, key, scope = nil, options = {})
+ flat_key = I18n::Backend::Flatten.normalize_flat_keys(locale,
+ key, scope, options[:separator]).to_sym
+ flat_hash = memoized_lookup[locale.to_sym]
+ flat_hash.key?(flat_key) ? flat_hash[flat_key] : (flat_hash[flat_key] = super)
+ end
+
+ def memoized_lookup
+ @memoized_lookup ||= Hash.new { |h, k| h[k] = {} }
+ end
+
+ def reset_memoizations!(locale=nil)
+ @memoized_locales = nil
+ (locale ? memoized_lookup[locale.to_sym] : memoized_lookup).clear
+ end
+ end
+ end
+end
--- /dev/null
+# I18n translation metadata is useful when you want to access information
+# about how a translation was looked up, pluralized or interpolated in
+# your application.
+#
+# msg = I18n.t(:message, :default => 'Hi!', :scope => :foo)
+# msg.translation_metadata
+# # => { :key => :message, :scope => :foo, :default => 'Hi!' }
+#
+# If a :count option was passed to #translate it will be set to the metadata.
+# Likewise, if any interpolation variables were passed they will also be set.
+#
+# To enable translation metadata you can simply include the Metadata module
+# into the Simple backend class - or whatever other backend you are using:
+#
+# I18n::Backend::Simple.include(I18n::Backend::Metadata)
+#
+module I18n
+ module Backend
+ module Metadata
+ class << self
+ def included(base)
+ Object.class_eval do
+ def translation_metadata
+ @translation_metadata ||= {}
+ end
+
+ def translation_metadata=(translation_metadata)
+ @translation_metadata = translation_metadata
+ end
+ end unless Object.method_defined?(:translation_metadata)
+ end
+ end
+
+ def translate(locale, key, options = {})
+ metadata = {
+ :locale => locale,
+ :key => key,
+ :scope => options[:scope],
+ :default => options[:default],
+ :separator => options[:separator],
+ :values => options.reject { |name, value| RESERVED_KEYS.include?(name) }
+ }
+ with_metadata(metadata) { super }
+ end
+
+ def interpolate(locale, entry, values = {})
+ metadata = entry.translation_metadata.merge(:original => entry)
+ with_metadata(metadata) { super }
+ end
+
+ def pluralize(locale, entry, count)
+ with_metadata(:count => count) { super }
+ end
+
+ protected
+
+ def with_metadata(metadata, &block)
+ result = yield
+ result.translation_metadata = result.translation_metadata.merge(metadata) if result
+ result
+ end
+
+ end
+ end
+end
--- /dev/null
+# I18n Pluralization are useful when you want your application to
+# customize pluralization rules.
+#
+# To enable locale specific pluralizations you can simply include the
+# Pluralization module to the Simple backend - or whatever other backend you
+# are using.
+#
+# I18n::Backend::Simple.include(I18n::Backend::Pluralization)
+#
+# You also need to make sure to provide pluralization algorithms to the
+# backend, i.e. include them to your I18n.load_path accordingly.
+module I18n
+ module Backend
+ module Pluralization
+ # Overwrites the Base backend translate method so that it will check the
+ # translation meta data space (:i18n) for a locale specific pluralization
+ # rule and use it to pluralize the given entry. I.e. the library expects
+ # pluralization rules to be stored at I18n.t(:'i18n.plural.rule')
+ #
+ # Pluralization rules are expected to respond to #call(count) and
+ # return a pluralization key. Valid keys depend on the translation data
+ # hash (entry) but it is generally recommended to follow CLDR's style,
+ # i.e., return one of the keys :zero, :one, :few, :many, :other.
+ #
+ # The :zero key is always picked directly when count equals 0 AND the
+ # translation data has the key :zero. This way translators are free to
+ # either pick a special :zero translation even for languages where the
+ # pluralizer does not return a :zero key.
+ def pluralize(locale, entry, count)
+ return entry unless entry.is_a?(Hash) and count
+
+ pluralizer = pluralizer(locale)
+ if pluralizer.respond_to?(:call)
+ key = count == 0 && entry.has_key?(:zero) ? :zero : pluralizer.call(count)
+ raise InvalidPluralizationData.new(entry, count) unless entry.has_key?(key)
+ entry[key]
+ else
+ super
+ end
+ end
+
+ protected
+
+ def pluralizers
+ @pluralizers ||= {}
+ end
+
+ def pluralizer(locale)
+ pluralizers[locale] ||= I18n.t(:'i18n.plural.rule', :locale => locale, :resolve => false)
+ end
+ end
+ end
+end
--- /dev/null
+module I18n
+ module Backend
+ # A simple backend that reads translations from YAML files and stores them in
+ # an in-memory hash. Relies on the Base backend.
+ #
+ # The implementation is provided by a Implementation module allowing to easily
+ # extend Simple backend's behavior by including modules. E.g.:
+ #
+ # module I18n::Backend::Pluralization
+ # def pluralize(*args)
+ # # extended pluralization logic
+ # super
+ # end
+ # end
+ #
+ # I18n::Backend::Simple.include(I18n::Backend::Pluralization)
+ class Simple
+ (class << self; self; end).class_eval { public :include }
+
+ module Implementation
+ include Base
+
+ def initialized?
+ @initialized ||= false
+ end
+
+ # Stores translations for the given locale in memory.
+ # This uses a deep merge for the translations hash, so existing
+ # translations will be overwritten by new ones only at the deepest
+ # level of the hash.
+ def store_translations(locale, data, options = {})
+ locale = locale.to_sym
+ translations[locale] ||= {}
+ data = data.deep_symbolize_keys
+ translations[locale].deep_merge!(data)
+ end
+
+ # Get available locales from the translations hash
+ def available_locales
+ init_translations unless initialized?
+ translations.inject([]) do |locales, (locale, data)|
+ locales << locale unless (data.keys - [:i18n]).empty?
+ locales
+ end
+ end
+
+ # Clean up translations hash and set initialized to false on reload!
+ def reload!
+ @initialized = false
+ @translations = nil
+ super
+ end
+
+ protected
+
+ def init_translations
+ load_translations
+ @initialized = true
+ end
+
+ def translations
+ @translations ||= {}
+ end
+
+ # Looks up a translation from the translations hash. Returns nil if
+ # eiher key is nil, or locale, scope or key do not exist as a key in the
+ # nested translations hash. Splits keys or scopes containing dots
+ # into multiple keys, i.e. <tt>currency.format</tt> is regarded the same as
+ # <tt>%w(currency format)</tt>.
+ def lookup(locale, key, scope = [], options = {})
+ init_translations unless initialized?
+ keys = I18n.normalize_keys(locale, key, scope, options[:separator])
+
+ keys.inject(translations) do |result, _key|
+ _key = _key.to_sym
+ return nil unless result.is_a?(Hash) && result.has_key?(_key)
+ result = result[_key]
+ result = resolve(locale, _key, result, options.merge(:scope => nil)) if result.is_a?(Symbol)
+ result
+ end
+ end
+ end
+
+ include Implementation
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+module I18n
+ module Backend
+ module Transliterator
+ DEFAULT_REPLACEMENT_CHAR = "?"
+
+ # Given a locale and a UTF-8 string, return the locale's ASCII
+ # approximation for the string.
+ def transliterate(locale, string, replacement = nil)
+ @transliterators ||= {}
+ @transliterators[locale] ||= Transliterator.get I18n.t(:'i18n.transliterate.rule',
+ :locale => locale, :resolve => false, :default => {})
+ @transliterators[locale].transliterate(string, replacement)
+ end
+
+ # Get a transliterator instance.
+ def self.get(rule = nil)
+ if !rule || rule.kind_of?(Hash)
+ HashTransliterator.new(rule)
+ elsif rule.kind_of? Proc
+ ProcTransliterator.new(rule)
+ else
+ raise I18n::ArgumentError, "Transliteration rule must be a proc or a hash."
+ end
+ end
+
+ # A transliterator which accepts a Proc as its transliteration rule.
+ class ProcTransliterator
+ def initialize(rule)
+ @rule = rule
+ end
+
+ def transliterate(string, replacement = nil)
+ @rule.call(string)
+ end
+ end
+
+ # A transliterator which accepts a Hash of characters as its translation
+ # rule.
+ class HashTransliterator
+ DEFAULT_APPROXIMATIONS = {
+ "À"=>"A", "Á"=>"A", "Â"=>"A", "Ã"=>"A", "Ä"=>"A", "Å"=>"A", "Æ"=>"AE",
+ "Ç"=>"C", "È"=>"E", "É"=>"E", "Ê"=>"E", "Ë"=>"E", "Ì"=>"I", "Í"=>"I",
+ "Î"=>"I", "Ï"=>"I", "Ð"=>"D", "Ñ"=>"N", "Ò"=>"O", "Ó"=>"O", "Ô"=>"O",
+ "Õ"=>"O", "Ö"=>"O", "×"=>"x", "Ø"=>"O", "Ù"=>"U", "Ú"=>"U", "Û"=>"U",
+ "Ü"=>"U", "Ý"=>"Y", "Þ"=>"Th", "ß"=>"ss", "à"=>"a", "á"=>"a", "â"=>"a",
+ "ã"=>"a", "ä"=>"a", "å"=>"a", "æ"=>"ae", "ç"=>"c", "è"=>"e", "é"=>"e",
+ "ê"=>"e", "ë"=>"e", "ì"=>"i", "í"=>"i", "î"=>"i", "ï"=>"i", "ð"=>"d",
+ "ñ"=>"n", "ò"=>"o", "ó"=>"o", "ô"=>"o", "õ"=>"o", "ö"=>"o", "ø"=>"o",
+ "ù"=>"u", "ú"=>"u", "û"=>"u", "ü"=>"u", "ý"=>"y", "þ"=>"th", "ÿ"=>"y",
+ "Ā"=>"A", "ā"=>"a", "Ă"=>"A", "ă"=>"a", "Ą"=>"A", "ą"=>"a", "Ć"=>"C",
+ "ć"=>"c", "Ĉ"=>"C", "ĉ"=>"c", "Ċ"=>"C", "ċ"=>"c", "Č"=>"C", "č"=>"c",
+ "Ď"=>"D", "ď"=>"d", "Đ"=>"D", "đ"=>"d", "Ē"=>"E", "ē"=>"e", "Ĕ"=>"E",
+ "ĕ"=>"e", "Ė"=>"E", "ė"=>"e", "Ę"=>"E", "ę"=>"e", "Ě"=>"E", "ě"=>"e",
+ "Ĝ"=>"G", "ĝ"=>"g", "Ğ"=>"G", "ğ"=>"g", "Ġ"=>"G", "ġ"=>"g", "Ģ"=>"G",
+ "ģ"=>"g", "Ĥ"=>"H", "ĥ"=>"h", "Ħ"=>"H", "ħ"=>"h", "Ĩ"=>"I", "ĩ"=>"i",
+ "Ī"=>"I", "ī"=>"i", "Ĭ"=>"I", "ĭ"=>"i", "Į"=>"I", "į"=>"i", "İ"=>"I",
+ "ı"=>"i", "IJ"=>"IJ", "ij"=>"ij", "Ĵ"=>"J", "ĵ"=>"j", "Ķ"=>"K", "ķ"=>"k",
+ "ĸ"=>"k", "Ĺ"=>"L", "ĺ"=>"l", "Ļ"=>"L", "ļ"=>"l", "Ľ"=>"L", "ľ"=>"l",
+ "Ŀ"=>"L", "ŀ"=>"l", "Ł"=>"L", "ł"=>"l", "Ń"=>"N", "ń"=>"n", "Ņ"=>"N",
+ "ņ"=>"n", "Ň"=>"N", "ň"=>"n", "ʼn"=>"'n", "Ŋ"=>"NG", "ŋ"=>"ng",
+ "Ō"=>"O", "ō"=>"o", "Ŏ"=>"O", "ŏ"=>"o", "Ő"=>"O", "ő"=>"o", "Œ"=>"OE",
+ "œ"=>"oe", "Ŕ"=>"R", "ŕ"=>"r", "Ŗ"=>"R", "ŗ"=>"r", "Ř"=>"R", "ř"=>"r",
+ "Ś"=>"S", "ś"=>"s", "Ŝ"=>"S", "ŝ"=>"s", "Ş"=>"S", "ş"=>"s", "Š"=>"S",
+ "š"=>"s", "Ţ"=>"T", "ţ"=>"t", "Ť"=>"T", "ť"=>"t", "Ŧ"=>"T", "ŧ"=>"t",
+ "Ũ"=>"U", "ũ"=>"u", "Ū"=>"U", "ū"=>"u", "Ŭ"=>"U", "ŭ"=>"u", "Ů"=>"U",
+ "ů"=>"u", "Ű"=>"U", "ű"=>"u", "Ų"=>"U", "ų"=>"u", "Ŵ"=>"W", "ŵ"=>"w",
+ "Ŷ"=>"Y", "ŷ"=>"y", "Ÿ"=>"Y", "Ź"=>"Z", "ź"=>"z", "Ż"=>"Z", "ż"=>"z",
+ "Ž"=>"Z", "ž"=>"z"
+ }
+
+ def initialize(rule = nil)
+ @rule = rule
+ add DEFAULT_APPROXIMATIONS
+ add rule if rule
+ end
+
+ def transliterate(string, replacement = nil)
+ string.gsub(/[^\x00-\x7f]/u) do |char|
+ approximations[char] || replacement || DEFAULT_REPLACEMENT_CHAR
+ end
+ end
+
+ private
+
+ def approximations
+ @approximations ||= {}
+ end
+
+ # Add transliteration rules to the approximations hash.
+ def add(hash)
+ hash.keys.each {|key| hash[key.to_s] = hash.delete(key).to_s}
+ approximations.merge! hash
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module I18n
+ class Config
+ # The only configuration value that is not global and scoped to thread is :locale.
+ # It defaults to the default_locale.
+ def locale
+ @locale ||= default_locale
+ end
+
+ # Sets the current locale pseudo-globally, i.e. in the Thread.current hash.
+ def locale=(locale)
+ @locale = locale.to_sym rescue nil
+ end
+
+ # Returns the current backend. Defaults to +Backend::Simple+.
+ def backend
+ @@backend ||= Backend::Simple.new
+ end
+
+ # Sets the current backend. Used to set a custom backend.
+ def backend=(backend)
+ @@backend = backend
+ end
+
+ # Returns the current default locale. Defaults to :'en'
+ def default_locale
+ @@default_locale ||= :en
+ end
+
+ # Sets the current default locale. Used to set a custom default locale.
+ def default_locale=(locale)
+ @@default_locale = locale.to_sym rescue nil
+ end
+
+ # Returns an array of locales for which translations are available.
+ # Unless you explicitely set these through I18n.available_locales=
+ # the call will be delegated to the backend.
+ def available_locales
+ @@available_locales ||= nil
+ @@available_locales || backend.available_locales
+ end
+
+ # Sets the available locales.
+ def available_locales=(locales)
+ @@available_locales = Array(locales).map { |locale| locale.to_sym }
+ @@available_locales = nil if @@available_locales.empty?
+ end
+
+ # Returns the current default scope separator. Defaults to '.'
+ def default_separator
+ @@default_separator ||= '.'
+ end
+
+ # Sets the current default scope separator.
+ def default_separator=(separator)
+ @@default_separator = separator
+ end
+
+ # Return the current exception handler. Defaults to :default_exception_handler.
+ def exception_handler
+ @@exception_handler ||= ExceptionHandler.new
+ end
+
+ # Sets the exception handler.
+ def exception_handler=(exception_handler)
+ @@exception_handler = exception_handler
+ end
+
+ # Allow clients to register paths providing translation data sources. The
+ # backend defines acceptable sources.
+ #
+ # E.g. the provided SimpleBackend accepts a list of paths to translation
+ # files which are either named *.rb and contain plain Ruby Hashes or are
+ # named *.yml and contain YAML data. So for the SimpleBackend clients may
+ # register translation files like this:
+ # I18n.load_path << 'path/to/locale/en.yml'
+ def load_path
+ @@load_path ||= []
+ end
+
+ # Sets the load path instance. Custom implementations are expected to
+ # behave like a Ruby Array.
+ def load_path=(load_path)
+ @@load_path = load_path
+ end
+ end
+end
--- /dev/null
+class Hash
+ def slice(*keep_keys)
+ h = {}
+ keep_keys.each { |key| h[key] = fetch(key) }
+ h
+ end unless Hash.method_defined?(:slice)
+
+ def except(*less_keys)
+ slice(*keys - less_keys)
+ end unless Hash.method_defined?(:except)
+
+ def deep_symbolize_keys
+ inject({}) { |result, (key, value)|
+ value = value.deep_symbolize_keys if value.is_a?(Hash)
+ result[(key.to_sym rescue key) || key] = value
+ result
+ }
+ end unless Hash.method_defined?(:deep_symbolize_keys)
+
+ # deep_merge_hash! by Stefan Rusterholz, see http://www.ruby-forum.com/topic/142809
+ MERGER = proc do |key, v1, v2|
+ Hash === v1 && Hash === v2 ? v1.merge(v2, &MERGER) : v2
+ end
+
+ def deep_merge!(data)
+ merge!(data, &MERGER)
+ end unless Hash.method_defined?(:deep_merge!)
+end
+
--- /dev/null
+module Kernel
+ def suppress_warnings
+ original_verbosity = $VERBOSE
+ $VERBOSE = nil
+ result = yield
+ $VERBOSE = original_verbosity
+ result
+ end
+end
--- /dev/null
+# This backports the Ruby 1.9 String interpolation syntax to Ruby 1.8.
+#
+# This backport has been shipped with I18n for a number of versions. Meanwhile
+# Rails has started to rely on it and we are going to move it to ActiveSupport.
+# See https://rails.lighthouseapp.com/projects/8994/tickets/6013-move-19-string-interpolation-syntax-backport-from-i18n-to-activesupport
+#
+# Once the above patch has been applied to Rails the following code will be
+# removed from I18n.
+
+=begin
+ heavily based on Masao Mutoh's gettext String interpolation extension
+ http://github.com/mutoh/gettext/blob/f6566738b981fe0952548c421042ad1e0cdfb31e/lib/gettext/core_ext/string.rb
+ Copyright (C) 2005-2009 Masao Mutoh
+ You may redistribute it and/or modify it under the same license terms as Ruby.
+=end
+
+begin
+ raise ArgumentError if ("a %{x}" % {:x=>'b'}) != 'a b'
+rescue ArgumentError
+ # KeyError is raised by String#% when the string contains a named placeholder
+ # that is not contained in the given arguments hash. Ruby 1.9 includes and
+ # raises this exception natively. We define it to mimic Ruby 1.9's behaviour
+ # in Ruby 1.8.x
+ class KeyError < IndexError
+ def initialize(message = nil)
+ super(message || "key not found")
+ end
+ end unless defined?(KeyError)
+
+ # Extension for String class. This feature is included in Ruby 1.9 or later but not occur TypeError.
+ #
+ # String#% method which accept "named argument". The translator can know
+ # the meaning of the msgids using "named argument" instead of %s/%d style.
+ class String
+ # For older ruby versions, such as ruby-1.8.5
+ alias :bytesize :size unless instance_methods.find {|m| m.to_s == 'bytesize'}
+ alias :interpolate_without_ruby_19_syntax :% # :nodoc:
+
+ INTERPOLATION_PATTERN = Regexp.union(
+ /%\{(\w+)\}/, # matches placeholders like "%{foo}"
+ /%<(\w+)>(.*?\d*\.?\d*[bBdiouxXeEfgGcps])/ # matches placeholders like "%<foo>.d"
+ )
+
+ INTERPOLATION_PATTERN_WITH_ESCAPE = Regexp.union(
+ /%%/,
+ INTERPOLATION_PATTERN
+ )
+
+ # % uses self (i.e. the String) as a format specification and returns the
+ # result of applying it to the given arguments. In other words it interpolates
+ # the given arguments to the string according to the formats the string
+ # defines.
+ #
+ # There are three ways to use it:
+ #
+ # * Using a single argument or Array of arguments.
+ #
+ # This is the default behaviour of the String class. See Kernel#sprintf for
+ # more details about the format string.
+ #
+ # Example:
+ #
+ # "%d %s" % [1, "message"]
+ # # => "1 message"
+ #
+ # * Using a Hash as an argument and unformatted, named placeholders.
+ #
+ # When you pass a Hash as an argument and specify placeholders with %{foo}
+ # it will interpret the hash values as named arguments.
+ #
+ # Example:
+ #
+ # "%{firstname}, %{lastname}" % {:firstname => "Masao", :lastname => "Mutoh"}
+ # # => "Masao Mutoh"
+ #
+ # * Using a Hash as an argument and formatted, named placeholders.
+ #
+ # When you pass a Hash as an argument and specify placeholders with %<foo>d
+ # it will interpret the hash values as named arguments and format the value
+ # according to the formatting instruction appended to the closing >.
+ #
+ # Example:
+ #
+ # "%<integer>d, %<float>.1f" % { :integer => 10, :float => 43.4 }
+ # # => "10, 43.3"
+ def %(args)
+ if args.kind_of?(Hash)
+ dup.gsub(INTERPOLATION_PATTERN_WITH_ESCAPE) do |match|
+ if match == '%%'
+ '%'
+ else
+ key = ($1 || $2).to_sym
+ raise KeyError unless args.has_key?(key)
+ $3 ? sprintf("%#{$3}", args[key]) : args[key]
+ end
+ end
+ elsif self =~ INTERPOLATION_PATTERN
+ raise ArgumentError.new('one hash required')
+ else
+ result = gsub(/%([{<])/, '%%\1')
+ result.send :'interpolate_without_ruby_19_syntax', args
+ end
+ end
+ end
+end
--- /dev/null
+module I18n
+ # Handles exceptions raised in the backend. All exceptions except for
+ # MissingTranslationData exceptions are re-thrown. When a MissingTranslationData
+ # was caught the handler returns an error message string containing the key/scope.
+ # Note that the exception handler is not called when the option :throw was given.
+ class ExceptionHandler
+ include Module.new {
+ def call(exception, locale, key, options)
+ if exception.is_a?(MissingTranslation)
+ options[:rescue_format] == :html ? exception.html_message : exception.message
+ elsif exception.is_a?(Exception)
+ raise exception
+ else
+ throw :exception, exception
+ end
+ end
+ }
+ end
+
+ class ArgumentError < ::ArgumentError; end
+
+ class InvalidLocale < ArgumentError
+ attr_reader :locale
+ def initialize(locale)
+ @locale = locale
+ super "#{locale.inspect} is not a valid locale"
+ end
+ end
+
+ class InvalidLocaleData < ArgumentError
+ attr_reader :filename
+ def initialize(filename)
+ @filename = filename
+ super "can not load translations from #{filename}, expected it to return a hash, but does not"
+ end
+ end
+
+ class MissingTranslation
+ module Base
+ attr_reader :locale, :key, :options
+
+ def initialize(locale, key, options = nil)
+ @key, @locale, @options = key, locale, options.dup || {}
+ options.each { |k, v| self.options[k] = v.inspect if v.is_a?(Proc) }
+ end
+
+ def html_message
+ key = keys.last.to_s.gsub('_', ' ').gsub(/\b('?[a-z])/) { $1.capitalize }
+ %(<span class="translation_missing" title="translation missing: #{keys.join('.')}">#{key}</span>)
+ end
+
+ def keys
+ @keys ||= I18n.normalize_keys(locale, key, options[:scope]).tap do |keys|
+ keys << 'no key' if keys.size < 2
+ end
+ end
+
+ def message
+ "translation missing: #{keys.join('.')}"
+ end
+ alias :to_s :message
+
+ def to_exception
+ MissingTranslationData.new(locale, key, options)
+ end
+ end
+
+ include Base
+ end
+
+ class MissingTranslationData < ArgumentError
+ include MissingTranslation::Base
+ end
+
+ class InvalidPluralizationData < ArgumentError
+ attr_reader :entry, :count
+ def initialize(entry, count)
+ @entry, @count = entry, count
+ super "translation data #{entry.inspect} can not be used with :count => #{count}"
+ end
+ end
+
+ class MissingInterpolationArgument < ArgumentError
+ attr_reader :values, :string
+ def initialize(values, string)
+ @values, @string = values, string
+ super "missing interpolation argument in #{string.inspect} (#{values.inspect} given)"
+ end
+ end
+
+ class ReservedInterpolationKey < ArgumentError
+ attr_reader :key, :string
+ def initialize(key, string)
+ @key, @string = key, string
+ super "reserved key #{key.inspect} used in #{string.inspect}"
+ end
+ end
+
+ class UnknownFileType < ArgumentError
+ attr_reader :type, :filename
+ def initialize(type, filename)
+ @type, @filename = type, filename
+ super "can not load translations from #{filename}, the file type #{type} is not known"
+ end
+ end
+end
--- /dev/null
+module I18n
+ module Gettext
+ PLURAL_SEPARATOR = "\001"
+ CONTEXT_SEPARATOR = "\004"
+
+ autoload :Helpers, 'i18n/gettext/helpers'
+
+ @@plural_keys = { :en => [:one, :other] }
+
+ class << self
+ # returns an array of plural keys for the given locale so that we can
+ # convert from gettext's integer-index based style
+ # TODO move this information to the pluralization module
+ def plural_keys(locale)
+ @@plural_keys[locale] || @@plural_keys[:en]
+ end
+
+ def extract_scope(msgid, separator)
+ scope = msgid.to_s.split(separator)
+ msgid = scope.pop
+ [scope, msgid]
+ end
+ end
+ end
+end
--- /dev/null
+require 'i18n/gettext'
+
+module I18n
+ module Gettext
+ # Implements classical Gettext style accessors. To use this include the
+ # module to the global namespace or wherever you want to use it.
+ #
+ # include I18n::Gettext::Helpers
+ module Helpers
+ def gettext(msgid, options = {})
+ I18n.t(msgid, { :default => msgid, :separator => '|' }.merge(options))
+ end
+ alias _ gettext
+
+ def sgettext(msgid, separator = '|')
+ scope, msgid = I18n::Gettext.extract_scope(msgid, separator)
+ I18n.t(msgid, :scope => scope, :default => msgid, :separator => separator)
+ end
+ alias s_ sgettext
+
+ def pgettext(msgctxt, msgid)
+ separator = I18n::Gettext::CONTEXT_SEPARATOR
+ sgettext([msgctxt, msgid].join(separator), separator)
+ end
+ alias p_ pgettext
+
+ def ngettext(msgid, msgid_plural, n = 1)
+ nsgettext(msgid, msgid_plural, n)
+ end
+ alias n_ ngettext
+
+ # Method signatures:
+ # nsgettext('Fruits|apple', 'apples', 2)
+ # nsgettext(['Fruits|apple', 'apples'], 2)
+ def nsgettext(msgid, msgid_plural, n = 1, separator = '|')
+ if msgid.is_a?(Array)
+ msgid, msgid_plural, n, separator = msgid[0], msgid[1], msgid_plural, n
+ separator = '|' unless separator.is_a?(::String)
+ end
+
+ scope, msgid = I18n::Gettext.extract_scope(msgid, separator)
+ default = { :one => msgid, :other => msgid_plural }
+ I18n.t(msgid, :default => default, :count => n, :scope => scope, :separator => separator)
+ end
+ alias ns_ nsgettext
+
+ # Method signatures:
+ # npgettext('Fruits', 'apple', 'apples', 2)
+ # npgettext('Fruits', ['apple', 'apples'], 2)
+ def npgettext(msgctxt, msgid, msgid_plural, n = 1)
+ separator = I18n::Gettext::CONTEXT_SEPARATOR
+
+ if msgid.is_a?(Array)
+ msgid_plural, msgid, n = msgid[1], [msgctxt, msgid[0]].join(separator), msgid_plural
+ else
+ msgid = [msgctxt, msgid].join(separator)
+ end
+
+ nsgettext(msgid, msgid_plural, n, separator)
+ end
+ alias np_ npgettext
+ end
+ end
+end
--- /dev/null
+=begin
+ poparser.rb - Generate a .mo
+
+ Copyright (C) 2003-2009 Masao Mutoh <mutoh at highway.ne.jp>
+
+ You may redistribute it and/or modify it under the same
+ license terms as Ruby.
+=end
+
+#MODIFIED
+# removed include GetText etc
+# added stub translation method _(x)
+require 'racc/parser'
+
+module GetText
+
+ class PoParser < Racc::Parser
+
+ def _(x)
+ x
+ end
+
+module_eval <<'..end src/poparser.ry modeval..id7a99570e05', 'src/poparser.ry', 108
+ def unescape(orig)
+ ret = orig.gsub(/\\n/, "\n")
+ ret.gsub!(/\\t/, "\t")
+ ret.gsub!(/\\r/, "\r")
+ ret.gsub!(/\\"/, "\"")
+ ret
+ end
+
+ def parse(str, data, ignore_fuzzy = true)
+ @comments = []
+ @data = data
+ @fuzzy = false
+ @msgctxt = ""
+ $ignore_fuzzy = ignore_fuzzy
+
+ str.strip!
+ @q = []
+ until str.empty? do
+ case str
+ when /\A\s+/
+ str = $'
+ when /\Amsgctxt/
+ @q.push [:MSGCTXT, $&]
+ str = $'
+ when /\Amsgid_plural/
+ @q.push [:MSGID_PLURAL, $&]
+ str = $'
+ when /\Amsgid/
+ @q.push [:MSGID, $&]
+ str = $'
+ when /\Amsgstr/
+ @q.push [:MSGSTR, $&]
+ str = $'
+ when /\A\[(\d+)\]/
+ @q.push [:PLURAL_NUM, $1]
+ str = $'
+ when /\A\#~(.*)/
+ $stderr.print _("Warning: obsolete msgid exists.\n")
+ $stderr.print " #{$&}\n"
+ @q.push [:COMMENT, $&]
+ str = $'
+ when /\A\#(.*)/
+ @q.push [:COMMENT, $&]
+ str = $'
+ when /\A\"(.*)\"/
+ @q.push [:STRING, $1]
+ str = $'
+ else
+ #c = str[0,1]
+ #@q.push [:STRING, c]
+ str = str[1..-1]
+ end
+ end
+ @q.push [false, '$end']
+ if $DEBUG
+ @q.each do |a,b|
+ puts "[#{a}, #{b}]"
+ end
+ end
+ @yydebug = true if $DEBUG
+ do_parse
+
+ if @comments.size > 0
+ @data.set_comment(:last, @comments.join("\n"))
+ end
+ @data
+ end
+
+ def next_token
+ @q.shift
+ end
+
+ def on_message(msgid, msgstr)
+ if msgstr.size > 0
+ @data[msgid] = msgstr
+ @data.set_comment(msgid, @comments.join("\n"))
+ end
+ @comments.clear
+ @msgctxt = ""
+ end
+
+ def on_comment(comment)
+ @fuzzy = true if (/fuzzy/ =~ comment)
+ @comments << comment
+ end
+
+
+..end src/poparser.ry modeval..id7a99570e05
+
+##### racc 1.4.5 generates ###
+
+racc_reduce_table = [
+ 0, 0, :racc_error,
+ 0, 10, :_reduce_none,
+ 2, 10, :_reduce_none,
+ 2, 10, :_reduce_none,
+ 2, 10, :_reduce_none,
+ 2, 12, :_reduce_5,
+ 1, 13, :_reduce_none,
+ 1, 13, :_reduce_none,
+ 4, 15, :_reduce_8,
+ 5, 16, :_reduce_9,
+ 2, 17, :_reduce_10,
+ 1, 17, :_reduce_none,
+ 3, 18, :_reduce_12,
+ 1, 11, :_reduce_13,
+ 2, 14, :_reduce_14,
+ 1, 14, :_reduce_15 ]
+
+racc_reduce_n = 16
+
+racc_shift_n = 26
+
+racc_action_table = [
+ 3, 13, 5, 7, 9, 15, 16, 17, 20, 17,
+ 13, 17, 13, 13, 11, 17, 23, 20, 13, 17 ]
+
+racc_action_check = [
+ 1, 16, 1, 1, 1, 12, 12, 12, 18, 18,
+ 7, 14, 15, 9, 3, 19, 20, 21, 23, 25 ]
+
+racc_action_pointer = [
+ nil, 0, nil, 14, nil, nil, nil, 3, nil, 6,
+ nil, nil, 0, nil, 4, 5, -6, nil, 2, 8,
+ 8, 11, nil, 11, nil, 12 ]
+
+racc_action_default = [
+ -1, -16, -2, -16, -3, -13, -4, -16, -6, -16,
+ -7, 26, -16, -15, -5, -16, -16, -14, -16, -8,
+ -16, -9, -11, -16, -10, -12 ]
+
+racc_goto_table = [
+ 12, 22, 14, 4, 24, 6, 2, 8, 18, 19,
+ 10, 21, 1, nil, nil, nil, 25 ]
+
+racc_goto_check = [
+ 5, 9, 5, 3, 9, 4, 2, 6, 5, 5,
+ 7, 8, 1, nil, nil, nil, 5 ]
+
+racc_goto_pointer = [
+ nil, 12, 5, 2, 4, -7, 6, 9, -7, -17 ]
+
+racc_goto_default = [
+ nil, nil, nil, nil, nil, nil, nil, nil, nil, nil ]
+
+racc_token_table = {
+ false => 0,
+ Object.new => 1,
+ :COMMENT => 2,
+ :MSGID => 3,
+ :MSGCTXT => 4,
+ :MSGID_PLURAL => 5,
+ :MSGSTR => 6,
+ :STRING => 7,
+ :PLURAL_NUM => 8 }
+
+racc_use_result_var = true
+
+racc_nt_base = 9
+
+Racc_arg = [
+ racc_action_table,
+ racc_action_check,
+ racc_action_default,
+ racc_action_pointer,
+ racc_goto_table,
+ racc_goto_check,
+ racc_goto_default,
+ racc_goto_pointer,
+ racc_nt_base,
+ racc_reduce_table,
+ racc_token_table,
+ racc_shift_n,
+ racc_reduce_n,
+ racc_use_result_var ]
+
+Racc_token_to_s_table = [
+'$end',
+'error',
+'COMMENT',
+'MSGID',
+'MSGCTXT',
+'MSGID_PLURAL',
+'MSGSTR',
+'STRING',
+'PLURAL_NUM',
+'$start',
+'msgfmt',
+'comment',
+'msgctxt',
+'message',
+'string_list',
+'single_message',
+'plural_message',
+'msgstr_plural',
+'msgstr_plural_line']
+
+Racc_debug_parser = true
+
+##### racc system variables end #####
+
+ # reduce 0 omitted
+
+ # reduce 1 omitted
+
+ # reduce 2 omitted
+
+ # reduce 3 omitted
+
+ # reduce 4 omitted
+
+module_eval <<'.,.,', 'src/poparser.ry', 25
+ def _reduce_5( val, _values, result )
+ @msgctxt = unescape(val[1]) + "\004"
+ result
+ end
+.,.,
+
+ # reduce 6 omitted
+
+ # reduce 7 omitted
+
+module_eval <<'.,.,', 'src/poparser.ry', 48
+ def _reduce_8( val, _values, result )
+ if @fuzzy and $ignore_fuzzy
+ if val[1] != ""
+ $stderr.print _("Warning: fuzzy message was ignored.\n")
+ $stderr.print " msgid '#{val[1]}'\n"
+ else
+ on_message('', unescape(val[3]))
+ end
+ @fuzzy = false
+ else
+ on_message(@msgctxt + unescape(val[1]), unescape(val[3]))
+ end
+ result = ""
+ result
+ end
+.,.,
+
+module_eval <<'.,.,', 'src/poparser.ry', 65
+ def _reduce_9( val, _values, result )
+ if @fuzzy and $ignore_fuzzy
+ if val[1] != ""
+ $stderr.print _("Warning: fuzzy message was ignored.\n")
+ $stderr.print "msgid = '#{val[1]}\n"
+ else
+ on_message('', unescape(val[3]))
+ end
+ @fuzzy = false
+ else
+ on_message(@msgctxt + unescape(val[1]) + "\000" + unescape(val[3]), unescape(val[4]))
+ end
+ result = ""
+ result
+ end
+.,.,
+
+module_eval <<'.,.,', 'src/poparser.ry', 76
+ def _reduce_10( val, _values, result )
+ if val[0].size > 0
+ result = val[0] + "\000" + val[1]
+ else
+ result = ""
+ end
+ result
+ end
+.,.,
+
+ # reduce 11 omitted
+
+module_eval <<'.,.,', 'src/poparser.ry', 84
+ def _reduce_12( val, _values, result )
+ result = val[2]
+ result
+ end
+.,.,
+
+module_eval <<'.,.,', 'src/poparser.ry', 91
+ def _reduce_13( val, _values, result )
+ on_comment(val[0])
+ result
+ end
+.,.,
+
+module_eval <<'.,.,', 'src/poparser.ry', 99
+ def _reduce_14( val, _values, result )
+ result = val.delete_if{|item| item == ""}.join
+ result
+ end
+.,.,
+
+module_eval <<'.,.,', 'src/poparser.ry', 103
+ def _reduce_15( val, _values, result )
+ result = val[0]
+ result
+ end
+.,.,
+
+ def _reduce_none( val, _values, result )
+ result
+ end
+
+ end # class PoParser
+
+end # module GetText
--- /dev/null
+# heavily based on Masao Mutoh's gettext String interpolation extension
+# http://github.com/mutoh/gettext/blob/f6566738b981fe0952548c421042ad1e0cdfb31e/lib/gettext/core_ext/string.rb
+
+module I18n
+ INTERPOLATION_PATTERN = Regexp.union(
+ /%%/,
+ /%\{(\w+)\}/, # matches placeholders like "%{foo}"
+ /%<(\w+)>(.*?\d*\.?\d*[bBdiouxXeEfgGcps])/ # matches placeholders like "%<foo>.d"
+ )
+
+ class << self
+ def interpolate(string, values)
+ raise ReservedInterpolationKey.new($1.to_sym, string) if string =~ RESERVED_KEYS_PATTERN
+ raise ArgumentError.new('Interpolation values must be a Hash.') unless values.kind_of?(Hash)
+ interpolate_hash(string, values)
+ end
+
+ def interpolate_hash(string, values)
+ string.gsub(INTERPOLATION_PATTERN) do |match|
+ if match == '%%'
+ '%'
+ else
+ key = ($1 || $2).to_sym
+ value = values.key?(key) ? values[key] : raise(MissingInterpolationArgument.new(values, string))
+ value = value.call(values) if value.respond_to?(:call)
+ $3 ? sprintf("%#{$3}", value) : value
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module I18n
+ module Locale
+ autoload :Fallbacks, 'i18n/locale/fallbacks'
+ autoload :Tag, 'i18n/locale/tag'
+ end
+end
--- /dev/null
+# Locale Fallbacks
+#
+# Extends the I18n module to hold a fallbacks instance which is set to an
+# instance of I18n::Locale::Fallbacks by default but can be swapped with a
+# different implementation.
+#
+# Locale fallbacks will compute a number of fallback locales for a given locale.
+# For example:
+#
+# <pre><code>
+# I18n.fallbacks[:"es-MX"] # => [:"es-MX", :es, :en] </code></pre>
+#
+# Locale fallbacks always fall back to
+#
+# * all parent locales of a given locale (e.g. :es for :"es-MX") first,
+# * the current default locales and all of their parents second
+#
+# The default locales are set to [I18n.default_locale] by default but can be
+# set to something else.
+#
+# One can additionally add any number of additional fallback locales manually.
+# These will be added before the default locales to the fallback chain. For
+# example:
+#
+# # using the default locale as default fallback locale
+#
+# I18n.default_locale = :"en-US"
+# I18n.fallbacks = I18n::Locale::Fallbacks.new(:"de-AT" => :"de-DE")
+# I18n.fallbacks[:"de-AT"] # => [:"de-AT", :"de-DE", :de, :"en-US", :en]
+#
+# # using a custom locale as default fallback locale
+#
+# I18n.fallbacks = I18n::Locale::Fallbacks.new(:"en-GB", :"de-AT" => :de, :"de-CH" => :de)
+# I18n.fallbacks[:"de-AT"] # => [:"de-AT", :de, :"en-GB", :en]
+# I18n.fallbacks[:"de-CH"] # => [:"de-CH", :de, :"en-GB", :en]
+#
+# # mapping fallbacks to an existing instance
+#
+# # people speaking Catalan also speak Spanish as spoken in Spain
+# fallbacks = I18n.fallbacks
+# fallbacks.map(:ca => :"es-ES")
+# fallbacks[:ca] # => [:ca, :"es-ES", :es, :"en-US", :en]
+#
+# # people speaking Arabian as spoken in Palestine also speak Hebrew as spoken in Israel
+# fallbacks.map(:"ar-PS" => :"he-IL")
+# fallbacks[:"ar-PS"] # => [:"ar-PS", :ar, :"he-IL", :he, :"en-US", :en]
+# fallbacks[:"ar-EG"] # => [:"ar-EG", :ar, :"en-US", :en]
+#
+# # people speaking Sami as spoken in Finnland also speak Swedish and Finnish as spoken in Finnland
+# fallbacks.map(:sms => [:"se-FI", :"fi-FI"])
+# fallbacks[:sms] # => [:sms, :"se-FI", :se, :"fi-FI", :fi, :"en-US", :en]
+
+module I18n
+ module Locale
+ class Fallbacks < Hash
+ def initialize(*mappings)
+ @map = {}
+ map(mappings.pop) if mappings.last.is_a?(Hash)
+ self.defaults = mappings.empty? ? [I18n.default_locale.to_sym] : mappings
+ end
+
+ def defaults=(defaults)
+ @defaults = defaults.map { |default| compute(default, false) }.flatten
+ end
+ attr_reader :defaults
+
+ def [](locale)
+ raise InvalidLocale.new(locale) if locale.nil?
+ locale = locale.to_sym
+ super || store(locale, compute(locale))
+ end
+
+ def map(mappings)
+ mappings.each do |from, to|
+ from, to = from.to_sym, Array(to)
+ to.each do |_to|
+ @map[from] ||= []
+ @map[from] << _to.to_sym
+ end
+ end
+ end
+
+ protected
+
+ def compute(tags, include_defaults = true, exclude = [])
+ result = Array(tags).collect do |tag|
+ tags = I18n::Locale::Tag.tag(tag).self_and_parents.map! { |t| t.to_sym } - exclude
+ tags.each { |_tag| tags += compute(@map[_tag], false, exclude + tags) if @map[_tag] }
+ tags
+ end.flatten
+ result.push(*defaults) if include_defaults
+ result.uniq.compact
+ end
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Locale
+ module Tag
+ autoload :Parents, 'i18n/locale/tag/parents'
+ autoload :Rfc4646, 'i18n/locale/tag/rfc4646'
+ autoload :Simple, 'i18n/locale/tag/simple'
+
+ class << self
+ # Returns the current locale tag implementation. Defaults to +I18n::Locale::Tag::Simple+.
+ def implementation
+ @@implementation ||= Simple
+ end
+
+ # Sets the current locale tag implementation. Use this to set a different locale tag implementation.
+ def implementation=(implementation)
+ @@implementation = implementation
+ end
+
+ # Factory method for locale tags. Delegates to the current locale tag implementation.
+ def tag(tag)
+ implementation.tag(tag)
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module I18n
+ module Locale
+ module Tag
+ module Parents
+ def parent
+ @parent ||= begin
+ segs = to_a.compact
+ segs.length > 1 ? self.class.tag(*segs[0..(segs.length-2)].join('-')) : nil
+ end
+ end
+
+ def self_and_parents
+ @self_and_parents ||= [self] + parents
+ end
+
+ def parents
+ @parents ||= ([parent] + (parent ? parent.parents : [])).compact
+ end
+ end
+ end
+ end
+end
--- /dev/null
+# RFC 4646/47 compliant Locale tag implementation that parses locale tags to
+# subtags such as language, script, region, variant etc.
+#
+# For more information see by http://en.wikipedia.org/wiki/IETF_language_tag
+#
+# Rfc4646::Parser does not implement grandfathered tags.
+
+module I18n
+ module Locale
+ module Tag
+ RFC4646_SUBTAGS = [ :language, :script, :region, :variant, :extension, :privateuse, :grandfathered ]
+ RFC4646_FORMATS = { :language => :downcase, :script => :capitalize, :region => :upcase, :variant => :downcase }
+
+ class Rfc4646 < Struct.new(*RFC4646_SUBTAGS)
+ class << self
+ # Parses the given tag and returns a Tag instance if it is valid.
+ # Returns false if the given tag is not valid according to RFC 4646.
+ def tag(tag)
+ matches = parser.match(tag)
+ new(*matches) if matches
+ end
+
+ def parser
+ @@parser ||= Rfc4646::Parser
+ end
+
+ def parser=(parser)
+ @@parser = parser
+ end
+ end
+
+ include Parents
+
+ RFC4646_FORMATS.each do |name, format|
+ define_method(name) { self[name].send(format) unless self[name].nil? }
+ end
+
+ def to_sym
+ to_s.to_sym
+ end
+
+ def to_s
+ @tag ||= to_a.compact.join("-")
+ end
+
+ def to_a
+ members.collect { |attr| self.send(attr) }
+ end
+
+ module Parser
+ PATTERN = %r{\A(?:
+ ([a-z]{2,3}(?:(?:-[a-z]{3}){0,3})?|[a-z]{4}|[a-z]{5,8}) # language
+ (?:-([a-z]{4}))? # script
+ (?:-([a-z]{2}|\d{3}))? # region
+ (?:-([0-9a-z]{5,8}|\d[0-9a-z]{3}))* # variant
+ (?:-([0-9a-wyz](?:-[0-9a-z]{2,8})+))* # extension
+ (?:-(x(?:-[0-9a-z]{1,8})+))?| # privateuse subtag
+ (x(?:-[0-9a-z]{1,8})+)| # privateuse tag
+ /* ([a-z]{1,3}(?:-[0-9a-z]{2,8}){1,2}) */ # grandfathered
+ )\z}xi
+
+ class << self
+ def match(tag)
+ c = PATTERN.match(tag.to_s).captures
+ c[0..4] << (c[5].nil? ? c[6] : c[5]) << c[7] # TODO c[7] is grandfathered, throw a NotImplemented exception here?
+ rescue
+ false
+ end
+ end
+ end
+ end
+ end
+ end
+end
--- /dev/null
+# Simple Locale tag implementation that computes subtags by simply splitting
+# the locale tag at '-' occurences.
+module I18n
+ module Locale
+ module Tag
+ class Simple
+ class << self
+ def tag(tag)
+ new(tag)
+ end
+ end
+
+ include Parents
+
+ attr_reader :tag
+
+ def initialize(*tag)
+ @tag = tag.join('-').to_sym
+ end
+
+ def subtags
+ @subtags = tag.to_s.split('-').map { |subtag| subtag.to_s }
+ end
+
+ def to_sym
+ tag
+ end
+
+ def to_s
+ tag.to_s
+ end
+
+ def to_a
+ subtags
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module I18n
+ module Tests
+ autoload :Basics, 'i18n/tests/basics'
+ autoload :Defaults, 'i18n/tests/defaults'
+ autoload :Interpolation, 'i18n/tests/interpolation'
+ autoload :Link, 'i18n/tests/link'
+ autoload :Localization, 'i18n/tests/localization'
+ autoload :Lookup, 'i18n/tests/lookup'
+ autoload :Pluralization, 'i18n/tests/pluralization'
+ autoload :Procs, 'i18n/tests/procs'
+ end
+end
--- /dev/null
+module I18n
+ module Tests
+ module Basics
+ def teardown
+ I18n.available_locales = nil
+ end
+
+ test "available_locales returns the locales stored to the backend by default" do
+ I18n.backend.store_translations('de', :foo => 'bar')
+ I18n.backend.store_translations('en', :foo => 'foo')
+
+ assert I18n.available_locales.include?(:de)
+ assert I18n.available_locales.include?(:en)
+ end
+
+ test "available_locales can be set to something else independently from the actual locale data" do
+ I18n.backend.store_translations('de', :foo => 'bar')
+ I18n.backend.store_translations('en', :foo => 'foo')
+
+ I18n.available_locales = :foo
+ assert_equal [:foo], I18n.available_locales
+
+ I18n.available_locales = [:foo, 'bar']
+ assert_equal [:foo, :bar], I18n.available_locales
+
+ I18n.available_locales = nil
+ assert I18n.available_locales.include?(:de)
+ assert I18n.available_locales.include?(:en)
+ end
+
+ test "available_locales memoizes when set explicitely" do
+ I18n.backend.expects(:available_locales).never
+ I18n.available_locales = [:foo]
+ I18n.backend.store_translations('de', :bar => 'baz')
+ I18n.reload!
+ assert_equal [:foo], I18n.available_locales
+ end
+
+ test "available_locales delegates to the backend when not set explicitely" do
+ I18n.backend.expects(:available_locales).twice
+ assert_equal I18n.available_locales, I18n.available_locales
+ end
+
+ test "storing a nil value as a translation removes it from the available locale data" do
+ I18n.backend.store_translations(:en, :to_be_deleted => 'bar')
+ assert_equal 'bar', I18n.t(:to_be_deleted, :default => 'baz')
+
+ I18n.cache_store.clear if I18n.respond_to?(:cache_store) && I18n.cache_store
+ I18n.backend.store_translations(:en, :to_be_deleted => nil)
+ assert_equal 'baz', I18n.t(:to_be_deleted, :default => 'baz')
+ end
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Tests
+ module Defaults
+ def setup
+ super
+ I18n.backend.store_translations(:en, :foo => { :bar => 'bar', :baz => 'baz' })
+ end
+
+ test "defaults: given nil as a key it returns the given default" do
+ assert_equal 'default', I18n.t(nil, :default => 'default')
+ end
+
+ test "defaults: given a symbol as a default it translates the symbol" do
+ assert_equal 'bar', I18n.t(nil, :default => :'foo.bar')
+ end
+
+ test "defaults: given a symbol as a default and a scope it stays inside the scope when looking up the symbol" do
+ assert_equal 'bar', I18n.t(:missing, :default => :bar, :scope => :foo)
+ end
+
+ test "defaults: given an array as a default it returns the first match" do
+ assert_equal 'bar', I18n.t(:does_not_exist, :default => [:does_not_exist_2, :'foo.bar'])
+ end
+
+ test "defaults: given an array of missing keys it raises a MissingTranslationData exception" do
+ assert_raise I18n::MissingTranslationData do
+ I18n.t(:does_not_exist, :default => [:does_not_exist_2, :does_not_exist_3], :raise => true)
+ end
+ end
+
+ test "defaults: using a custom scope separator" do
+ # data must have been stored using the custom separator when using the ActiveRecord backend
+ I18n.backend.store_translations(:en, { :foo => { :bar => 'bar' } }, { :separator => '|' })
+ assert_equal 'bar', I18n.t(nil, :default => :'foo|bar', :separator => '|')
+ end
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Tests
+ module Interpolation
+ # If no interpolation parameter is not given, I18n should not alter the string.
+ # This behavior is due to three reasons:
+ #
+ # * Checking interpolation keys in all strings hits performance, badly;
+ #
+ # * This allows us to retrieve untouched values through I18n. For example
+ # I could have a middleware that returns I18n lookup results in JSON
+ # to be processed through Javascript. Leaving the keys untouched allows
+ # the interpolation to happen at the javascript level;
+ #
+ # * Security concerns: if I allow users to translate a web site, they can
+ # insert %{} in messages causing the I18n lookup to fail in every request.
+ #
+ test "interpolation: given no values it does not alter the string" do
+ assert_equal 'Hi %{name}!', interpolate(:default => 'Hi %{name}!')
+ end
+
+ test "interpolation: given values it interpolates them into the string" do
+ assert_equal 'Hi David!', interpolate(:default => 'Hi %{name}!', :name => 'David')
+ end
+
+ test "interpolation: given a nil value it still interpolates it into the string" do
+ assert_equal 'Hi !', interpolate(:default => 'Hi %{name}!', :name => nil)
+ end
+
+ test "interpolation: given a lambda as a value it calls it if the string contains the key" do
+ assert_equal 'Hi David!', interpolate(:default => 'Hi %{name}!', :name => lambda { |*args| 'David' })
+ end
+
+ test "interpolation: given a lambda as a value it does not call it if the string does not contain the key" do
+ assert_nothing_raised { interpolate(:default => 'Hi!', :name => lambda { |*args| raise 'fail' }) }
+ end
+
+ test "interpolation: given values but missing a key it raises I18n::MissingInterpolationArgument" do
+ assert_raise(I18n::MissingInterpolationArgument) do
+ interpolate(:default => '%{foo}', :bar => 'bar')
+ end
+ end
+
+ test "interpolation: it does not raise I18n::MissingInterpolationArgument for escaped variables" do
+ assert_nothing_raised(I18n::MissingInterpolationArgument) do
+ assert_equal 'Barr %{foo}', interpolate(:default => '%{bar} %%{foo}', :bar => 'Barr')
+ end
+ end
+
+ test "interpolation: it does not change the original, stored translation string" do
+ I18n.backend.store_translations(:en, :interpolate => 'Hi %{name}!')
+ assert_equal 'Hi David!', interpolate(:interpolate, :name => 'David')
+ assert_equal 'Hi Yehuda!', interpolate(:interpolate, :name => 'Yehuda')
+ end
+
+ test "interpolation: given the translation is in utf-8 it still works" do
+ assert_equal 'Häi David!', interpolate(:default => 'Häi %{name}!', :name => 'David')
+ end
+
+ test "interpolation: given the value is in utf-8 it still works" do
+ assert_equal 'Hi ゆきひろ!', interpolate(:default => 'Hi %{name}!', :name => 'ゆきひろ')
+ end
+
+ test "interpolation: given the translation and the value are in utf-8 it still works" do
+ assert_equal 'こんにちは、ゆきひろさん!', interpolate(:default => 'こんにちは、%{name}さん!', :name => 'ゆきひろ')
+ end
+
+ if Kernel.const_defined?(:Encoding)
+ test "interpolation: given a euc-jp translation and a utf-8 value it raises Encoding::CompatibilityError" do
+ assert_raise(Encoding::CompatibilityError) do
+ interpolate(:default => euc_jp('こんにちは、%{name}さん!'), :name => 'ゆきひろ')
+ end
+ end
+
+ test "interpolation: given a utf-8 translation and a euc-jp value it raises Encoding::CompatibilityError" do
+ assert_raise(Encoding::CompatibilityError) do
+ interpolate(:default => 'こんにちは、%{name}さん!', :name => euc_jp('ゆきひろ'))
+ end
+ end
+
+ test "interpolation: ASCII strings in the backend should be encoded to UTF8 if interpolation options are in UTF8" do
+ I18n.backend.store_translations 'en', 'encoding' => ('%{who} let me go'.force_encoding("ASCII"))
+ result = I18n.t 'encoding', :who => "måmmå miå"
+ assert_equal Encoding::UTF_8, result.encoding
+ end
+
+ test "interpolation: UTF8 strings in the backend are still returned as UTF8 with ASCII interpolation" do
+ I18n.backend.store_translations 'en', 'encoding' => 'måmmå miå %{what}'
+ result = I18n.t 'encoding', :what => 'let me go'.force_encoding("ASCII")
+ assert_equal Encoding::UTF_8, result.encoding
+ end
+
+ test "interpolation: UTF8 strings in the backend are still returned as UTF8 even with numbers interpolation" do
+ I18n.backend.store_translations 'en', 'encoding' => '%{count} times: måmmå miå'
+ result = I18n.t 'encoding', :count => 3
+ assert_equal Encoding::UTF_8, result.encoding
+ end
+ end
+
+ test "interpolation: given a translations containing a reserved key it raises I18n::ReservedInterpolationKey" do
+ assert_raise(I18n::ReservedInterpolationKey) { interpolate(:default => '%{default}', :foo => :bar) }
+ assert_raise(I18n::ReservedInterpolationKey) { interpolate(:default => '%{scope}', :foo => :bar) }
+ assert_raise(I18n::ReservedInterpolationKey) { interpolate(:default => '%{separator}', :foo => :bar) }
+ end
+
+ protected
+
+ def capture(stream)
+ begin
+ stream = stream.to_s
+ eval "$#{stream} = StringIO.new"
+ yield
+ result = eval("$#{stream}").string
+ ensure
+ eval("$#{stream} = #{stream.upcase}")
+ end
+
+ result
+ end
+
+ def euc_jp(string)
+ string.encode!(Encoding::EUC_JP)
+ end
+
+ def interpolate(*args)
+ options = args.last.is_a?(Hash) ? args.pop : {}
+ key = args.pop
+ I18n.backend.translate('en', key, options)
+ end
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Tests
+ module Link
+ test "linked lookup: if a key resolves to a symbol it looks up the symbol" do
+ I18n.backend.store_translations 'en', {
+ :link => :linked,
+ :linked => 'linked'
+ }
+ assert_equal 'linked', I18n.backend.translate('en', :link)
+ end
+
+ test "linked lookup: if a key resolves to a dot-separated symbol it looks up the symbol" do
+ I18n.backend.store_translations 'en', {
+ :link => :"foo.linked",
+ :foo => { :linked => 'linked' }
+ }
+ assert_equal('linked', I18n.backend.translate('en', :link))
+ end
+
+ test "linked lookup: if a dot-separated key resolves to a symbol it looks up the symbol" do
+ I18n.backend.store_translations 'en', {
+ :foo => { :link => :linked },
+ :linked => 'linked'
+ }
+ assert_equal('linked', I18n.backend.translate('en', :'foo.link'))
+ end
+
+ test "linked lookup: if a dot-separated key resolves to a dot-separated symbol it looks up the symbol" do
+ I18n.backend.store_translations 'en', {
+ :foo => { :link => :"bar.linked" },
+ :bar => { :linked => 'linked' }
+ }
+ assert_equal('linked', I18n.backend.translate('en', :'foo.link'))
+ end
+
+ test "linked lookup: links always refer to the absolute key" do
+ I18n.backend.store_translations 'en', {
+ :foo => { :link => :linked, :linked => 'linked in foo' },
+ :linked => 'linked absolutely'
+ }
+ assert_equal 'linked absolutely', I18n.backend.translate('en', :link, :scope => :foo)
+ end
+
+ test "linked lookup: a link can resolve to a namespace in the middle of a dot-separated key" do
+ I18n.backend.store_translations 'en', {
+ :activemodel => { :errors => { :messages => { :blank => "can't be blank" } } },
+ :activerecord => { :errors => { :messages => :"activemodel.errors.messages" } }
+ }
+ assert_equal "can't be blank", I18n.t(:"activerecord.errors.messages.blank")
+ assert_equal "can't be blank", I18n.t(:"activerecord.errors.messages.blank")
+ end
+ end
+ end
+end
--- /dev/null
+module I18n
+ module Tests
+ module Localization
+ autoload :Date, 'i18n/tests/localization/date'
+ autoload :DateTime, 'i18n/tests/localization/date_time'
+ autoload :Time, 'i18n/tests/localization/time'
+ autoload :Procs, 'i18n/tests/localization/procs'
+
+ def self.included(base)
+ base.class_eval do
+ include I18n::Tests::Localization::Date
+ include I18n::Tests::Localization::DateTime
+ include I18n::Tests::Localization::Procs
+ include I18n::Tests::Localization::Time
+ end
+ end
+ end
+ end
+end
\ No newline at end of file
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Tests
+ module Localization
+ module Date
+ def setup
+ super
+ setup_date_translations
+ @date = ::Date.new(2008, 3, 1)
+ end
+
+ test "localize Date: given the short format it uses it" do
+ # TODO should be Mrz, shouldn't it?
+ assert_equal '01. Mar', I18n.l(@date, :format => :short, :locale => :de)
+ end
+
+ test "localize Date: given the long format it uses it" do
+ assert_equal '01. März 2008', I18n.l(@date, :format => :long, :locale => :de)
+ end
+
+ test "localize Date: given the default format it uses it" do
+ assert_equal '01.03.2008', I18n.l(@date, :format => :default, :locale => :de)
+ end
+
+ test "localize Date: given a day name format it returns the correct day name" do
+ assert_equal 'Samstag', I18n.l(@date, :format => '%A', :locale => :de)
+ end
+
+ test "localize Date: given an abbreviated day name format it returns the correct abbreviated day name" do
+ assert_equal 'Sa', I18n.l(@date, :format => '%a', :locale => :de)
+ end
+
+ test "localize Date: given a month name format it returns the correct month name" do
+ assert_equal 'März', I18n.l(@date, :format => '%B', :locale => :de)
+ end
+
+ test "localize Date: given an abbreviated month name format it returns the correct abbreviated month name" do
+ # TODO should be Mrz, shouldn't it?
+ assert_equal 'Mar', I18n.l(@date, :format => '%b', :locale => :de)
+ end
+
+ test "localize Date: given an unknown format it does not fail" do
+ assert_nothing_raised { I18n.l(@date, :format => '%x') }
+ end
+
+ test "localize Date: given nil it raises I18n::ArgumentError" do
+ assert_raise(I18n::ArgumentError) { I18n.l(nil) }
+ end
+
+ test "localize Date: given a plain Object it raises I18n::ArgumentError" do
+ assert_raise(I18n::ArgumentError) { I18n.l(Object.new) }
+ end
+
+ test "localize Date: given a format is missing it raises I18n::MissingTranslationData" do
+ assert_raise(I18n::MissingTranslationData) { I18n.l(@date, :format => :missing) }
+ end
+
+ test "localize Date: it does not alter the format string" do
+ assert_equal '01. Februar 2009', I18n.l(::Date.parse('2009-02-01'), :format => :long, :locale => :de)
+ assert_equal '01. Oktober 2009', I18n.l(::Date.parse('2009-10-01'), :format => :long, :locale => :de)
+ end
+
+ protected
+
+ def setup_date_translations
+ I18n.backend.store_translations :de, {
+ :date => {
+ :formats => {
+ :default => "%d.%m.%Y",
+ :short => "%d. %b",
+ :long => "%d. %B %Y",
+ },
+ :day_names => %w(Sonntag Montag Dienstag Mittwoch Donnerstag Freitag Samstag),
+ :abbr_day_names => %w(So Mo Di Mi Do Fr Sa),
+ :month_names => %w(Januar Februar März April Mai Juni Juli August September Oktober November Dezember).unshift(nil),
+ :abbr_month_names => %w(Jan Feb Mar Apr Mai Jun Jul Aug Sep Okt Nov Dez).unshift(nil)
+ }
+ }
+ end
+ end
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Tests
+ module Localization
+ module DateTime
+ def setup
+ super
+ setup_datetime_translations
+ @datetime = ::DateTime.new(2008, 3, 1, 6)
+ @other_datetime = ::DateTime.new(2008, 3, 1, 18)
+ end
+
+ test "localize DateTime: given the short format it uses it" do
+ # TODO should be Mrz, shouldn't it?
+ assert_equal '01. Mar 06:00', I18n.l(@datetime, :format => :short, :locale => :de)
+ end
+
+ test "localize DateTime: given the long format it uses it" do
+ assert_equal '01. März 2008 06:00', I18n.l(@datetime, :format => :long, :locale => :de)
+ end
+
+ test "localize DateTime: given the default format it uses it" do
+ # TODO should be Mrz, shouldn't it?
+ assert_equal 'Sa, 01. Mar 2008 06:00:00 +0000', I18n.l(@datetime, :format => :default, :locale => :de)
+ end
+
+ test "localize DateTime: given a day name format it returns the correct day name" do
+ assert_equal 'Samstag', I18n.l(@datetime, :format => '%A', :locale => :de)
+ end
+
+ test "localize DateTime: given an abbreviated day name format it returns the correct abbreviated day name" do
+ assert_equal 'Sa', I18n.l(@datetime, :format => '%a', :locale => :de)
+ end
+
+ test "localize DateTime: given a month name format it returns the correct month name" do
+ assert_equal 'März', I18n.l(@datetime, :format => '%B', :locale => :de)
+ end
+
+ test "localize DateTime: given an abbreviated month name format it returns the correct abbreviated month name" do
+ # TODO should be Mrz, shouldn't it?
+ assert_equal 'Mar', I18n.l(@datetime, :format => '%b', :locale => :de)
+ end
+
+ test "localize DateTime: given a meridian indicator format it returns the correct meridian indicator" do
+ assert_equal 'am', I18n.l(@datetime, :format => '%p', :locale => :de)
+ assert_equal 'pm', I18n.l(@other_datetime, :format => '%p', :locale => :de)
+ end
+
+ test "localize DateTime: given an unknown format it does not fail" do
+ assert_nothing_raised { I18n.l(@datetime, :format => '%x') }
+ end
+
+ test "localize DateTime: given a format is missing it raises I18n::MissingTranslationData" do
+ assert_raise(I18n::MissingTranslationData) { I18n.l(@datetime, :format => :missing) }
+ end
+
+ protected
+
+ def setup_datetime_translations
+ # time translations might have been set up in Tests::Api::Localization::Time
+ I18n.backend.store_translations :de, {
+ :time => {
+ :formats => {
+ :default => "%a, %d. %b %Y %H:%M:%S %z",
+ :short => "%d. %b %H:%M",
+ :long => "%d. %B %Y %H:%M"
+ },
+ :am => 'am',
+ :pm => 'pm'
+ }
+ }
+ end
+ end
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Tests
+ module Localization
+ module Procs
+ test "localize: using day names from lambdas" do
+ setup_time_proc_translations
+ time = ::Time.utc(2008, 3, 1, 6, 0)
+ assert_match(/Суббота/, I18n.l(time, :format => "%A, %d %B", :locale => :ru))
+ assert_match(/суббота/, I18n.l(time, :format => "%d %B (%A)", :locale => :ru))
+ end
+
+ test "localize: using month names from lambdas" do
+ setup_time_proc_translations
+ time = ::Time.utc(2008, 3, 1, 6, 0)
+ assert_match(/марта/, I18n.l(time, :format => "%d %B %Y", :locale => :ru))
+ assert_match(/Март /, I18n.l(time, :format => "%B %Y", :locale => :ru))
+ end
+
+ test "localize: using abbreviated day names from lambdas" do
+ setup_time_proc_translations
+ time = ::Time.utc(2008, 3, 1, 6, 0)
+ assert_match(/марта/, I18n.l(time, :format => "%d %b %Y", :locale => :ru))
+ assert_match(/март /, I18n.l(time, :format => "%b %Y", :locale => :ru))
+ end
+
+ test "localize Date: given a format that resolves to a Proc it calls the Proc with the object" do
+ setup_time_proc_translations
+ date = ::Date.new(2008, 3, 1, 6)
+ assert_equal '[Sat, 01 Mar 2008, {}]', I18n.l(date, :format => :proc, :locale => :ru)
+ end
+
+ test "localize Date: given a format that resolves to a Proc it calls the Proc with the object and extra options" do
+ setup_time_proc_translations
+ date = ::Date.new(2008, 3, 1, 6)
+ assert_equal '[Sat, 01 Mar 2008, {:foo=>"foo"}]', I18n.l(date, :format => :proc, :foo => 'foo', :locale => :ru)
+ end
+
+ test "localize DateTime: given a format that resolves to a Proc it calls the Proc with the object" do
+ setup_time_proc_translations
+ datetime = ::DateTime.new(2008, 3, 1, 6)
+ assert_equal '[Sat, 01 Mar 2008 06:00:00 +00:00, {}]', I18n.l(datetime, :format => :proc, :locale => :ru)
+ end
+
+ test "localize DateTime: given a format that resolves to a Proc it calls the Proc with the object and extra options" do
+ setup_time_proc_translations
+ datetime = ::DateTime.new(2008, 3, 1, 6)
+ assert_equal '[Sat, 01 Mar 2008 06:00:00 +00:00, {:foo=>"foo"}]', I18n.l(datetime, :format => :proc, :foo => 'foo', :locale => :ru)
+ end
+
+ test "localize Time: given a format that resolves to a Proc it calls the Proc with the object" do
+ setup_time_proc_translations
+ time = ::Time.utc(2008, 3, 1, 6, 0)
+ assert_equal inspect_args([time, {}]), I18n.l(time, :format => :proc, :locale => :ru)
+ end
+
+ test "localize Time: given a format that resolves to a Proc it calls the Proc with the object and extra options" do
+ setup_time_proc_translations
+ time = ::Time.utc(2008, 3, 1, 6, 0)
+ options = { :foo => 'foo' }
+ assert_equal inspect_args([time, options]), I18n.l(time, options.merge(:format => :proc, :locale => :ru))
+ end
+
+ protected
+
+ def inspect_args(args)
+ args = args.map do |arg|
+ case arg
+ when ::Time, ::DateTime
+ arg.strftime('%a, %d %b %Y %H:%M:%S %Z').sub('+0000', '+00:00')
+ when ::Date
+ arg.strftime('%a, %d %b %Y')
+ when Hash
+ arg.delete(:fallback)
+ arg.inspect
+ else
+ arg.inspect
+ end
+ end
+ "[#{args.join(', ')}]"
+ end
+
+ def setup_time_proc_translations
+ I18n.backend.store_translations :ru, {
+ :time => {
+ :formats => {
+ :proc => lambda { |*args| inspect_args(args) }
+ }
+ },
+ :date => {
+ :formats => {
+ :proc => lambda { |*args| inspect_args(args) }
+ },
+ :'day_names' => lambda { |key, options|
+ (options[:format] =~ /^%A/) ?
+ %w(Воскресенье Понедельник Вторник Среда Четверг Пятница Суббота) :
+ %w(воскресенье понедельник вторник среда четверг пятница суббота)
+ },
+ :'month_names' => lambda { |key, options|
+ (options[:format] =~ /(%d|%e)(\s*)?(%B)/) ?
+ %w(января февраля марта апреля мая июня июля августа сентября октября ноября декабря).unshift(nil) :
+ %w(Январь Февраль Март Апрель Май Июнь Июль Август Сентябрь Октябрь Ноябрь Декабрь).unshift(nil)
+ },
+ :'abbr_month_names' => lambda { |key, options|
+ (options[:format] =~ /(%d|%e)(\s*)(%b)/) ?
+ %w(янв. февр. марта апр. мая июня июля авг. сент. окт. нояб. дек.).unshift(nil) :
+ %w(янв. февр. март апр. май июнь июль авг. сент. окт. нояб. дек.).unshift(nil)
+ },
+ }
+ }
+ end
+ end
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Tests
+ module Localization
+ module Time
+ def setup
+ super
+ setup_time_translations
+ @time = ::Time.utc(2008, 3, 1, 6, 0)
+ @other_time = ::Time.utc(2008, 3, 1, 18, 0)
+ end
+
+ test "localize Time: given the short format it uses it" do
+ # TODO should be Mrz, shouldn't it?
+ assert_equal '01. Mar 06:00', I18n.l(@time, :format => :short, :locale => :de)
+ end
+
+ test "localize Time: given the long format it uses it" do
+ assert_equal '01. März 2008 06:00', I18n.l(@time, :format => :long, :locale => :de)
+ end
+
+ # TODO Seems to break on Windows because ENV['TZ'] is ignored. What's a better way to do this?
+ # def test_localize_given_the_default_format_it_uses_it
+ # assert_equal 'Sa, 01. Mar 2008 06:00:00 +0000', I18n.l(@time, :format => :default, :locale => :de)
+ # end
+
+ test "localize Time: given a day name format it returns the correct day name" do
+ assert_equal 'Samstag', I18n.l(@time, :format => '%A', :locale => :de)
+ end
+
+ test "localize Time: given an abbreviated day name format it returns the correct abbreviated day name" do
+ assert_equal 'Sa', I18n.l(@time, :format => '%a', :locale => :de)
+ end
+
+ test "localize Time: given a month name format it returns the correct month name" do
+ assert_equal 'März', I18n.l(@time, :format => '%B', :locale => :de)
+ end
+
+ test "localize Time: given an abbreviated month name format it returns the correct abbreviated month name" do
+ # TODO should be Mrz, shouldn't it?
+ assert_equal 'Mar', I18n.l(@time, :format => '%b', :locale => :de)
+ end
+
+ test "localize Time: given a meridian indicator format it returns the correct meridian indicator" do
+ assert_equal 'am', I18n.l(@time, :format => '%p', :locale => :de)
+ assert_equal 'pm', I18n.l(@other_time, :format => '%p', :locale => :de)
+ end
+
+ test "localize Time: given an unknown format it does not fail" do
+ assert_nothing_raised { I18n.l(@time, :format => '%x') }
+ end
+
+ test "localize Time: given a format is missing it raises I18n::MissingTranslationData" do
+ assert_raise(I18n::MissingTranslationData) { I18n.l(@time, :format => :missing) }
+ end
+
+ protected
+
+ def setup_time_translations
+ I18n.backend.store_translations :de, {
+ :time => {
+ :formats => {
+ :default => "%a, %d. %b %Y %H:%M:%S %z",
+ :short => "%d. %b %H:%M",
+ :long => "%d. %B %Y %H:%M",
+ },
+ :am => 'am',
+ :pm => 'pm'
+ }
+ }
+ end
+ end
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Tests
+ module Lookup
+ def setup
+ super
+ I18n.backend.store_translations(:en, :foo => { :bar => 'bar', :baz => 'baz' }, :falsy => false, :truthy => true,
+ :string => "a", :array => %w(a b c), :hash => { "a" => "b" })
+ end
+
+ test "lookup: it returns a string" do
+ assert_equal("a", I18n.t(:string))
+ end
+
+ test "lookup: it returns hash" do
+ assert_equal({ :a => "b" }, I18n.t(:hash))
+ end
+
+ test "lookup: it returns a array" do
+ assert_equal(%w(a b c), I18n.t(:array))
+ end
+
+ test "lookup: it returns a native true" do
+ assert I18n.t(:truthy) === true
+ end
+
+ test "lookup: it returns a native false" do
+ assert I18n.t(:falsy) === false
+ end
+
+ test "lookup: given a missing key, no default and no raise option it returns an error message" do
+ assert_equal "translation missing: en.missing", I18n.t(:missing)
+ end
+
+ test "lookup: given a missing key, no default and the raise option it raises MissingTranslationData" do
+ assert_raise(I18n::MissingTranslationData) { I18n.t(:missing, :raise => true) }
+ end
+
+ test "lookup: does not raise an exception if no translation data is present for the given locale" do
+ assert_nothing_raised { I18n.t(:foo, :locale => :xx) }
+ end
+
+ test "lookup: given an array of keys it translates all of them" do
+ assert_equal %w(bar baz), I18n.t([:bar, :baz], :scope => [:foo])
+ end
+
+ test "lookup: using a custom scope separator" do
+ # data must have been stored using the custom separator when using the ActiveRecord backend
+ I18n.backend.store_translations(:en, { :foo => { :bar => 'bar' } }, { :separator => '|' })
+ assert_equal 'bar', I18n.t('foo|bar', :separator => '|')
+ end
+
+ # In fact it probably *should* fail but Rails currently relies on using the default locale instead.
+ # So we'll stick to this for now until we get it fixed in Rails.
+ test "lookup: given nil as a locale it does not raise but use the default locale" do
+ # assert_raise(I18n::InvalidLocale) { I18n.t(:bar, :locale => nil) }
+ assert_nothing_raised { I18n.t(:bar, :locale => nil) }
+ end
+
+ test "lookup: a resulting String is not frozen" do
+ assert !I18n.t(:string).frozen?
+ end
+
+ test "lookup: a resulting Array is not frozen" do
+ assert !I18n.t(:array).frozen?
+ end
+
+ test "lookup: a resulting Hash is not frozen" do
+ assert !I18n.t(:hash).frozen?
+ end
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Tests
+ module Pluralization
+ test "pluralization: given 0 it returns the :zero translation if it is defined" do
+ assert_equal 'zero', I18n.t(:default => { :zero => 'zero' }, :count => 0)
+ end
+
+ test "pluralization: given 0 it returns the :other translation if :zero is not defined" do
+ assert_equal 'bars', I18n.t(:default => { :other => 'bars' }, :count => 0)
+ end
+
+ test "pluralization: given 1 it returns the singular translation" do
+ assert_equal 'bar', I18n.t(:default => { :one => 'bar' }, :count => 1)
+ end
+
+ test "pluralization: given 2 it returns the :other translation" do
+ assert_equal 'bars', I18n.t(:default => { :other => 'bars' }, :count => 2)
+ end
+
+ test "pluralization: given 3 it returns the :other translation" do
+ assert_equal 'bars', I18n.t(:default => { :other => 'bars' }, :count => 3)
+ end
+
+ test "pluralization: given nil it returns the whole entry" do
+ assert_equal({ :one => 'bar' }, I18n.t(:default => { :one => 'bar' }, :count => nil))
+ end
+
+ test "pluralization: given incomplete pluralization data it raises I18n::InvalidPluralizationData" do
+ assert_raise(I18n::InvalidPluralizationData) { I18n.t(:default => { :one => 'bar' }, :count => 2) }
+ end
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+
+module I18n
+ module Tests
+ module Procs
+ test "lookup: given a translation is a proc it calls the proc with the key and interpolation values" do
+ I18n.backend.store_translations(:en, :a_lambda => lambda { |*args| filter_args(*args) })
+ assert_equal '[:a_lambda, {:foo=>"foo"}]', I18n.t(:a_lambda, :foo => 'foo')
+ end
+
+ test "defaults: given a default is a Proc it calls it with the key and interpolation values" do
+ proc = lambda { |*args| filter_args(*args) }
+ assert_equal '[nil, {:foo=>"foo"}]', I18n.t(nil, :default => proc, :foo => 'foo')
+ end
+
+ test "defaults: given a default is a key that resolves to a Proc it calls it with the key and interpolation values" do
+ I18n.backend.store_translations(:en, :a_lambda => lambda { |*args| filter_args(*args) })
+ assert_equal '[:a_lambda, {:foo=>"foo"}]', I18n.t(nil, :default => :a_lambda, :foo => 'foo')
+ assert_equal '[:a_lambda, {:foo=>"foo"}]', I18n.t(nil, :default => [nil, :a_lambda], :foo => 'foo')
+ end
+
+ test "interpolation: given an interpolation value is a lambda it calls it with key and values before interpolating it" do
+ proc = lambda { |*args| filter_args(*args) }
+ assert_match %r(\[\{:foo=>#<Proc.*>\}\]), I18n.t(nil, :default => '%{foo}', :foo => proc)
+ end
+
+ test "interpolation: given a key resolves to a Proc that returns a string then interpolation still works" do
+ proc = lambda { |*args| "%{foo}: " + filter_args(*args) }
+ assert_equal 'foo: [nil, {:foo=>"foo"}]', I18n.t(nil, :default => proc, :foo => 'foo')
+ end
+
+ test "pluralization: given a key resolves to a Proc that returns valid data then pluralization still works" do
+ proc = lambda { |*args| { :zero => 'zero', :one => 'one', :other => 'other' } }
+ assert_equal 'zero', I18n.t(:default => proc, :count => 0)
+ assert_equal 'one', I18n.t(:default => proc, :count => 1)
+ assert_equal 'other', I18n.t(:default => proc, :count => 2)
+ end
+
+ test "lookup: given the option :resolve => false was passed it does not resolve proc translations" do
+ I18n.backend.store_translations(:en, :a_lambda => lambda { |*args| filter_args(*args) })
+ assert_equal Proc, I18n.t(:a_lambda, :resolve => false).class
+ end
+
+ test "lookup: given the option :resolve => false was passed it does not resolve proc default" do
+ assert_equal Proc, I18n.t(nil, :default => lambda { |*args| filter_args(*args) }, :resolve => false).class
+ end
+
+ protected
+
+ def filter_args(*args)
+ args.map {|arg| arg.delete(:fallback) if arg.is_a?(Hash) ; arg }.inspect
+ end
+ end
+ end
+end
--- /dev/null
+module I18n
+ VERSION = "0.6.1"
+end
--- /dev/null
+.*.sw[pon]
+coverage
+pkg
+.nfs.*
+.idea
+java/Json.iml
+Gemfile.lock
+.rvmrc
--- /dev/null
+# Passes arguments to bundle install (http://gembundler.com/man/bundle-install.1.html)
+bundler_args: --binstubs
+
+# Specify which ruby versions you wish to run your tests on, each version will be used
+rvm:
+ - 1.8.7
+ - 1.9.2
+ - 1.9.3
+ - rbx
+ - rbx-2.0
+ - ree
+ - jruby
+ - ruby-head
+
+script: "bundle exec rake"
--- /dev/null
+2013-02-04 (1.5.5)
+ * Security fix for JSON create_additions default value. It should not be
+ possible to create additions unless
+ explicitely requested by setting the create_additions argument to true or
+ using the JSON.load/dump interface.
+ * Backport change that corrects Time serialisation/deserialisation on some
+ platforms.
+2011-08-31 (1.5.4)
+ * Fix memory leak when used from multiple JRuby. (Patch by
+ jfirebaugh@github).
+ * Apply patch by Eric Wong <nocode@yhbt.net> that fixes garbage collection problem
+ reported in https://github.com/flori/json/issues/46.
+ * Add :quirks_mode option to parser and generator.
+ * Add support for Rational and Complex number additions via json/add/complex
+ and json/add/rational requires.
+2011-06-20 (1.5.3)
+ * Alias State#configure method as State#merge to increase duck type synonymy with Hash.
+ * Add as_json methods in json/add/core, so rails can create its json objects
+ the new way.
+2011-05-11 (1.5.2)
+ * Apply documentation patch by Cory Monty <cory.monty@gmail.com>.
+ * Add gemspecs for json and json_pure.
+ * Fix bug in jruby pretty printing.
+ * Fix bug in object_class and array_class when inheriting from Hash or Array.
+2011-01-24 (1.5.1)
+ * Made rake-compiler build a fat binary gem. This should fix issue
+ https://github.com/flori/json/issues#issue/54.
+2011-01-22 (1.5.0)
+ * Included Java source codes for the Jruby extension made by Daniel Luz
+ <dev@mernen.com>.
+ * Output full exception message of deep_const_get to aid debugging.
+ * Fixed an issue with ruby 1.9 Module#const_defined? method, that was
+ reported by Riley Goodside.
+2010-08-09 (1.4.6)
+ * Fixed oversight reported in http://github.com/flori/json/issues/closed#issue/23,
+ always create a new object from the state prototype.
+ * Made pure and ext api more similar again.
+2010-08-07 (1.4.5)
+ * Manage data structure nesting depth in state object during generation. This
+ should reduce problems with to_json method definіtions that only have one
+ argument.
+ * Some fixes in the state objects and additional tests.
+2010-08-06 (1.4.4)
+ * Fixes build problem for rubinius under OS X, http://github.com/flori/json/issues/closed#issue/25
+ * Fixes crashes described in http://github.com/flori/json/issues/closed#issue/21 and
+ http://github.com/flori/json/issues/closed#issue/23
+2010-05-05 (1.4.3)
+ * Fixed some test assertions, from Ruby r27587 and r27590, patch by nobu.
+ * Fixed issue http://github.com/flori/json/issues/#issue/20 reported by
+ electronicwhisper@github. Thx!
+2010-04-26 (1.4.2)
+ * Applied patch from naruse Yui NARUSE <naruse@airemix.com> to make building with
+ Microsoft Visual C possible again.
+ * Applied patch from devrandom <c1.github@niftybox.net> in order to allow building of
+ json_pure if extensiontask is not present.
+ * Thanks to Dustin Schneider <dustin@stocktwits.com>, who reported a memory
+ leak, which is fixed in this release.
+ * Applied 993f261ccb8f911d2ae57e9db48ec7acd0187283 patch from josh@github.
+2010-04-25 (1.4.1)
+ * Fix for a bug reported by Dan DeLeo <dan@kallistec.com>, caused by T_FIXNUM
+ being different on 32bit/64bit architectures.
+2010-04-23 (1.4.0)
+ * Major speed improvements and building with simplified
+ directory/file-structure.
+ * Extension should at least be comapatible with MRI, YARV and Rubinius.
+2010-04-07 (1.2.4)
+ * Triger const_missing callback to make Rails' dynamic class loading work.
+2010-03-11 (1.2.3)
+ * Added a State#[] method which returns an attribute's value in order to
+ increase duck type compatibility to Hash.
+2010-02-27 (1.2.2)
+ * Made some changes to make the building of the parser/generator compatible
+ to Rubinius.
+2009-11-25 (1.2.1)
+ * Added :symbolize_names option to Parser, which returns symbols instead of
+ strings in object names/keys.
+2009-10-01 (1.2.0)
+ * fast_generate now raises an exeception for nan and infinite floats.
+ * On Ruby 1.8 json supports parsing of UTF-8, UTF-16BE, UTF-16LE, UTF-32BE,
+ and UTF-32LE JSON documents now. Under Ruby 1.9 the M17n conversion
+ functions are used to convert from all supported encodings. ASCII-8BIT
+ encoded strings are handled like all strings under Ruby 1.8 were.
+ * Better documentation
+2009-08-23 (1.1.9)
+ * Added forgotten main doc file extra_rdoc_files.
+2009-08-23 (1.1.8)
+ * Applied a patch by OZAWA Sakuro <sakuro@2238club.org> to make json/pure
+ work in environments that don't provide iconv.
+ * Applied patch by okkez_ in order to fix Ruby Bug #1768:
+ http://redmine.ruby-lang.org/issues/show/1768.
+ * Finally got around to avoid the rather paranoid escaping of ?/ characters
+ in the generator's output. The parsers aren't affected by this change.
+ Thanks to Rich Apodaca <rapodaca@metamolecular.com> for the suggestion.
+2009-06-29 (1.1.7)
+ * Security Fix for JSON::Pure::Parser. A specially designed string could
+ cause catastrophic backtracking in one of the parser's regular expressions
+ in earlier 1.1.x versions. JSON::Ext::Parser isn't affected by this issue.
+ Thanks to Bartosz Blimke <bartosz@new-bamboo.co.uk> for reporting this
+ problem.
+ * This release also uses a less strict ruby version requirement for the
+ creation of the mswin32 native gem.
+2009-05-10 (1.1.6)
+ * No changes. І tested native linux gems in the last release and they don't
+ play well with different ruby versions other than the one the gem was built
+ with. This release is just to bump the version number in order to skip the
+ native gem on rubyforge.
+2009-05-10 (1.1.5)
+ * Started to build gems with rake-compiler gem.
+ * Applied patch object/array class patch from Brian Candler
+ <B.Candler@pobox.com> and fixes.
+2009-04-01 (1.1.4)
+ * Fixed a bug in the creation of serialized generic rails objects reported by
+ Friedrich Graeter <graeter@hydrixos.org>.
+ * Deleted tests/runner.rb, we're using testrb instead.
+ * Editor supports Infinity in numbers now.
+ * Made some changes in order to get the library to compile/run under Ruby
+ 1.9.
+ * Improved speed of the code path for the fast_generate method in the pure
+ variant.
+2008-07-10 (1.1.3)
+ * Wesley Beary <monki@geemus.com> reported a bug in json/add/core's DateTime
+ handling: If the nominator and denominator of the offset were divisible by
+ each other Ruby's Rational#to_s returns them as an integer not a fraction
+ with '/'. This caused a ZeroDivisionError during parsing.
+ * Use Date#start and DateTime#start instead of sg method, while
+ remaining backwards compatible.
+ * Supports ragel >= 6.0 now.
+ * Corrected some tests.
+ * Some minor changes.
+2007-11-27 (1.1.2)
+ * Remember default dir (last used directory) in editor.
+ * JSON::Editor.edit method added, the editor can now receive json texts from
+ the clipboard via C-v.
+ * Load json texts from an URL pasted via middle button press.
+ * Added :create_additions option to Parser. This makes it possible to disable
+ the creation of additions by force, in order to treat json texts as data
+ while having additions loaded.
+ * Jacob Maine <jmaine@blurb.com> reported, that JSON(:foo) outputs a JSON
+ object if the rails addition is enabled, which is wrong. It now outputs a
+ JSON string "foo" instead, like suggested by Jacob Maine.
+ * Discovered a bug in the Ruby Bugs Tracker on rubyforge, that was reported
+ by John Evans lgastako@gmail.com. He could produce a crash in the JSON
+ generator by returning something other than a String instance from a
+ to_json method. I now guard against this by doing a rather crude type
+ check, which raises an exception instead of crashing.
+2007-07-06 (1.1.1)
+ * Yui NARUSE <naruse@airemix.com> sent some patches to fix tests for Ruby
+ 1.9. I applied them and adapted some of them a bit to run both on 1.8 and
+ 1.9.
+ * Introduced a JSON.parse! method without depth checking for people who like
+ danger.
+ * Made generate and pretty_generate methods configurable by an options hash.
+ * Added :allow_nan option to parser and generator in order to handle NaN,
+ Infinity, and -Infinity correctly - if requested. Floats, which aren't numbers,
+ aren't valid JSON according to RFC4627, so by default an exception will be
+ raised if any of these symbols are encountered. Thanks to Andrea Censi
+ <andrea.censi@dis.uniroma1.it> for his hint about this.
+ * Fixed some more tests for Ruby 1.9.
+ * Implemented dump/load interface of Marshal as suggested in ruby-core:11405
+ by murphy <murphy@rubychan.de>.
+ * Implemented the max_nesting feature for generate methods, too.
+ * Added some implementations for ruby core's custom objects for
+ serialisation/deserialisation purposes.
+2007-05-21 (1.1.0)
+ * Implemented max_nesting feature for parser to avoid stack overflows for
+ data from untrusted sources. If you trust the source, you can disable it
+ with the option max_nesting => false.
+ * Piers Cawley <pdcawley@bofh.org.uk> reported a bug, that not every
+ character can be escaped by ?\ as required by RFC4627. There's a
+ contradiction between David Crockford's JSON checker test vectors (in
+ tests/fixtures) and RFC4627, though. I decided to stick to the RFC, because
+ the JSON checker seems to be a bit older than the RFC.
+ * Extended license to Ruby License, which includes the GPL.
+ * Added keyboard shortcuts, and 'Open location' menu item to edit_json.rb.
+2007-05-09 (1.0.4)
+ * Applied a patch from Yui NARUSE <naruse@airemix.com> to make JSON compile
+ under Ruby 1.9. Thank you very much for mailing it to me!
+ * Made binary variants of JSON fail early, instead of falling back to the
+ pure version. This should avoid overshadowing of eventual problems while
+ loading of the binary.
+2007-03-24 (1.0.3)
+ * Improved performance of pure variant a bit.
+ * The ext variant of this release supports the mswin32 platform. Ugh!
+2007-03-24 (1.0.2)
+ * Ext Parser didn't parse 0e0 correctly into 0.0: Fixed!
+2007-03-24 (1.0.1)
+ * Forgot some object files in the build dir. I really like that - not!
+2007-03-24 (1.0.0)
+ * Added C implementations for the JSON generator and a ragel based JSON
+ parser in C.
+ * Much more tests, especially fixtures from json.org.
+ * Further improved conformance to RFC4627.
+2007-02-09 (0.4.3)
+ * Conform more to RFC4627 for JSON: This means JSON strings
+ now always must contain exactly one object "{ ... }" or array "[ ... ]" in
+ order to be parsed without raising an exception. The definition of what
+ constitutes a whitespace is narrower in JSON than in Ruby ([ \t\r\n]), and
+ there are differences in floats and integers (no octals or hexadecimals) as
+ well.
+ * Added aliases generate and pretty_generate of unparse and pretty_unparse.
+ * Fixed a test case.
+ * Catch an Iconv::InvalidEncoding exception, that seems to occur on some Sun
+ boxes with SunOS 5.8, if iconv doesn't support utf16 conversions. This was
+ reported by Andrew R Jackson <andrewj@bcm.tmc.edu>, thanks a bunch!
+2006-08-25 (0.4.2)
+ * Fixed a bug in handling solidi (/-characters), that was reported by
+ Kevin Gilpin <kevin.gilpin@alum.mit.edu>.
+2006-02-06 (0.4.1)
+ * Fixed a bug related to escaping with backslashes. Thanks for the report go
+ to Florian Munz <surf@theflow.de>.
+2005-09-23 (0.4.0)
+ * Initial Rubyforge Version
--- /dev/null
+Ruby is copyrighted free software by Yukihiro Matsumoto <matz@netlab.co.jp>.
+You can redistribute it and/or modify it under either the terms of the GPL
+(see GPL file), or the conditions below:
+
+ 1. You may make and give away verbatim copies of the source form of the
+ software without restriction, provided that you duplicate all of the
+ original copyright notices and associated disclaimers.
+
+ 2. You may modify your copy of the software in any way, provided that
+ you do at least ONE of the following:
+
+ a) place your modifications in the Public Domain or otherwise
+ make them Freely Available, such as by posting said
+ modifications to Usenet or an equivalent medium, or by allowing
+ the author to include your modifications in the software.
+
+ b) use the modified software only within your corporation or
+ organization.
+
+ c) rename any non-standard executables so the names do not conflict
+ with standard executables, which must also be provided.
+
+ d) make other distribution arrangements with the author.
+
+ 3. You may distribute the software in object code or executable
+ form, provided that you do at least ONE of the following:
+
+ a) distribute the executables and library files of the software,
+ together with instructions (in the manual page or equivalent)
+ on where to get the original distribution.
+
+ b) accompany the distribution with the machine-readable source of
+ the software.
+
+ c) give non-standard executables non-standard names, with
+ instructions on where to get the original software distribution.
+
+ d) make other distribution arrangements with the author.
+
+ 4. You may modify and include the part of the software into any other
+ software (possibly commercial). But some files in the distribution
+ are not written by the author, so that they are not under this terms.
+
+ They are gc.c(partly), utils.c(partly), regex.[ch], st.[ch] and some
+ files under the ./missing directory. See each file for the copying
+ condition.
+
+ 5. The scripts and library files supplied as input to or produced as
+ output from the software do not automatically fall under the
+ copyright of the software, but belong to whomever generated them,
+ and may be sold commercially, and may be aggregated with this
+ software.
+
+ 6. THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR
+ IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+ WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ PURPOSE.
+
--- /dev/null
+JSON-JRuby is copyrighted free software by Daniel Luz <mernen at gmail dot com>,
+and is a derivative work of Florian Frank's json library <flori at ping dot de>.
+You can redistribute it and/or modify it under either the terms of the GPL
+version 2 (see the file GPL), or the conditions below:
+
+ 1. You may make and give away verbatim copies of the source form of the
+ software without restriction, provided that you duplicate all of the
+ original copyright notices and associated disclaimers.
+
+ 2. You may modify your copy of the software in any way, provided that
+ you do at least ONE of the following:
+
+ a) place your modifications in the Public Domain or otherwise
+ make them Freely Available, such as by posting said
+ modifications to Usenet or an equivalent medium, or by allowing
+ the author to include your modifications in the software.
+
+ b) use the modified software only within your corporation or
+ organization.
+
+ c) give non-standard binaries non-standard names, with
+ instructions on where to get the original software distribution.
+
+ d) make other distribution arrangements with the author.
+
+ 3. You may distribute the software in object code or binary form,
+ provided that you do at least ONE of the following:
+
+ a) distribute the binaries and library files of the software,
+ together with instructions (in the manual page or equivalent)
+ on where to get the original distribution.
+
+ b) accompany the distribution with the machine-readable source of
+ the software.
+
+ c) give non-standard binaries non-standard names, with
+ instructions on where to get the original software distribution.
+
+ d) make other distribution arrangements with the author.
+
+ 4. You may modify and include the part of the software into any other
+ software (possibly commercial). But some files in the distribution
+ are not written by the author, so that they are not under these terms.
+
+ For the list of those files and their copying conditions, see the
+ file LEGAL.
+
+ 5. The scripts and library files supplied as input to or produced as
+ output from the software do not automatically fall under the
+ copyright of the software, but belong to whomever generated them,
+ and may be sold commercially, and may be aggregated with this
+ software.
+
+ 6. THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR
+ IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
+ WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ PURPOSE.
--- /dev/null
+ GNU GENERAL PUBLIC LICENSE
+ Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.
+ 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+ Preamble
+
+ The licenses for most software are designed to take away your
+freedom to share and change it. By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users. This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it. (Some other Free Software Foundation software is covered by
+the GNU Library General Public License instead.) You can apply it to
+your programs, too.
+
+ When we speak of free software, we are referring to freedom, not
+price. Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+ To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+ For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have. You must make sure that they, too, receive or can get the
+source code. And you must show them these terms so they know their
+rights.
+
+ We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+ Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software. If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+ Finally, any free program is threatened constantly by software
+patents. We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary. To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+ The precise terms and conditions for copying, distribution and
+modification follow.
+\f
+ GNU GENERAL PUBLIC LICENSE
+ TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+ 0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License. The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language. (Hereinafter, translation is included without limitation in
+the term "modification".) Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope. The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+ 1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+ 2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+ a) You must cause the modified files to carry prominent notices
+ stating that you changed the files and the date of any change.
+
+ b) You must cause any work that you distribute or publish, that in
+ whole or in part contains or is derived from the Program or any
+ part thereof, to be licensed as a whole at no charge to all third
+ parties under the terms of this License.
+
+ c) If the modified program normally reads commands interactively
+ when run, you must cause it, when started running for such
+ interactive use in the most ordinary way, to print or display an
+ announcement including an appropriate copyright notice and a
+ notice that there is no warranty (or else, saying that you provide
+ a warranty) and that users may redistribute the program under
+ these conditions, and telling the user how to view a copy of this
+ License. (Exception: if the Program itself is interactive but
+ does not normally print such an announcement, your work based on
+ the Program is not required to print an announcement.)
+\f
+These requirements apply to the modified work as a whole. If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works. But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+ 3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+ a) Accompany it with the complete corresponding machine-readable
+ source code, which must be distributed under the terms of Sections
+ 1 and 2 above on a medium customarily used for software interchange; or,
+
+ b) Accompany it with a written offer, valid for at least three
+ years, to give any third party, for a charge no more than your
+ cost of physically performing source distribution, a complete
+ machine-readable copy of the corresponding source code, to be
+ distributed under the terms of Sections 1 and 2 above on a medium
+ customarily used for software interchange; or,
+
+ c) Accompany it with the information you received as to the offer
+ to distribute corresponding source code. (This alternative is
+ allowed only for noncommercial distribution and only if you
+ received the program in object code or executable form with such
+ an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it. For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable. However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+\f
+ 4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License. Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+ 5. You are not required to accept this License, since you have not
+signed it. However, nothing else grants you permission to modify or
+distribute the Program or its derivative works. These actions are
+prohibited by law if you do not accept this License. Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+ 6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions. You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+ 7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all. For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices. Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+\f
+ 8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded. In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+ 9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time. Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number. If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation. If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+ 10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission. For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this. Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+ NO WARRANTY
+
+ 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+ 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+ END OF TERMS AND CONDITIONS
+\f
+ How to Apply These Terms to Your New Programs
+
+ If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+ To do so, attach the following notices to the program. It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+ <one line to give the program's name and a brief idea of what it does.>
+ Copyright (C) <year> <name of author>
+
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation; either version 2 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, write to the Free Software
+ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+ Gnomovision version 69, Copyright (C) year name of author
+ Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+ This is free software, and you are welcome to redistribute it
+ under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License. Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary. Here is a sample; alter the names:
+
+ Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+ `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+ <signature of Ty Coon>, 1 April 1989
+ Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs. If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library. If this is what you want to do, use the GNU Library General
+Public License instead of this License.
--- /dev/null
+# vim: set ft=ruby:
+
+source :rubygems
+
+gemspec :name => 'json'
+gemspec :name => 'json_pure'
+gemspec :name => 'json-java'
+
+gem 'utils'
+gem 'test-unit'
+gem 'debugger', :platform => :mri_19
--- /dev/null
+JSON-JRuby
+==========
+
+JSON-JRuby is a port of Florian Frank's native
+[`json` library](http://json.rubyforge.org/) to JRuby.
+It aims to be a perfect drop-in replacement for `json_pure`.
+
+
+Development version
+===================
+
+The latest version is available from the
+[Git repository](http://github.com/mernen/json-jruby/tree):
+
+ git clone git://github.com/mernen/json-jruby.git
+
+
+Compiling
+=========
+
+You'll need JRuby version 1.2 or greater to build JSON-JRuby.
+Its path must be set on the `jruby.dir` property of
+`nbproject/project.properties` (defaults to `../jruby`).
+
+Additionally, you'll need [Ant](http://ant.apache.org/), and
+[Ragel](http://www.cs.queensu.ca/~thurston/ragel/) 6.4 or greater.
+
+Then, from the folder where the sources are located, type:
+
+ ant clean jar
+
+to clean any leftovers from previous builds and generate the `.jar` files.
+To generate a RubyGem, specify the `gem` action rather than `jar`.
--- /dev/null
+= JSON implementation for Ruby http://travis-ci.org/flori/json.png
+
+== Description
+
+This is a implementation of the JSON specification according to RFC 4627
+http://www.ietf.org/rfc/rfc4627.txt . Starting from version 1.0.0 on there
+will be two variants available:
+
+* A pure ruby variant, that relies on the iconv and the stringscan
+ extensions, which are both part of the ruby standard library.
+* The quite a bit faster C extension variant, which is in parts implemented
+ in C and comes with its own unicode conversion functions and a parser
+ generated by the ragel state machine compiler
+ http://www.cs.queensu.ca/~thurston/ragel .
+
+Both variants of the JSON generator generate UTF-8 character sequences by
+default. If an :ascii_only option with a true value is given, they escape all
+non-ASCII and control characters with \uXXXX escape sequences, and support
+UTF-16 surrogate pairs in order to be able to generate the whole range of
+unicode code points.
+
+All strings, that are to be encoded as JSON strings, should be UTF-8 byte
+sequences on the Ruby side. To encode raw binary strings, that aren't UTF-8
+encoded, please use the to_json_raw_object method of String (which produces
+an object, that contains a byte array) and decode the result on the receiving
+endpoint.
+
+The JSON parsers can parse UTF-8, UTF-16BE, UTF-16LE, UTF-32BE, and UTF-32LE
+JSON documents under Ruby 1.8. Under Ruby 1.9 they take advantage of Ruby's
+M17n features and can parse all documents which have the correct
+String#encoding set. If a document string has ASCII-8BIT as an encoding the
+parser attempts to figure out which of the UTF encodings from above it is and
+trys to parse it.
+
+== Installation
+
+It's recommended to use the extension variant of JSON, because it's faster than
+the pure ruby variant. If you cannot build it on your system, you can settle
+for the latter.
+
+Just type into the command line as root:
+
+ # rake install
+
+The above command will build the extensions and install them on your system.
+
+ # rake install_pure
+
+or
+
+ # ruby install.rb
+
+will just install the pure ruby implementation of JSON.
+
+If you use Rubygems you can type
+
+ # gem install json
+
+instead, to install the newest JSON version.
+
+There is also a pure ruby json only variant of the gem, that can be installed
+with:
+
+ # gem install json_pure
+
+== Compiling the extensions yourself
+
+If you want to build the extensions yourself you need rake:
+
+ You can get it from rubyforge:
+ http://rubyforge.org/projects/rake
+
+ or just type
+
+ # gem install rake
+
+ for the installation via rubygems.
+
+If you want to create the parser.c file from its parser.rl file or draw nice
+graphviz images of the state machines, you need ragel from: http://www.cs.queensu.ca/~thurston/ragel
+
+
+== Usage
+
+To use JSON you can
+ require 'json'
+to load the installed variant (either the extension 'json' or the pure
+variant 'json_pure'). If you have installed the extension variant, you can
+pick either the extension variant or the pure variant by typing
+ require 'json/ext'
+or
+ require 'json/pure'
+
+Now you can parse a JSON document into a ruby data structure by calling
+
+ JSON.parse(document)
+
+If you want to generate a JSON document from a ruby data structure call
+ JSON.generate(data)
+
+You can also use the pretty_generate method (which formats the output more
+verbosely and nicely) or fast_generate (which doesn't do any of the security
+checks generate performs, e. g. nesting deepness checks).
+
+To create a valid JSON document you have to make sure, that the output is
+embedded in either a JSON array [] or a JSON object {}. The easiest way to do
+this, is by putting your values in a Ruby Array or Hash instance.
+
+There are also the JSON and JSON[] methods which use parse on a String or
+generate a JSON document from an array or hash:
+
+ document = JSON 'test' => 23 # => "{\"test\":23}"
+ document = JSON['test'] => 23 # => "{\"test\":23}"
+
+and
+
+ data = JSON '{"test":23}' # => {"test"=>23}
+ data = JSON['{"test":23}'] # => {"test"=>23}
+
+You can choose to load a set of common additions to ruby core's objects if
+you
+ require 'json/add/core'
+
+After requiring this you can, e. g., serialise/deserialise Ruby ranges:
+
+ JSON JSON(1..10) # => 1..10
+
+To find out how to add JSON support to other or your own classes, read the
+section "More Examples" below.
+
+To get the best compatibility to rails' JSON implementation, you can
+ require 'json/add/rails'
+
+Both of the additions attempt to require 'json' (like above) first, if it has
+not been required yet.
+
+== More Examples
+
+To create a JSON document from a ruby data structure, you can call
+JSON.generate like that:
+
+ json = JSON.generate [1, 2, {"a"=>3.141}, false, true, nil, 4..10]
+ # => "[1,2,{\"a\":3.141},false,true,null,\"4..10\"]"
+
+To get back a ruby data structure from a JSON document, you have to call
+JSON.parse on it:
+
+ JSON.parse json
+ # => [1, 2, {"a"=>3.141}, false, true, nil, "4..10"]
+
+Note, that the range from the original data structure is a simple
+string now. The reason for this is, that JSON doesn't support ranges
+or arbitrary classes. In this case the json library falls back to call
+Object#to_json, which is the same as #to_s.to_json.
+
+It's possible to add JSON support serialization to arbitrary classes by
+simply implementing a more specialized version of the #to_json method, that
+should return a JSON object (a hash converted to JSON with #to_json) like
+this (don't forget the *a for all the arguments):
+
+ class Range
+ def to_json(*a)
+ {
+ 'json_class' => self.class.name, # = 'Range'
+ 'data' => [ first, last, exclude_end? ]
+ }.to_json(*a)
+ end
+ end
+
+The hash key 'json_class' is the class, that will be asked to deserialise the
+JSON representation later. In this case it's 'Range', but any namespace of
+the form 'A::B' or '::A::B' will do. All other keys are arbitrary and can be
+used to store the necessary data to configure the object to be deserialised.
+
+If a the key 'json_class' is found in a JSON object, the JSON parser checks
+if the given class responds to the json_create class method. If so, it is
+called with the JSON object converted to a Ruby hash. So a range can
+be deserialised by implementing Range.json_create like this:
+
+ class Range
+ def self.json_create(o)
+ new(*o['data'])
+ end
+ end
+
+Now it possible to serialise/deserialise ranges as well:
+
+ json = JSON.generate [1, 2, {"a"=>3.141}, false, true, nil, 4..10]
+ # => "[1,2,{\"a\":3.141},false,true,null,{\"json_class\":\"Range\",\"data\":[4,10,false]}]"
+ JSON.parse json
+ # => [1, 2, {"a"=>3.141}, false, true, nil, 4..10]
+
+JSON.generate always creates the shortest possible string representation of a
+ruby data structure in one line. This is good for data storage or network
+protocols, but not so good for humans to read. Fortunately there's also
+JSON.pretty_generate (or JSON.pretty_generate) that creates a more readable
+output:
+
+ puts JSON.pretty_generate([1, 2, {"a"=>3.141}, false, true, nil, 4..10])
+ [
+ 1,
+ 2,
+ {
+ "a": 3.141
+ },
+ false,
+ true,
+ null,
+ {
+ "json_class": "Range",
+ "data": [
+ 4,
+ 10,
+ false
+ ]
+ }
+ ]
+
+There are also the methods Kernel#j for generate, and Kernel#jj for
+pretty_generate output to the console, that work analogous to Core Ruby's p and
+the pp library's pp methods.
+
+The script tools/server.rb contains a small example if you want to test, how
+receiving a JSON object from a webrick server in your browser with the
+javasript prototype library http://www.prototypejs.org works.
+
+== Speed Comparisons
+
+I have created some benchmark results (see the benchmarks/data-p4-3Ghz
+subdir of the package) for the JSON-parser to estimate the speed up in the C
+extension:
+
+ Comparing times (call_time_mean):
+ 1 ParserBenchmarkExt#parser 900 repeats:
+ 553.922304770 ( real) -> 21.500x
+ 0.001805307
+ 2 ParserBenchmarkYAML#parser 1000 repeats:
+ 224.513358139 ( real) -> 8.714x
+ 0.004454078
+ 3 ParserBenchmarkPure#parser 1000 repeats:
+ 26.755020642 ( real) -> 1.038x
+ 0.037376163
+ 4 ParserBenchmarkRails#parser 1000 repeats:
+ 25.763381731 ( real) -> 1.000x
+ 0.038814780
+ calls/sec ( time) -> speed covers
+ secs/call
+
+In the table above 1 is JSON::Ext::Parser, 2 is YAML.load with YAML
+compatbile JSON document, 3 is is JSON::Pure::Parser, and 4 is
+ActiveSupport::JSON.decode. The ActiveSupport JSON-decoder converts the
+input first to YAML and then uses the YAML-parser, the conversion seems to
+slow it down so much that it is only as fast as the JSON::Pure::Parser!
+
+If you look at the benchmark data you can see that this is mostly caused by
+the frequent high outliers - the median of the Rails-parser runs is still
+overall smaller than the median of the JSON::Pure::Parser runs:
+
+ Comparing times (call_time_median):
+ 1 ParserBenchmarkExt#parser 900 repeats:
+ 800.592479481 ( real) -> 26.936x
+ 0.001249075
+ 2 ParserBenchmarkYAML#parser 1000 repeats:
+ 271.002390644 ( real) -> 9.118x
+ 0.003690004
+ 3 ParserBenchmarkRails#parser 1000 repeats:
+ 30.227910865 ( real) -> 1.017x
+ 0.033082008
+ 4 ParserBenchmarkPure#parser 1000 repeats:
+ 29.722384421 ( real) -> 1.000x
+ 0.033644676
+ calls/sec ( time) -> speed covers
+ secs/call
+
+I have benchmarked the JSON-Generator as well. This generated a few more
+values, because there are different modes that also influence the achieved
+speed:
+
+ Comparing times (call_time_mean):
+ 1 GeneratorBenchmarkExt#generator_fast 1000 repeats:
+ 547.354332608 ( real) -> 15.090x
+ 0.001826970
+ 2 GeneratorBenchmarkExt#generator_safe 1000 repeats:
+ 443.968212317 ( real) -> 12.240x
+ 0.002252414
+ 3 GeneratorBenchmarkExt#generator_pretty 900 repeats:
+ 375.104545883 ( real) -> 10.341x
+ 0.002665923
+ 4 GeneratorBenchmarkPure#generator_fast 1000 repeats:
+ 49.978706968 ( real) -> 1.378x
+ 0.020008521
+ 5 GeneratorBenchmarkRails#generator 1000 repeats:
+ 38.531868759 ( real) -> 1.062x
+ 0.025952543
+ 6 GeneratorBenchmarkPure#generator_safe 1000 repeats:
+ 36.927649925 ( real) -> 1.018x 7 (>=3859)
+ 0.027079979
+ 7 GeneratorBenchmarkPure#generator_pretty 1000 repeats:
+ 36.272134441 ( real) -> 1.000x 6 (>=3859)
+ 0.027569373
+ calls/sec ( time) -> speed covers
+ secs/call
+
+In the table above 1-3 are JSON::Ext::Generator methods. 4, 6, and 7 are
+JSON::Pure::Generator methods and 5 is the Rails JSON generator. It is now a
+bit faster than the generator_safe and generator_pretty methods of the pure
+variant but slower than the others.
+
+To achieve the fastest JSON document output, you can use the fast_generate
+method. Beware, that this will disable the checking for circular Ruby data
+structures, which may cause JSON to go into an infinite loop.
+
+Here are the median comparisons for completeness' sake:
+
+ Comparing times (call_time_median):
+ 1 GeneratorBenchmarkExt#generator_fast 1000 repeats:
+ 708.258020939 ( real) -> 16.547x
+ 0.001411915
+ 2 GeneratorBenchmarkExt#generator_safe 1000 repeats:
+ 569.105020353 ( real) -> 13.296x
+ 0.001757145
+ 3 GeneratorBenchmarkExt#generator_pretty 900 repeats:
+ 482.825371244 ( real) -> 11.280x
+ 0.002071142
+ 4 GeneratorBenchmarkPure#generator_fast 1000 repeats:
+ 62.717626652 ( real) -> 1.465x
+ 0.015944481
+ 5 GeneratorBenchmarkRails#generator 1000 repeats:
+ 43.965681162 ( real) -> 1.027x
+ 0.022745013
+ 6 GeneratorBenchmarkPure#generator_safe 1000 repeats:
+ 43.929073409 ( real) -> 1.026x 7 (>=3859)
+ 0.022763968
+ 7 GeneratorBenchmarkPure#generator_pretty 1000 repeats:
+ 42.802514491 ( real) -> 1.000x 6 (>=3859)
+ 0.023363113
+ calls/sec ( time) -> speed covers
+ secs/call
+
+== Author
+
+Florian Frank <mailto:flori@ping.de>
+
+== License
+
+Ruby License, see the COPYING file included in the source distribution. The
+Ruby License includes the GNU General Public License (GPL), Version 2, so see
+the file GPL as well.
+
+== Download
+
+The latest version of this library can be downloaded at
+
+* http://rubyforge.org/frs?group_id=953
+
+Online Documentation should be located at
+
+* http://json.rubyforge.org
--- /dev/null
+begin
+ require 'rubygems/package_task'
+rescue LoadError
+end
+
+require 'rbconfig'
+begin
+ include RbConfig
+rescue NameError
+ include Config
+end
+
+
+require 'rake/clean'
+CLOBBER.include Dir['benchmarks/data/*.{dat,log}'], 'doc', 'Gemfile.lock'
+CLEAN.include FileList['diagrams/*.*'], 'doc', 'coverage', 'tmp',
+ FileList["ext/**/{Makefile,mkmf.log}"], 'build', 'dist', FileList['**/*.rbc'],
+ FileList["{ext,lib}/**/*.{so,bundle,#{CONFIG['DLEXT']},o,obj,pdb,lib,manifest,exp,def,jar,class,dSYM}"],
+ FileList['java/src/**/*.class']
+
+MAKE = ENV['MAKE'] || %w[gmake make].find { |c| system(c, '-v') }
+PKG_NAME = 'json'
+PKG_TITLE = 'JSON Implementation for Ruby'
+PKG_VERSION = File.read('VERSION').chomp
+PKG_FILES = FileList["**/*"].exclude(/CVS|pkg|tmp|coverage|Makefile|\.nfs\.|\.iml\Z/).exclude(/\.(so|bundle|o|class|#{CONFIG['DLEXT']})$/)
+
+EXT_ROOT_DIR = 'ext/json/ext'
+EXT_PARSER_DIR = "#{EXT_ROOT_DIR}/parser"
+EXT_PARSER_DL = "#{EXT_PARSER_DIR}/parser.#{CONFIG['DLEXT']}"
+RAGEL_PATH = "#{EXT_PARSER_DIR}/parser.rl"
+EXT_PARSER_SRC = "#{EXT_PARSER_DIR}/parser.c"
+PKG_FILES << EXT_PARSER_SRC
+EXT_GENERATOR_DIR = "#{EXT_ROOT_DIR}/generator"
+EXT_GENERATOR_DL = "#{EXT_GENERATOR_DIR}/generator.#{CONFIG['DLEXT']}"
+EXT_GENERATOR_SRC = "#{EXT_GENERATOR_DIR}/generator.c"
+
+JAVA_DIR = "java/src/json/ext"
+JAVA_RAGEL_PATH = "#{JAVA_DIR}/Parser.rl"
+JAVA_PARSER_SRC = "#{JAVA_DIR}/Parser.java"
+JAVA_SOURCES = FileList["#{JAVA_DIR}/*.java"]
+JAVA_CLASSES = []
+JRUBY_PARSER_JAR = File.expand_path("lib/json/ext/parser.jar")
+JRUBY_GENERATOR_JAR = File.expand_path("lib/json/ext/generator.jar")
+
+RAGEL_CODEGEN = %w[rlcodegen rlgen-cd ragel].find { |c| system(c, '-v') }
+RAGEL_DOTGEN = %w[rlgen-dot rlgen-cd ragel].find { |c| system(c, '-v') }
+
+def myruby(*args, &block)
+ @myruby ||= File.join(CONFIG['bindir'], CONFIG['ruby_install_name'])
+ options = (Hash === args.last) ? args.pop : {}
+ if args.length > 1 then
+ sh(*([@myruby] + args + [options]), &block)
+ else
+ sh("#{@myruby} #{args.first}", options, &block)
+ end
+end
+
+desc "Installing library (pure)"
+task :install_pure => :version do
+ myruby 'install.rb'
+end
+
+task :install_ext_really do
+ sitearchdir = CONFIG["sitearchdir"]
+ cd 'ext' do
+ for file in Dir["json/ext/*.#{CONFIG['DLEXT']}"]
+ d = File.join(sitearchdir, file)
+ mkdir_p File.dirname(d)
+ install(file, d)
+ end
+ warn " *** Installed EXT ruby library."
+ end
+end
+
+desc "Installing library (extension)"
+task :install_ext => [ :compile, :install_pure, :install_ext_really ]
+
+desc "Installing library (extension)"
+if RUBY_PLATFORM =~ /java/
+ task :install => :install_pure
+else
+ task :install => :install_ext
+end
+
+if defined?(Gem) and defined?(Gem::PackageTask)
+ spec_pure = Gem::Specification.new do |s|
+ s.name = 'json_pure'
+ s.version = PKG_VERSION
+ s.summary = PKG_TITLE
+ s.description = "This is a JSON implementation in pure Ruby."
+
+ s.files = PKG_FILES
+
+ s.require_path = 'lib'
+ s.add_development_dependency 'permutation'
+ s.add_development_dependency 'bullshit'
+ s.add_development_dependency 'sdoc'
+ s.add_development_dependency 'rake', '~>0.9.2'
+ s.add_dependency 'spruz', '~>0.2.8'
+
+ s.bindir = "bin"
+ s.executables = [ "edit_json.rb", "prettify_json.rb" ]
+
+ s.extra_rdoc_files << 'README.rdoc'
+ s.rdoc_options <<
+ '--title' << 'JSON implemention for ruby' << '--main' << 'README.rdoc'
+ s.test_files.concat Dir['./tests/test_*.rb']
+
+ s.author = "Florian Frank"
+ s.email = "flori@ping.de"
+ s.homepage = "http://flori.github.com/#{PKG_NAME}"
+ s.rubyforge_project = "json"
+ end
+
+ desc 'Creates a json_pure.gemspec file'
+ task :gemspec_pure => :version do
+ File.open('json_pure.gemspec', 'w') do |gemspec|
+ gemspec.write spec_pure.to_ruby
+ end
+ end
+
+ Gem::PackageTask.new(spec_pure) do |pkg|
+ pkg.need_tar = true
+ pkg.package_files = PKG_FILES
+ end
+
+ spec_ext = Gem::Specification.new do |s|
+ s.name = 'json'
+ s.version = PKG_VERSION
+ s.summary = PKG_TITLE
+ s.description = "This is a JSON implementation as a Ruby extension in C."
+
+ s.files = PKG_FILES
+
+ s.extensions = FileList['ext/**/extconf.rb']
+
+ s.require_path = EXT_ROOT_DIR
+ s.require_paths << 'ext'
+ s.require_paths << 'lib'
+ s.add_development_dependency 'permutation'
+ s.add_development_dependency 'bullshit'
+ s.add_development_dependency 'sdoc'
+
+ s.bindir = "bin"
+ s.executables = [ "edit_json.rb", "prettify_json.rb" ]
+
+ s.extra_rdoc_files << 'README.rdoc'
+ s.rdoc_options <<
+ '--title' << 'JSON implemention for Ruby' << '--main' << 'README.rdoc'
+ s.test_files.concat Dir['./tests/test_*.rb']
+
+ s.author = "Florian Frank"
+ s.email = "flori@ping.de"
+ s.homepage = "http://flori.github.com/#{PKG_NAME}"
+ s.rubyforge_project = "json"
+ end
+
+ desc 'Creates a json.gemspec file'
+ task :gemspec_ext => :version do
+ File.open('json.gemspec', 'w') do |gemspec|
+ gemspec.write spec_ext.to_ruby
+ end
+ end
+
+ Gem::PackageTask.new(spec_ext) do |pkg|
+ pkg.need_tar = true
+ pkg.package_files = PKG_FILES
+ end
+
+
+ desc 'Create all gemspec files'
+ task :gemspec => [ :gemspec_pure, :gemspec_ext ]
+end
+
+desc m = "Writing version information for #{PKG_VERSION}"
+task :version do
+ puts m
+ File.open(File.join('lib', 'json', 'version.rb'), 'w') do |v|
+ v.puts <<EOT
+module JSON
+ # JSON version
+ VERSION = '#{PKG_VERSION}'
+ VERSION_ARRAY = VERSION.split(/\\./).map { |x| x.to_i } # :nodoc:
+ VERSION_MAJOR = VERSION_ARRAY[0] # :nodoc:
+ VERSION_MINOR = VERSION_ARRAY[1] # :nodoc:
+ VERSION_BUILD = VERSION_ARRAY[2] # :nodoc:
+end
+EOT
+ end
+end
+
+desc "Testing library (pure ruby)"
+task :test_pure => :clean do
+ ENV['JSON'] = 'pure'
+ ENV['RUBYOPT'] = "-Ilib #{ENV['RUBYOPT']}"
+ myruby '-S', 'testrb', *Dir['./tests/test_*.rb']
+end
+
+desc "Testing library (pure ruby and extension)"
+task :test => [ :test_pure, :test_ext ]
+
+namespace :gems do
+ task :install do
+ sh 'bundle'
+ end
+end
+
+if defined?(RUBY_ENGINE) and RUBY_ENGINE == 'jruby'
+ if ENV.key?('JAVA_HOME')
+ warn " *** JAVA_HOME was set to #{ENV['JAVA_HOME'].inspect}"
+ else File.directory?(local_java = '/usr/local/java/jdk')
+ ENV['JAVA_HOME'] = local_java
+ warn " *** JAVA_HOME is set to #{ENV['JAVA_HOME'].inspect}"
+ ENV['PATH'] = ENV['PATH'].split(/:/).unshift(java_path = "#{ENV['JAVA_HOME']}/bin") * ':'
+ warn " *** java binaries are assumed to be in #{java_path.inspect}"
+ end
+
+ file JAVA_PARSER_SRC => JAVA_RAGEL_PATH do
+ cd JAVA_DIR do
+ if RAGEL_CODEGEN == 'ragel'
+ sh "ragel Parser.rl -J -o Parser.java"
+ else
+ sh "ragel -x Parser.rl | #{RAGEL_CODEGEN} -J"
+ end
+ end
+ end
+
+ desc "Generate parser for java with ragel"
+ task :ragel => JAVA_PARSER_SRC
+
+ desc "Delete the ragel generated Java source"
+ task :ragel_clean do
+ rm_rf JAVA_PARSER_SRC
+ end
+
+ JRUBY_JAR = File.join(CONFIG["libdir"], "jruby.jar")
+ if File.exist?(JRUBY_JAR)
+ JAVA_SOURCES.each do |src|
+ classpath = (Dir['java/lib/*.jar'] << 'java/src' << JRUBY_JAR) * ':'
+ obj = src.sub(/\.java\Z/, '.class')
+ file obj => src do
+ sh 'javac', '-classpath', classpath, '-source', '1.5', src
+ end
+ JAVA_CLASSES << obj
+ end
+ else
+ warn "WARNING: Cannot find jruby in path => Cannot build jruby extension!"
+ end
+
+ desc "Compiling jruby extension"
+ task :compile => JAVA_CLASSES
+
+ desc "Package the jruby gem"
+ task :jruby_gem => :create_jar do
+ sh 'gem build json-java.gemspec'
+ mkdir_p 'pkg'
+ mv "json-#{PKG_VERSION}-java.gem", 'pkg'
+ end
+
+ desc "Testing library (jruby)"
+ task :test_ext => :create_jar do
+ ENV['JSON'] = 'ext'
+ myruby '-S', 'testrb', '-Ilib', *Dir['./tests/test_*.rb']
+ end
+
+ file JRUBY_PARSER_JAR => :compile do
+ cd 'java/src' do
+ parser_classes = FileList[
+ "json/ext/ByteListTranscoder*.class",
+ "json/ext/OptionsReader*.class",
+ "json/ext/Parser*.class",
+ "json/ext/RuntimeInfo*.class",
+ "json/ext/StringDecoder*.class",
+ "json/ext/Utils*.class"
+ ]
+ sh 'jar', 'cf', File.basename(JRUBY_PARSER_JAR), *parser_classes
+ mv File.basename(JRUBY_PARSER_JAR), File.dirname(JRUBY_PARSER_JAR)
+ end
+ end
+
+ desc "Create parser jar"
+ task :create_parser_jar => JRUBY_PARSER_JAR
+
+ file JRUBY_GENERATOR_JAR => :compile do
+ cd 'java/src' do
+ generator_classes = FileList[
+ "json/ext/ByteListTranscoder*.class",
+ "json/ext/OptionsReader*.class",
+ "json/ext/Generator*.class",
+ "json/ext/RuntimeInfo*.class",
+ "json/ext/StringEncoder*.class",
+ "json/ext/Utils*.class"
+ ]
+ sh 'jar', 'cf', File.basename(JRUBY_GENERATOR_JAR), *generator_classes
+ mv File.basename(JRUBY_GENERATOR_JAR), File.dirname(JRUBY_GENERATOR_JAR)
+ end
+ end
+
+ desc "Create generator jar"
+ task :create_generator_jar => JRUBY_GENERATOR_JAR
+
+ desc "Create parser and generator jars"
+ task :create_jar => [ :create_parser_jar, :create_generator_jar ]
+
+ desc "Build all gems and archives for a new release of the jruby extension."
+ task :build => [ :clean, :version, :jruby_gem ]
+
+ task :release => :build
+else
+ desc "Compiling extension"
+ task :compile => [ EXT_PARSER_DL, EXT_GENERATOR_DL ]
+
+ file EXT_PARSER_DL => EXT_PARSER_SRC do
+ cd EXT_PARSER_DIR do
+ myruby 'extconf.rb'
+ sh MAKE
+ end
+ cp "#{EXT_PARSER_DIR}/parser.#{CONFIG['DLEXT']}", EXT_ROOT_DIR
+ end
+
+ file EXT_GENERATOR_DL => EXT_GENERATOR_SRC do
+ cd EXT_GENERATOR_DIR do
+ myruby 'extconf.rb'
+ sh MAKE
+ end
+ cp "#{EXT_GENERATOR_DIR}/generator.#{CONFIG['DLEXT']}", EXT_ROOT_DIR
+ end
+
+ desc "Testing library (extension)"
+ task :test_ext => :compile do
+ ENV['JSON'] = 'ext'
+ ENV['RUBYOPT'] = "-Iext:lib #{ENV['RUBYOPT']}"
+ myruby '-S', 'testrb', *Dir['./tests/test_*.rb']
+ end
+
+ desc "Benchmarking parser"
+ task :benchmark_parser do
+ ENV['RUBYOPT'] = "-Ilib:ext #{ENV['RUBYOPT']}"
+ myruby 'benchmarks/parser_benchmark.rb'
+ myruby 'benchmarks/parser2_benchmark.rb'
+ end
+
+ desc "Benchmarking generator"
+ task :benchmark_generator do
+ ENV['RUBYOPT'] = "-Ilib:ext #{ENV['RUBYOPT']}"
+ myruby 'benchmarks/generator_benchmark.rb'
+ myruby 'benchmarks/generator2_benchmark.rb'
+ end
+
+ desc "Benchmarking library"
+ task :benchmark => [ :benchmark_parser, :benchmark_generator ]
+
+ desc "Create RDOC documentation"
+ task :doc => [ :version, EXT_PARSER_SRC ] do
+ sh "sdoc -o doc -t '#{PKG_TITLE}' -m README.rdoc README.rdoc lib/json.rb #{FileList['lib/json/**/*.rb']} #{EXT_PARSER_SRC} #{EXT_GENERATOR_SRC}"
+ end
+
+ desc "Generate parser with ragel"
+ task :ragel => EXT_PARSER_SRC
+
+ desc "Delete the ragel generated C source"
+ task :ragel_clean do
+ rm_rf EXT_PARSER_SRC
+ end
+
+ file EXT_PARSER_SRC => RAGEL_PATH do
+ cd EXT_PARSER_DIR do
+ if RAGEL_CODEGEN == 'ragel'
+ sh "ragel parser.rl -G2 -o parser.c"
+ else
+ sh "ragel -x parser.rl | #{RAGEL_CODEGEN} -G2"
+ end
+ src = File.read("parser.c").gsub(/[ \t]+$/, '')
+ File.open("parser.c", "w") {|f| f.print src}
+ end
+ end
+
+ desc "Generate diagrams of ragel parser (ps)"
+ task :ragel_dot_ps do
+ root = 'diagrams'
+ specs = []
+ File.new(RAGEL_PATH).grep(/^\s*machine\s*(\S+);\s*$/) { specs << $1 }
+ for s in specs
+ if RAGEL_DOTGEN == 'ragel'
+ sh "ragel #{RAGEL_PATH} -S#{s} -p -V | dot -Tps -o#{root}/#{s}.ps"
+ else
+ sh "ragel -x #{RAGEL_PATH} -S#{s} | #{RAGEL_DOTGEN} -p|dot -Tps -o#{root}/#{s}.ps"
+ end
+ end
+ end
+
+ desc "Generate diagrams of ragel parser (png)"
+ task :ragel_dot_png do
+ root = 'diagrams'
+ specs = []
+ File.new(RAGEL_PATH).grep(/^\s*machine\s*(\S+);\s*$/) { specs << $1 }
+ for s in specs
+ if RAGEL_DOTGEN == 'ragel'
+ sh "ragel #{RAGEL_PATH} -S#{s} -p -V | dot -Tpng -o#{root}/#{s}.png"
+ else
+ sh "ragel -x #{RAGEL_PATH} -S#{s} | #{RAGEL_DOTGEN} -p|dot -Tpng -o#{root}/#{s}.png"
+ end
+ end
+ end
+
+ desc "Generate diagrams of ragel parser"
+ task :ragel_dot => [ :ragel_dot_png, :ragel_dot_ps ]
+
+ desc "Build all gems and archives for a new release of json and json_pure."
+ task :build => [ :clean, :gemspec, :package ]
+
+ task :release => :build
+end
+
+desc "Compile in the the source directory"
+task :default => [ :clean, :gemspec, :test ]
--- /dev/null
+#!/usr/bin/env ruby
+require 'json/editor'
+
+filename, encoding = ARGV
+JSON::Editor.start(encoding) do |window|
+ if filename
+ window.file_open(filename)
+ end
+end
--- /dev/null
+#!/usr/bin/env ruby
+
+require 'json'
+require 'fileutils'
+include FileUtils
+require 'spruz/go'
+include Spruz::GO
+
+opts = go 'slhi:', args = ARGV.dup
+if opts['h'] || opts['l'] && opts['s']
+ puts <<EOT
+Usage: #{File.basename($0)} [OPTION] [FILE]
+
+If FILE is skipped, this scripts waits for input from STDIN. Otherwise
+FILE is opened, read, and used as input for the prettifier.
+
+OPTION can be
+ -s to output the shortest possible JSON (precludes -l)
+ -l to output a longer, better formatted JSON (precludes -s)
+ -i EXT prettifies FILE in place, saving a backup to FILE.EXT
+ -h this help
+EOT
+ exit 0
+end
+
+json_opts = { :max_nesting => false, :create_additions => false }
+
+document =
+ if filename = args.first or filename == '-'
+ File.read(filename)
+ else
+ STDIN.read
+ end
+
+json = JSON.parse document, json_opts
+
+output = if opts['s']
+ JSON.fast_generate json, json_opts
+else # default is -l
+ JSON.pretty_generate json, json_opts
+end
+
+if opts['i'] && filename
+ cp filename, "#{filename}.#{opts['i']}"
+ File.open(filename, 'w') { |f| f.puts output }
+else
+ puts output
+end
--- /dev/null
+{"a":2,"b":3.141,"TIME":"2007-03-14T11:52:40","c":"c","d":[1,"b",3.14],"COUNT":666,"e":{"foo":"bar"},"foo":"B\u00e4r","g":"\u677e\u672c\u884c\u5f18","h":1000.0,"bar":"\u00a9 \u2260 \u20ac!","i":0.001,"j":"\ud840\udc01"}
--- /dev/null
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
+ <head>
+ <title>Javascript Example</title>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
+ <script src="prototype.js" type="text/javascript"></script>
+ </head>
+
+ <body>
+ <h1>Fetching object from server</h1>
+ <div id="list">
+ Wait...<br/>
+ <noscript><p>Switch on Javascript!</p></noscript>
+ </div>
+ <script type="text/javascript">
+ <!--
+ function pollJSON() {
+ new Ajax.Request('/json',
+ {
+ method: 'get',
+ onSuccess: function(transport) {
+ var response = transport.responseText || "no response text";
+ response = eval("(" + response + ")");
+ var text = "";
+ for (var k in response) {
+ text = text + "<b>" + k + "</b>: " + response[k] + "<br/>"
+ }
+ $("list").update(text);
+ },
+ onFailure: function() { alert('Something went wrong...') }
+ });
+ }
+ new PeriodicalExecuter(pollJSON, 1);
+ -->
+ </script>
+ </body>
+</html>
--- /dev/null
+/* Prototype JavaScript framework, version 1.6.0
+ * (c) 2005-2007 Sam Stephenson
+ *
+ * Prototype is freely distributable under the terms of an MIT-style license.
+ * For details, see the Prototype web site: http://www.prototypejs.org/
+ *
+ *--------------------------------------------------------------------------*/
+
+var Prototype = {
+ Version: '1.6.0',
+
+ Browser: {
+ IE: !!(window.attachEvent && !window.opera),
+ Opera: !!window.opera,
+ WebKit: navigator.userAgent.indexOf('AppleWebKit/') > -1,
+ Gecko: navigator.userAgent.indexOf('Gecko') > -1 && navigator.userAgent.indexOf('KHTML') == -1,
+ MobileSafari: !!navigator.userAgent.match(/Apple.*Mobile.*Safari/)
+ },
+
+ BrowserFeatures: {
+ XPath: !!document.evaluate,
+ ElementExtensions: !!window.HTMLElement,
+ SpecificElementExtensions:
+ document.createElement('div').__proto__ &&
+ document.createElement('div').__proto__ !==
+ document.createElement('form').__proto__
+ },
+
+ ScriptFragment: '<script[^>]*>([\\S\\s]*?)<\/script>',
+ JSONFilter: /^\/\*-secure-([\s\S]*)\*\/\s*$/,
+
+ emptyFunction: function() { },
+ K: function(x) { return x }
+};
+
+if (Prototype.Browser.MobileSafari)
+ Prototype.BrowserFeatures.SpecificElementExtensions = false;
+
+if (Prototype.Browser.WebKit)
+ Prototype.BrowserFeatures.XPath = false;
+
+/* Based on Alex Arnell's inheritance implementation. */
+var Class = {
+ create: function() {
+ var parent = null, properties = $A(arguments);
+ if (Object.isFunction(properties[0]))
+ parent = properties.shift();
+
+ function klass() {
+ this.initialize.apply(this, arguments);
+ }
+
+ Object.extend(klass, Class.Methods);
+ klass.superclass = parent;
+ klass.subclasses = [];
+
+ if (parent) {
+ var subclass = function() { };
+ subclass.prototype = parent.prototype;
+ klass.prototype = new subclass;
+ parent.subclasses.push(klass);
+ }
+
+ for (var i = 0; i < properties.length; i++)
+ klass.addMethods(properties[i]);
+
+ if (!klass.prototype.initialize)
+ klass.prototype.initialize = Prototype.emptyFunction;
+
+ klass.prototype.constructor = klass;
+
+ return klass;
+ }
+};
+
+Class.Methods = {
+ addMethods: function(source) {
+ var ancestor = this.superclass && this.superclass.prototype;
+ var properties = Object.keys(source);
+
+ if (!Object.keys({ toString: true }).length)
+ properties.push("toString", "valueOf");
+
+ for (var i = 0, length = properties.length; i < length; i++) {
+ var property = properties[i], value = source[property];
+ if (ancestor && Object.isFunction(value) &&
+ value.argumentNames().first() == "$super") {
+ var method = value, value = Object.extend((function(m) {
+ return function() { return ancestor[m].apply(this, arguments) };
+ })(property).wrap(method), {
+ valueOf: function() { return method },
+ toString: function() { return method.toString() }
+ });
+ }
+ this.prototype[property] = value;
+ }
+
+ return this;
+ }
+};
+
+var Abstract = { };
+
+Object.extend = function(destination, source) {
+ for (var property in source)
+ destination[property] = source[property];
+ return destination;
+};
+
+Object.extend(Object, {
+ inspect: function(object) {
+ try {
+ if (object === undefined) return 'undefined';
+ if (object === null) return 'null';
+ return object.inspect ? object.inspect() : object.toString();
+ } catch (e) {
+ if (e instanceof RangeError) return '...';
+ throw e;
+ }
+ },
+
+ toJSON: function(object) {
+ var type = typeof object;
+ switch (type) {
+ case 'undefined':
+ case 'function':
+ case 'unknown': return;
+ case 'boolean': return object.toString();
+ }
+
+ if (object === null) return 'null';
+ if (object.toJSON) return object.toJSON();
+ if (Object.isElement(object)) return;
+
+ var results = [];
+ for (var property in object) {
+ var value = Object.toJSON(object[property]);
+ if (value !== undefined)
+ results.push(property.toJSON() + ': ' + value);
+ }
+
+ return '{' + results.join(', ') + '}';
+ },
+
+ toQueryString: function(object) {
+ return $H(object).toQueryString();
+ },
+
+ toHTML: function(object) {
+ return object && object.toHTML ? object.toHTML() : String.interpret(object);
+ },
+
+ keys: function(object) {
+ var keys = [];
+ for (var property in object)
+ keys.push(property);
+ return keys;
+ },
+
+ values: function(object) {
+ var values = [];
+ for (var property in object)
+ values.push(object[property]);
+ return values;
+ },
+
+ clone: function(object) {
+ return Object.extend({ }, object);
+ },
+
+ isElement: function(object) {
+ return object && object.nodeType == 1;
+ },
+
+ isArray: function(object) {
+ return object && object.constructor === Array;
+ },
+
+ isHash: function(object) {
+ return object instanceof Hash;
+ },
+
+ isFunction: function(object) {
+ return typeof object == "function";
+ },
+
+ isString: function(object) {
+ return typeof object == "string";
+ },
+
+ isNumber: function(object) {
+ return typeof object == "number";
+ },
+
+ isUndefined: function(object) {
+ return typeof object == "undefined";
+ }
+});
+
+Object.extend(Function.prototype, {
+ argumentNames: function() {
+ var names = this.toString().match(/^[\s\(]*function[^(]*\((.*?)\)/)[1].split(",").invoke("strip");
+ return names.length == 1 && !names[0] ? [] : names;
+ },
+
+ bind: function() {
+ if (arguments.length < 2 && arguments[0] === undefined) return this;
+ var __method = this, args = $A(arguments), object = args.shift();
+ return function() {
+ return __method.apply(object, args.concat($A(arguments)));
+ }
+ },
+
+ bindAsEventListener: function() {
+ var __method = this, args = $A(arguments), object = args.shift();
+ return function(event) {
+ return __method.apply(object, [event || window.event].concat(args));
+ }
+ },
+
+ curry: function() {
+ if (!arguments.length) return this;
+ var __method = this, args = $A(arguments);
+ return function() {
+ return __method.apply(this, args.concat($A(arguments)));
+ }
+ },
+
+ delay: function() {
+ var __method = this, args = $A(arguments), timeout = args.shift() * 1000;
+ return window.setTimeout(function() {
+ return __method.apply(__method, args);
+ }, timeout);
+ },
+
+ wrap: function(wrapper) {
+ var __method = this;
+ return function() {
+ return wrapper.apply(this, [__method.bind(this)].concat($A(arguments)));
+ }
+ },
+
+ methodize: function() {
+ if (this._methodized) return this._methodized;
+ var __method = this;
+ return this._methodized = function() {
+ return __method.apply(null, [this].concat($A(arguments)));
+ };
+ }
+});
+
+Function.prototype.defer = Function.prototype.delay.curry(0.01);
+
+Date.prototype.toJSON = function() {
+ return '"' + this.getUTCFullYear() + '-' +
+ (this.getUTCMonth() + 1).toPaddedString(2) + '-' +
+ this.getUTCDate().toPaddedString(2) + 'T' +
+ this.getUTCHours().toPaddedString(2) + ':' +
+ this.getUTCMinutes().toPaddedString(2) + ':' +
+ this.getUTCSeconds().toPaddedString(2) + 'Z"';
+};
+
+var Try = {
+ these: function() {
+ var returnValue;
+
+ for (var i = 0, length = arguments.length; i < length; i++) {
+ var lambda = arguments[i];
+ try {
+ returnValue = lambda();
+ break;
+ } catch (e) { }
+ }
+
+ return returnValue;
+ }
+};
+
+RegExp.prototype.match = RegExp.prototype.test;
+
+RegExp.escape = function(str) {
+ return String(str).replace(/([.*+?^=!:${}()|[\]\/\\])/g, '\\$1');
+};
+
+/*--------------------------------------------------------------------------*/
+
+var PeriodicalExecuter = Class.create({
+ initialize: function(callback, frequency) {
+ this.callback = callback;
+ this.frequency = frequency;
+ this.currentlyExecuting = false;
+
+ this.registerCallback();
+ },
+
+ registerCallback: function() {
+ this.timer = setInterval(this.onTimerEvent.bind(this), this.frequency * 1000);
+ },
+
+ execute: function() {
+ this.callback(this);
+ },
+
+ stop: function() {
+ if (!this.timer) return;
+ clearInterval(this.timer);
+ this.timer = null;
+ },
+
+ onTimerEvent: function() {
+ if (!this.currentlyExecuting) {
+ try {
+ this.currentlyExecuting = true;
+ this.execute();
+ } finally {
+ this.currentlyExecuting = false;
+ }
+ }
+ }
+});
+Object.extend(String, {
+ interpret: function(value) {
+ return value == null ? '' : String(value);
+ },
+ specialChar: {
+ '\b': '\\b',
+ '\t': '\\t',
+ '\n': '\\n',
+ '\f': '\\f',
+ '\r': '\\r',
+ '\\': '\\\\'
+ }
+});
+
+Object.extend(String.prototype, {
+ gsub: function(pattern, replacement) {
+ var result = '', source = this, match;
+ replacement = arguments.callee.prepareReplacement(replacement);
+
+ while (source.length > 0) {
+ if (match = source.match(pattern)) {
+ result += source.slice(0, match.index);
+ result += String.interpret(replacement(match));
+ source = source.slice(match.index + match[0].length);
+ } else {
+ result += source, source = '';
+ }
+ }
+ return result;
+ },
+
+ sub: function(pattern, replacement, count) {
+ replacement = this.gsub.prepareReplacement(replacement);
+ count = count === undefined ? 1 : count;
+
+ return this.gsub(pattern, function(match) {
+ if (--count < 0) return match[0];
+ return replacement(match);
+ });
+ },
+
+ scan: function(pattern, iterator) {
+ this.gsub(pattern, iterator);
+ return String(this);
+ },
+
+ truncate: function(length, truncation) {
+ length = length || 30;
+ truncation = truncation === undefined ? '...' : truncation;
+ return this.length > length ?
+ this.slice(0, length - truncation.length) + truncation : String(this);
+ },
+
+ strip: function() {
+ return this.replace(/^\s+/, '').replace(/\s+$/, '');
+ },
+
+ stripTags: function() {
+ return this.replace(/<\/?[^>]+>/gi, '');
+ },
+
+ stripScripts: function() {
+ return this.replace(new RegExp(Prototype.ScriptFragment, 'img'), '');
+ },
+
+ extractScripts: function() {
+ var matchAll = new RegExp(Prototype.ScriptFragment, 'img');
+ var matchOne = new RegExp(Prototype.ScriptFragment, 'im');
+ return (this.match(matchAll) || []).map(function(scriptTag) {
+ return (scriptTag.match(matchOne) || ['', ''])[1];
+ });
+ },
+
+ evalScripts: function() {
+ return this.extractScripts().map(function(script) { return eval(script) });
+ },
+
+ escapeHTML: function() {
+ var self = arguments.callee;
+ self.text.data = this;
+ return self.div.innerHTML;
+ },
+
+ unescapeHTML: function() {
+ var div = new Element('div');
+ div.innerHTML = this.stripTags();
+ return div.childNodes[0] ? (div.childNodes.length > 1 ?
+ $A(div.childNodes).inject('', function(memo, node) { return memo+node.nodeValue }) :
+ div.childNodes[0].nodeValue) : '';
+ },
+
+ toQueryParams: function(separator) {
+ var match = this.strip().match(/([^?#]*)(#.*)?$/);
+ if (!match) return { };
+
+ return match[1].split(separator || '&').inject({ }, function(hash, pair) {
+ if ((pair = pair.split('='))[0]) {
+ var key = decodeURIComponent(pair.shift());
+ var value = pair.length > 1 ? pair.join('=') : pair[0];
+ if (value != undefined) value = decodeURIComponent(value);
+
+ if (key in hash) {
+ if (!Object.isArray(hash[key])) hash[key] = [hash[key]];
+ hash[key].push(value);
+ }
+ else hash[key] = value;
+ }
+ return hash;
+ });
+ },
+
+ toArray: function() {
+ return this.split('');
+ },
+
+ succ: function() {
+ return this.slice(0, this.length - 1) +
+ String.fromCharCode(this.charCodeAt(this.length - 1) + 1);
+ },
+
+ times: function(count) {
+ return count < 1 ? '' : new Array(count + 1).join(this);
+ },
+
+ camelize: function() {
+ var parts = this.split('-'), len = parts.length;
+ if (len == 1) return parts[0];
+
+ var camelized = this.charAt(0) == '-'
+ ? parts[0].charAt(0).toUpperCase() + parts[0].substring(1)
+ : parts[0];
+
+ for (var i = 1; i < len; i++)
+ camelized += parts[i].charAt(0).toUpperCase() + parts[i].substring(1);
+
+ return camelized;
+ },
+
+ capitalize: function() {
+ return this.charAt(0).toUpperCase() + this.substring(1).toLowerCase();
+ },
+
+ underscore: function() {
+ return this.gsub(/::/, '/').gsub(/([A-Z]+)([A-Z][a-z])/,'#{1}_#{2}').gsub(/([a-z\d])([A-Z])/,'#{1}_#{2}').gsub(/-/,'_').toLowerCase();
+ },
+
+ dasherize: function() {
+ return this.gsub(/_/,'-');
+ },
+
+ inspect: function(useDoubleQuotes) {
+ var escapedString = this.gsub(/[\x00-\x1f\\]/, function(match) {
+ var character = String.specialChar[match[0]];
+ return character ? character : '\\u00' + match[0].charCodeAt().toPaddedString(2, 16);
+ });
+ if (useDoubleQuotes) return '"' + escapedString.replace(/"/g, '\\"') + '"';
+ return "'" + escapedString.replace(/'/g, '\\\'') + "'";
+ },
+
+ toJSON: function() {
+ return this.inspect(true);
+ },
+
+ unfilterJSON: function(filter) {
+ return this.sub(filter || Prototype.JSONFilter, '#{1}');
+ },
+
+ isJSON: function() {
+ var str = this.replace(/\\./g, '@').replace(/"[^"\\\n\r]*"/g, '');
+ return (/^[,:{}\[\]0-9.\-+Eaeflnr-u \n\r\t]*$/).test(str);
+ },
+
+ evalJSON: function(sanitize) {
+ var json = this.unfilterJSON();
+ try {
+ if (!sanitize || json.isJSON()) return eval('(' + json + ')');
+ } catch (e) { }
+ throw new SyntaxError('Badly formed JSON string: ' + this.inspect());
+ },
+
+ include: function(pattern) {
+ return this.indexOf(pattern) > -1;
+ },
+
+ startsWith: function(pattern) {
+ return this.indexOf(pattern) === 0;
+ },
+
+ endsWith: function(pattern) {
+ var d = this.length - pattern.length;
+ return d >= 0 && this.lastIndexOf(pattern) === d;
+ },
+
+ empty: function() {
+ return this == '';
+ },
+
+ blank: function() {
+ return /^\s*$/.test(this);
+ },
+
+ interpolate: function(object, pattern) {
+ return new Template(this, pattern).evaluate(object);
+ }
+});
+
+if (Prototype.Browser.WebKit || Prototype.Browser.IE) Object.extend(String.prototype, {
+ escapeHTML: function() {
+ return this.replace(/&/g,'&').replace(/</g,'<').replace(/>/g,'>');
+ },
+ unescapeHTML: function() {
+ return this.replace(/&/g,'&').replace(/</g,'<').replace(/>/g,'>');
+ }
+});
+
+String.prototype.gsub.prepareReplacement = function(replacement) {
+ if (Object.isFunction(replacement)) return replacement;
+ var template = new Template(replacement);
+ return function(match) { return template.evaluate(match) };
+};
+
+String.prototype.parseQuery = String.prototype.toQueryParams;
+
+Object.extend(String.prototype.escapeHTML, {
+ div: document.createElement('div'),
+ text: document.createTextNode('')
+});
+
+with (String.prototype.escapeHTML) div.appendChild(text);
+
+var Template = Class.create({
+ initialize: function(template, pattern) {
+ this.template = template.toString();
+ this.pattern = pattern || Template.Pattern;
+ },
+
+ evaluate: function(object) {
+ if (Object.isFunction(object.toTemplateReplacements))
+ object = object.toTemplateReplacements();
+
+ return this.template.gsub(this.pattern, function(match) {
+ if (object == null) return '';
+
+ var before = match[1] || '';
+ if (before == '\\') return match[2];
+
+ var ctx = object, expr = match[3];
+ var pattern = /^([^.[]+|\[((?:.*?[^\\])?)\])(\.|\[|$)/, match = pattern.exec(expr);
+ if (match == null) return before;
+
+ while (match != null) {
+ var comp = match[1].startsWith('[') ? match[2].gsub('\\\\]', ']') : match[1];
+ ctx = ctx[comp];
+ if (null == ctx || '' == match[3]) break;
+ expr = expr.substring('[' == match[3] ? match[1].length : match[0].length);
+ match = pattern.exec(expr);
+ }
+
+ return before + String.interpret(ctx);
+ }.bind(this));
+ }
+});
+Template.Pattern = /(^|.|\r|\n)(#\{(.*?)\})/;
+
+var $break = { };
+
+var Enumerable = {
+ each: function(iterator, context) {
+ var index = 0;
+ iterator = iterator.bind(context);
+ try {
+ this._each(function(value) {
+ iterator(value, index++);
+ });
+ } catch (e) {
+ if (e != $break) throw e;
+ }
+ return this;
+ },
+
+ eachSlice: function(number, iterator, context) {
+ iterator = iterator ? iterator.bind(context) : Prototype.K;
+ var index = -number, slices = [], array = this.toArray();
+ while ((index += number) < array.length)
+ slices.push(array.slice(index, index+number));
+ return slices.collect(iterator, context);
+ },
+
+ all: function(iterator, context) {
+ iterator = iterator ? iterator.bind(context) : Prototype.K;
+ var result = true;
+ this.each(function(value, index) {
+ result = result && !!iterator(value, index);
+ if (!result) throw $break;
+ });
+ return result;
+ },
+
+ any: function(iterator, context) {
+ iterator = iterator ? iterator.bind(context) : Prototype.K;
+ var result = false;
+ this.each(function(value, index) {
+ if (result = !!iterator(value, index))
+ throw $break;
+ });
+ return result;
+ },
+
+ collect: function(iterator, context) {
+ iterator = iterator ? iterator.bind(context) : Prototype.K;
+ var results = [];
+ this.each(function(value, index) {
+ results.push(iterator(value, index));
+ });
+ return results;
+ },
+
+ detect: function(iterator, context) {
+ iterator = iterator.bind(context);
+ var result;
+ this.each(function(value, index) {
+ if (iterator(value, index)) {
+ result = value;
+ throw $break;
+ }
+ });
+ return result;
+ },
+
+ findAll: function(iterator, context) {
+ iterator = iterator.bind(context);
+ var results = [];
+ this.each(function(value, index) {
+ if (iterator(value, index))
+ results.push(value);
+ });
+ return results;
+ },
+
+ grep: function(filter, iterator, context) {
+ iterator = iterator ? iterator.bind(context) : Prototype.K;
+ var results = [];
+
+ if (Object.isString(filter))
+ filter = new RegExp(filter);
+
+ this.each(function(value, index) {
+ if (filter.match(value))
+ results.push(iterator(value, index));
+ });
+ return results;
+ },
+
+ include: function(object) {
+ if (Object.isFunction(this.indexOf))
+ if (this.indexOf(object) != -1) return true;
+
+ var found = false;
+ this.each(function(value) {
+ if (value == object) {
+ found = true;
+ throw $break;
+ }
+ });
+ return found;
+ },
+
+ inGroupsOf: function(number, fillWith) {
+ fillWith = fillWith === undefined ? null : fillWith;
+ return this.eachSlice(number, function(slice) {
+ while(slice.length < number) slice.push(fillWith);
+ return slice;
+ });
+ },
+
+ inject: function(memo, iterator, context) {
+ iterator = iterator.bind(context);
+ this.each(function(value, index) {
+ memo = iterator(memo, value, index);
+ });
+ return memo;
+ },
+
+ invoke: function(method) {
+ var args = $A(arguments).slice(1);
+ return this.map(function(value) {
+ return value[method].apply(value, args);
+ });
+ },
+
+ max: function(iterator, context) {
+ iterator = iterator ? iterator.bind(context) : Prototype.K;
+ var result;
+ this.each(function(value, index) {
+ value = iterator(value, index);
+ if (result == undefined || value >= result)
+ result = value;
+ });
+ return result;
+ },
+
+ min: function(iterator, context) {
+ iterator = iterator ? iterator.bind(context) : Prototype.K;
+ var result;
+ this.each(function(value, index) {
+ value = iterator(value, index);
+ if (result == undefined || value < result)
+ result = value;
+ });
+ return result;
+ },
+
+ partition: function(iterator, context) {
+ iterator = iterator ? iterator.bind(context) : Prototype.K;
+ var trues = [], falses = [];
+ this.each(function(value, index) {
+ (iterator(value, index) ?
+ trues : falses).push(value);
+ });
+ return [trues, falses];
+ },
+
+ pluck: function(property) {
+ var results = [];
+ this.each(function(value) {
+ results.push(value[property]);
+ });
+ return results;
+ },
+
+ reject: function(iterator, context) {
+ iterator = iterator.bind(context);
+ var results = [];
+ this.each(function(value, index) {
+ if (!iterator(value, index))
+ results.push(value);
+ });
+ return results;
+ },
+
+ sortBy: function(iterator, context) {
+ iterator = iterator.bind(context);
+ return this.map(function(value, index) {
+ return {value: value, criteria: iterator(value, index)};
+ }).sort(function(left, right) {
+ var a = left.criteria, b = right.criteria;
+ return a < b ? -1 : a > b ? 1 : 0;
+ }).pluck('value');
+ },
+
+ toArray: function() {
+ return this.map();
+ },
+
+ zip: function() {
+ var iterator = Prototype.K, args = $A(arguments);
+ if (Object.isFunction(args.last()))
+ iterator = args.pop();
+
+ var collections = [this].concat(args).map($A);
+ return this.map(function(value, index) {
+ return iterator(collections.pluck(index));
+ });
+ },
+
+ size: function() {
+ return this.toArray().length;
+ },
+
+ inspect: function() {
+ return '#<Enumerable:' + this.toArray().inspect() + '>';
+ }
+};
+
+Object.extend(Enumerable, {
+ map: Enumerable.collect,
+ find: Enumerable.detect,
+ select: Enumerable.findAll,
+ filter: Enumerable.findAll,
+ member: Enumerable.include,
+ entries: Enumerable.toArray,
+ every: Enumerable.all,
+ some: Enumerable.any
+});
+function $A(iterable) {
+ if (!iterable) return [];
+ if (iterable.toArray) return iterable.toArray();
+ var length = iterable.length, results = new Array(length);
+ while (length--) results[length] = iterable[length];
+ return results;
+}
+
+if (Prototype.Browser.WebKit) {
+ function $A(iterable) {
+ if (!iterable) return [];
+ if (!(Object.isFunction(iterable) && iterable == '[object NodeList]') &&
+ iterable.toArray) return iterable.toArray();
+ var length = iterable.length, results = new Array(length);
+ while (length--) results[length] = iterable[length];
+ return results;
+ }
+}
+
+Array.from = $A;
+
+Object.extend(Array.prototype, Enumerable);
+
+if (!Array.prototype._reverse) Array.prototype._reverse = Array.prototype.reverse;
+
+Object.extend(Array.prototype, {
+ _each: function(iterator) {
+ for (var i = 0, length = this.length; i < length; i++)
+ iterator(this[i]);
+ },
+
+ clear: function() {
+ this.length = 0;
+ return this;
+ },
+
+ first: function() {
+ return this[0];
+ },
+
+ last: function() {
+ return this[this.length - 1];
+ },
+
+ compact: function() {
+ return this.select(function(value) {
+ return value != null;
+ });
+ },
+
+ flatten: function() {
+ return this.inject([], function(array, value) {
+ return array.concat(Object.isArray(value) ?
+ value.flatten() : [value]);
+ });
+ },
+
+ without: function() {
+ var values = $A(arguments);
+ return this.select(function(value) {
+ return !values.include(value);
+ });
+ },
+
+ reverse: function(inline) {
+ return (inline !== false ? this : this.toArray())._reverse();
+ },
+
+ reduce: function() {
+ return this.length > 1 ? this : this[0];
+ },
+
+ uniq: function(sorted) {
+ return this.inject([], function(array, value, index) {
+ if (0 == index || (sorted ? array.last() != value : !array.include(value)))
+ array.push(value);
+ return array;
+ });
+ },
+
+ intersect: function(array) {
+ return this.uniq().findAll(function(item) {
+ return array.detect(function(value) { return item === value });
+ });
+ },
+
+ clone: function() {
+ return [].concat(this);
+ },
+
+ size: function() {
+ return this.length;
+ },
+
+ inspect: function() {
+ return '[' + this.map(Object.inspect).join(', ') + ']';
+ },
+
+ toJSON: function() {
+ var results = [];
+ this.each(function(object) {
+ var value = Object.toJSON(object);
+ if (value !== undefined) results.push(value);
+ });
+ return '[' + results.join(', ') + ']';
+ }
+});
+
+// use native browser JS 1.6 implementation if available
+if (Object.isFunction(Array.prototype.forEach))
+ Array.prototype._each = Array.prototype.forEach;
+
+if (!Array.prototype.indexOf) Array.prototype.indexOf = function(item, i) {
+ i || (i = 0);
+ var length = this.length;
+ if (i < 0) i = length + i;
+ for (; i < length; i++)
+ if (this[i] === item) return i;
+ return -1;
+};
+
+if (!Array.prototype.lastIndexOf) Array.prototype.lastIndexOf = function(item, i) {
+ i = isNaN(i) ? this.length : (i < 0 ? this.length + i : i) + 1;
+ var n = this.slice(0, i).reverse().indexOf(item);
+ return (n < 0) ? n : i - n - 1;
+};
+
+Array.prototype.toArray = Array.prototype.clone;
+
+function $w(string) {
+ if (!Object.isString(string)) return [];
+ string = string.strip();
+ return string ? string.split(/\s+/) : [];
+}
+
+if (Prototype.Browser.Opera){
+ Array.prototype.concat = function() {
+ var array = [];
+ for (var i = 0, length = this.length; i < length; i++) array.push(this[i]);
+ for (var i = 0, length = arguments.length; i < length; i++) {
+ if (Object.isArray(arguments[i])) {
+ for (var j = 0, arrayLength = arguments[i].length; j < arrayLength; j++)
+ array.push(arguments[i][j]);
+ } else {
+ array.push(arguments[i]);
+ }
+ }
+ return array;
+ };
+}
+Object.extend(Number.prototype, {
+ toColorPart: function() {
+ return this.toPaddedString(2, 16);
+ },
+
+ succ: function() {
+ return this + 1;
+ },
+
+ times: function(iterator) {
+ $R(0, this, true).each(iterator);
+ return this;
+ },
+
+ toPaddedString: function(length, radix) {
+ var string = this.toString(radix || 10);
+ return '0'.times(length - string.length) + string;
+ },
+
+ toJSON: function() {
+ return isFinite(this) ? this.toString() : 'null';
+ }
+});
+
+$w('abs round ceil floor').each(function(method){
+ Number.prototype[method] = Math[method].methodize();
+});
+function $H(object) {
+ return new Hash(object);
+};
+
+var Hash = Class.create(Enumerable, (function() {
+ if (function() {
+ var i = 0, Test = function(value) { this.key = value };
+ Test.prototype.key = 'foo';
+ for (var property in new Test('bar')) i++;
+ return i > 1;
+ }()) {
+ function each(iterator) {
+ var cache = [];
+ for (var key in this._object) {
+ var value = this._object[key];
+ if (cache.include(key)) continue;
+ cache.push(key);
+ var pair = [key, value];
+ pair.key = key;
+ pair.value = value;
+ iterator(pair);
+ }
+ }
+ } else {
+ function each(iterator) {
+ for (var key in this._object) {
+ var value = this._object[key], pair = [key, value];
+ pair.key = key;
+ pair.value = value;
+ iterator(pair);
+ }
+ }
+ }
+
+ function toQueryPair(key, value) {
+ if (Object.isUndefined(value)) return key;
+ return key + '=' + encodeURIComponent(String.interpret(value));
+ }
+
+ return {
+ initialize: function(object) {
+ this._object = Object.isHash(object) ? object.toObject() : Object.clone(object);
+ },
+
+ _each: each,
+
+ set: function(key, value) {
+ return this._object[key] = value;
+ },
+
+ get: function(key) {
+ return this._object[key];
+ },
+
+ unset: function(key) {
+ var value = this._object[key];
+ delete this._object[key];
+ return value;
+ },
+
+ toObject: function() {
+ return Object.clone(this._object);
+ },
+
+ keys: function() {
+ return this.pluck('key');
+ },
+
+ values: function() {
+ return this.pluck('value');
+ },
+
+ index: function(value) {
+ var match = this.detect(function(pair) {
+ return pair.value === value;
+ });
+ return match && match.key;
+ },
+
+ merge: function(object) {
+ return this.clone().update(object);
+ },
+
+ update: function(object) {
+ return new Hash(object).inject(this, function(result, pair) {
+ result.set(pair.key, pair.value);
+ return result;
+ });
+ },
+
+ toQueryString: function() {
+ return this.map(function(pair) {
+ var key = encodeURIComponent(pair.key), values = pair.value;
+
+ if (values && typeof values == 'object') {
+ if (Object.isArray(values))
+ return values.map(toQueryPair.curry(key)).join('&');
+ }
+ return toQueryPair(key, values);
+ }).join('&');
+ },
+
+ inspect: function() {
+ return '#<Hash:{' + this.map(function(pair) {
+ return pair.map(Object.inspect).join(': ');
+ }).join(', ') + '}>';
+ },
+
+ toJSON: function() {
+ return Object.toJSON(this.toObject());
+ },
+
+ clone: function() {
+ return new Hash(this);
+ }
+ }
+})());
+
+Hash.prototype.toTemplateReplacements = Hash.prototype.toObject;
+Hash.from = $H;
+var ObjectRange = Class.create(Enumerable, {
+ initialize: function(start, end, exclusive) {
+ this.start = start;
+ this.end = end;
+ this.exclusive = exclusive;
+ },
+
+ _each: function(iterator) {
+ var value = this.start;
+ while (this.include(value)) {
+ iterator(value);
+ value = value.succ();
+ }
+ },
+
+ include: function(value) {
+ if (value < this.start)
+ return false;
+ if (this.exclusive)
+ return value < this.end;
+ return value <= this.end;
+ }
+});
+
+var $R = function(start, end, exclusive) {
+ return new ObjectRange(start, end, exclusive);
+};
+
+var Ajax = {
+ getTransport: function() {
+ return Try.these(
+ function() {return new XMLHttpRequest()},
+ function() {return new ActiveXObject('Msxml2.XMLHTTP')},
+ function() {return new ActiveXObject('Microsoft.XMLHTTP')}
+ ) || false;
+ },
+
+ activeRequestCount: 0
+};
+
+Ajax.Responders = {
+ responders: [],
+
+ _each: function(iterator) {
+ this.responders._each(iterator);
+ },
+
+ register: function(responder) {
+ if (!this.include(responder))
+ this.responders.push(responder);
+ },
+
+ unregister: function(responder) {
+ this.responders = this.responders.without(responder);
+ },
+
+ dispatch: function(callback, request, transport, json) {
+ this.each(function(responder) {
+ if (Object.isFunction(responder[callback])) {
+ try {
+ responder[callback].apply(responder, [request, transport, json]);
+ } catch (e) { }
+ }
+ });
+ }
+};
+
+Object.extend(Ajax.Responders, Enumerable);
+
+Ajax.Responders.register({
+ onCreate: function() { Ajax.activeRequestCount++ },
+ onComplete: function() { Ajax.activeRequestCount-- }
+});
+
+Ajax.Base = Class.create({
+ initialize: function(options) {
+ this.options = {
+ method: 'post',
+ asynchronous: true,
+ contentType: 'application/x-www-form-urlencoded',
+ encoding: 'UTF-8',
+ parameters: '',
+ evalJSON: true,
+ evalJS: true
+ };
+ Object.extend(this.options, options || { });
+
+ this.options.method = this.options.method.toLowerCase();
+ if (Object.isString(this.options.parameters))
+ this.options.parameters = this.options.parameters.toQueryParams();
+ }
+});
+
+Ajax.Request = Class.create(Ajax.Base, {
+ _complete: false,
+
+ initialize: function($super, url, options) {
+ $super(options);
+ this.transport = Ajax.getTransport();
+ this.request(url);
+ },
+
+ request: function(url) {
+ this.url = url;
+ this.method = this.options.method;
+ var params = Object.clone(this.options.parameters);
+
+ if (!['get', 'post'].include(this.method)) {
+ // simulate other verbs over post
+ params['_method'] = this.method;
+ this.method = 'post';
+ }
+
+ this.parameters = params;
+
+ if (params = Object.toQueryString(params)) {
+ // when GET, append parameters to URL
+ if (this.method == 'get')
+ this.url += (this.url.include('?') ? '&' : '?') + params;
+ else if (/Konqueror|Safari|KHTML/.test(navigator.userAgent))
+ params += '&_=';
+ }
+
+ try {
+ var response = new Ajax.Response(this);
+ if (this.options.onCreate) this.options.onCreate(response);
+ Ajax.Responders.dispatch('onCreate', this, response);
+
+ this.transport.open(this.method.toUpperCase(), this.url,
+ this.options.asynchronous);
+
+ if (this.options.asynchronous) this.respondToReadyState.bind(this).defer(1);
+
+ this.transport.onreadystatechange = this.onStateChange.bind(this);
+ this.setRequestHeaders();
+
+ this.body = this.method == 'post' ? (this.options.postBody || params) : null;
+ this.transport.send(this.body);
+
+ /* Force Firefox to handle ready state 4 for synchronous requests */
+ if (!this.options.asynchronous && this.transport.overrideMimeType)
+ this.onStateChange();
+
+ }
+ catch (e) {
+ this.dispatchException(e);
+ }
+ },
+
+ onStateChange: function() {
+ var readyState = this.transport.readyState;
+ if (readyState > 1 && !((readyState == 4) && this._complete))
+ this.respondToReadyState(this.transport.readyState);
+ },
+
+ setRequestHeaders: function() {
+ var headers = {
+ 'X-Requested-With': 'XMLHttpRequest',
+ 'X-Prototype-Version': Prototype.Version,
+ 'Accept': 'text/javascript, text/html, application/xml, text/xml, */*'
+ };
+
+ if (this.method == 'post') {
+ headers['Content-type'] = this.options.contentType +
+ (this.options.encoding ? '; charset=' + this.options.encoding : '');
+
+ /* Force "Connection: close" for older Mozilla browsers to work
+ * around a bug where XMLHttpRequest sends an incorrect
+ * Content-length header. See Mozilla Bugzilla #246651.
+ */
+ if (this.transport.overrideMimeType &&
+ (navigator.userAgent.match(/Gecko\/(\d{4})/) || [0,2005])[1] < 2005)
+ headers['Connection'] = 'close';
+ }
+
+ // user-defined headers
+ if (typeof this.options.requestHeaders == 'object') {
+ var extras = this.options.requestHeaders;
+
+ if (Object.isFunction(extras.push))
+ for (var i = 0, length = extras.length; i < length; i += 2)
+ headers[extras[i]] = extras[i+1];
+ else
+ $H(extras).each(function(pair) { headers[pair.key] = pair.value });
+ }
+
+ for (var name in headers)
+ this.transport.setRequestHeader(name, headers[name]);
+ },
+
+ success: function() {
+ var status = this.getStatus();
+ return !status || (status >= 200 && status < 300);
+ },
+
+ getStatus: function() {
+ try {
+ return this.transport.status || 0;
+ } catch (e) { return 0 }
+ },
+
+ respondToReadyState: function(readyState) {
+ var state = Ajax.Request.Events[readyState], response = new Ajax.Response(this);
+
+ if (state == 'Complete') {
+ try {
+ this._complete = true;
+ (this.options['on' + response.status]
+ || this.options['on' + (this.success() ? 'Success' : 'Failure')]
+ || Prototype.emptyFunction)(response, response.headerJSON);
+ } catch (e) {
+ this.dispatchException(e);
+ }
+
+ var contentType = response.getHeader('Content-type');
+ if (this.options.evalJS == 'force'
+ || (this.options.evalJS && contentType
+ && contentType.match(/^\s*(text|application)\/(x-)?(java|ecma)script(;.*)?\s*$/i)))
+ this.evalResponse();
+ }
+
+ try {
+ (this.options['on' + state] || Prototype.emptyFunction)(response, response.headerJSON);
+ Ajax.Responders.dispatch('on' + state, this, response, response.headerJSON);
+ } catch (e) {
+ this.dispatchException(e);
+ }
+
+ if (state == 'Complete') {
+ // avoid memory leak in MSIE: clean up
+ this.transport.onreadystatechange = Prototype.emptyFunction;
+ }
+ },
+
+ getHeader: function(name) {
+ try {
+ return this.transport.getResponseHeader(name);
+ } catch (e) { return null }
+ },
+
+ evalResponse: function() {
+ try {
+ return eval((this.transport.responseText || '').unfilterJSON());
+ } catch (e) {
+ this.dispatchException(e);
+ }
+ },
+
+ dispatchException: function(exception) {
+ (this.options.onException || Prototype.emptyFunction)(this, exception);
+ Ajax.Responders.dispatch('onException', this, exception);
+ }
+});
+
+Ajax.Request.Events =
+ ['Uninitialized', 'Loading', 'Loaded', 'Interactive', 'Complete'];
+
+Ajax.Response = Class.create({
+ initialize: function(request){
+ this.request = request;
+ var transport = this.transport = request.transport,
+ readyState = this.readyState = transport.readyState;
+
+ if((readyState > 2 && !Prototype.Browser.IE) || readyState == 4) {
+ this.status = this.getStatus();
+ this.statusText = this.getStatusText();
+ this.responseText = String.interpret(transport.responseText);
+ this.headerJSON = this._getHeaderJSON();
+ }
+
+ if(readyState == 4) {
+ var xml = transport.responseXML;
+ this.responseXML = xml === undefined ? null : xml;
+ this.responseJSON = this._getResponseJSON();
+ }
+ },
+
+ status: 0,
+ statusText: '',
+
+ getStatus: Ajax.Request.prototype.getStatus,
+
+ getStatusText: function() {
+ try {
+ return this.transport.statusText || '';
+ } catch (e) { return '' }
+ },
+
+ getHeader: Ajax.Request.prototype.getHeader,
+
+ getAllHeaders: function() {
+ try {
+ return this.getAllResponseHeaders();
+ } catch (e) { return null }
+ },
+
+ getResponseHeader: function(name) {
+ return this.transport.getResponseHeader(name);
+ },
+
+ getAllResponseHeaders: function() {
+ return this.transport.getAllResponseHeaders();
+ },
+
+ _getHeaderJSON: function() {
+ var json = this.getHeader('X-JSON');
+ if (!json) return null;
+ json = decodeURIComponent(escape(json));
+ try {
+ return json.evalJSON(this.request.options.sanitizeJSON);
+ } catch (e) {
+ this.request.dispatchException(e);
+ }
+ },
+
+ _getResponseJSON: function() {
+ var options = this.request.options;
+ if (!options.evalJSON || (options.evalJSON != 'force' &&
+ !(this.getHeader('Content-type') || '').include('application/json')))
+ return null;
+ try {
+ return this.transport.responseText.evalJSON(options.sanitizeJSON);
+ } catch (e) {
+ this.request.dispatchException(e);
+ }
+ }
+});
+
+Ajax.Updater = Class.create(Ajax.Request, {
+ initialize: function($super, container, url, options) {
+ this.container = {
+ success: (container.success || container),
+ failure: (container.failure || (container.success ? null : container))
+ };
+
+ options = options || { };
+ var onComplete = options.onComplete;
+ options.onComplete = (function(response, param) {
+ this.updateContent(response.responseText);
+ if (Object.isFunction(onComplete)) onComplete(response, param);
+ }).bind(this);
+
+ $super(url, options);
+ },
+
+ updateContent: function(responseText) {
+ var receiver = this.container[this.success() ? 'success' : 'failure'],
+ options = this.options;
+
+ if (!options.evalScripts) responseText = responseText.stripScripts();
+
+ if (receiver = $(receiver)) {
+ if (options.insertion) {
+ if (Object.isString(options.insertion)) {
+ var insertion = { }; insertion[options.insertion] = responseText;
+ receiver.insert(insertion);
+ }
+ else options.insertion(receiver, responseText);
+ }
+ else receiver.update(responseText);
+ }
+
+ if (this.success()) {
+ if (this.onComplete) this.onComplete.bind(this).defer();
+ }
+ }
+});
+
+Ajax.PeriodicalUpdater = Class.create(Ajax.Base, {
+ initialize: function($super, container, url, options) {
+ $super(options);
+ this.onComplete = this.options.onComplete;
+
+ this.frequency = (this.options.frequency || 2);
+ this.decay = (this.options.decay || 1);
+
+ this.updater = { };
+ this.container = container;
+ this.url = url;
+
+ this.start();
+ },
+
+ start: function() {
+ this.options.onComplete = this.updateComplete.bind(this);
+ this.onTimerEvent();
+ },
+
+ stop: function() {
+ this.updater.options.onComplete = undefined;
+ clearTimeout(this.timer);
+ (this.onComplete || Prototype.emptyFunction).apply(this, arguments);
+ },
+
+ updateComplete: function(response) {
+ if (this.options.decay) {
+ this.decay = (response.responseText == this.lastText ?
+ this.decay * this.options.decay : 1);
+
+ this.lastText = response.responseText;
+ }
+ this.timer = this.onTimerEvent.bind(this).delay(this.decay * this.frequency);
+ },
+
+ onTimerEvent: function() {
+ this.updater = new Ajax.Updater(this.container, this.url, this.options);
+ }
+});
+function $(element) {
+ if (arguments.length > 1) {
+ for (var i = 0, elements = [], length = arguments.length; i < length; i++)
+ elements.push($(arguments[i]));
+ return elements;
+ }
+ if (Object.isString(element))
+ element = document.getElementById(element);
+ return Element.extend(element);
+}
+
+if (Prototype.BrowserFeatures.XPath) {
+ document._getElementsByXPath = function(expression, parentElement) {
+ var results = [];
+ var query = document.evaluate(expression, $(parentElement) || document,
+ null, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
+ for (var i = 0, length = query.snapshotLength; i < length; i++)
+ results.push(Element.extend(query.snapshotItem(i)));
+ return results;
+ };
+}
+
+/*--------------------------------------------------------------------------*/
+
+if (!window.Node) var Node = { };
+
+if (!Node.ELEMENT_NODE) {
+ // DOM level 2 ECMAScript Language Binding
+ Object.extend(Node, {
+ ELEMENT_NODE: 1,
+ ATTRIBUTE_NODE: 2,
+ TEXT_NODE: 3,
+ CDATA_SECTION_NODE: 4,
+ ENTITY_REFERENCE_NODE: 5,
+ ENTITY_NODE: 6,
+ PROCESSING_INSTRUCTION_NODE: 7,
+ COMMENT_NODE: 8,
+ DOCUMENT_NODE: 9,
+ DOCUMENT_TYPE_NODE: 10,
+ DOCUMENT_FRAGMENT_NODE: 11,
+ NOTATION_NODE: 12
+ });
+}
+
+(function() {
+ var element = this.Element;
+ this.Element = function(tagName, attributes) {
+ attributes = attributes || { };
+ tagName = tagName.toLowerCase();
+ var cache = Element.cache;
+ if (Prototype.Browser.IE && attributes.name) {
+ tagName = '<' + tagName + ' name="' + attributes.name + '">';
+ delete attributes.name;
+ return Element.writeAttribute(document.createElement(tagName), attributes);
+ }
+ if (!cache[tagName]) cache[tagName] = Element.extend(document.createElement(tagName));
+ return Element.writeAttribute(cache[tagName].cloneNode(false), attributes);
+ };
+ Object.extend(this.Element, element || { });
+}).call(window);
+
+Element.cache = { };
+
+Element.Methods = {
+ visible: function(element) {
+ return $(element).style.display != 'none';
+ },
+
+ toggle: function(element) {
+ element = $(element);
+ Element[Element.visible(element) ? 'hide' : 'show'](element);
+ return element;
+ },
+
+ hide: function(element) {
+ $(element).style.display = 'none';
+ return element;
+ },
+
+ show: function(element) {
+ $(element).style.display = '';
+ return element;
+ },
+
+ remove: function(element) {
+ element = $(element);
+ element.parentNode.removeChild(element);
+ return element;
+ },
+
+ update: function(element, content) {
+ element = $(element);
+ if (content && content.toElement) content = content.toElement();
+ if (Object.isElement(content)) return element.update().insert(content);
+ content = Object.toHTML(content);
+ element.innerHTML = content.stripScripts();
+ content.evalScripts.bind(content).defer();
+ return element;
+ },
+
+ replace: function(element, content) {
+ element = $(element);
+ if (content && content.toElement) content = content.toElement();
+ else if (!Object.isElement(content)) {
+ content = Object.toHTML(content);
+ var range = element.ownerDocument.createRange();
+ range.selectNode(element);
+ content.evalScripts.bind(content).defer();
+ content = range.createContextualFragment(content.stripScripts());
+ }
+ element.parentNode.replaceChild(content, element);
+ return element;
+ },
+
+ insert: function(element, insertions) {
+ element = $(element);
+
+ if (Object.isString(insertions) || Object.isNumber(insertions) ||
+ Object.isElement(insertions) || (insertions && (insertions.toElement || insertions.toHTML)))
+ insertions = {bottom:insertions};
+
+ var content, t, range;
+
+ for (position in insertions) {
+ content = insertions[position];
+ position = position.toLowerCase();
+ t = Element._insertionTranslations[position];
+
+ if (content && content.toElement) content = content.toElement();
+ if (Object.isElement(content)) {
+ t.insert(element, content);
+ continue;
+ }
+
+ content = Object.toHTML(content);
+
+ range = element.ownerDocument.createRange();
+ t.initializeRange(element, range);
+ t.insert(element, range.createContextualFragment(content.stripScripts()));
+
+ content.evalScripts.bind(content).defer();
+ }
+
+ return element;
+ },
+
+ wrap: function(element, wrapper, attributes) {
+ element = $(element);
+ if (Object.isElement(wrapper))
+ $(wrapper).writeAttribute(attributes || { });
+ else if (Object.isString(wrapper)) wrapper = new Element(wrapper, attributes);
+ else wrapper = new Element('div', wrapper);
+ if (element.parentNode)
+ element.parentNode.replaceChild(wrapper, element);
+ wrapper.appendChild(element);
+ return wrapper;
+ },
+
+ inspect: function(element) {
+ element = $(element);
+ var result = '<' + element.tagName.toLowerCase();
+ $H({'id': 'id', 'className': 'class'}).each(function(pair) {
+ var property = pair.first(), attribute = pair.last();
+ var value = (element[property] || '').toString();
+ if (value) result += ' ' + attribute + '=' + value.inspect(true);
+ });
+ return result + '>';
+ },
+
+ recursivelyCollect: function(element, property) {
+ element = $(element);
+ var elements = [];
+ while (element = element[property])
+ if (element.nodeType == 1)
+ elements.push(Element.extend(element));
+ return elements;
+ },
+
+ ancestors: function(element) {
+ return $(element).recursivelyCollect('parentNode');
+ },
+
+ descendants: function(element) {
+ return $A($(element).getElementsByTagName('*')).each(Element.extend);
+ },
+
+ firstDescendant: function(element) {
+ element = $(element).firstChild;
+ while (element && element.nodeType != 1) element = element.nextSibling;
+ return $(element);
+ },
+
+ immediateDescendants: function(element) {
+ if (!(element = $(element).firstChild)) return [];
+ while (element && element.nodeType != 1) element = element.nextSibling;
+ if (element) return [element].concat($(element).nextSiblings());
+ return [];
+ },
+
+ previousSiblings: function(element) {
+ return $(element).recursivelyCollect('previousSibling');
+ },
+
+ nextSiblings: function(element) {
+ return $(element).recursivelyCollect('nextSibling');
+ },
+
+ siblings: function(element) {
+ element = $(element);
+ return element.previousSiblings().reverse().concat(element.nextSiblings());
+ },
+
+ match: function(element, selector) {
+ if (Object.isString(selector))
+ selector = new Selector(selector);
+ return selector.match($(element));
+ },
+
+ up: function(element, expression, index) {
+ element = $(element);
+ if (arguments.length == 1) return $(element.parentNode);
+ var ancestors = element.ancestors();
+ return expression ? Selector.findElement(ancestors, expression, index) :
+ ancestors[index || 0];
+ },
+
+ down: function(element, expression, index) {
+ element = $(element);
+ if (arguments.length == 1) return element.firstDescendant();
+ var descendants = element.descendants();
+ return expression ? Selector.findElement(descendants, expression, index) :
+ descendants[index || 0];
+ },
+
+ previous: function(element, expression, index) {
+ element = $(element);
+ if (arguments.length == 1) return $(Selector.handlers.previousElementSibling(element));
+ var previousSiblings = element.previousSiblings();
+ return expression ? Selector.findElement(previousSiblings, expression, index) :
+ previousSiblings[index || 0];
+ },
+
+ next: function(element, expression, index) {
+ element = $(element);
+ if (arguments.length == 1) return $(Selector.handlers.nextElementSibling(element));
+ var nextSiblings = element.nextSiblings();
+ return expression ? Selector.findElement(nextSiblings, expression, index) :
+ nextSiblings[index || 0];
+ },
+
+ select: function() {
+ var args = $A(arguments), element = $(args.shift());
+ return Selector.findChildElements(element, args);
+ },
+
+ adjacent: function() {
+ var args = $A(arguments), element = $(args.shift());
+ return Selector.findChildElements(element.parentNode, args).without(element);
+ },
+
+ identify: function(element) {
+ element = $(element);
+ var id = element.readAttribute('id'), self = arguments.callee;
+ if (id) return id;
+ do { id = 'anonymous_element_' + self.counter++ } while ($(id));
+ element.writeAttribute('id', id);
+ return id;
+ },
+
+ readAttribute: function(element, name) {
+ element = $(element);
+ if (Prototype.Browser.IE) {
+ var t = Element._attributeTranslations.read;
+ if (t.values[name]) return t.values[name](element, name);
+ if (t.names[name]) name = t.names[name];
+ if (name.include(':')) {
+ return (!element.attributes || !element.attributes[name]) ? null :
+ element.attributes[name].value;
+ }
+ }
+ return element.getAttribute(name);
+ },
+
+ writeAttribute: function(element, name, value) {
+ element = $(element);
+ var attributes = { }, t = Element._attributeTranslations.write;
+
+ if (typeof name == 'object') attributes = name;
+ else attributes[name] = value === undefined ? true : value;
+
+ for (var attr in attributes) {
+ var name = t.names[attr] || attr, value = attributes[attr];
+ if (t.values[attr]) name = t.values[attr](element, value);
+ if (value === false || value === null)
+ element.removeAttribute(name);
+ else if (value === true)
+ element.setAttribute(name, name);
+ else element.setAttribute(name, value);
+ }
+ return element;
+ },
+
+ getHeight: function(element) {
+ return $(element).getDimensions().height;
+ },
+
+ getWidth: function(element) {
+ return $(element).getDimensions().width;
+ },
+
+ classNames: function(element) {
+ return new Element.ClassNames(element);
+ },
+
+ hasClassName: function(element, className) {
+ if (!(element = $(element))) return;
+ var elementClassName = element.className;
+ return (elementClassName.length > 0 && (elementClassName == className ||
+ new RegExp("(^|\\s)" + className + "(\\s|$)").test(elementClassName)));
+ },
+
+ addClassName: function(element, className) {
+ if (!(element = $(element))) return;
+ if (!element.hasClassName(className))
+ element.className += (element.className ? ' ' : '') + className;
+ return element;
+ },
+
+ removeClassName: function(element, className) {
+ if (!(element = $(element))) return;
+ element.className = element.className.replace(
+ new RegExp("(^|\\s+)" + className + "(\\s+|$)"), ' ').strip();
+ return element;
+ },
+
+ toggleClassName: function(element, className) {
+ if (!(element = $(element))) return;
+ return element[element.hasClassName(className) ?
+ 'removeClassName' : 'addClassName'](className);
+ },
+
+ // removes whitespace-only text node children
+ cleanWhitespace: function(element) {
+ element = $(element);
+ var node = element.firstChild;
+ while (node) {
+ var nextNode = node.nextSibling;
+ if (node.nodeType == 3 && !/\S/.test(node.nodeValue))
+ element.removeChild(node);
+ node = nextNode;
+ }
+ return element;
+ },
+
+ empty: function(element) {
+ return $(element).innerHTML.blank();
+ },
+
+ descendantOf: function(element, ancestor) {
+ element = $(element), ancestor = $(ancestor);
+
+ if (element.compareDocumentPosition)
+ return (element.compareDocumentPosition(ancestor) & 8) === 8;
+
+ if (element.sourceIndex && !Prototype.Browser.Opera) {
+ var e = element.sourceIndex, a = ancestor.sourceIndex,
+ nextAncestor = ancestor.nextSibling;
+ if (!nextAncestor) {
+ do { ancestor = ancestor.parentNode; }
+ while (!(nextAncestor = ancestor.nextSibling) && ancestor.parentNode);
+ }
+ if (nextAncestor) return (e > a && e < nextAncestor.sourceIndex);
+ }
+
+ while (element = element.parentNode)
+ if (element == ancestor) return true;
+ return false;
+ },
+
+ scrollTo: function(element) {
+ element = $(element);
+ var pos = element.cumulativeOffset();
+ window.scrollTo(pos[0], pos[1]);
+ return element;
+ },
+
+ getStyle: function(element, style) {
+ element = $(element);
+ style = style == 'float' ? 'cssFloat' : style.camelize();
+ var value = element.style[style];
+ if (!value) {
+ var css = document.defaultView.getComputedStyle(element, null);
+ value = css ? css[style] : null;
+ }
+ if (style == 'opacity') return value ? parseFloat(value) : 1.0;
+ return value == 'auto' ? null : value;
+ },
+
+ getOpacity: function(element) {
+ return $(element).getStyle('opacity');
+ },
+
+ setStyle: function(element, styles) {
+ element = $(element);
+ var elementStyle = element.style, match;
+ if (Object.isString(styles)) {
+ element.style.cssText += ';' + styles;
+ return styles.include('opacity') ?
+ element.setOpacity(styles.match(/opacity:\s*(\d?\.?\d*)/)[1]) : element;
+ }
+ for (var property in styles)
+ if (property == 'opacity') element.setOpacity(styles[property]);
+ else
+ elementStyle[(property == 'float' || property == 'cssFloat') ?
+ (elementStyle.styleFloat === undefined ? 'cssFloat' : 'styleFloat') :
+ property] = styles[property];
+
+ return element;
+ },
+
+ setOpacity: function(element, value) {
+ element = $(element);
+ element.style.opacity = (value == 1 || value === '') ? '' :
+ (value < 0.00001) ? 0 : value;
+ return element;
+ },
+
+ getDimensions: function(element) {
+ element = $(element);
+ var display = $(element).getStyle('display');
+ if (display != 'none' && display != null) // Safari bug
+ return {width: element.offsetWidth, height: element.offsetHeight};
+
+ // All *Width and *Height properties give 0 on elements with display none,
+ // so enable the element temporarily
+ var els = element.style;
+ var originalVisibility = els.visibility;
+ var originalPosition = els.position;
+ var originalDisplay = els.display;
+ els.visibility = 'hidden';
+ els.position = 'absolute';
+ els.display = 'block';
+ var originalWidth = element.clientWidth;
+ var originalHeight = element.clientHeight;
+ els.display = originalDisplay;
+ els.position = originalPosition;
+ els.visibility = originalVisibility;
+ return {width: originalWidth, height: originalHeight};
+ },
+
+ makePositioned: function(element) {
+ element = $(element);
+ var pos = Element.getStyle(element, 'position');
+ if (pos == 'static' || !pos) {
+ element._madePositioned = true;
+ element.style.position = 'relative';
+ // Opera returns the offset relative to the positioning context, when an
+ // element is position relative but top and left have not been defined
+ if (window.opera) {
+ element.style.top = 0;
+ element.style.left = 0;
+ }
+ }
+ return element;
+ },
+
+ undoPositioned: function(element) {
+ element = $(element);
+ if (element._madePositioned) {
+ element._madePositioned = undefined;
+ element.style.position =
+ element.style.top =
+ element.style.left =
+ element.style.bottom =
+ element.style.right = '';
+ }
+ return element;
+ },
+
+ makeClipping: function(element) {
+ element = $(element);
+ if (element._overflow) return element;
+ element._overflow = Element.getStyle(element, 'overflow') || 'auto';
+ if (element._overflow !== 'hidden')
+ element.style.overflow = 'hidden';
+ return element;
+ },
+
+ undoClipping: function(element) {
+ element = $(element);
+ if (!element._overflow) return element;
+ element.style.overflow = element._overflow == 'auto' ? '' : element._overflow;
+ element._overflow = null;
+ return element;
+ },
+
+ cumulativeOffset: function(element) {
+ var valueT = 0, valueL = 0;
+ do {
+ valueT += element.offsetTop || 0;
+ valueL += element.offsetLeft || 0;
+ element = element.offsetParent;
+ } while (element);
+ return Element._returnOffset(valueL, valueT);
+ },
+
+ positionedOffset: function(element) {
+ var valueT = 0, valueL = 0;
+ do {
+ valueT += element.offsetTop || 0;
+ valueL += element.offsetLeft || 0;
+ element = element.offsetParent;
+ if (element) {
+ if (element.tagName == 'BODY') break;
+ var p = Element.getStyle(element, 'position');
+ if (p == 'relative' || p == 'absolute') break;
+ }
+ } while (element);
+ return Element._returnOffset(valueL, valueT);
+ },
+
+ absolutize: function(element) {
+ element = $(element);
+ if (element.getStyle('position') == 'absolute') return;
+ // Position.prepare(); // To be done manually by Scripty when it needs it.
+
+ var offsets = element.positionedOffset();
+ var top = offsets[1];
+ var left = offsets[0];
+ var width = element.clientWidth;
+ var height = element.clientHeight;
+
+ element._originalLeft = left - parseFloat(element.style.left || 0);
+ element._originalTop = top - parseFloat(element.style.top || 0);
+ element._originalWidth = element.style.width;
+ element._originalHeight = element.style.height;
+
+ element.style.position = 'absolute';
+ element.style.top = top + 'px';
+ element.style.left = left + 'px';
+ element.style.width = width + 'px';
+ element.style.height = height + 'px';
+ return element;
+ },
+
+ relativize: function(element) {
+ element = $(element);
+ if (element.getStyle('position') == 'relative') return;
+ // Position.prepare(); // To be done manually by Scripty when it needs it.
+
+ element.style.position = 'relative';
+ var top = parseFloat(element.style.top || 0) - (element._originalTop || 0);
+ var left = parseFloat(element.style.left || 0) - (element._originalLeft || 0);
+
+ element.style.top = top + 'px';
+ element.style.left = left + 'px';
+ element.style.height = element._originalHeight;
+ element.style.width = element._originalWidth;
+ return element;
+ },
+
+ cumulativeScrollOffset: function(element) {
+ var valueT = 0, valueL = 0;
+ do {
+ valueT += element.scrollTop || 0;
+ valueL += element.scrollLeft || 0;
+ element = element.parentNode;
+ } while (element);
+ return Element._returnOffset(valueL, valueT);
+ },
+
+ getOffsetParent: function(element) {
+ if (element.offsetParent) return $(element.offsetParent);
+ if (element == document.body) return $(element);
+
+ while ((element = element.parentNode) && element != document.body)
+ if (Element.getStyle(element, 'position') != 'static')
+ return $(element);
+
+ return $(document.body);
+ },
+
+ viewportOffset: function(forElement) {
+ var valueT = 0, valueL = 0;
+
+ var element = forElement;
+ do {
+ valueT += element.offsetTop || 0;
+ valueL += element.offsetLeft || 0;
+
+ // Safari fix
+ if (element.offsetParent == document.body &&
+ Element.getStyle(element, 'position') == 'absolute') break;
+
+ } while (element = element.offsetParent);
+
+ element = forElement;
+ do {
+ if (!Prototype.Browser.Opera || element.tagName == 'BODY') {
+ valueT -= element.scrollTop || 0;
+ valueL -= element.scrollLeft || 0;
+ }
+ } while (element = element.parentNode);
+
+ return Element._returnOffset(valueL, valueT);
+ },
+
+ clonePosition: function(element, source) {
+ var options = Object.extend({
+ setLeft: true,
+ setTop: true,
+ setWidth: true,
+ setHeight: true,
+ offsetTop: 0,
+ offsetLeft: 0
+ }, arguments[2] || { });
+
+ // find page position of source
+ source = $(source);
+ var p = source.viewportOffset();
+
+ // find coordinate system to use
+ element = $(element);
+ var delta = [0, 0];
+ var parent = null;
+ // delta [0,0] will do fine with position: fixed elements,
+ // position:absolute needs offsetParent deltas
+ if (Element.getStyle(element, 'position') == 'absolute') {
+ parent = element.getOffsetParent();
+ delta = parent.viewportOffset();
+ }
+
+ // correct by body offsets (fixes Safari)
+ if (parent == document.body) {
+ delta[0] -= document.body.offsetLeft;
+ delta[1] -= document.body.offsetTop;
+ }
+
+ // set position
+ if (options.setLeft) element.style.left = (p[0] - delta[0] + options.offsetLeft) + 'px';
+ if (options.setTop) element.style.top = (p[1] - delta[1] + options.offsetTop) + 'px';
+ if (options.setWidth) element.style.width = source.offsetWidth + 'px';
+ if (options.setHeight) element.style.height = source.offsetHeight + 'px';
+ return element;
+ }
+};
+
+Element.Methods.identify.counter = 1;
+
+Object.extend(Element.Methods, {
+ getElementsBySelector: Element.Methods.select,
+ childElements: Element.Methods.immediateDescendants
+});
+
+Element._attributeTranslations = {
+ write: {
+ names: {
+ className: 'class',
+ htmlFor: 'for'
+ },
+ values: { }
+ }
+};
+
+
+if (!document.createRange || Prototype.Browser.Opera) {
+ Element.Methods.insert = function(element, insertions) {
+ element = $(element);
+
+ if (Object.isString(insertions) || Object.isNumber(insertions) ||
+ Object.isElement(insertions) || (insertions && (insertions.toElement || insertions.toHTML)))
+ insertions = { bottom: insertions };
+
+ var t = Element._insertionTranslations, content, position, pos, tagName;
+
+ for (position in insertions) {
+ content = insertions[position];
+ position = position.toLowerCase();
+ pos = t[position];
+
+ if (content && content.toElement) content = content.toElement();
+ if (Object.isElement(content)) {
+ pos.insert(element, content);
+ continue;
+ }
+
+ content = Object.toHTML(content);
+ tagName = ((position == 'before' || position == 'after')
+ ? element.parentNode : element).tagName.toUpperCase();
+
+ if (t.tags[tagName]) {
+ var fragments = Element._getContentFromAnonymousElement(tagName, content.stripScripts());
+ if (position == 'top' || position == 'after') fragments.reverse();
+ fragments.each(pos.insert.curry(element));
+ }
+ else element.insertAdjacentHTML(pos.adjacency, content.stripScripts());
+
+ content.evalScripts.bind(content).defer();
+ }
+
+ return element;
+ };
+}
+
+if (Prototype.Browser.Opera) {
+ Element.Methods._getStyle = Element.Methods.getStyle;
+ Element.Methods.getStyle = function(element, style) {
+ switch(style) {
+ case 'left':
+ case 'top':
+ case 'right':
+ case 'bottom':
+ if (Element._getStyle(element, 'position') == 'static') return null;
+ default: return Element._getStyle(element, style);
+ }
+ };
+ Element.Methods._readAttribute = Element.Methods.readAttribute;
+ Element.Methods.readAttribute = function(element, attribute) {
+ if (attribute == 'title') return element.title;
+ return Element._readAttribute(element, attribute);
+ };
+}
+
+else if (Prototype.Browser.IE) {
+ $w('positionedOffset getOffsetParent viewportOffset').each(function(method) {
+ Element.Methods[method] = Element.Methods[method].wrap(
+ function(proceed, element) {
+ element = $(element);
+ var position = element.getStyle('position');
+ if (position != 'static') return proceed(element);
+ element.setStyle({ position: 'relative' });
+ var value = proceed(element);
+ element.setStyle({ position: position });
+ return value;
+ }
+ );
+ });
+
+ Element.Methods.getStyle = function(element, style) {
+ element = $(element);
+ style = (style == 'float' || style == 'cssFloat') ? 'styleFloat' : style.camelize();
+ var value = element.style[style];
+ if (!value && element.currentStyle) value = element.currentStyle[style];
+
+ if (style == 'opacity') {
+ if (value = (element.getStyle('filter') || '').match(/alpha\(opacity=(.*)\)/))
+ if (value[1]) return parseFloat(value[1]) / 100;
+ return 1.0;
+ }
+
+ if (value == 'auto') {
+ if ((style == 'width' || style == 'height') && (element.getStyle('display') != 'none'))
+ return element['offset' + style.capitalize()] + 'px';
+ return null;
+ }
+ return value;
+ };
+
+ Element.Methods.setOpacity = function(element, value) {
+ function stripAlpha(filter){
+ return filter.replace(/alpha\([^\)]*\)/gi,'');
+ }
+ element = $(element);
+ var currentStyle = element.currentStyle;
+ if ((currentStyle && !currentStyle.hasLayout) ||
+ (!currentStyle && element.style.zoom == 'normal'))
+ element.style.zoom = 1;
+
+ var filter = element.getStyle('filter'), style = element.style;
+ if (value == 1 || value === '') {
+ (filter = stripAlpha(filter)) ?
+ style.filter = filter : style.removeAttribute('filter');
+ return element;
+ } else if (value < 0.00001) value = 0;
+ style.filter = stripAlpha(filter) +
+ 'alpha(opacity=' + (value * 100) + ')';
+ return element;
+ };
+
+ Element._attributeTranslations = {
+ read: {
+ names: {
+ 'class': 'className',
+ 'for': 'htmlFor'
+ },
+ values: {
+ _getAttr: function(element, attribute) {
+ return element.getAttribute(attribute, 2);
+ },
+ _getAttrNode: function(element, attribute) {
+ var node = element.getAttributeNode(attribute);
+ return node ? node.value : "";
+ },
+ _getEv: function(element, attribute) {
+ var attribute = element.getAttribute(attribute);
+ return attribute ? attribute.toString().slice(23, -2) : null;
+ },
+ _flag: function(element, attribute) {
+ return $(element).hasAttribute(attribute) ? attribute : null;
+ },
+ style: function(element) {
+ return element.style.cssText.toLowerCase();
+ },
+ title: function(element) {
+ return element.title;
+ }
+ }
+ }
+ };
+
+ Element._attributeTranslations.write = {
+ names: Object.clone(Element._attributeTranslations.read.names),
+ values: {
+ checked: function(element, value) {
+ element.checked = !!value;
+ },
+
+ style: function(element, value) {
+ element.style.cssText = value ? value : '';
+ }
+ }
+ };
+
+ Element._attributeTranslations.has = {};
+
+ $w('colSpan rowSpan vAlign dateTime accessKey tabIndex ' +
+ 'encType maxLength readOnly longDesc').each(function(attr) {
+ Element._attributeTranslations.write.names[attr.toLowerCase()] = attr;
+ Element._attributeTranslations.has[attr.toLowerCase()] = attr;
+ });
+
+ (function(v) {
+ Object.extend(v, {
+ href: v._getAttr,
+ src: v._getAttr,
+ type: v._getAttr,
+ action: v._getAttrNode,
+ disabled: v._flag,
+ checked: v._flag,
+ readonly: v._flag,
+ multiple: v._flag,
+ onload: v._getEv,
+ onunload: v._getEv,
+ onclick: v._getEv,
+ ondblclick: v._getEv,
+ onmousedown: v._getEv,
+ onmouseup: v._getEv,
+ onmouseover: v._getEv,
+ onmousemove: v._getEv,
+ onmouseout: v._getEv,
+ onfocus: v._getEv,
+ onblur: v._getEv,
+ onkeypress: v._getEv,
+ onkeydown: v._getEv,
+ onkeyup: v._getEv,
+ onsubmit: v._getEv,
+ onreset: v._getEv,
+ onselect: v._getEv,
+ onchange: v._getEv
+ });
+ })(Element._attributeTranslations.read.values);
+}
+
+else if (Prototype.Browser.Gecko && /rv:1\.8\.0/.test(navigator.userAgent)) {
+ Element.Methods.setOpacity = function(element, value) {
+ element = $(element);
+ element.style.opacity = (value == 1) ? 0.999999 :
+ (value === '') ? '' : (value < 0.00001) ? 0 : value;
+ return element;
+ };
+}
+
+else if (Prototype.Browser.WebKit) {
+ Element.Methods.setOpacity = function(element, value) {
+ element = $(element);
+ element.style.opacity = (value == 1 || value === '') ? '' :
+ (value < 0.00001) ? 0 : value;
+
+ if (value == 1)
+ if(element.tagName == 'IMG' && element.width) {
+ element.width++; element.width--;
+ } else try {
+ var n = document.createTextNode(' ');
+ element.appendChild(n);
+ element.removeChild(n);
+ } catch (e) { }
+
+ return element;
+ };
+
+ // Safari returns margins on body which is incorrect if the child is absolutely
+ // positioned. For performance reasons, redefine Position.cumulativeOffset for
+ // KHTML/WebKit only.
+ Element.Methods.cumulativeOffset = function(element) {
+ var valueT = 0, valueL = 0;
+ do {
+ valueT += element.offsetTop || 0;
+ valueL += element.offsetLeft || 0;
+ if (element.offsetParent == document.body)
+ if (Element.getStyle(element, 'position') == 'absolute') break;
+
+ element = element.offsetParent;
+ } while (element);
+
+ return Element._returnOffset(valueL, valueT);
+ };
+}
+
+if (Prototype.Browser.IE || Prototype.Browser.Opera) {
+ // IE and Opera are missing .innerHTML support for TABLE-related and SELECT elements
+ Element.Methods.update = function(element, content) {
+ element = $(element);
+
+ if (content && content.toElement) content = content.toElement();
+ if (Object.isElement(content)) return element.update().insert(content);
+
+ content = Object.toHTML(content);
+ var tagName = element.tagName.toUpperCase();
+
+ if (tagName in Element._insertionTranslations.tags) {
+ $A(element.childNodes).each(function(node) { element.removeChild(node) });
+ Element._getContentFromAnonymousElement(tagName, content.stripScripts())
+ .each(function(node) { element.appendChild(node) });
+ }
+ else element.innerHTML = content.stripScripts();
+
+ content.evalScripts.bind(content).defer();
+ return element;
+ };
+}
+
+if (document.createElement('div').outerHTML) {
+ Element.Methods.replace = function(element, content) {
+ element = $(element);
+
+ if (content && content.toElement) content = content.toElement();
+ if (Object.isElement(content)) {
+ element.parentNode.replaceChild(content, element);
+ return element;
+ }
+
+ content = Object.toHTML(content);
+ var parent = element.parentNode, tagName = parent.tagName.toUpperCase();
+
+ if (Element._insertionTranslations.tags[tagName]) {
+ var nextSibling = element.next();
+ var fragments = Element._getContentFromAnonymousElement(tagName, content.stripScripts());
+ parent.removeChild(element);
+ if (nextSibling)
+ fragments.each(function(node) { parent.insertBefore(node, nextSibling) });
+ else
+ fragments.each(function(node) { parent.appendChild(node) });
+ }
+ else element.outerHTML = content.stripScripts();
+
+ content.evalScripts.bind(content).defer();
+ return element;
+ };
+}
+
+Element._returnOffset = function(l, t) {
+ var result = [l, t];
+ result.left = l;
+ result.top = t;
+ return result;
+};
+
+Element._getContentFromAnonymousElement = function(tagName, html) {
+ var div = new Element('div'), t = Element._insertionTranslations.tags[tagName];
+ div.innerHTML = t[0] + html + t[1];
+ t[2].times(function() { div = div.firstChild });
+ return $A(div.childNodes);
+};
+
+Element._insertionTranslations = {
+ before: {
+ adjacency: 'beforeBegin',
+ insert: function(element, node) {
+ element.parentNode.insertBefore(node, element);
+ },
+ initializeRange: function(element, range) {
+ range.setStartBefore(element);
+ }
+ },
+ top: {
+ adjacency: 'afterBegin',
+ insert: function(element, node) {
+ element.insertBefore(node, element.firstChild);
+ },
+ initializeRange: function(element, range) {
+ range.selectNodeContents(element);
+ range.collapse(true);
+ }
+ },
+ bottom: {
+ adjacency: 'beforeEnd',
+ insert: function(element, node) {
+ element.appendChild(node);
+ }
+ },
+ after: {
+ adjacency: 'afterEnd',
+ insert: function(element, node) {
+ element.parentNode.insertBefore(node, element.nextSibling);
+ },
+ initializeRange: function(element, range) {
+ range.setStartAfter(element);
+ }
+ },
+ tags: {
+ TABLE: ['<table>', '</table>', 1],
+ TBODY: ['<table><tbody>', '</tbody></table>', 2],
+ TR: ['<table><tbody><tr>', '</tr></tbody></table>', 3],
+ TD: ['<table><tbody><tr><td>', '</td></tr></tbody></table>', 4],
+ SELECT: ['<select>', '</select>', 1]
+ }
+};
+
+(function() {
+ this.bottom.initializeRange = this.top.initializeRange;
+ Object.extend(this.tags, {
+ THEAD: this.tags.TBODY,
+ TFOOT: this.tags.TBODY,
+ TH: this.tags.TD
+ });
+}).call(Element._insertionTranslations);
+
+Element.Methods.Simulated = {
+ hasAttribute: function(element, attribute) {
+ attribute = Element._attributeTranslations.has[attribute] || attribute;
+ var node = $(element).getAttributeNode(attribute);
+ return node && node.specified;
+ }
+};
+
+Element.Methods.ByTag = { };
+
+Object.extend(Element, Element.Methods);
+
+if (!Prototype.BrowserFeatures.ElementExtensions &&
+ document.createElement('div').__proto__) {
+ window.HTMLElement = { };
+ window.HTMLElement.prototype = document.createElement('div').__proto__;
+ Prototype.BrowserFeatures.ElementExtensions = true;
+}
+
+Element.extend = (function() {
+ if (Prototype.BrowserFeatures.SpecificElementExtensions)
+ return Prototype.K;
+
+ var Methods = { }, ByTag = Element.Methods.ByTag;
+
+ var extend = Object.extend(function(element) {
+ if (!element || element._extendedByPrototype ||
+ element.nodeType != 1 || element == window) return element;
+
+ var methods = Object.clone(Methods),
+ tagName = element.tagName, property, value;
+
+ // extend methods for specific tags
+ if (ByTag[tagName]) Object.extend(methods, ByTag[tagName]);
+
+ for (property in methods) {
+ value = methods[property];
+ if (Object.isFunction(value) && !(property in element))
+ element[property] = value.methodize();
+ }
+
+ element._extendedByPrototype = Prototype.emptyFunction;
+ return element;
+
+ }, {
+ refresh: function() {
+ // extend methods for all tags (Safari doesn't need this)
+ if (!Prototype.BrowserFeatures.ElementExtensions) {
+ Object.extend(Methods, Element.Methods);
+ Object.extend(Methods, Element.Methods.Simulated);
+ }
+ }
+ });
+
+ extend.refresh();
+ return extend;
+})();
+
+Element.hasAttribute = function(element, attribute) {
+ if (element.hasAttribute) return element.hasAttribute(attribute);
+ return Element.Methods.Simulated.hasAttribute(element, attribute);
+};
+
+Element.addMethods = function(methods) {
+ var F = Prototype.BrowserFeatures, T = Element.Methods.ByTag;
+
+ if (!methods) {
+ Object.extend(Form, Form.Methods);
+ Object.extend(Form.Element, Form.Element.Methods);
+ Object.extend(Element.Methods.ByTag, {
+ "FORM": Object.clone(Form.Methods),
+ "INPUT": Object.clone(Form.Element.Methods),
+ "SELECT": Object.clone(Form.Element.Methods),
+ "TEXTAREA": Object.clone(Form.Element.Methods)
+ });
+ }
+
+ if (arguments.length == 2) {
+ var tagName = methods;
+ methods = arguments[1];
+ }
+
+ if (!tagName) Object.extend(Element.Methods, methods || { });
+ else {
+ if (Object.isArray(tagName)) tagName.each(extend);
+ else extend(tagName);
+ }
+
+ function extend(tagName) {
+ tagName = tagName.toUpperCase();
+ if (!Element.Methods.ByTag[tagName])
+ Element.Methods.ByTag[tagName] = { };
+ Object.extend(Element.Methods.ByTag[tagName], methods);
+ }
+
+ function copy(methods, destination, onlyIfAbsent) {
+ onlyIfAbsent = onlyIfAbsent || false;
+ for (var property in methods) {
+ var value = methods[property];
+ if (!Object.isFunction(value)) continue;
+ if (!onlyIfAbsent || !(property in destination))
+ destination[property] = value.methodize();
+ }
+ }
+
+ function findDOMClass(tagName) {
+ var klass;
+ var trans = {
+ "OPTGROUP": "OptGroup", "TEXTAREA": "TextArea", "P": "Paragraph",
+ "FIELDSET": "FieldSet", "UL": "UList", "OL": "OList", "DL": "DList",
+ "DIR": "Directory", "H1": "Heading", "H2": "Heading", "H3": "Heading",
+ "H4": "Heading", "H5": "Heading", "H6": "Heading", "Q": "Quote",
+ "INS": "Mod", "DEL": "Mod", "A": "Anchor", "IMG": "Image", "CAPTION":
+ "TableCaption", "COL": "TableCol", "COLGROUP": "TableCol", "THEAD":
+ "TableSection", "TFOOT": "TableSection", "TBODY": "TableSection", "TR":
+ "TableRow", "TH": "TableCell", "TD": "TableCell", "FRAMESET":
+ "FrameSet", "IFRAME": "IFrame"
+ };
+ if (trans[tagName]) klass = 'HTML' + trans[tagName] + 'Element';
+ if (window[klass]) return window[klass];
+ klass = 'HTML' + tagName + 'Element';
+ if (window[klass]) return window[klass];
+ klass = 'HTML' + tagName.capitalize() + 'Element';
+ if (window[klass]) return window[klass];
+
+ window[klass] = { };
+ window[klass].prototype = document.createElement(tagName).__proto__;
+ return window[klass];
+ }
+
+ if (F.ElementExtensions) {
+ copy(Element.Methods, HTMLElement.prototype);
+ copy(Element.Methods.Simulated, HTMLElement.prototype, true);
+ }
+
+ if (F.SpecificElementExtensions) {
+ for (var tag in Element.Methods.ByTag) {
+ var klass = findDOMClass(tag);
+ if (Object.isUndefined(klass)) continue;
+ copy(T[tag], klass.prototype);
+ }
+ }
+
+ Object.extend(Element, Element.Methods);
+ delete Element.ByTag;
+
+ if (Element.extend.refresh) Element.extend.refresh();
+ Element.cache = { };
+};
+
+document.viewport = {
+ getDimensions: function() {
+ var dimensions = { };
+ $w('width height').each(function(d) {
+ var D = d.capitalize();
+ dimensions[d] = self['inner' + D] ||
+ (document.documentElement['client' + D] || document.body['client' + D]);
+ });
+ return dimensions;
+ },
+
+ getWidth: function() {
+ return this.getDimensions().width;
+ },
+
+ getHeight: function() {
+ return this.getDimensions().height;
+ },
+
+ getScrollOffsets: function() {
+ return Element._returnOffset(
+ window.pageXOffset || document.documentElement.scrollLeft || document.body.scrollLeft,
+ window.pageYOffset || document.documentElement.scrollTop || document.body.scrollTop);
+ }
+};
+/* Portions of the Selector class are derived from Jack Slocum’s DomQuery,
+ * part of YUI-Ext version 0.40, distributed under the terms of an MIT-style
+ * license. Please see http://www.yui-ext.com/ for more information. */
+
+var Selector = Class.create({
+ initialize: function(expression) {
+ this.expression = expression.strip();
+ this.compileMatcher();
+ },
+
+ compileMatcher: function() {
+ // Selectors with namespaced attributes can't use the XPath version
+ if (Prototype.BrowserFeatures.XPath && !(/(\[[\w-]*?:|:checked)/).test(this.expression))
+ return this.compileXPathMatcher();
+
+ var e = this.expression, ps = Selector.patterns, h = Selector.handlers,
+ c = Selector.criteria, le, p, m;
+
+ if (Selector._cache[e]) {
+ this.matcher = Selector._cache[e];
+ return;
+ }
+
+ this.matcher = ["this.matcher = function(root) {",
+ "var r = root, h = Selector.handlers, c = false, n;"];
+
+ while (e && le != e && (/\S/).test(e)) {
+ le = e;
+ for (var i in ps) {
+ p = ps[i];
+ if (m = e.match(p)) {
+ this.matcher.push(Object.isFunction(c[i]) ? c[i](m) :
+ new Template(c[i]).evaluate(m));
+ e = e.replace(m[0], '');
+ break;
+ }
+ }
+ }
+
+ this.matcher.push("return h.unique(n);\n}");
+ eval(this.matcher.join('\n'));
+ Selector._cache[this.expression] = this.matcher;
+ },
+
+ compileXPathMatcher: function() {
+ var e = this.expression, ps = Selector.patterns,
+ x = Selector.xpath, le, m;
+
+ if (Selector._cache[e]) {
+ this.xpath = Selector._cache[e]; return;
+ }
+
+ this.matcher = ['.//*'];
+ while (e && le != e && (/\S/).test(e)) {
+ le = e;
+ for (var i in ps) {
+ if (m = e.match(ps[i])) {
+ this.matcher.push(Object.isFunction(x[i]) ? x[i](m) :
+ new Template(x[i]).evaluate(m));
+ e = e.replace(m[0], '');
+ break;
+ }
+ }
+ }
+
+ this.xpath = this.matcher.join('');
+ Selector._cache[this.expression] = this.xpath;
+ },
+
+ findElements: function(root) {
+ root = root || document;
+ if (this.xpath) return document._getElementsByXPath(this.xpath, root);
+ return this.matcher(root);
+ },
+
+ match: function(element) {
+ this.tokens = [];
+
+ var e = this.expression, ps = Selector.patterns, as = Selector.assertions;
+ var le, p, m;
+
+ while (e && le !== e && (/\S/).test(e)) {
+ le = e;
+ for (var i in ps) {
+ p = ps[i];
+ if (m = e.match(p)) {
+ // use the Selector.assertions methods unless the selector
+ // is too complex.
+ if (as[i]) {
+ this.tokens.push([i, Object.clone(m)]);
+ e = e.replace(m[0], '');
+ } else {
+ // reluctantly do a document-wide search
+ // and look for a match in the array
+ return this.findElements(document).include(element);
+ }
+ }
+ }
+ }
+
+ var match = true, name, matches;
+ for (var i = 0, token; token = this.tokens[i]; i++) {
+ name = token[0], matches = token[1];
+ if (!Selector.assertions[name](element, matches)) {
+ match = false; break;
+ }
+ }
+
+ return match;
+ },
+
+ toString: function() {
+ return this.expression;
+ },
+
+ inspect: function() {
+ return "#<Selector:" + this.expression.inspect() + ">";
+ }
+});
+
+Object.extend(Selector, {
+ _cache: { },
+
+ xpath: {
+ descendant: "//*",
+ child: "/*",
+ adjacent: "/following-sibling::*[1]",
+ laterSibling: '/following-sibling::*',
+ tagName: function(m) {
+ if (m[1] == '*') return '';
+ return "[local-name()='" + m[1].toLowerCase() +
+ "' or local-name()='" + m[1].toUpperCase() + "']";
+ },
+ className: "[contains(concat(' ', @class, ' '), ' #{1} ')]",
+ id: "[@id='#{1}']",
+ attrPresence: "[@#{1}]",
+ attr: function(m) {
+ m[3] = m[5] || m[6];
+ return new Template(Selector.xpath.operators[m[2]]).evaluate(m);
+ },
+ pseudo: function(m) {
+ var h = Selector.xpath.pseudos[m[1]];
+ if (!h) return '';
+ if (Object.isFunction(h)) return h(m);
+ return new Template(Selector.xpath.pseudos[m[1]]).evaluate(m);
+ },
+ operators: {
+ '=': "[@#{1}='#{3}']",
+ '!=': "[@#{1}!='#{3}']",
+ '^=': "[starts-with(@#{1}, '#{3}')]",
+ '$=': "[substring(@#{1}, (string-length(@#{1}) - string-length('#{3}') + 1))='#{3}']",
+ '*=': "[contains(@#{1}, '#{3}')]",
+ '~=': "[contains(concat(' ', @#{1}, ' '), ' #{3} ')]",
+ '|=': "[contains(concat('-', @#{1}, '-'), '-#{3}-')]"
+ },
+ pseudos: {
+ 'first-child': '[not(preceding-sibling::*)]',
+ 'last-child': '[not(following-sibling::*)]',
+ 'only-child': '[not(preceding-sibling::* or following-sibling::*)]',
+ 'empty': "[count(*) = 0 and (count(text()) = 0 or translate(text(), ' \t\r\n', '') = '')]",
+ 'checked': "[@checked]",
+ 'disabled': "[@disabled]",
+ 'enabled': "[not(@disabled)]",
+ 'not': function(m) {
+ var e = m[6], p = Selector.patterns,
+ x = Selector.xpath, le, m, v;
+
+ var exclusion = [];
+ while (e && le != e && (/\S/).test(e)) {
+ le = e;
+ for (var i in p) {
+ if (m = e.match(p[i])) {
+ v = Object.isFunction(x[i]) ? x[i](m) : new Template(x[i]).evaluate(m);
+ exclusion.push("(" + v.substring(1, v.length - 1) + ")");
+ e = e.replace(m[0], '');
+ break;
+ }
+ }
+ }
+ return "[not(" + exclusion.join(" and ") + ")]";
+ },
+ 'nth-child': function(m) {
+ return Selector.xpath.pseudos.nth("(count(./preceding-sibling::*) + 1) ", m);
+ },
+ 'nth-last-child': function(m) {
+ return Selector.xpath.pseudos.nth("(count(./following-sibling::*) + 1) ", m);
+ },
+ 'nth-of-type': function(m) {
+ return Selector.xpath.pseudos.nth("position() ", m);
+ },
+ 'nth-last-of-type': function(m) {
+ return Selector.xpath.pseudos.nth("(last() + 1 - position()) ", m);
+ },
+ 'first-of-type': function(m) {
+ m[6] = "1"; return Selector.xpath.pseudos['nth-of-type'](m);
+ },
+ 'last-of-type': function(m) {
+ m[6] = "1"; return Selector.xpath.pseudos['nth-last-of-type'](m);
+ },
+ 'only-of-type': function(m) {
+ var p = Selector.xpath.pseudos; return p['first-of-type'](m) + p['last-of-type'](m);
+ },
+ nth: function(fragment, m) {
+ var mm, formula = m[6], predicate;
+ if (formula == 'even') formula = '2n+0';
+ if (formula == 'odd') formula = '2n+1';
+ if (mm = formula.match(/^(\d+)$/)) // digit only
+ return '[' + fragment + "= " + mm[1] + ']';
+ if (mm = formula.match(/^(-?\d*)?n(([+-])(\d+))?/)) { // an+b
+ if (mm[1] == "-") mm[1] = -1;
+ var a = mm[1] ? Number(mm[1]) : 1;
+ var b = mm[2] ? Number(mm[2]) : 0;
+ predicate = "[((#{fragment} - #{b}) mod #{a} = 0) and " +
+ "((#{fragment} - #{b}) div #{a} >= 0)]";
+ return new Template(predicate).evaluate({
+ fragment: fragment, a: a, b: b });
+ }
+ }
+ }
+ },
+
+ criteria: {
+ tagName: 'n = h.tagName(n, r, "#{1}", c); c = false;',
+ className: 'n = h.className(n, r, "#{1}", c); c = false;',
+ id: 'n = h.id(n, r, "#{1}", c); c = false;',
+ attrPresence: 'n = h.attrPresence(n, r, "#{1}"); c = false;',
+ attr: function(m) {
+ m[3] = (m[5] || m[6]);
+ return new Template('n = h.attr(n, r, "#{1}", "#{3}", "#{2}"); c = false;').evaluate(m);
+ },
+ pseudo: function(m) {
+ if (m[6]) m[6] = m[6].replace(/"/g, '\\"');
+ return new Template('n = h.pseudo(n, "#{1}", "#{6}", r, c); c = false;').evaluate(m);
+ },
+ descendant: 'c = "descendant";',
+ child: 'c = "child";',
+ adjacent: 'c = "adjacent";',
+ laterSibling: 'c = "laterSibling";'
+ },
+
+ patterns: {
+ // combinators must be listed first
+ // (and descendant needs to be last combinator)
+ laterSibling: /^\s*~\s*/,
+ child: /^\s*>\s*/,
+ adjacent: /^\s*\+\s*/,
+ descendant: /^\s/,
+
+ // selectors follow
+ tagName: /^\s*(\*|[\w\-]+)(\b|$)?/,
+ id: /^#([\w\-\*]+)(\b|$)/,
+ className: /^\.([\w\-\*]+)(\b|$)/,
+ pseudo: /^:((first|last|nth|nth-last|only)(-child|-of-type)|empty|checked|(en|dis)abled|not)(\((.*?)\))?(\b|$|(?=\s)|(?=:))/,
+ attrPresence: /^\[([\w]+)\]/,
+ attr: /\[((?:[\w-]*:)?[\w-]+)\s*(?:([!^$*~|]?=)\s*((['"])([^\4]*?)\4|([^'"][^\]]*?)))?\]/
+ },
+
+ // for Selector.match and Element#match
+ assertions: {
+ tagName: function(element, matches) {
+ return matches[1].toUpperCase() == element.tagName.toUpperCase();
+ },
+
+ className: function(element, matches) {
+ return Element.hasClassName(element, matches[1]);
+ },
+
+ id: function(element, matches) {
+ return element.id === matches[1];
+ },
+
+ attrPresence: function(element, matches) {
+ return Element.hasAttribute(element, matches[1]);
+ },
+
+ attr: function(element, matches) {
+ var nodeValue = Element.readAttribute(element, matches[1]);
+ return Selector.operators[matches[2]](nodeValue, matches[3]);
+ }
+ },
+
+ handlers: {
+ // UTILITY FUNCTIONS
+ // joins two collections
+ concat: function(a, b) {
+ for (var i = 0, node; node = b[i]; i++)
+ a.push(node);
+ return a;
+ },
+
+ // marks an array of nodes for counting
+ mark: function(nodes) {
+ for (var i = 0, node; node = nodes[i]; i++)
+ node._counted = true;
+ return nodes;
+ },
+
+ unmark: function(nodes) {
+ for (var i = 0, node; node = nodes[i]; i++)
+ node._counted = undefined;
+ return nodes;
+ },
+
+ // mark each child node with its position (for nth calls)
+ // "ofType" flag indicates whether we're indexing for nth-of-type
+ // rather than nth-child
+ index: function(parentNode, reverse, ofType) {
+ parentNode._counted = true;
+ if (reverse) {
+ for (var nodes = parentNode.childNodes, i = nodes.length - 1, j = 1; i >= 0; i--) {
+ var node = nodes[i];
+ if (node.nodeType == 1 && (!ofType || node._counted)) node.nodeIndex = j++;
+ }
+ } else {
+ for (var i = 0, j = 1, nodes = parentNode.childNodes; node = nodes[i]; i++)
+ if (node.nodeType == 1 && (!ofType || node._counted)) node.nodeIndex = j++;
+ }
+ },
+
+ // filters out duplicates and extends all nodes
+ unique: function(nodes) {
+ if (nodes.length == 0) return nodes;
+ var results = [], n;
+ for (var i = 0, l = nodes.length; i < l; i++)
+ if (!(n = nodes[i])._counted) {
+ n._counted = true;
+ results.push(Element.extend(n));
+ }
+ return Selector.handlers.unmark(results);
+ },
+
+ // COMBINATOR FUNCTIONS
+ descendant: function(nodes) {
+ var h = Selector.handlers;
+ for (var i = 0, results = [], node; node = nodes[i]; i++)
+ h.concat(results, node.getElementsByTagName('*'));
+ return results;
+ },
+
+ child: function(nodes) {
+ var h = Selector.handlers;
+ for (var i = 0, results = [], node; node = nodes[i]; i++) {
+ for (var j = 0, children = [], child; child = node.childNodes[j]; j++)
+ if (child.nodeType == 1 && child.tagName != '!') results.push(child);
+ }
+ return results;
+ },
+
+ adjacent: function(nodes) {
+ for (var i = 0, results = [], node; node = nodes[i]; i++) {
+ var next = this.nextElementSibling(node);
+ if (next) results.push(next);
+ }
+ return results;
+ },
+
+ laterSibling: function(nodes) {
+ var h = Selector.handlers;
+ for (var i = 0, results = [], node; node = nodes[i]; i++)
+ h.concat(results, Element.nextSiblings(node));
+ return results;
+ },
+
+ nextElementSibling: function(node) {
+ while (node = node.nextSibling)
+ if (node.nodeType == 1) return node;
+ return null;
+ },
+
+ previousElementSibling: function(node) {
+ while (node = node.previousSibling)
+ if (node.nodeType == 1) return node;
+ return null;
+ },
+
+ // TOKEN FUNCTIONS
+ tagName: function(nodes, root, tagName, combinator) {
+ tagName = tagName.toUpperCase();
+ var results = [], h = Selector.handlers;
+ if (nodes) {
+ if (combinator) {
+ // fastlane for ordinary descendant combinators
+ if (combinator == "descendant") {
+ for (var i = 0, node; node = nodes[i]; i++)
+ h.concat(results, node.getElementsByTagName(tagName));
+ return results;
+ } else nodes = this[combinator](nodes);
+ if (tagName == "*") return nodes;
+ }
+ for (var i = 0, node; node = nodes[i]; i++)
+ if (node.tagName.toUpperCase() == tagName) results.push(node);
+ return results;
+ } else return root.getElementsByTagName(tagName);
+ },
+
+ id: function(nodes, root, id, combinator) {
+ var targetNode = $(id), h = Selector.handlers;
+ if (!targetNode) return [];
+ if (!nodes && root == document) return [targetNode];
+ if (nodes) {
+ if (combinator) {
+ if (combinator == 'child') {
+ for (var i = 0, node; node = nodes[i]; i++)
+ if (targetNode.parentNode == node) return [targetNode];
+ } else if (combinator == 'descendant') {
+ for (var i = 0, node; node = nodes[i]; i++)
+ if (Element.descendantOf(targetNode, node)) return [targetNode];
+ } else if (combinator == 'adjacent') {
+ for (var i = 0, node; node = nodes[i]; i++)
+ if (Selector.handlers.previousElementSibling(targetNode) == node)
+ return [targetNode];
+ } else nodes = h[combinator](nodes);
+ }
+ for (var i = 0, node; node = nodes[i]; i++)
+ if (node == targetNode) return [targetNode];
+ return [];
+ }
+ return (targetNode && Element.descendantOf(targetNode, root)) ? [targetNode] : [];
+ },
+
+ className: function(nodes, root, className, combinator) {
+ if (nodes && combinator) nodes = this[combinator](nodes);
+ return Selector.handlers.byClassName(nodes, root, className);
+ },
+
+ byClassName: function(nodes, root, className) {
+ if (!nodes) nodes = Selector.handlers.descendant([root]);
+ var needle = ' ' + className + ' ';
+ for (var i = 0, results = [], node, nodeClassName; node = nodes[i]; i++) {
+ nodeClassName = node.className;
+ if (nodeClassName.length == 0) continue;
+ if (nodeClassName == className || (' ' + nodeClassName + ' ').include(needle))
+ results.push(node);
+ }
+ return results;
+ },
+
+ attrPresence: function(nodes, root, attr) {
+ if (!nodes) nodes = root.getElementsByTagName("*");
+ var results = [];
+ for (var i = 0, node; node = nodes[i]; i++)
+ if (Element.hasAttribute(node, attr)) results.push(node);
+ return results;
+ },
+
+ attr: function(nodes, root, attr, value, operator) {
+ if (!nodes) nodes = root.getElementsByTagName("*");
+ var handler = Selector.operators[operator], results = [];
+ for (var i = 0, node; node = nodes[i]; i++) {
+ var nodeValue = Element.readAttribute(node, attr);
+ if (nodeValue === null) continue;
+ if (handler(nodeValue, value)) results.push(node);
+ }
+ return results;
+ },
+
+ pseudo: function(nodes, name, value, root, combinator) {
+ if (nodes && combinator) nodes = this[combinator](nodes);
+ if (!nodes) nodes = root.getElementsByTagName("*");
+ return Selector.pseudos[name](nodes, value, root);
+ }
+ },
+
+ pseudos: {
+ 'first-child': function(nodes, value, root) {
+ for (var i = 0, results = [], node; node = nodes[i]; i++) {
+ if (Selector.handlers.previousElementSibling(node)) continue;
+ results.push(node);
+ }
+ return results;
+ },
+ 'last-child': function(nodes, value, root) {
+ for (var i = 0, results = [], node; node = nodes[i]; i++) {
+ if (Selector.handlers.nextElementSibling(node)) continue;
+ results.push(node);
+ }
+ return results;
+ },
+ 'only-child': function(nodes, value, root) {
+ var h = Selector.handlers;
+ for (var i = 0, results = [], node; node = nodes[i]; i++)
+ if (!h.previousElementSibling(node) && !h.nextElementSibling(node))
+ results.push(node);
+ return results;
+ },
+ 'nth-child': function(nodes, formula, root) {
+ return Selector.pseudos.nth(nodes, formula, root);
+ },
+ 'nth-last-child': function(nodes, formula, root) {
+ return Selector.pseudos.nth(nodes, formula, root, true);
+ },
+ 'nth-of-type': function(nodes, formula, root) {
+ return Selector.pseudos.nth(nodes, formula, root, false, true);
+ },
+ 'nth-last-of-type': function(nodes, formula, root) {
+ return Selector.pseudos.nth(nodes, formula, root, true, true);
+ },
+ 'first-of-type': function(nodes, formula, root) {
+ return Selector.pseudos.nth(nodes, "1", root, false, true);
+ },
+ 'last-of-type': function(nodes, formula, root) {
+ return Selector.pseudos.nth(nodes, "1", root, true, true);
+ },
+ 'only-of-type': function(nodes, formula, root) {
+ var p = Selector.pseudos;
+ return p['last-of-type'](p['first-of-type'](nodes, formula, root), formula, root);
+ },
+
+ // handles the an+b logic
+ getIndices: function(a, b, total) {
+ if (a == 0) return b > 0 ? [b] : [];
+ return $R(1, total).inject([], function(memo, i) {
+ if (0 == (i - b) % a && (i - b) / a >= 0) memo.push(i);
+ return memo;
+ });
+ },
+
+ // handles nth(-last)-child, nth(-last)-of-type, and (first|last)-of-type
+ nth: function(nodes, formula, root, reverse, ofType) {
+ if (nodes.length == 0) return [];
+ if (formula == 'even') formula = '2n+0';
+ if (formula == 'odd') formula = '2n+1';
+ var h = Selector.handlers, results = [], indexed = [], m;
+ h.mark(nodes);
+ for (var i = 0, node; node = nodes[i]; i++) {
+ if (!node.parentNode._counted) {
+ h.index(node.parentNode, reverse, ofType);
+ indexed.push(node.parentNode);
+ }
+ }
+ if (formula.match(/^\d+$/)) { // just a number
+ formula = Number(formula);
+ for (var i = 0, node; node = nodes[i]; i++)
+ if (node.nodeIndex == formula) results.push(node);
+ } else if (m = formula.match(/^(-?\d*)?n(([+-])(\d+))?/)) { // an+b
+ if (m[1] == "-") m[1] = -1;
+ var a = m[1] ? Number(m[1]) : 1;
+ var b = m[2] ? Number(m[2]) : 0;
+ var indices = Selector.pseudos.getIndices(a, b, nodes.length);
+ for (var i = 0, node, l = indices.length; node = nodes[i]; i++) {
+ for (var j = 0; j < l; j++)
+ if (node.nodeIndex == indices[j]) results.push(node);
+ }
+ }
+ h.unmark(nodes);
+ h.unmark(indexed);
+ return results;
+ },
+
+ 'empty': function(nodes, value, root) {
+ for (var i = 0, results = [], node; node = nodes[i]; i++) {
+ // IE treats comments as element nodes
+ if (node.tagName == '!' || (node.firstChild && !node.innerHTML.match(/^\s*$/))) continue;
+ results.push(node);
+ }
+ return results;
+ },
+
+ 'not': function(nodes, selector, root) {
+ var h = Selector.handlers, selectorType, m;
+ var exclusions = new Selector(selector).findElements(root);
+ h.mark(exclusions);
+ for (var i = 0, results = [], node; node = nodes[i]; i++)
+ if (!node._counted) results.push(node);
+ h.unmark(exclusions);
+ return results;
+ },
+
+ 'enabled': function(nodes, value, root) {
+ for (var i = 0, results = [], node; node = nodes[i]; i++)
+ if (!node.disabled) results.push(node);
+ return results;
+ },
+
+ 'disabled': function(nodes, value, root) {
+ for (var i = 0, results = [], node; node = nodes[i]; i++)
+ if (node.disabled) results.push(node);
+ return results;
+ },
+
+ 'checked': function(nodes, value, root) {
+ for (var i = 0, results = [], node; node = nodes[i]; i++)
+ if (node.checked) results.push(node);
+ return results;
+ }
+ },
+
+ operators: {
+ '=': function(nv, v) { return nv == v; },
+ '!=': function(nv, v) { return nv != v; },
+ '^=': function(nv, v) { return nv.startsWith(v); },
+ '$=': function(nv, v) { return nv.endsWith(v); },
+ '*=': function(nv, v) { return nv.include(v); },
+ '~=': function(nv, v) { return (' ' + nv + ' ').include(' ' + v + ' '); },
+ '|=': function(nv, v) { return ('-' + nv.toUpperCase() + '-').include('-' + v.toUpperCase() + '-'); }
+ },
+
+ matchElements: function(elements, expression) {
+ var matches = new Selector(expression).findElements(), h = Selector.handlers;
+ h.mark(matches);
+ for (var i = 0, results = [], element; element = elements[i]; i++)
+ if (element._counted) results.push(element);
+ h.unmark(matches);
+ return results;
+ },
+
+ findElement: function(elements, expression, index) {
+ if (Object.isNumber(expression)) {
+ index = expression; expression = false;
+ }
+ return Selector.matchElements(elements, expression || '*')[index || 0];
+ },
+
+ findChildElements: function(element, expressions) {
+ var exprs = expressions.join(','), expressions = [];
+ exprs.scan(/(([\w#:.~>+()\s-]+|\*|\[.*?\])+)\s*(,|$)/, function(m) {
+ expressions.push(m[1].strip());
+ });
+ var results = [], h = Selector.handlers;
+ for (var i = 0, l = expressions.length, selector; i < l; i++) {
+ selector = new Selector(expressions[i].strip());
+ h.concat(results, selector.findElements(element));
+ }
+ return (l > 1) ? h.unique(results) : results;
+ }
+});
+
+function $$() {
+ return Selector.findChildElements(document, $A(arguments));
+}
+var Form = {
+ reset: function(form) {
+ $(form).reset();
+ return form;
+ },
+
+ serializeElements: function(elements, options) {
+ if (typeof options != 'object') options = { hash: !!options };
+ else if (options.hash === undefined) options.hash = true;
+ var key, value, submitted = false, submit = options.submit;
+
+ var data = elements.inject({ }, function(result, element) {
+ if (!element.disabled && element.name) {
+ key = element.name; value = $(element).getValue();
+ if (value != null && (element.type != 'submit' || (!submitted &&
+ submit !== false && (!submit || key == submit) && (submitted = true)))) {
+ if (key in result) {
+ // a key is already present; construct an array of values
+ if (!Object.isArray(result[key])) result[key] = [result[key]];
+ result[key].push(value);
+ }
+ else result[key] = value;
+ }
+ }
+ return result;
+ });
+
+ return options.hash ? data : Object.toQueryString(data);
+ }
+};
+
+Form.Methods = {
+ serialize: function(form, options) {
+ return Form.serializeElements(Form.getElements(form), options);
+ },
+
+ getElements: function(form) {
+ return $A($(form).getElementsByTagName('*')).inject([],
+ function(elements, child) {
+ if (Form.Element.Serializers[child.tagName.toLowerCase()])
+ elements.push(Element.extend(child));
+ return elements;
+ }
+ );
+ },
+
+ getInputs: function(form, typeName, name) {
+ form = $(form);
+ var inputs = form.getElementsByTagName('input');
+
+ if (!typeName && !name) return $A(inputs).map(Element.extend);
+
+ for (var i = 0, matchingInputs = [], length = inputs.length; i < length; i++) {
+ var input = inputs[i];
+ if ((typeName && input.type != typeName) || (name && input.name != name))
+ continue;
+ matchingInputs.push(Element.extend(input));
+ }
+
+ return matchingInputs;
+ },
+
+ disable: function(form) {
+ form = $(form);
+ Form.getElements(form).invoke('disable');
+ return form;
+ },
+
+ enable: function(form) {
+ form = $(form);
+ Form.getElements(form).invoke('enable');
+ return form;
+ },
+
+ findFirstElement: function(form) {
+ var elements = $(form).getElements().findAll(function(element) {
+ return 'hidden' != element.type && !element.disabled;
+ });
+ var firstByIndex = elements.findAll(function(element) {
+ return element.hasAttribute('tabIndex') && element.tabIndex >= 0;
+ }).sortBy(function(element) { return element.tabIndex }).first();
+
+ return firstByIndex ? firstByIndex : elements.find(function(element) {
+ return ['input', 'select', 'textarea'].include(element.tagName.toLowerCase());
+ });
+ },
+
+ focusFirstElement: function(form) {
+ form = $(form);
+ form.findFirstElement().activate();
+ return form;
+ },
+
+ request: function(form, options) {
+ form = $(form), options = Object.clone(options || { });
+
+ var params = options.parameters, action = form.readAttribute('action') || '';
+ if (action.blank()) action = window.location.href;
+ options.parameters = form.serialize(true);
+
+ if (params) {
+ if (Object.isString(params)) params = params.toQueryParams();
+ Object.extend(options.parameters, params);
+ }
+
+ if (form.hasAttribute('method') && !options.method)
+ options.method = form.method;
+
+ return new Ajax.Request(action, options);
+ }
+};
+
+/*--------------------------------------------------------------------------*/
+
+Form.Element = {
+ focus: function(element) {
+ $(element).focus();
+ return element;
+ },
+
+ select: function(element) {
+ $(element).select();
+ return element;
+ }
+};
+
+Form.Element.Methods = {
+ serialize: function(element) {
+ element = $(element);
+ if (!element.disabled && element.name) {
+ var value = element.getValue();
+ if (value != undefined) {
+ var pair = { };
+ pair[element.name] = value;
+ return Object.toQueryString(pair);
+ }
+ }
+ return '';
+ },
+
+ getValue: function(element) {
+ element = $(element);
+ var method = element.tagName.toLowerCase();
+ return Form.Element.Serializers[method](element);
+ },
+
+ setValue: function(element, value) {
+ element = $(element);
+ var method = element.tagName.toLowerCase();
+ Form.Element.Serializers[method](element, value);
+ return element;
+ },
+
+ clear: function(element) {
+ $(element).value = '';
+ return element;
+ },
+
+ present: function(element) {
+ return $(element).value != '';
+ },
+
+ activate: function(element) {
+ element = $(element);
+ try {
+ element.focus();
+ if (element.select && (element.tagName.toLowerCase() != 'input' ||
+ !['button', 'reset', 'submit'].include(element.type)))
+ element.select();
+ } catch (e) { }
+ return element;
+ },
+
+ disable: function(element) {
+ element = $(element);
+ element.blur();
+ element.disabled = true;
+ return element;
+ },
+
+ enable: function(element) {
+ element = $(element);
+ element.disabled = false;
+ return element;
+ }
+};
+
+/*--------------------------------------------------------------------------*/
+
+var Field = Form.Element;
+var $F = Form.Element.Methods.getValue;
+
+/*--------------------------------------------------------------------------*/
+
+Form.Element.Serializers = {
+ input: function(element, value) {
+ switch (element.type.toLowerCase()) {
+ case 'checkbox':
+ case 'radio':
+ return Form.Element.Serializers.inputSelector(element, value);
+ default:
+ return Form.Element.Serializers.textarea(element, value);
+ }
+ },
+
+ inputSelector: function(element, value) {
+ if (value === undefined) return element.checked ? element.value : null;
+ else element.checked = !!value;
+ },
+
+ textarea: function(element, value) {
+ if (value === undefined) return element.value;
+ else element.value = value;
+ },
+
+ select: function(element, index) {
+ if (index === undefined)
+ return this[element.type == 'select-one' ?
+ 'selectOne' : 'selectMany'](element);
+ else {
+ var opt, value, single = !Object.isArray(index);
+ for (var i = 0, length = element.length; i < length; i++) {
+ opt = element.options[i];
+ value = this.optionValue(opt);
+ if (single) {
+ if (value == index) {
+ opt.selected = true;
+ return;
+ }
+ }
+ else opt.selected = index.include(value);
+ }
+ }
+ },
+
+ selectOne: function(element) {
+ var index = element.selectedIndex;
+ return index >= 0 ? this.optionValue(element.options[index]) : null;
+ },
+
+ selectMany: function(element) {
+ var values, length = element.length;
+ if (!length) return null;
+
+ for (var i = 0, values = []; i < length; i++) {
+ var opt = element.options[i];
+ if (opt.selected) values.push(this.optionValue(opt));
+ }
+ return values;
+ },
+
+ optionValue: function(opt) {
+ // extend element because hasAttribute may not be native
+ return Element.extend(opt).hasAttribute('value') ? opt.value : opt.text;
+ }
+};
+
+/*--------------------------------------------------------------------------*/
+
+Abstract.TimedObserver = Class.create(PeriodicalExecuter, {
+ initialize: function($super, element, frequency, callback) {
+ $super(callback, frequency);
+ this.element = $(element);
+ this.lastValue = this.getValue();
+ },
+
+ execute: function() {
+ var value = this.getValue();
+ if (Object.isString(this.lastValue) && Object.isString(value) ?
+ this.lastValue != value : String(this.lastValue) != String(value)) {
+ this.callback(this.element, value);
+ this.lastValue = value;
+ }
+ }
+});
+
+Form.Element.Observer = Class.create(Abstract.TimedObserver, {
+ getValue: function() {
+ return Form.Element.getValue(this.element);
+ }
+});
+
+Form.Observer = Class.create(Abstract.TimedObserver, {
+ getValue: function() {
+ return Form.serialize(this.element);
+ }
+});
+
+/*--------------------------------------------------------------------------*/
+
+Abstract.EventObserver = Class.create({
+ initialize: function(element, callback) {
+ this.element = $(element);
+ this.callback = callback;
+
+ this.lastValue = this.getValue();
+ if (this.element.tagName.toLowerCase() == 'form')
+ this.registerFormCallbacks();
+ else
+ this.registerCallback(this.element);
+ },
+
+ onElementEvent: function() {
+ var value = this.getValue();
+ if (this.lastValue != value) {
+ this.callback(this.element, value);
+ this.lastValue = value;
+ }
+ },
+
+ registerFormCallbacks: function() {
+ Form.getElements(this.element).each(this.registerCallback, this);
+ },
+
+ registerCallback: function(element) {
+ if (element.type) {
+ switch (element.type.toLowerCase()) {
+ case 'checkbox':
+ case 'radio':
+ Event.observe(element, 'click', this.onElementEvent.bind(this));
+ break;
+ default:
+ Event.observe(element, 'change', this.onElementEvent.bind(this));
+ break;
+ }
+ }
+ }
+});
+
+Form.Element.EventObserver = Class.create(Abstract.EventObserver, {
+ getValue: function() {
+ return Form.Element.getValue(this.element);
+ }
+});
+
+Form.EventObserver = Class.create(Abstract.EventObserver, {
+ getValue: function() {
+ return Form.serialize(this.element);
+ }
+});
+if (!window.Event) var Event = { };
+
+Object.extend(Event, {
+ KEY_BACKSPACE: 8,
+ KEY_TAB: 9,
+ KEY_RETURN: 13,
+ KEY_ESC: 27,
+ KEY_LEFT: 37,
+ KEY_UP: 38,
+ KEY_RIGHT: 39,
+ KEY_DOWN: 40,
+ KEY_DELETE: 46,
+ KEY_HOME: 36,
+ KEY_END: 35,
+ KEY_PAGEUP: 33,
+ KEY_PAGEDOWN: 34,
+ KEY_INSERT: 45,
+
+ cache: { },
+
+ relatedTarget: function(event) {
+ var element;
+ switch(event.type) {
+ case 'mouseover': element = event.fromElement; break;
+ case 'mouseout': element = event.toElement; break;
+ default: return null;
+ }
+ return Element.extend(element);
+ }
+});
+
+Event.Methods = (function() {
+ var isButton;
+
+ if (Prototype.Browser.IE) {
+ var buttonMap = { 0: 1, 1: 4, 2: 2 };
+ isButton = function(event, code) {
+ return event.button == buttonMap[code];
+ };
+
+ } else if (Prototype.Browser.WebKit) {
+ isButton = function(event, code) {
+ switch (code) {
+ case 0: return event.which == 1 && !event.metaKey;
+ case 1: return event.which == 1 && event.metaKey;
+ default: return false;
+ }
+ };
+
+ } else {
+ isButton = function(event, code) {
+ return event.which ? (event.which === code + 1) : (event.button === code);
+ };
+ }
+
+ return {
+ isLeftClick: function(event) { return isButton(event, 0) },
+ isMiddleClick: function(event) { return isButton(event, 1) },
+ isRightClick: function(event) { return isButton(event, 2) },
+
+ element: function(event) {
+ var node = Event.extend(event).target;
+ return Element.extend(node.nodeType == Node.TEXT_NODE ? node.parentNode : node);
+ },
+
+ findElement: function(event, expression) {
+ var element = Event.element(event);
+ return element.match(expression) ? element : element.up(expression);
+ },
+
+ pointer: function(event) {
+ return {
+ x: event.pageX || (event.clientX +
+ (document.documentElement.scrollLeft || document.body.scrollLeft)),
+ y: event.pageY || (event.clientY +
+ (document.documentElement.scrollTop || document.body.scrollTop))
+ };
+ },
+
+ pointerX: function(event) { return Event.pointer(event).x },
+ pointerY: function(event) { return Event.pointer(event).y },
+
+ stop: function(event) {
+ Event.extend(event);
+ event.preventDefault();
+ event.stopPropagation();
+ event.stopped = true;
+ }
+ };
+})();
+
+Event.extend = (function() {
+ var methods = Object.keys(Event.Methods).inject({ }, function(m, name) {
+ m[name] = Event.Methods[name].methodize();
+ return m;
+ });
+
+ if (Prototype.Browser.IE) {
+ Object.extend(methods, {
+ stopPropagation: function() { this.cancelBubble = true },
+ preventDefault: function() { this.returnValue = false },
+ inspect: function() { return "[object Event]" }
+ });
+
+ return function(event) {
+ if (!event) return false;
+ if (event._extendedByPrototype) return event;
+
+ event._extendedByPrototype = Prototype.emptyFunction;
+ var pointer = Event.pointer(event);
+ Object.extend(event, {
+ target: event.srcElement,
+ relatedTarget: Event.relatedTarget(event),
+ pageX: pointer.x,
+ pageY: pointer.y
+ });
+ return Object.extend(event, methods);
+ };
+
+ } else {
+ Event.prototype = Event.prototype || document.createEvent("HTMLEvents").__proto__;
+ Object.extend(Event.prototype, methods);
+ return Prototype.K;
+ }
+})();
+
+Object.extend(Event, (function() {
+ var cache = Event.cache;
+
+ function getEventID(element) {
+ if (element._eventID) return element._eventID;
+ arguments.callee.id = arguments.callee.id || 1;
+ return element._eventID = ++arguments.callee.id;
+ }
+
+ function getDOMEventName(eventName) {
+ if (eventName && eventName.include(':')) return "dataavailable";
+ return eventName;
+ }
+
+ function getCacheForID(id) {
+ return cache[id] = cache[id] || { };
+ }
+
+ function getWrappersForEventName(id, eventName) {
+ var c = getCacheForID(id);
+ return c[eventName] = c[eventName] || [];
+ }
+
+ function createWrapper(element, eventName, handler) {
+ var id = getEventID(element);
+ var c = getWrappersForEventName(id, eventName);
+ if (c.pluck("handler").include(handler)) return false;
+
+ var wrapper = function(event) {
+ if (!Event || !Event.extend ||
+ (event.eventName && event.eventName != eventName))
+ return false;
+
+ Event.extend(event);
+ handler.call(element, event)
+ };
+
+ wrapper.handler = handler;
+ c.push(wrapper);
+ return wrapper;
+ }
+
+ function findWrapper(id, eventName, handler) {
+ var c = getWrappersForEventName(id, eventName);
+ return c.find(function(wrapper) { return wrapper.handler == handler });
+ }
+
+ function destroyWrapper(id, eventName, handler) {
+ var c = getCacheForID(id);
+ if (!c[eventName]) return false;
+ c[eventName] = c[eventName].without(findWrapper(id, eventName, handler));
+ }
+
+ function destroyCache() {
+ for (var id in cache)
+ for (var eventName in cache[id])
+ cache[id][eventName] = null;
+ }
+
+ if (window.attachEvent) {
+ window.attachEvent("onunload", destroyCache);
+ }
+
+ return {
+ observe: function(element, eventName, handler) {
+ element = $(element);
+ var name = getDOMEventName(eventName);
+
+ var wrapper = createWrapper(element, eventName, handler);
+ if (!wrapper) return element;
+
+ if (element.addEventListener) {
+ element.addEventListener(name, wrapper, false);
+ } else {
+ element.attachEvent("on" + name, wrapper);
+ }
+
+ return element;
+ },
+
+ stopObserving: function(element, eventName, handler) {
+ element = $(element);
+ var id = getEventID(element), name = getDOMEventName(eventName);
+
+ if (!handler && eventName) {
+ getWrappersForEventName(id, eventName).each(function(wrapper) {
+ element.stopObserving(eventName, wrapper.handler);
+ });
+ return element;
+
+ } else if (!eventName) {
+ Object.keys(getCacheForID(id)).each(function(eventName) {
+ element.stopObserving(eventName);
+ });
+ return element;
+ }
+
+ var wrapper = findWrapper(id, eventName, handler);
+ if (!wrapper) return element;
+
+ if (element.removeEventListener) {
+ element.removeEventListener(name, wrapper, false);
+ } else {
+ element.detachEvent("on" + name, wrapper);
+ }
+
+ destroyWrapper(id, eventName, handler);
+
+ return element;
+ },
+
+ fire: function(element, eventName, memo) {
+ element = $(element);
+ if (element == document && document.createEvent && !element.dispatchEvent)
+ element = document.documentElement;
+
+ if (document.createEvent) {
+ var event = document.createEvent("HTMLEvents");
+ event.initEvent("dataavailable", true, true);
+ } else {
+ var event = document.createEventObject();
+ event.eventType = "ondataavailable";
+ }
+
+ event.eventName = eventName;
+ event.memo = memo || { };
+
+ if (document.createEvent) {
+ element.dispatchEvent(event);
+ } else {
+ element.fireEvent(event.eventType, event);
+ }
+
+ return event;
+ }
+ };
+})());
+
+Object.extend(Event, Event.Methods);
+
+Element.addMethods({
+ fire: Event.fire,
+ observe: Event.observe,
+ stopObserving: Event.stopObserving
+});
+
+Object.extend(document, {
+ fire: Element.Methods.fire.methodize(),
+ observe: Element.Methods.observe.methodize(),
+ stopObserving: Element.Methods.stopObserving.methodize()
+});
+
+(function() {
+ /* Support for the DOMContentLoaded event is based on work by Dan Webb,
+ Matthias Miller, Dean Edwards and John Resig. */
+
+ var timer, fired = false;
+
+ function fireContentLoadedEvent() {
+ if (fired) return;
+ if (timer) window.clearInterval(timer);
+ document.fire("dom:loaded");
+ fired = true;
+ }
+
+ if (document.addEventListener) {
+ if (Prototype.Browser.WebKit) {
+ timer = window.setInterval(function() {
+ if (/loaded|complete/.test(document.readyState))
+ fireContentLoadedEvent();
+ }, 0);
+
+ Event.observe(window, "load", fireContentLoadedEvent);
+
+ } else {
+ document.addEventListener("DOMContentLoaded",
+ fireContentLoadedEvent, false);
+ }
+
+ } else {
+ document.write("<script id=__onDOMContentLoaded defer src=//:><\/script>");
+ $("__onDOMContentLoaded").onreadystatechange = function() {
+ if (this.readyState == "complete") {
+ this.onreadystatechange = null;
+ fireContentLoadedEvent();
+ }
+ };
+ }
+})();
+/*------------------------------- DEPRECATED -------------------------------*/
+
+Hash.toQueryString = Object.toQueryString;
+
+var Toggle = { display: Element.toggle };
+
+Element.Methods.childOf = Element.Methods.descendantOf;
+
+var Insertion = {
+ Before: function(element, content) {
+ return Element.insert(element, {before:content});
+ },
+
+ Top: function(element, content) {
+ return Element.insert(element, {top:content});
+ },
+
+ Bottom: function(element, content) {
+ return Element.insert(element, {bottom:content});
+ },
+
+ After: function(element, content) {
+ return Element.insert(element, {after:content});
+ }
+};
+
+var $continue = new Error('"throw $continue" is deprecated, use "return" instead');
+
+// This should be moved to script.aculo.us; notice the deprecated methods
+// further below, that map to the newer Element methods.
+var Position = {
+ // set to true if needed, warning: firefox performance problems
+ // NOT neeeded for page scrolling, only if draggable contained in
+ // scrollable elements
+ includeScrollOffsets: false,
+
+ // must be called before calling withinIncludingScrolloffset, every time the
+ // page is scrolled
+ prepare: function() {
+ this.deltaX = window.pageXOffset
+ || document.documentElement.scrollLeft
+ || document.body.scrollLeft
+ || 0;
+ this.deltaY = window.pageYOffset
+ || document.documentElement.scrollTop
+ || document.body.scrollTop
+ || 0;
+ },
+
+ // caches x/y coordinate pair to use with overlap
+ within: function(element, x, y) {
+ if (this.includeScrollOffsets)
+ return this.withinIncludingScrolloffsets(element, x, y);
+ this.xcomp = x;
+ this.ycomp = y;
+ this.offset = Element.cumulativeOffset(element);
+
+ return (y >= this.offset[1] &&
+ y < this.offset[1] + element.offsetHeight &&
+ x >= this.offset[0] &&
+ x < this.offset[0] + element.offsetWidth);
+ },
+
+ withinIncludingScrolloffsets: function(element, x, y) {
+ var offsetcache = Element.cumulativeScrollOffset(element);
+
+ this.xcomp = x + offsetcache[0] - this.deltaX;
+ this.ycomp = y + offsetcache[1] - this.deltaY;
+ this.offset = Element.cumulativeOffset(element);
+
+ return (this.ycomp >= this.offset[1] &&
+ this.ycomp < this.offset[1] + element.offsetHeight &&
+ this.xcomp >= this.offset[0] &&
+ this.xcomp < this.offset[0] + element.offsetWidth);
+ },
+
+ // within must be called directly before
+ overlap: function(mode, element) {
+ if (!mode) return 0;
+ if (mode == 'vertical')
+ return ((this.offset[1] + element.offsetHeight) - this.ycomp) /
+ element.offsetHeight;
+ if (mode == 'horizontal')
+ return ((this.offset[0] + element.offsetWidth) - this.xcomp) /
+ element.offsetWidth;
+ },
+
+ // Deprecation layer -- use newer Element methods now (1.5.2).
+
+ cumulativeOffset: Element.Methods.cumulativeOffset,
+
+ positionedOffset: Element.Methods.positionedOffset,
+
+ absolutize: function(element) {
+ Position.prepare();
+ return Element.absolutize(element);
+ },
+
+ relativize: function(element) {
+ Position.prepare();
+ return Element.relativize(element);
+ },
+
+ realOffset: Element.Methods.cumulativeScrollOffset,
+
+ offsetParent: Element.Methods.getOffsetParent,
+
+ page: Element.Methods.viewportOffset,
+
+ clone: function(source, target, options) {
+ options = options || { };
+ return Element.clonePosition(target, source, options);
+ }
+};
+
+/*--------------------------------------------------------------------------*/
+
+if (!document.getElementsByClassName) document.getElementsByClassName = function(instanceMethods){
+ function iter(name) {
+ return name.blank() ? null : "[contains(concat(' ', @class, ' '), ' " + name + " ')]";
+ }
+
+ instanceMethods.getElementsByClassName = Prototype.BrowserFeatures.XPath ?
+ function(element, className) {
+ className = className.toString().strip();
+ var cond = /\s/.test(className) ? $w(className).map(iter).join('') : iter(className);
+ return cond ? document._getElementsByXPath('.//*' + cond, element) : [];
+ } : function(element, className) {
+ className = className.toString().strip();
+ var elements = [], classNames = (/\s/.test(className) ? $w(className) : null);
+ if (!classNames && !className) return elements;
+
+ var nodes = $(element).getElementsByTagName('*');
+ className = ' ' + className + ' ';
+
+ for (var i = 0, child, cn; child = nodes[i]; i++) {
+ if (child.className && (cn = ' ' + child.className + ' ') && (cn.include(className) ||
+ (classNames && classNames.all(function(name) {
+ return !name.toString().blank() && cn.include(' ' + name + ' ');
+ }))))
+ elements.push(Element.extend(child));
+ }
+ return elements;
+ };
+
+ return function(className, parentElement) {
+ return $(parentElement || document.body).getElementsByClassName(className);
+ };
+}(Element.Methods);
+
+/*--------------------------------------------------------------------------*/
+
+Element.ClassNames = Class.create();
+Element.ClassNames.prototype = {
+ initialize: function(element) {
+ this.element = $(element);
+ },
+
+ _each: function(iterator) {
+ this.element.className.split(/\s+/).select(function(name) {
+ return name.length > 0;
+ })._each(iterator);
+ },
+
+ set: function(className) {
+ this.element.className = className;
+ },
+
+ add: function(classNameToAdd) {
+ if (this.include(classNameToAdd)) return;
+ this.set($A(this).concat(classNameToAdd).join(' '));
+ },
+
+ remove: function(classNameToRemove) {
+ if (!this.include(classNameToRemove)) return;
+ this.set($A(this).without(classNameToRemove).join(' '));
+ },
+
+ toString: function() {
+ return $A(this).join(' ');
+ }
+};
+
+Object.extend(Element.ClassNames.prototype, Enumerable);
+
+/*--------------------------------------------------------------------------*/
+
+Element.addMethods();
\ No newline at end of file
--- /dev/null
+require 'mkmf'
+require 'rbconfig'
+
+unless $CFLAGS.gsub!(/ -O[\dsz]?/, ' -O3')
+ $CFLAGS << ' -O3'
+end
+if CONFIG['CC'] =~ /gcc/
+ $CFLAGS << ' -Wall'
+ #unless $CFLAGS.gsub!(/ -O[\dsz]?/, ' -O0 -ggdb')
+ # $CFLAGS << ' -O0 -ggdb'
+ #end
+end
+
+if RUBY_VERSION < "1.9"
+ have_header("re.h")
+else
+ have_header("ruby/re.h")
+ have_header("ruby/encoding.h")
+end
+create_makefile 'json/ext/generator'
--- /dev/null
+#include "generator.h"
+
+#ifdef HAVE_RUBY_ENCODING_H
+static VALUE CEncoding_UTF_8;
+static ID i_encoding, i_encode;
+#endif
+
+static VALUE mJSON, mExt, mGenerator, cState, mGeneratorMethods, mObject,
+ mHash, mArray, mFixnum, mBignum, mFloat, mString, mString_Extend,
+ mTrueClass, mFalseClass, mNilClass, eGeneratorError,
+ eNestingError, CRegexp_MULTILINE, CJSON_SAFE_STATE_PROTOTYPE,
+ i_SAFE_STATE_PROTOTYPE;
+
+static ID i_to_s, i_to_json, i_new, i_indent, i_space, i_space_before,
+ i_object_nl, i_array_nl, i_max_nesting, i_allow_nan, i_ascii_only,
+ i_quirks_mode, i_pack, i_unpack, i_create_id, i_extend, i_key_p,
+ i_aref, i_send, i_respond_to_p, i_match, i_keys, i_depth, i_dup;
+
+/*
+ * Copyright 2001-2004 Unicode, Inc.
+ *
+ * Disclaimer
+ *
+ * This source code is provided as is by Unicode, Inc. No claims are
+ * made as to fitness for any particular purpose. No warranties of any
+ * kind are expressed or implied. The recipient agrees to determine
+ * applicability of information provided. If this file has been
+ * purchased on magnetic or optical media from Unicode, Inc., the
+ * sole remedy for any claim will be exchange of defective media
+ * within 90 days of receipt.
+ *
+ * Limitations on Rights to Redistribute This Code
+ *
+ * Unicode, Inc. hereby grants the right to freely use the information
+ * supplied in this file in the creation of products supporting the
+ * Unicode Standard, and to make copies of this file in any form
+ * for internal or external distribution as long as this notice
+ * remains attached.
+ */
+
+/*
+ * Index into the table below with the first byte of a UTF-8 sequence to
+ * get the number of trailing bytes that are supposed to follow it.
+ * Note that *legal* UTF-8 values can't have 4 or 5-bytes. The table is
+ * left as-is for anyone who may want to do such conversion, which was
+ * allowed in earlier algorithms.
+ */
+static const char trailingBytesForUTF8[256] = {
+ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+ 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,
+ 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2, 3,3,3,3,3,3,3,3,4,4,4,4,5,5,5,5
+};
+
+/*
+ * Magic values subtracted from a buffer value during UTF8 conversion.
+ * This table contains as many values as there might be trailing bytes
+ * in a UTF-8 sequence.
+ */
+static const UTF32 offsetsFromUTF8[6] = { 0x00000000UL, 0x00003080UL, 0x000E2080UL,
+ 0x03C82080UL, 0xFA082080UL, 0x82082080UL };
+
+/*
+ * Utility routine to tell whether a sequence of bytes is legal UTF-8.
+ * This must be called with the length pre-determined by the first byte.
+ * If not calling this from ConvertUTF8to*, then the length can be set by:
+ * length = trailingBytesForUTF8[*source]+1;
+ * and the sequence is illegal right away if there aren't that many bytes
+ * available.
+ * If presented with a length > 4, this returns 0. The Unicode
+ * definition of UTF-8 goes up to 4-byte sequences.
+ */
+static unsigned char isLegalUTF8(const UTF8 *source, unsigned long length)
+{
+ UTF8 a;
+ const UTF8 *srcptr = source+length;
+ switch (length) {
+ default: return 0;
+ /* Everything else falls through when "1"... */
+ case 4: if ((a = (*--srcptr)) < 0x80 || a > 0xBF) return 0;
+ case 3: if ((a = (*--srcptr)) < 0x80 || a > 0xBF) return 0;
+ case 2: if ((a = (*--srcptr)) > 0xBF) return 0;
+
+ switch (*source) {
+ /* no fall-through in this inner switch */
+ case 0xE0: if (a < 0xA0) return 0; break;
+ case 0xED: if (a > 0x9F) return 0; break;
+ case 0xF0: if (a < 0x90) return 0; break;
+ case 0xF4: if (a > 0x8F) return 0; break;
+ default: if (a < 0x80) return 0;
+ }
+
+ case 1: if (*source >= 0x80 && *source < 0xC2) return 0;
+ }
+ if (*source > 0xF4) return 0;
+ return 1;
+}
+
+/* Escapes the UTF16 character and stores the result in the buffer buf. */
+static void unicode_escape(char *buf, UTF16 character)
+{
+ const char *digits = "0123456789abcdef";
+
+ buf[2] = digits[character >> 12];
+ buf[3] = digits[(character >> 8) & 0xf];
+ buf[4] = digits[(character >> 4) & 0xf];
+ buf[5] = digits[character & 0xf];
+}
+
+/* Escapes the UTF16 character and stores the result in the buffer buf, then
+ * the buffer buf іs appended to the FBuffer buffer. */
+static void unicode_escape_to_buffer(FBuffer *buffer, char buf[6], UTF16
+ character)
+{
+ unicode_escape(buf, character);
+ fbuffer_append(buffer, buf, 6);
+}
+
+/* Converts string to a JSON string in FBuffer buffer, where all but the ASCII
+ * and control characters are JSON escaped. */
+static void convert_UTF8_to_JSON_ASCII(FBuffer *buffer, VALUE string)
+{
+ const UTF8 *source = (UTF8 *) RSTRING_PTR(string);
+ const UTF8 *sourceEnd = source + RSTRING_LEN(string);
+ char buf[6] = { '\\', 'u' };
+
+ while (source < sourceEnd) {
+ UTF32 ch = 0;
+ unsigned short extraBytesToRead = trailingBytesForUTF8[*source];
+ if (source + extraBytesToRead >= sourceEnd) {
+ rb_raise(rb_path2class("JSON::GeneratorError"),
+ "partial character in source, but hit end");
+ }
+ if (!isLegalUTF8(source, extraBytesToRead+1)) {
+ rb_raise(rb_path2class("JSON::GeneratorError"),
+ "source sequence is illegal/malformed utf-8");
+ }
+ /*
+ * The cases all fall through. See "Note A" below.
+ */
+ switch (extraBytesToRead) {
+ case 5: ch += *source++; ch <<= 6; /* remember, illegal UTF-8 */
+ case 4: ch += *source++; ch <<= 6; /* remember, illegal UTF-8 */
+ case 3: ch += *source++; ch <<= 6;
+ case 2: ch += *source++; ch <<= 6;
+ case 1: ch += *source++; ch <<= 6;
+ case 0: ch += *source++;
+ }
+ ch -= offsetsFromUTF8[extraBytesToRead];
+
+ if (ch <= UNI_MAX_BMP) { /* Target is a character <= 0xFFFF */
+ /* UTF-16 surrogate values are illegal in UTF-32 */
+ if (ch >= UNI_SUR_HIGH_START && ch <= UNI_SUR_LOW_END) {
+#if UNI_STRICT_CONVERSION
+ source -= (extraBytesToRead+1); /* return to the illegal value itself */
+ rb_raise(rb_path2class("JSON::GeneratorError"),
+ "source sequence is illegal/malformed utf-8");
+#else
+ unicode_escape_to_buffer(buffer, buf, UNI_REPLACEMENT_CHAR);
+#endif
+ } else {
+ /* normal case */
+ if (ch >= 0x20 && ch <= 0x7f) {
+ switch (ch) {
+ case '\\':
+ fbuffer_append(buffer, "\\\\", 2);
+ break;
+ case '"':
+ fbuffer_append(buffer, "\\\"", 2);
+ break;
+ default:
+ fbuffer_append_char(buffer, (char)ch);
+ break;
+ }
+ } else {
+ switch (ch) {
+ case '\n':
+ fbuffer_append(buffer, "\\n", 2);
+ break;
+ case '\r':
+ fbuffer_append(buffer, "\\r", 2);
+ break;
+ case '\t':
+ fbuffer_append(buffer, "\\t", 2);
+ break;
+ case '\f':
+ fbuffer_append(buffer, "\\f", 2);
+ break;
+ case '\b':
+ fbuffer_append(buffer, "\\b", 2);
+ break;
+ default:
+ unicode_escape_to_buffer(buffer, buf, (UTF16) ch);
+ break;
+ }
+ }
+ }
+ } else if (ch > UNI_MAX_UTF16) {
+#if UNI_STRICT_CONVERSION
+ source -= (extraBytesToRead+1); /* return to the start */
+ rb_raise(rb_path2class("JSON::GeneratorError"),
+ "source sequence is illegal/malformed utf8");
+#else
+ unicode_escape_to_buffer(buffer, buf, UNI_REPLACEMENT_CHAR);
+#endif
+ } else {
+ /* target is a character in range 0xFFFF - 0x10FFFF. */
+ ch -= halfBase;
+ unicode_escape_to_buffer(buffer, buf, (UTF16)((ch >> halfShift) + UNI_SUR_HIGH_START));
+ unicode_escape_to_buffer(buffer, buf, (UTF16)((ch & halfMask) + UNI_SUR_LOW_START));
+ }
+ }
+}
+
+/* Converts string to a JSON string in FBuffer buffer, where only the
+ * characters required by the JSON standard are JSON escaped. The remaining
+ * characters (should be UTF8) are just passed through and appended to the
+ * result. */
+static void convert_UTF8_to_JSON(FBuffer *buffer, VALUE string)
+{
+ const char *ptr = RSTRING_PTR(string), *p;
+ unsigned long len = RSTRING_LEN(string), start = 0, end = 0;
+ const char *escape = NULL;
+ int escape_len;
+ unsigned char c;
+ char buf[6] = { '\\', 'u' };
+
+ for (start = 0, end = 0; end < len;) {
+ p = ptr + end;
+ c = (unsigned char) *p;
+ if (c < 0x20) {
+ switch (c) {
+ case '\n':
+ escape = "\\n";
+ escape_len = 2;
+ break;
+ case '\r':
+ escape = "\\r";
+ escape_len = 2;
+ break;
+ case '\t':
+ escape = "\\t";
+ escape_len = 2;
+ break;
+ case '\f':
+ escape = "\\f";
+ escape_len = 2;
+ break;
+ case '\b':
+ escape = "\\b";
+ escape_len = 2;
+ break;
+ default:
+ unicode_escape(buf, (UTF16) *p);
+ escape = buf;
+ escape_len = 6;
+ break;
+ }
+ } else {
+ switch (c) {
+ case '\\':
+ escape = "\\\\";
+ escape_len = 2;
+ break;
+ case '"':
+ escape = "\\\"";
+ escape_len = 2;
+ break;
+ default:
+ end++;
+ continue;
+ break;
+ }
+ }
+ fbuffer_append(buffer, ptr + start, end - start);
+ fbuffer_append(buffer, escape, escape_len);
+ start = ++end;
+ escape = NULL;
+ }
+ fbuffer_append(buffer, ptr + start, end - start);
+}
+
+static char *fstrndup(const char *ptr, unsigned long len) {
+ char *result;
+ if (len <= 0) return NULL;
+ result = ALLOC_N(char, len);
+ memccpy(result, ptr, 0, len);
+ return result;
+}
+
+/* fbuffer implementation */
+
+static FBuffer *fbuffer_alloc()
+{
+ FBuffer *fb = ALLOC(FBuffer);
+ memset((void *) fb, 0, sizeof(FBuffer));
+ fb->initial_length = FBUFFER_INITIAL_LENGTH;
+ return fb;
+}
+
+static FBuffer *fbuffer_alloc_with_length(unsigned long initial_length)
+{
+ FBuffer *fb;
+ assert(initial_length > 0);
+ fb = ALLOC(FBuffer);
+ memset((void *) fb, 0, sizeof(FBuffer));
+ fb->initial_length = initial_length;
+ return fb;
+}
+
+static void fbuffer_free(FBuffer *fb)
+{
+ if (fb->ptr) ruby_xfree(fb->ptr);
+ ruby_xfree(fb);
+}
+
+static void fbuffer_clear(FBuffer *fb)
+{
+ fb->len = 0;
+}
+
+static void fbuffer_inc_capa(FBuffer *fb, unsigned long requested)
+{
+ unsigned long required;
+
+ if (!fb->ptr) {
+ fb->ptr = ALLOC_N(char, fb->initial_length);
+ fb->capa = fb->initial_length;
+ }
+
+ for (required = fb->capa; requested > required - fb->len; required <<= 1);
+
+ if (required > fb->capa) {
+ REALLOC_N(fb->ptr, char, required);
+ fb->capa = required;
+ }
+}
+
+static void fbuffer_append(FBuffer *fb, const char *newstr, unsigned long len)
+{
+ if (len > 0) {
+ fbuffer_inc_capa(fb, len);
+ MEMCPY(fb->ptr + fb->len, newstr, char, len);
+ fb->len += len;
+ }
+}
+
+static void fbuffer_append_str(FBuffer *fb, VALUE str)
+{
+ const char *newstr = StringValuePtr(str);
+ unsigned long len = RSTRING_LEN(str);
+
+ RB_GC_GUARD(str);
+
+ fbuffer_append(fb, newstr, len);
+}
+
+static void fbuffer_append_char(FBuffer *fb, char newchr)
+{
+ fbuffer_inc_capa(fb, 1);
+ *(fb->ptr + fb->len) = newchr;
+ fb->len++;
+}
+
+static void freverse(char *start, char *end)
+{
+ char c;
+
+ while (end > start) {
+ c = *end, *end-- = *start, *start++ = c;
+ }
+}
+
+static long fltoa(long number, char *buf)
+{
+ static char digits[] = "0123456789";
+ long sign = number;
+ char* tmp = buf;
+
+ if (sign < 0) number = -number;
+ do *tmp++ = digits[number % 10]; while (number /= 10);
+ if (sign < 0) *tmp++ = '-';
+ freverse(buf, tmp - 1);
+ return tmp - buf;
+}
+
+static void fbuffer_append_long(FBuffer *fb, long number)
+{
+ char buf[20];
+ unsigned long len = fltoa(number, buf);
+ fbuffer_append(fb, buf, len);
+}
+
+static FBuffer *fbuffer_dup(FBuffer *fb)
+{
+ unsigned long len = fb->len;
+ FBuffer *result;
+
+ if (len > 0) {
+ result = fbuffer_alloc_with_length(len);
+ fbuffer_append(result, FBUFFER_PAIR(fb));
+ } else {
+ result = fbuffer_alloc();
+ }
+ return result;
+}
+
+/*
+ * Document-module: JSON::Ext::Generator
+ *
+ * This is the JSON generator implemented as a C extension. It can be
+ * configured to be used by setting
+ *
+ * JSON.generator = JSON::Ext::Generator
+ *
+ * with the method generator= in JSON.
+ *
+ */
+
+/*
+ * call-seq: to_json(state = nil)
+ *
+ * Returns a JSON string containing a JSON object, that is generated from
+ * this Hash instance.
+ * _state_ is a JSON::State object, that can also be used to configure the
+ * produced JSON string output further.
+ */
+static VALUE mHash_to_json(int argc, VALUE *argv, VALUE self)
+{
+ GENERATE_JSON(object);
+}
+
+/*
+ * call-seq: to_json(state = nil)
+ *
+ * Returns a JSON string containing a JSON array, that is generated from
+ * this Array instance.
+ * _state_ is a JSON::State object, that can also be used to configure the
+ * produced JSON string output further.
+ */
+static VALUE mArray_to_json(int argc, VALUE *argv, VALUE self) {
+ GENERATE_JSON(array);
+}
+
+/*
+ * call-seq: to_json(*)
+ *
+ * Returns a JSON string representation for this Integer number.
+ */
+static VALUE mFixnum_to_json(int argc, VALUE *argv, VALUE self)
+{
+ GENERATE_JSON(fixnum);
+}
+
+/*
+ * call-seq: to_json(*)
+ *
+ * Returns a JSON string representation for this Integer number.
+ */
+static VALUE mBignum_to_json(int argc, VALUE *argv, VALUE self)
+{
+ GENERATE_JSON(bignum);
+}
+
+/*
+ * call-seq: to_json(*)
+ *
+ * Returns a JSON string representation for this Float number.
+ */
+static VALUE mFloat_to_json(int argc, VALUE *argv, VALUE self)
+{
+ GENERATE_JSON(float);
+}
+
+/*
+ * call-seq: String.included(modul)
+ *
+ * Extends _modul_ with the String::Extend module.
+ */
+static VALUE mString_included_s(VALUE self, VALUE modul) {
+ VALUE result = rb_funcall(modul, i_extend, 1, mString_Extend);
+ return result;
+}
+
+/*
+ * call-seq: to_json(*)
+ *
+ * This string should be encoded with UTF-8 A call to this method
+ * returns a JSON string encoded with UTF16 big endian characters as
+ * \u????.
+ */
+static VALUE mString_to_json(int argc, VALUE *argv, VALUE self)
+{
+ GENERATE_JSON(string);
+}
+
+/*
+ * call-seq: to_json_raw_object()
+ *
+ * This method creates a raw object hash, that can be nested into
+ * other data structures and will be generated as a raw string. This
+ * method should be used, if you want to convert raw strings to JSON
+ * instead of UTF-8 strings, e. g. binary data.
+ */
+static VALUE mString_to_json_raw_object(VALUE self)
+{
+ VALUE ary;
+ VALUE result = rb_hash_new();
+ rb_hash_aset(result, rb_funcall(mJSON, i_create_id, 0), rb_class_name(rb_obj_class(self)));
+ ary = rb_funcall(self, i_unpack, 1, rb_str_new2("C*"));
+ rb_hash_aset(result, rb_str_new2("raw"), ary);
+ return result;
+}
+
+/*
+ * call-seq: to_json_raw(*args)
+ *
+ * This method creates a JSON text from the result of a call to
+ * to_json_raw_object of this String.
+ */
+static VALUE mString_to_json_raw(int argc, VALUE *argv, VALUE self)
+{
+ VALUE obj = mString_to_json_raw_object(self);
+ Check_Type(obj, T_HASH);
+ return mHash_to_json(argc, argv, obj);
+}
+
+/*
+ * call-seq: json_create(o)
+ *
+ * Raw Strings are JSON Objects (the raw bytes are stored in an array for the
+ * key "raw"). The Ruby String can be created by this module method.
+ */
+static VALUE mString_Extend_json_create(VALUE self, VALUE o)
+{
+ VALUE ary;
+ Check_Type(o, T_HASH);
+ ary = rb_hash_aref(o, rb_str_new2("raw"));
+ return rb_funcall(ary, i_pack, 1, rb_str_new2("C*"));
+}
+
+/*
+ * call-seq: to_json(*)
+ *
+ * Returns a JSON string for true: 'true'.
+ */
+static VALUE mTrueClass_to_json(int argc, VALUE *argv, VALUE self)
+{
+ GENERATE_JSON(true);
+}
+
+/*
+ * call-seq: to_json(*)
+ *
+ * Returns a JSON string for false: 'false'.
+ */
+static VALUE mFalseClass_to_json(int argc, VALUE *argv, VALUE self)
+{
+ GENERATE_JSON(false);
+}
+
+/*
+ * call-seq: to_json(*)
+ *
+ * Returns a JSON string for nil: 'null'.
+ */
+static VALUE mNilClass_to_json(int argc, VALUE *argv, VALUE self)
+{
+ GENERATE_JSON(null);
+}
+
+/*
+ * call-seq: to_json(*)
+ *
+ * Converts this object to a string (calling #to_s), converts
+ * it to a JSON string, and returns the result. This is a fallback, if no
+ * special method #to_json was defined for some object.
+ */
+static VALUE mObject_to_json(int argc, VALUE *argv, VALUE self)
+{
+ VALUE state;
+ VALUE string = rb_funcall(self, i_to_s, 0);
+ rb_scan_args(argc, argv, "01", &state);
+ Check_Type(string, T_STRING);
+ state = cState_from_state_s(cState, state);
+ return cState_partial_generate(state, string);
+}
+
+static void State_free(JSON_Generator_State *state)
+{
+ if (state->indent) ruby_xfree(state->indent);
+ if (state->space) ruby_xfree(state->space);
+ if (state->space_before) ruby_xfree(state->space_before);
+ if (state->object_nl) ruby_xfree(state->object_nl);
+ if (state->array_nl) ruby_xfree(state->array_nl);
+ if (state->array_delim) fbuffer_free(state->array_delim);
+ if (state->object_delim) fbuffer_free(state->object_delim);
+ if (state->object_delim2) fbuffer_free(state->object_delim2);
+ ruby_xfree(state);
+}
+
+static JSON_Generator_State *State_allocate()
+{
+ JSON_Generator_State *state = ALLOC(JSON_Generator_State);
+ MEMZERO(state, JSON_Generator_State, 1);
+ return state;
+}
+
+static VALUE cState_s_allocate(VALUE klass)
+{
+ JSON_Generator_State *state = State_allocate();
+ return Data_Wrap_Struct(klass, NULL, State_free, state);
+}
+
+/*
+ * call-seq: configure(opts)
+ *
+ * Configure this State instance with the Hash _opts_, and return
+ * itself.
+ */
+static VALUE cState_configure(VALUE self, VALUE opts)
+{
+ VALUE tmp;
+ GET_STATE(self);
+ tmp = rb_convert_type(opts, T_HASH, "Hash", "to_hash");
+ if (NIL_P(tmp)) tmp = rb_convert_type(opts, T_HASH, "Hash", "to_h");
+ if (NIL_P(tmp)) {
+ rb_raise(rb_eArgError, "opts has to be hash like or convertable into a hash");
+ }
+ opts = tmp;
+ tmp = rb_hash_aref(opts, ID2SYM(i_indent));
+ if (RTEST(tmp)) {
+ unsigned long len;
+ Check_Type(tmp, T_STRING);
+ len = RSTRING_LEN(tmp);
+ state->indent = fstrndup(RSTRING_PTR(tmp), len);
+ state->indent_len = len;
+ }
+ tmp = rb_hash_aref(opts, ID2SYM(i_space));
+ if (RTEST(tmp)) {
+ unsigned long len;
+ Check_Type(tmp, T_STRING);
+ len = RSTRING_LEN(tmp);
+ state->space = fstrndup(RSTRING_PTR(tmp), len);
+ state->space_len = len;
+ }
+ tmp = rb_hash_aref(opts, ID2SYM(i_space_before));
+ if (RTEST(tmp)) {
+ unsigned long len;
+ Check_Type(tmp, T_STRING);
+ len = RSTRING_LEN(tmp);
+ state->space_before = fstrndup(RSTRING_PTR(tmp), len);
+ state->space_before_len = len;
+ }
+ tmp = rb_hash_aref(opts, ID2SYM(i_array_nl));
+ if (RTEST(tmp)) {
+ unsigned long len;
+ Check_Type(tmp, T_STRING);
+ len = RSTRING_LEN(tmp);
+ state->array_nl = fstrndup(RSTRING_PTR(tmp), len);
+ state->array_nl_len = len;
+ }
+ tmp = rb_hash_aref(opts, ID2SYM(i_object_nl));
+ if (RTEST(tmp)) {
+ unsigned long len;
+ Check_Type(tmp, T_STRING);
+ len = RSTRING_LEN(tmp);
+ state->object_nl = fstrndup(RSTRING_PTR(tmp), len);
+ state->object_nl_len = len;
+ }
+ tmp = ID2SYM(i_max_nesting);
+ state->max_nesting = 19;
+ if (option_given_p(opts, tmp)) {
+ VALUE max_nesting = rb_hash_aref(opts, tmp);
+ if (RTEST(max_nesting)) {
+ Check_Type(max_nesting, T_FIXNUM);
+ state->max_nesting = FIX2LONG(max_nesting);
+ } else {
+ state->max_nesting = 0;
+ }
+ }
+ tmp = ID2SYM(i_depth);
+ state->depth = 0;
+ if (option_given_p(opts, tmp)) {
+ VALUE depth = rb_hash_aref(opts, tmp);
+ if (RTEST(depth)) {
+ Check_Type(depth, T_FIXNUM);
+ state->depth = FIX2LONG(depth);
+ } else {
+ state->depth = 0;
+ }
+ }
+ tmp = rb_hash_aref(opts, ID2SYM(i_allow_nan));
+ state->allow_nan = RTEST(tmp);
+ tmp = rb_hash_aref(opts, ID2SYM(i_ascii_only));
+ state->ascii_only = RTEST(tmp);
+ tmp = rb_hash_aref(opts, ID2SYM(i_quirks_mode));
+ state->quirks_mode = RTEST(tmp);
+ return self;
+}
+
+/*
+ * call-seq: to_h
+ *
+ * Returns the configuration instance variables as a hash, that can be
+ * passed to the configure method.
+ */
+static VALUE cState_to_h(VALUE self)
+{
+ VALUE result = rb_hash_new();
+ GET_STATE(self);
+ rb_hash_aset(result, ID2SYM(i_indent), rb_str_new(state->indent, state->indent_len));
+ rb_hash_aset(result, ID2SYM(i_space), rb_str_new(state->space, state->space_len));
+ rb_hash_aset(result, ID2SYM(i_space_before), rb_str_new(state->space_before, state->space_before_len));
+ rb_hash_aset(result, ID2SYM(i_object_nl), rb_str_new(state->object_nl, state->object_nl_len));
+ rb_hash_aset(result, ID2SYM(i_array_nl), rb_str_new(state->array_nl, state->array_nl_len));
+ rb_hash_aset(result, ID2SYM(i_allow_nan), state->allow_nan ? Qtrue : Qfalse);
+ rb_hash_aset(result, ID2SYM(i_ascii_only), state->ascii_only ? Qtrue : Qfalse);
+ rb_hash_aset(result, ID2SYM(i_quirks_mode), state->quirks_mode ? Qtrue : Qfalse);
+ rb_hash_aset(result, ID2SYM(i_max_nesting), LONG2FIX(state->max_nesting));
+ rb_hash_aset(result, ID2SYM(i_depth), LONG2FIX(state->depth));
+ return result;
+}
+
+/*
+* call-seq: [](name)
+*
+* Return the value returned by method +name+.
+*/
+static VALUE cState_aref(VALUE self, VALUE name)
+{
+ GET_STATE(self);
+ if (RTEST(rb_funcall(self, i_respond_to_p, 1, name))) {
+ return rb_funcall(self, i_send, 1, name);
+ } else {
+ return Qnil;
+ }
+}
+
+static void generate_json_object(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj)
+{
+ char *object_nl = state->object_nl;
+ long object_nl_len = state->object_nl_len;
+ char *indent = state->indent;
+ long indent_len = state->indent_len;
+ long max_nesting = state->max_nesting;
+ char *delim = FBUFFER_PTR(state->object_delim);
+ long delim_len = FBUFFER_LEN(state->object_delim);
+ char *delim2 = FBUFFER_PTR(state->object_delim2);
+ long delim2_len = FBUFFER_LEN(state->object_delim2);
+ long depth = ++state->depth;
+ int i, j;
+ VALUE key, key_to_s, keys;
+ if (max_nesting != 0 && depth > max_nesting) {
+ fbuffer_free(buffer);
+ rb_raise(eNestingError, "nesting of %ld is too deep", --state->depth);
+ }
+ fbuffer_append_char(buffer, '{');
+ keys = rb_funcall(obj, i_keys, 0);
+ for(i = 0; i < RARRAY_LEN(keys); i++) {
+ if (i > 0) fbuffer_append(buffer, delim, delim_len);
+ if (object_nl) {
+ fbuffer_append(buffer, object_nl, object_nl_len);
+ }
+ if (indent) {
+ for (j = 0; j < depth; j++) {
+ fbuffer_append(buffer, indent, indent_len);
+ }
+ }
+ key = rb_ary_entry(keys, i);
+ key_to_s = rb_funcall(key, i_to_s, 0);
+ Check_Type(key_to_s, T_STRING);
+ generate_json(buffer, Vstate, state, key_to_s);
+ fbuffer_append(buffer, delim2, delim2_len);
+ generate_json(buffer, Vstate, state, rb_hash_aref(obj, key));
+ }
+ depth = --state->depth;
+ if (object_nl) {
+ fbuffer_append(buffer, object_nl, object_nl_len);
+ if (indent) {
+ for (j = 0; j < depth; j++) {
+ fbuffer_append(buffer, indent, indent_len);
+ }
+ }
+ }
+ fbuffer_append_char(buffer, '}');
+}
+
+static void generate_json_array(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj)
+{
+ char *array_nl = state->array_nl;
+ long array_nl_len = state->array_nl_len;
+ char *indent = state->indent;
+ long indent_len = state->indent_len;
+ long max_nesting = state->max_nesting;
+ char *delim = FBUFFER_PTR(state->array_delim);
+ long delim_len = FBUFFER_LEN(state->array_delim);
+ long depth = ++state->depth;
+ int i, j;
+ if (max_nesting != 0 && depth > max_nesting) {
+ fbuffer_free(buffer);
+ rb_raise(eNestingError, "nesting of %ld is too deep", --state->depth);
+ }
+ fbuffer_append_char(buffer, '[');
+ if (array_nl) fbuffer_append(buffer, array_nl, array_nl_len);
+ for(i = 0; i < RARRAY_LEN(obj); i++) {
+ if (i > 0) fbuffer_append(buffer, delim, delim_len);
+ if (indent) {
+ for (j = 0; j < depth; j++) {
+ fbuffer_append(buffer, indent, indent_len);
+ }
+ }
+ generate_json(buffer, Vstate, state, rb_ary_entry(obj, i));
+ }
+ state->depth = --depth;
+ if (array_nl) {
+ fbuffer_append(buffer, array_nl, array_nl_len);
+ if (indent) {
+ for (j = 0; j < depth; j++) {
+ fbuffer_append(buffer, indent, indent_len);
+ }
+ }
+ }
+ fbuffer_append_char(buffer, ']');
+}
+
+static void generate_json_string(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj)
+{
+ fbuffer_append_char(buffer, '"');
+#ifdef HAVE_RUBY_ENCODING_H
+ obj = rb_funcall(obj, i_encode, 1, CEncoding_UTF_8);
+#endif
+ if (state->ascii_only) {
+ convert_UTF8_to_JSON_ASCII(buffer, obj);
+ } else {
+ convert_UTF8_to_JSON(buffer, obj);
+ }
+ fbuffer_append_char(buffer, '"');
+}
+
+static void generate_json_null(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj)
+{
+ fbuffer_append(buffer, "null", 4);
+}
+
+static void generate_json_false(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj)
+{
+ fbuffer_append(buffer, "false", 5);
+}
+
+static void generate_json_true(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj)
+{
+ fbuffer_append(buffer, "true", 4);
+}
+
+static void generate_json_fixnum(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj)
+{
+ fbuffer_append_long(buffer, FIX2LONG(obj));
+}
+
+static void generate_json_bignum(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj)
+{
+ VALUE tmp = rb_funcall(obj, i_to_s, 0);
+ fbuffer_append_str(buffer, tmp);
+}
+
+static void generate_json_float(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj)
+{
+ double value = RFLOAT_VALUE(obj);
+ char allow_nan = state->allow_nan;
+ VALUE tmp = rb_funcall(obj, i_to_s, 0);
+ if (!allow_nan) {
+ if (isinf(value)) {
+ fbuffer_free(buffer);
+ rb_raise(eGeneratorError, "%u: %s not allowed in JSON", __LINE__, StringValueCStr(tmp));
+ } else if (isnan(value)) {
+ fbuffer_free(buffer);
+ rb_raise(eGeneratorError, "%u: %s not allowed in JSON", __LINE__, StringValueCStr(tmp));
+ }
+ }
+ fbuffer_append_str(buffer, tmp);
+}
+
+static void generate_json(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj)
+{
+ VALUE tmp;
+ VALUE klass = CLASS_OF(obj);
+ if (klass == rb_cHash) {
+ generate_json_object(buffer, Vstate, state, obj);
+ } else if (klass == rb_cArray) {
+ generate_json_array(buffer, Vstate, state, obj);
+ } else if (klass == rb_cString) {
+ generate_json_string(buffer, Vstate, state, obj);
+ } else if (obj == Qnil) {
+ generate_json_null(buffer, Vstate, state, obj);
+ } else if (obj == Qfalse) {
+ generate_json_false(buffer, Vstate, state, obj);
+ } else if (obj == Qtrue) {
+ generate_json_true(buffer, Vstate, state, obj);
+ } else if (klass == rb_cFixnum) {
+ generate_json_fixnum(buffer, Vstate, state, obj);
+ } else if (klass == rb_cBignum) {
+ generate_json_bignum(buffer, Vstate, state, obj);
+ } else if (klass == rb_cFloat) {
+ generate_json_float(buffer, Vstate, state, obj);
+ } else if (rb_respond_to(obj, i_to_json)) {
+ tmp = rb_funcall(obj, i_to_json, 1, Vstate);
+ Check_Type(tmp, T_STRING);
+ fbuffer_append_str(buffer, tmp);
+ } else {
+ tmp = rb_funcall(obj, i_to_s, 0);
+ Check_Type(tmp, T_STRING);
+ generate_json(buffer, Vstate, state, tmp);
+ }
+}
+
+static FBuffer *cState_prepare_buffer(VALUE self)
+{
+ FBuffer *buffer = fbuffer_alloc();
+ GET_STATE(self);
+
+ if (state->object_delim) {
+ fbuffer_clear(state->object_delim);
+ } else {
+ state->object_delim = fbuffer_alloc_with_length(16);
+ }
+ fbuffer_append_char(state->object_delim, ',');
+ if (state->object_delim2) {
+ fbuffer_clear(state->object_delim2);
+ } else {
+ state->object_delim2 = fbuffer_alloc_with_length(16);
+ }
+ fbuffer_append_char(state->object_delim2, ':');
+ if (state->space) fbuffer_append(state->object_delim2, state->space, state->space_len);
+
+ if (state->array_delim) {
+ fbuffer_clear(state->array_delim);
+ } else {
+ state->array_delim = fbuffer_alloc_with_length(16);
+ }
+ fbuffer_append_char(state->array_delim, ',');
+ if (state->array_nl) fbuffer_append(state->array_delim, state->array_nl, state->array_nl_len);
+ return buffer;
+}
+
+static VALUE fbuffer_to_s(FBuffer *fb)
+{
+ VALUE result = rb_str_new(FBUFFER_PAIR(fb));
+ fbuffer_free(fb);
+ FORCE_UTF8(result);
+ return result;
+}
+
+static VALUE cState_partial_generate(VALUE self, VALUE obj)
+{
+ FBuffer *buffer = cState_prepare_buffer(self);
+ GET_STATE(self);
+ generate_json(buffer, self, state, obj);
+ return fbuffer_to_s(buffer);
+}
+
+/*
+ * call-seq: generate(obj)
+ *
+ * Generates a valid JSON document from object +obj+ and returns the
+ * result. If no valid JSON document can be created this method raises a
+ * GeneratorError exception.
+ */
+static VALUE cState_generate(VALUE self, VALUE obj)
+{
+ VALUE result = cState_partial_generate(self, obj);
+ VALUE re, args[2];
+ GET_STATE(self);
+ if (!state->quirks_mode) {
+ args[0] = rb_str_new2("\\A\\s*(?:\\[.*\\]|\\{.*\\})\\s*\\Z");
+ args[1] = CRegexp_MULTILINE;
+ re = rb_class_new_instance(2, args, rb_cRegexp);
+ if (NIL_P(rb_funcall(re, i_match, 1, result))) {
+ rb_raise(eGeneratorError, "only generation of JSON objects or arrays allowed");
+ }
+ }
+ return result;
+}
+
+/*
+ * call-seq: new(opts = {})
+ *
+ * Instantiates a new State object, configured by _opts_.
+ *
+ * _opts_ can have the following keys:
+ *
+ * * *indent*: a string used to indent levels (default: ''),
+ * * *space*: a string that is put after, a : or , delimiter (default: ''),
+ * * *space_before*: a string that is put before a : pair delimiter (default: ''),
+ * * *object_nl*: a string that is put at the end of a JSON object (default: ''),
+ * * *array_nl*: a string that is put at the end of a JSON array (default: ''),
+ * * *allow_nan*: true if NaN, Infinity, and -Infinity should be
+ * generated, otherwise an exception is thrown, if these values are
+ * encountered. This options defaults to false.
+ * * *quirks_mode*: Enables quirks_mode for parser, that is for example
+ * generating single JSON values instead of documents is possible.
+ */
+static VALUE cState_initialize(int argc, VALUE *argv, VALUE self)
+{
+ VALUE opts;
+ GET_STATE(self);
+ state->max_nesting = 19;
+ rb_scan_args(argc, argv, "01", &opts);
+ if (!NIL_P(opts)) cState_configure(self, opts);
+ return self;
+}
+
+/*
+ * call-seq: initialize_copy(orig)
+ *
+ * Initializes this object from orig if it to be duplicated/cloned and returns
+ * it.
+*/
+static VALUE cState_init_copy(VALUE obj, VALUE orig)
+{
+ JSON_Generator_State *objState, *origState;
+
+ Data_Get_Struct(obj, JSON_Generator_State, objState);
+ Data_Get_Struct(orig, JSON_Generator_State, origState);
+ if (!objState) rb_raise(rb_eArgError, "unallocated JSON::State");
+
+ MEMCPY(objState, origState, JSON_Generator_State, 1);
+ objState->indent = fstrndup(origState->indent, origState->indent_len);
+ objState->space = fstrndup(origState->space, origState->space_len);
+ objState->space_before = fstrndup(origState->space_before, origState->space_before_len);
+ objState->object_nl = fstrndup(origState->object_nl, origState->object_nl_len);
+ objState->array_nl = fstrndup(origState->array_nl, origState->array_nl_len);
+ if (origState->array_delim) objState->array_delim = fbuffer_dup(origState->array_delim);
+ if (origState->object_delim) objState->object_delim = fbuffer_dup(origState->object_delim);
+ if (origState->object_delim2) objState->object_delim2 = fbuffer_dup(origState->object_delim2);
+ return obj;
+}
+
+/*
+ * call-seq: from_state(opts)
+ *
+ * Creates a State object from _opts_, which ought to be Hash to create a
+ * new State instance configured by _opts_, something else to create an
+ * unconfigured instance. If _opts_ is a State object, it is just returned.
+ */
+static VALUE cState_from_state_s(VALUE self, VALUE opts)
+{
+ if (rb_obj_is_kind_of(opts, self)) {
+ return opts;
+ } else if (rb_obj_is_kind_of(opts, rb_cHash)) {
+ return rb_funcall(self, i_new, 1, opts);
+ } else {
+ if (NIL_P(CJSON_SAFE_STATE_PROTOTYPE)) {
+ CJSON_SAFE_STATE_PROTOTYPE = rb_const_get(mJSON, i_SAFE_STATE_PROTOTYPE);
+ }
+ return rb_funcall(CJSON_SAFE_STATE_PROTOTYPE, i_dup, 0);
+ }
+}
+
+/*
+ * call-seq: indent()
+ *
+ * This string is used to indent levels in the JSON text.
+ */
+static VALUE cState_indent(VALUE self)
+{
+ GET_STATE(self);
+ return state->indent ? rb_str_new2(state->indent) : rb_str_new2("");
+}
+
+/*
+ * call-seq: indent=(indent)
+ *
+ * This string is used to indent levels in the JSON text.
+ */
+static VALUE cState_indent_set(VALUE self, VALUE indent)
+{
+ unsigned long len;
+ GET_STATE(self);
+ Check_Type(indent, T_STRING);
+ len = RSTRING_LEN(indent);
+ if (len == 0) {
+ if (state->indent) {
+ ruby_xfree(state->indent);
+ state->indent = NULL;
+ state->indent_len = 0;
+ }
+ } else {
+ if (state->indent) ruby_xfree(state->indent);
+ state->indent = strdup(RSTRING_PTR(indent));
+ state->indent_len = len;
+ }
+ return Qnil;
+}
+
+/*
+ * call-seq: space()
+ *
+ * This string is used to insert a space between the tokens in a JSON
+ * string.
+ */
+static VALUE cState_space(VALUE self)
+{
+ GET_STATE(self);
+ return state->space ? rb_str_new2(state->space) : rb_str_new2("");
+}
+
+/*
+ * call-seq: space=(space)
+ *
+ * This string is used to insert a space between the tokens in a JSON
+ * string.
+ */
+static VALUE cState_space_set(VALUE self, VALUE space)
+{
+ unsigned long len;
+ GET_STATE(self);
+ Check_Type(space, T_STRING);
+ len = RSTRING_LEN(space);
+ if (len == 0) {
+ if (state->space) {
+ ruby_xfree(state->space);
+ state->space = NULL;
+ state->space_len = 0;
+ }
+ } else {
+ if (state->space) ruby_xfree(state->space);
+ state->space = strdup(RSTRING_PTR(space));
+ state->space_len = len;
+ }
+ return Qnil;
+}
+
+/*
+ * call-seq: space_before()
+ *
+ * This string is used to insert a space before the ':' in JSON objects.
+ */
+static VALUE cState_space_before(VALUE self)
+{
+ GET_STATE(self);
+ return state->space_before ? rb_str_new2(state->space_before) : rb_str_new2("");
+}
+
+/*
+ * call-seq: space_before=(space_before)
+ *
+ * This string is used to insert a space before the ':' in JSON objects.
+ */
+static VALUE cState_space_before_set(VALUE self, VALUE space_before)
+{
+ unsigned long len;
+ GET_STATE(self);
+ Check_Type(space_before, T_STRING);
+ len = RSTRING_LEN(space_before);
+ if (len == 0) {
+ if (state->space_before) {
+ ruby_xfree(state->space_before);
+ state->space_before = NULL;
+ state->space_before_len = 0;
+ }
+ } else {
+ if (state->space_before) ruby_xfree(state->space_before);
+ state->space_before = strdup(RSTRING_PTR(space_before));
+ state->space_before_len = len;
+ }
+ return Qnil;
+}
+
+/*
+ * call-seq: object_nl()
+ *
+ * This string is put at the end of a line that holds a JSON object (or
+ * Hash).
+ */
+static VALUE cState_object_nl(VALUE self)
+{
+ GET_STATE(self);
+ return state->object_nl ? rb_str_new2(state->object_nl) : rb_str_new2("");
+}
+
+/*
+ * call-seq: object_nl=(object_nl)
+ *
+ * This string is put at the end of a line that holds a JSON object (or
+ * Hash).
+ */
+static VALUE cState_object_nl_set(VALUE self, VALUE object_nl)
+{
+ unsigned long len;
+ GET_STATE(self);
+ Check_Type(object_nl, T_STRING);
+ len = RSTRING_LEN(object_nl);
+ if (len == 0) {
+ if (state->object_nl) {
+ ruby_xfree(state->object_nl);
+ state->object_nl = NULL;
+ }
+ } else {
+ if (state->object_nl) ruby_xfree(state->object_nl);
+ state->object_nl = strdup(RSTRING_PTR(object_nl));
+ state->object_nl_len = len;
+ }
+ return Qnil;
+}
+
+/*
+ * call-seq: array_nl()
+ *
+ * This string is put at the end of a line that holds a JSON array.
+ */
+static VALUE cState_array_nl(VALUE self)
+{
+ GET_STATE(self);
+ return state->array_nl ? rb_str_new2(state->array_nl) : rb_str_new2("");
+}
+
+/*
+ * call-seq: array_nl=(array_nl)
+ *
+ * This string is put at the end of a line that holds a JSON array.
+ */
+static VALUE cState_array_nl_set(VALUE self, VALUE array_nl)
+{
+ unsigned long len;
+ GET_STATE(self);
+ Check_Type(array_nl, T_STRING);
+ len = RSTRING_LEN(array_nl);
+ if (len == 0) {
+ if (state->array_nl) {
+ ruby_xfree(state->array_nl);
+ state->array_nl = NULL;
+ }
+ } else {
+ if (state->array_nl) ruby_xfree(state->array_nl);
+ state->array_nl = strdup(RSTRING_PTR(array_nl));
+ state->array_nl_len = len;
+ }
+ return Qnil;
+}
+
+
+/*
+* call-seq: check_circular?
+*
+* Returns true, if circular data structures should be checked,
+* otherwise returns false.
+*/
+static VALUE cState_check_circular_p(VALUE self)
+{
+ GET_STATE(self);
+ return state->max_nesting ? Qtrue : Qfalse;
+}
+
+/*
+ * call-seq: max_nesting
+ *
+ * This integer returns the maximum level of data structure nesting in
+ * the generated JSON, max_nesting = 0 if no maximum is checked.
+ */
+static VALUE cState_max_nesting(VALUE self)
+{
+ GET_STATE(self);
+ return LONG2FIX(state->max_nesting);
+}
+
+/*
+ * call-seq: max_nesting=(depth)
+ *
+ * This sets the maximum level of data structure nesting in the generated JSON
+ * to the integer depth, max_nesting = 0 if no maximum should be checked.
+ */
+static VALUE cState_max_nesting_set(VALUE self, VALUE depth)
+{
+ GET_STATE(self);
+ Check_Type(depth, T_FIXNUM);
+ return state->max_nesting = FIX2LONG(depth);
+}
+
+/*
+ * call-seq: allow_nan?
+ *
+ * Returns true, if NaN, Infinity, and -Infinity should be generated, otherwise
+ * returns false.
+ */
+static VALUE cState_allow_nan_p(VALUE self)
+{
+ GET_STATE(self);
+ return state->allow_nan ? Qtrue : Qfalse;
+}
+
+/*
+ * call-seq: ascii_only?
+ *
+ * Returns true, if NaN, Infinity, and -Infinity should be generated, otherwise
+ * returns false.
+ */
+static VALUE cState_ascii_only_p(VALUE self)
+{
+ GET_STATE(self);
+ return state->ascii_only ? Qtrue : Qfalse;
+}
+
+/*
+ * call-seq: quirks_mode?
+ *
+ * Returns true, if quirks mode is enabled. Otherwise returns false.
+ */
+static VALUE cState_quirks_mode_p(VALUE self)
+{
+ GET_STATE(self);
+ return state->quirks_mode ? Qtrue : Qfalse;
+}
+
+/*
+ * call-seq: quirks_mode=(enable)
+ *
+ * If set to true, enables the quirks_mode mode.
+ */
+static VALUE cState_quirks_mode_set(VALUE self, VALUE enable)
+{
+ GET_STATE(self);
+ state->quirks_mode = RTEST(enable);
+ return Qnil;
+}
+
+/*
+ * call-seq: depth
+ *
+ * This integer returns the current depth of data structure nesting.
+ */
+static VALUE cState_depth(VALUE self)
+{
+ GET_STATE(self);
+ return LONG2FIX(state->depth);
+}
+
+/*
+ * call-seq: depth=(depth)
+ *
+ * This sets the maximum level of data structure nesting in the generated JSON
+ * to the integer depth, max_nesting = 0 if no maximum should be checked.
+ */
+static VALUE cState_depth_set(VALUE self, VALUE depth)
+{
+ GET_STATE(self);
+ Check_Type(depth, T_FIXNUM);
+ return state->depth = FIX2LONG(depth);
+}
+
+/*
+ *
+ */
+void Init_generator()
+{
+ rb_require("json/common");
+
+ mJSON = rb_define_module("JSON");
+ mExt = rb_define_module_under(mJSON, "Ext");
+ mGenerator = rb_define_module_under(mExt, "Generator");
+
+ eGeneratorError = rb_path2class("JSON::GeneratorError");
+ eNestingError = rb_path2class("JSON::NestingError");
+
+ cState = rb_define_class_under(mGenerator, "State", rb_cObject);
+ rb_define_alloc_func(cState, cState_s_allocate);
+ rb_define_singleton_method(cState, "from_state", cState_from_state_s, 1);
+ rb_define_method(cState, "initialize", cState_initialize, -1);
+ rb_define_method(cState, "initialize_copy", cState_init_copy, 1);
+ rb_define_method(cState, "indent", cState_indent, 0);
+ rb_define_method(cState, "indent=", cState_indent_set, 1);
+ rb_define_method(cState, "space", cState_space, 0);
+ rb_define_method(cState, "space=", cState_space_set, 1);
+ rb_define_method(cState, "space_before", cState_space_before, 0);
+ rb_define_method(cState, "space_before=", cState_space_before_set, 1);
+ rb_define_method(cState, "object_nl", cState_object_nl, 0);
+ rb_define_method(cState, "object_nl=", cState_object_nl_set, 1);
+ rb_define_method(cState, "array_nl", cState_array_nl, 0);
+ rb_define_method(cState, "array_nl=", cState_array_nl_set, 1);
+ rb_define_method(cState, "max_nesting", cState_max_nesting, 0);
+ rb_define_method(cState, "max_nesting=", cState_max_nesting_set, 1);
+ rb_define_method(cState, "check_circular?", cState_check_circular_p, 0);
+ rb_define_method(cState, "allow_nan?", cState_allow_nan_p, 0);
+ rb_define_method(cState, "ascii_only?", cState_ascii_only_p, 0);
+ rb_define_method(cState, "quirks_mode?", cState_quirks_mode_p, 0);
+ rb_define_method(cState, "quirks_mode", cState_quirks_mode_p, 0);
+ rb_define_method(cState, "quirks_mode=", cState_quirks_mode_set, 1);
+ rb_define_method(cState, "depth", cState_depth, 0);
+ rb_define_method(cState, "depth=", cState_depth_set, 1);
+ rb_define_method(cState, "configure", cState_configure, 1);
+ rb_define_alias(cState, "merge", "configure");
+ rb_define_method(cState, "to_h", cState_to_h, 0);
+ rb_define_method(cState, "[]", cState_aref, 1);
+ rb_define_method(cState, "generate", cState_generate, 1);
+
+ mGeneratorMethods = rb_define_module_under(mGenerator, "GeneratorMethods");
+ mObject = rb_define_module_under(mGeneratorMethods, "Object");
+ rb_define_method(mObject, "to_json", mObject_to_json, -1);
+ mHash = rb_define_module_under(mGeneratorMethods, "Hash");
+ rb_define_method(mHash, "to_json", mHash_to_json, -1);
+ mArray = rb_define_module_under(mGeneratorMethods, "Array");
+ rb_define_method(mArray, "to_json", mArray_to_json, -1);
+ mFixnum = rb_define_module_under(mGeneratorMethods, "Fixnum");
+ rb_define_method(mFixnum, "to_json", mFixnum_to_json, -1);
+ mBignum = rb_define_module_under(mGeneratorMethods, "Bignum");
+ rb_define_method(mBignum, "to_json", mBignum_to_json, -1);
+ mFloat = rb_define_module_under(mGeneratorMethods, "Float");
+ rb_define_method(mFloat, "to_json", mFloat_to_json, -1);
+ mString = rb_define_module_under(mGeneratorMethods, "String");
+ rb_define_singleton_method(mString, "included", mString_included_s, 1);
+ rb_define_method(mString, "to_json", mString_to_json, -1);
+ rb_define_method(mString, "to_json_raw", mString_to_json_raw, -1);
+ rb_define_method(mString, "to_json_raw_object", mString_to_json_raw_object, 0);
+ mString_Extend = rb_define_module_under(mString, "Extend");
+ rb_define_method(mString_Extend, "json_create", mString_Extend_json_create, 1);
+ mTrueClass = rb_define_module_under(mGeneratorMethods, "TrueClass");
+ rb_define_method(mTrueClass, "to_json", mTrueClass_to_json, -1);
+ mFalseClass = rb_define_module_under(mGeneratorMethods, "FalseClass");
+ rb_define_method(mFalseClass, "to_json", mFalseClass_to_json, -1);
+ mNilClass = rb_define_module_under(mGeneratorMethods, "NilClass");
+ rb_define_method(mNilClass, "to_json", mNilClass_to_json, -1);
+
+ CRegexp_MULTILINE = rb_const_get(rb_cRegexp, rb_intern("MULTILINE"));
+ i_to_s = rb_intern("to_s");
+ i_to_json = rb_intern("to_json");
+ i_new = rb_intern("new");
+ i_indent = rb_intern("indent");
+ i_space = rb_intern("space");
+ i_space_before = rb_intern("space_before");
+ i_object_nl = rb_intern("object_nl");
+ i_array_nl = rb_intern("array_nl");
+ i_max_nesting = rb_intern("max_nesting");
+ i_allow_nan = rb_intern("allow_nan");
+ i_ascii_only = rb_intern("ascii_only");
+ i_quirks_mode = rb_intern("quirks_mode");
+ i_depth = rb_intern("depth");
+ i_pack = rb_intern("pack");
+ i_unpack = rb_intern("unpack");
+ i_create_id = rb_intern("create_id");
+ i_extend = rb_intern("extend");
+ i_key_p = rb_intern("key?");
+ i_aref = rb_intern("[]");
+ i_send = rb_intern("__send__");
+ i_respond_to_p = rb_intern("respond_to?");
+ i_match = rb_intern("match");
+ i_keys = rb_intern("keys");
+ i_dup = rb_intern("dup");
+#ifdef HAVE_RUBY_ENCODING_H
+ CEncoding_UTF_8 = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-8"));
+ i_encoding = rb_intern("encoding");
+ i_encode = rb_intern("encode");
+#endif
+ i_SAFE_STATE_PROTOTYPE = rb_intern("SAFE_STATE_PROTOTYPE");
+ CJSON_SAFE_STATE_PROTOTYPE = Qnil;
+}
--- /dev/null
+#ifndef _GENERATOR_H_
+#define _GENERATOR_H_
+
+#include <string.h>
+#include <assert.h>
+#include <math.h>
+
+#include "ruby.h"
+
+#if HAVE_RUBY_RE_H
+#include "ruby/re.h"
+#endif
+
+#if HAVE_RE_H
+#include "re.h"
+#endif
+
+#ifdef HAVE_RUBY_ENCODING_H
+#include "ruby/encoding.h"
+#define FORCE_UTF8(obj) rb_enc_associate((obj), rb_utf8_encoding())
+#else
+#define FORCE_UTF8(obj)
+#endif
+
+#define option_given_p(opts, key) RTEST(rb_funcall(opts, i_key_p, 1, key))
+
+#ifndef RHASH_SIZE
+#define RHASH_SIZE(hsh) (RHASH(hsh)->tbl->num_entries)
+#endif
+
+#ifndef RFLOAT_VALUE
+#define RFLOAT_VALUE(val) (RFLOAT(val)->value)
+#endif
+
+#ifndef RARRAY_PTR
+#define RARRAY_PTR(ARRAY) RARRAY(ARRAY)->ptr
+#endif
+#ifndef RARRAY_LEN
+#define RARRAY_LEN(ARRAY) RARRAY(ARRAY)->len
+#endif
+#ifndef RSTRING_PTR
+#define RSTRING_PTR(string) RSTRING(string)->ptr
+#endif
+#ifndef RSTRING_LEN
+#define RSTRING_LEN(string) RSTRING(string)->len
+#endif
+
+/* We don't need to guard objects for rbx, so let's do nothing at all. */
+#ifndef RB_GC_GUARD
+#define RB_GC_GUARD(object)
+#endif
+
+/* fbuffer implementation */
+
+typedef struct FBufferStruct {
+ unsigned long initial_length;
+ char *ptr;
+ unsigned long len;
+ unsigned long capa;
+} FBuffer;
+
+#define FBUFFER_INITIAL_LENGTH 4096
+
+#define FBUFFER_PTR(fb) (fb->ptr)
+#define FBUFFER_LEN(fb) (fb->len)
+#define FBUFFER_CAPA(fb) (fb->capa)
+#define FBUFFER_PAIR(fb) FBUFFER_PTR(fb), FBUFFER_LEN(fb)
+
+static char *fstrndup(const char *ptr, unsigned long len);
+static FBuffer *fbuffer_alloc();
+static FBuffer *fbuffer_alloc_with_length(unsigned long initial_length);
+static void fbuffer_free(FBuffer *fb);
+static void fbuffer_clear(FBuffer *fb);
+static void fbuffer_append(FBuffer *fb, const char *newstr, unsigned long len);
+static void fbuffer_append_long(FBuffer *fb, long number);
+static void fbuffer_append_char(FBuffer *fb, char newchr);
+static FBuffer *fbuffer_dup(FBuffer *fb);
+static VALUE fbuffer_to_s(FBuffer *fb);
+
+/* unicode defintions */
+
+#define UNI_STRICT_CONVERSION 1
+
+typedef unsigned long UTF32; /* at least 32 bits */
+typedef unsigned short UTF16; /* at least 16 bits */
+typedef unsigned char UTF8; /* typically 8 bits */
+
+#define UNI_REPLACEMENT_CHAR (UTF32)0x0000FFFD
+#define UNI_MAX_BMP (UTF32)0x0000FFFF
+#define UNI_MAX_UTF16 (UTF32)0x0010FFFF
+#define UNI_MAX_UTF32 (UTF32)0x7FFFFFFF
+#define UNI_MAX_LEGAL_UTF32 (UTF32)0x0010FFFF
+
+#define UNI_SUR_HIGH_START (UTF32)0xD800
+#define UNI_SUR_HIGH_END (UTF32)0xDBFF
+#define UNI_SUR_LOW_START (UTF32)0xDC00
+#define UNI_SUR_LOW_END (UTF32)0xDFFF
+
+static const int halfShift = 10; /* used for shifting by 10 bits */
+
+static const UTF32 halfBase = 0x0010000UL;
+static const UTF32 halfMask = 0x3FFUL;
+
+static unsigned char isLegalUTF8(const UTF8 *source, unsigned long length);
+static void unicode_escape(char *buf, UTF16 character);
+static void unicode_escape_to_buffer(FBuffer *buffer, char buf[6], UTF16 character);
+static void convert_UTF8_to_JSON_ASCII(FBuffer *buffer, VALUE string);
+static void convert_UTF8_to_JSON(FBuffer *buffer, VALUE string);
+
+/* ruby api and some helpers */
+
+typedef struct JSON_Generator_StateStruct {
+ char *indent;
+ long indent_len;
+ char *space;
+ long space_len;
+ char *space_before;
+ long space_before_len;
+ char *object_nl;
+ long object_nl_len;
+ char *array_nl;
+ long array_nl_len;
+ FBuffer *array_delim;
+ FBuffer *object_delim;
+ FBuffer *object_delim2;
+ long max_nesting;
+ char allow_nan;
+ char ascii_only;
+ char quirks_mode;
+ long depth;
+} JSON_Generator_State;
+
+#define GET_STATE(self) \
+ JSON_Generator_State *state; \
+ Data_Get_Struct(self, JSON_Generator_State, state)
+
+#define GENERATE_JSON(type) \
+ FBuffer *buffer; \
+ VALUE Vstate; \
+ JSON_Generator_State *state; \
+ \
+ rb_scan_args(argc, argv, "01", &Vstate); \
+ Vstate = cState_from_state_s(cState, Vstate); \
+ Data_Get_Struct(Vstate, JSON_Generator_State, state); \
+ buffer = cState_prepare_buffer(Vstate); \
+ generate_json_##type(buffer, Vstate, state, self); \
+ return fbuffer_to_s(buffer)
+
+static VALUE mHash_to_json(int argc, VALUE *argv, VALUE self);
+static VALUE mArray_to_json(int argc, VALUE *argv, VALUE self);
+static VALUE mFixnum_to_json(int argc, VALUE *argv, VALUE self);
+static VALUE mBignum_to_json(int argc, VALUE *argv, VALUE self);
+static VALUE mFloat_to_json(int argc, VALUE *argv, VALUE self);
+static VALUE mString_included_s(VALUE self, VALUE modul);
+static VALUE mString_to_json(int argc, VALUE *argv, VALUE self);
+static VALUE mString_to_json_raw_object(VALUE self);
+static VALUE mString_to_json_raw(int argc, VALUE *argv, VALUE self);
+static VALUE mString_Extend_json_create(VALUE self, VALUE o);
+static VALUE mTrueClass_to_json(int argc, VALUE *argv, VALUE self);
+static VALUE mFalseClass_to_json(int argc, VALUE *argv, VALUE self);
+static VALUE mNilClass_to_json(int argc, VALUE *argv, VALUE self);
+static VALUE mObject_to_json(int argc, VALUE *argv, VALUE self);
+static void State_free(JSON_Generator_State *state);
+static JSON_Generator_State *State_allocate();
+static VALUE cState_s_allocate(VALUE klass);
+static VALUE cState_configure(VALUE self, VALUE opts);
+static VALUE cState_to_h(VALUE self);
+static void generate_json(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj);
+static void generate_json_object(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj);
+static void generate_json_array(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj);
+static void generate_json_string(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj);
+static void generate_json_null(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj);
+static void generate_json_false(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj);
+static void generate_json_true(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj);
+static void generate_json_fixnum(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj);
+static void generate_json_bignum(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj);
+static void generate_json_float(FBuffer *buffer, VALUE Vstate, JSON_Generator_State *state, VALUE obj);
+static VALUE cState_partial_generate(VALUE self, VALUE obj);
+static VALUE cState_generate(VALUE self, VALUE obj);
+static VALUE cState_initialize(int argc, VALUE *argv, VALUE self);
+static VALUE cState_from_state_s(VALUE self, VALUE opts);
+static VALUE cState_indent(VALUE self);
+static VALUE cState_indent_set(VALUE self, VALUE indent);
+static VALUE cState_space(VALUE self);
+static VALUE cState_space_set(VALUE self, VALUE space);
+static VALUE cState_space_before(VALUE self);
+static VALUE cState_space_before_set(VALUE self, VALUE space_before);
+static VALUE cState_object_nl(VALUE self);
+static VALUE cState_object_nl_set(VALUE self, VALUE object_nl);
+static VALUE cState_array_nl(VALUE self);
+static VALUE cState_array_nl_set(VALUE self, VALUE array_nl);
+static VALUE cState_max_nesting(VALUE self);
+static VALUE cState_max_nesting_set(VALUE self, VALUE depth);
+static VALUE cState_allow_nan_p(VALUE self);
+static VALUE cState_ascii_only_p(VALUE self);
+static VALUE cState_depth(VALUE self);
+static VALUE cState_depth_set(VALUE self, VALUE depth);
+static FBuffer *cState_prepare_buffer(VALUE self);
+
+#endif
--- /dev/null
+require 'mkmf'
+require 'rbconfig'
+
+unless $CFLAGS.gsub!(/ -O[\dsz]?/, ' -O3')
+ $CFLAGS << ' -O3'
+end
+if CONFIG['CC'] =~ /gcc/
+ $CFLAGS << ' -Wall'
+ #unless $CFLAGS.gsub!(/ -O[\dsz]?/, ' -O0 -ggdb')
+ # $CFLAGS << ' -O0 -ggdb'
+ #end
+end
+
+have_header("re.h")
+have_header("ruby/st.h")
+create_makefile 'json/ext/parser'
--- /dev/null
+
+#line 1 "parser.rl"
+#include "parser.h"
+
+/* unicode */
+
+static const char digit_values[256] = {
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -1,
+ -1, -1, -1, -1, -1, -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1
+};
+
+static UTF32 unescape_unicode(const unsigned char *p)
+{
+ char b;
+ UTF32 result = 0;
+ b = digit_values[p[0]];
+ if (b < 0) return UNI_REPLACEMENT_CHAR;
+ result = (result << 4) | b;
+ b = digit_values[p[1]];
+ result = (result << 4) | b;
+ if (b < 0) return UNI_REPLACEMENT_CHAR;
+ b = digit_values[p[2]];
+ result = (result << 4) | b;
+ if (b < 0) return UNI_REPLACEMENT_CHAR;
+ b = digit_values[p[3]];
+ result = (result << 4) | b;
+ if (b < 0) return UNI_REPLACEMENT_CHAR;
+ return result;
+}
+
+static int convert_UTF32_to_UTF8(char *buf, UTF32 ch)
+{
+ int len = 1;
+ if (ch <= 0x7F) {
+ buf[0] = (char) ch;
+ } else if (ch <= 0x07FF) {
+ buf[0] = (char) ((ch >> 6) | 0xC0);
+ buf[1] = (char) ((ch & 0x3F) | 0x80);
+ len++;
+ } else if (ch <= 0xFFFF) {
+ buf[0] = (char) ((ch >> 12) | 0xE0);
+ buf[1] = (char) (((ch >> 6) & 0x3F) | 0x80);
+ buf[2] = (char) ((ch & 0x3F) | 0x80);
+ len += 2;
+ } else if (ch <= 0x1fffff) {
+ buf[0] =(char) ((ch >> 18) | 0xF0);
+ buf[1] =(char) (((ch >> 12) & 0x3F) | 0x80);
+ buf[2] =(char) (((ch >> 6) & 0x3F) | 0x80);
+ buf[3] =(char) ((ch & 0x3F) | 0x80);
+ len += 3;
+ } else {
+ buf[0] = '?';
+ }
+ return len;
+}
+
+#ifdef HAVE_RUBY_ENCODING_H
+static VALUE CEncoding_ASCII_8BIT, CEncoding_UTF_8, CEncoding_UTF_16BE,
+ CEncoding_UTF_16LE, CEncoding_UTF_32BE, CEncoding_UTF_32LE;
+static ID i_encoding, i_encode;
+#else
+static ID i_iconv;
+#endif
+
+static VALUE mJSON, mExt, cParser, eParserError, eNestingError;
+static VALUE CNaN, CInfinity, CMinusInfinity;
+
+static ID i_json_creatable_p, i_json_create, i_create_id, i_create_additions,
+ i_chr, i_max_nesting, i_allow_nan, i_symbolize_names, i_quirks_mode,
+ i_object_class, i_array_class, i_key_p, i_deep_const_get, i_match,
+ i_match_string, i_aset, i_leftshift;
+
+
+#line 109 "parser.rl"
+
+
+
+#line 91 "parser.c"
+static const int JSON_object_start = 1;
+static const int JSON_object_first_final = 27;
+static const int JSON_object_error = 0;
+
+static const int JSON_object_en_main = 1;
+
+
+#line 150 "parser.rl"
+
+
+static char *JSON_parse_object(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+ VALUE last_name = Qnil;
+ VALUE object_class = json->object_class;
+
+ if (json->max_nesting && json->current_nesting > json->max_nesting) {
+ rb_raise(eNestingError, "nesting of %d is too deep", json->current_nesting);
+ }
+
+ *result = NIL_P(object_class) ? rb_hash_new() : rb_class_new_instance(0, 0, object_class);
+
+
+#line 115 "parser.c"
+ {
+ cs = JSON_object_start;
+ }
+
+#line 165 "parser.rl"
+
+#line 122 "parser.c"
+ {
+ if ( p == pe )
+ goto _test_eof;
+ switch ( cs )
+ {
+case 1:
+ if ( (*p) == 123 )
+ goto st2;
+ goto st0;
+st0:
+cs = 0;
+ goto _out;
+st2:
+ if ( ++p == pe )
+ goto _test_eof2;
+case 2:
+ switch( (*p) ) {
+ case 13: goto st2;
+ case 32: goto st2;
+ case 34: goto tr2;
+ case 47: goto st23;
+ case 125: goto tr4;
+ }
+ if ( 9 <= (*p) && (*p) <= 10 )
+ goto st2;
+ goto st0;
+tr2:
+#line 132 "parser.rl"
+ {
+ char *np;
+ json->parsing_name = 1;
+ np = JSON_parse_string(json, p, pe, &last_name);
+ json->parsing_name = 0;
+ if (np == NULL) { p--; {p++; cs = 3; goto _out;} } else {p = (( np))-1;}
+ }
+ goto st3;
+st3:
+ if ( ++p == pe )
+ goto _test_eof3;
+case 3:
+#line 163 "parser.c"
+ switch( (*p) ) {
+ case 13: goto st3;
+ case 32: goto st3;
+ case 47: goto st4;
+ case 58: goto st8;
+ }
+ if ( 9 <= (*p) && (*p) <= 10 )
+ goto st3;
+ goto st0;
+st4:
+ if ( ++p == pe )
+ goto _test_eof4;
+case 4:
+ switch( (*p) ) {
+ case 42: goto st5;
+ case 47: goto st7;
+ }
+ goto st0;
+st5:
+ if ( ++p == pe )
+ goto _test_eof5;
+case 5:
+ if ( (*p) == 42 )
+ goto st6;
+ goto st5;
+st6:
+ if ( ++p == pe )
+ goto _test_eof6;
+case 6:
+ switch( (*p) ) {
+ case 42: goto st6;
+ case 47: goto st3;
+ }
+ goto st5;
+st7:
+ if ( ++p == pe )
+ goto _test_eof7;
+case 7:
+ if ( (*p) == 10 )
+ goto st3;
+ goto st7;
+st8:
+ if ( ++p == pe )
+ goto _test_eof8;
+case 8:
+ switch( (*p) ) {
+ case 13: goto st8;
+ case 32: goto st8;
+ case 34: goto tr11;
+ case 45: goto tr11;
+ case 47: goto st19;
+ case 73: goto tr11;
+ case 78: goto tr11;
+ case 91: goto tr11;
+ case 102: goto tr11;
+ case 110: goto tr11;
+ case 116: goto tr11;
+ case 123: goto tr11;
+ }
+ if ( (*p) > 10 ) {
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto tr11;
+ } else if ( (*p) >= 9 )
+ goto st8;
+ goto st0;
+tr11:
+#line 117 "parser.rl"
+ {
+ VALUE v = Qnil;
+ char *np = JSON_parse_value(json, p, pe, &v);
+ if (np == NULL) {
+ p--; {p++; cs = 9; goto _out;}
+ } else {
+ if (NIL_P(json->object_class)) {
+ rb_hash_aset(*result, last_name, v);
+ } else {
+ rb_funcall(*result, i_aset, 2, last_name, v);
+ }
+ {p = (( np))-1;}
+ }
+ }
+ goto st9;
+st9:
+ if ( ++p == pe )
+ goto _test_eof9;
+case 9:
+#line 250 "parser.c"
+ switch( (*p) ) {
+ case 13: goto st9;
+ case 32: goto st9;
+ case 44: goto st10;
+ case 47: goto st15;
+ case 125: goto tr4;
+ }
+ if ( 9 <= (*p) && (*p) <= 10 )
+ goto st9;
+ goto st0;
+st10:
+ if ( ++p == pe )
+ goto _test_eof10;
+case 10:
+ switch( (*p) ) {
+ case 13: goto st10;
+ case 32: goto st10;
+ case 34: goto tr2;
+ case 47: goto st11;
+ }
+ if ( 9 <= (*p) && (*p) <= 10 )
+ goto st10;
+ goto st0;
+st11:
+ if ( ++p == pe )
+ goto _test_eof11;
+case 11:
+ switch( (*p) ) {
+ case 42: goto st12;
+ case 47: goto st14;
+ }
+ goto st0;
+st12:
+ if ( ++p == pe )
+ goto _test_eof12;
+case 12:
+ if ( (*p) == 42 )
+ goto st13;
+ goto st12;
+st13:
+ if ( ++p == pe )
+ goto _test_eof13;
+case 13:
+ switch( (*p) ) {
+ case 42: goto st13;
+ case 47: goto st10;
+ }
+ goto st12;
+st14:
+ if ( ++p == pe )
+ goto _test_eof14;
+case 14:
+ if ( (*p) == 10 )
+ goto st10;
+ goto st14;
+st15:
+ if ( ++p == pe )
+ goto _test_eof15;
+case 15:
+ switch( (*p) ) {
+ case 42: goto st16;
+ case 47: goto st18;
+ }
+ goto st0;
+st16:
+ if ( ++p == pe )
+ goto _test_eof16;
+case 16:
+ if ( (*p) == 42 )
+ goto st17;
+ goto st16;
+st17:
+ if ( ++p == pe )
+ goto _test_eof17;
+case 17:
+ switch( (*p) ) {
+ case 42: goto st17;
+ case 47: goto st9;
+ }
+ goto st16;
+st18:
+ if ( ++p == pe )
+ goto _test_eof18;
+case 18:
+ if ( (*p) == 10 )
+ goto st9;
+ goto st18;
+tr4:
+#line 140 "parser.rl"
+ { p--; {p++; cs = 27; goto _out;} }
+ goto st27;
+st27:
+ if ( ++p == pe )
+ goto _test_eof27;
+case 27:
+#line 346 "parser.c"
+ goto st0;
+st19:
+ if ( ++p == pe )
+ goto _test_eof19;
+case 19:
+ switch( (*p) ) {
+ case 42: goto st20;
+ case 47: goto st22;
+ }
+ goto st0;
+st20:
+ if ( ++p == pe )
+ goto _test_eof20;
+case 20:
+ if ( (*p) == 42 )
+ goto st21;
+ goto st20;
+st21:
+ if ( ++p == pe )
+ goto _test_eof21;
+case 21:
+ switch( (*p) ) {
+ case 42: goto st21;
+ case 47: goto st8;
+ }
+ goto st20;
+st22:
+ if ( ++p == pe )
+ goto _test_eof22;
+case 22:
+ if ( (*p) == 10 )
+ goto st8;
+ goto st22;
+st23:
+ if ( ++p == pe )
+ goto _test_eof23;
+case 23:
+ switch( (*p) ) {
+ case 42: goto st24;
+ case 47: goto st26;
+ }
+ goto st0;
+st24:
+ if ( ++p == pe )
+ goto _test_eof24;
+case 24:
+ if ( (*p) == 42 )
+ goto st25;
+ goto st24;
+st25:
+ if ( ++p == pe )
+ goto _test_eof25;
+case 25:
+ switch( (*p) ) {
+ case 42: goto st25;
+ case 47: goto st2;
+ }
+ goto st24;
+st26:
+ if ( ++p == pe )
+ goto _test_eof26;
+case 26:
+ if ( (*p) == 10 )
+ goto st2;
+ goto st26;
+ }
+ _test_eof2: cs = 2; goto _test_eof;
+ _test_eof3: cs = 3; goto _test_eof;
+ _test_eof4: cs = 4; goto _test_eof;
+ _test_eof5: cs = 5; goto _test_eof;
+ _test_eof6: cs = 6; goto _test_eof;
+ _test_eof7: cs = 7; goto _test_eof;
+ _test_eof8: cs = 8; goto _test_eof;
+ _test_eof9: cs = 9; goto _test_eof;
+ _test_eof10: cs = 10; goto _test_eof;
+ _test_eof11: cs = 11; goto _test_eof;
+ _test_eof12: cs = 12; goto _test_eof;
+ _test_eof13: cs = 13; goto _test_eof;
+ _test_eof14: cs = 14; goto _test_eof;
+ _test_eof15: cs = 15; goto _test_eof;
+ _test_eof16: cs = 16; goto _test_eof;
+ _test_eof17: cs = 17; goto _test_eof;
+ _test_eof18: cs = 18; goto _test_eof;
+ _test_eof27: cs = 27; goto _test_eof;
+ _test_eof19: cs = 19; goto _test_eof;
+ _test_eof20: cs = 20; goto _test_eof;
+ _test_eof21: cs = 21; goto _test_eof;
+ _test_eof22: cs = 22; goto _test_eof;
+ _test_eof23: cs = 23; goto _test_eof;
+ _test_eof24: cs = 24; goto _test_eof;
+ _test_eof25: cs = 25; goto _test_eof;
+ _test_eof26: cs = 26; goto _test_eof;
+
+ _test_eof: {}
+ _out: {}
+ }
+
+#line 166 "parser.rl"
+
+ if (cs >= JSON_object_first_final) {
+ if (json->create_additions) {
+ VALUE klassname = rb_hash_aref(*result, json->create_id);
+ if (!NIL_P(klassname)) {
+ VALUE klass = rb_funcall(mJSON, i_deep_const_get, 1, klassname);
+ if (RTEST(rb_funcall(klass, i_json_creatable_p, 0))) {
+ *result = rb_funcall(klass, i_json_create, 1, *result);
+ }
+ }
+ }
+ return p + 1;
+ } else {
+ return NULL;
+ }
+}
+
+
+
+#line 464 "parser.c"
+static const int JSON_value_start = 1;
+static const int JSON_value_first_final = 21;
+static const int JSON_value_error = 0;
+
+static const int JSON_value_en_main = 1;
+
+
+#line 265 "parser.rl"
+
+
+static char *JSON_parse_value(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+
+
+#line 480 "parser.c"
+ {
+ cs = JSON_value_start;
+ }
+
+#line 272 "parser.rl"
+
+#line 487 "parser.c"
+ {
+ if ( p == pe )
+ goto _test_eof;
+ switch ( cs )
+ {
+case 1:
+ switch( (*p) ) {
+ case 34: goto tr0;
+ case 45: goto tr2;
+ case 73: goto st2;
+ case 78: goto st9;
+ case 91: goto tr5;
+ case 102: goto st11;
+ case 110: goto st15;
+ case 116: goto st18;
+ case 123: goto tr9;
+ }
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto tr2;
+ goto st0;
+st0:
+cs = 0;
+ goto _out;
+tr0:
+#line 213 "parser.rl"
+ {
+ char *np = JSON_parse_string(json, p, pe, result);
+ if (np == NULL) { p--; {p++; cs = 21; goto _out;} } else {p = (( np))-1;}
+ }
+ goto st21;
+tr2:
+#line 218 "parser.rl"
+ {
+ char *np;
+ if(pe > p + 9 - json->quirks_mode && !strncmp(MinusInfinity, p, 9)) {
+ if (json->allow_nan) {
+ *result = CMinusInfinity;
+ {p = (( p + 10))-1;}
+ p--; {p++; cs = 21; goto _out;}
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p);
+ }
+ }
+ np = JSON_parse_float(json, p, pe, result);
+ if (np != NULL) {p = (( np))-1;}
+ np = JSON_parse_integer(json, p, pe, result);
+ if (np != NULL) {p = (( np))-1;}
+ p--; {p++; cs = 21; goto _out;}
+ }
+ goto st21;
+tr5:
+#line 236 "parser.rl"
+ {
+ char *np;
+ json->current_nesting++;
+ np = JSON_parse_array(json, p, pe, result);
+ json->current_nesting--;
+ if (np == NULL) { p--; {p++; cs = 21; goto _out;} } else {p = (( np))-1;}
+ }
+ goto st21;
+tr9:
+#line 244 "parser.rl"
+ {
+ char *np;
+ json->current_nesting++;
+ np = JSON_parse_object(json, p, pe, result);
+ json->current_nesting--;
+ if (np == NULL) { p--; {p++; cs = 21; goto _out;} } else {p = (( np))-1;}
+ }
+ goto st21;
+tr16:
+#line 206 "parser.rl"
+ {
+ if (json->allow_nan) {
+ *result = CInfinity;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p - 8);
+ }
+ }
+ goto st21;
+tr18:
+#line 199 "parser.rl"
+ {
+ if (json->allow_nan) {
+ *result = CNaN;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p - 2);
+ }
+ }
+ goto st21;
+tr22:
+#line 193 "parser.rl"
+ {
+ *result = Qfalse;
+ }
+ goto st21;
+tr25:
+#line 190 "parser.rl"
+ {
+ *result = Qnil;
+ }
+ goto st21;
+tr28:
+#line 196 "parser.rl"
+ {
+ *result = Qtrue;
+ }
+ goto st21;
+st21:
+ if ( ++p == pe )
+ goto _test_eof21;
+case 21:
+#line 252 "parser.rl"
+ { p--; {p++; cs = 21; goto _out;} }
+#line 602 "parser.c"
+ goto st0;
+st2:
+ if ( ++p == pe )
+ goto _test_eof2;
+case 2:
+ if ( (*p) == 110 )
+ goto st3;
+ goto st0;
+st3:
+ if ( ++p == pe )
+ goto _test_eof3;
+case 3:
+ if ( (*p) == 102 )
+ goto st4;
+ goto st0;
+st4:
+ if ( ++p == pe )
+ goto _test_eof4;
+case 4:
+ if ( (*p) == 105 )
+ goto st5;
+ goto st0;
+st5:
+ if ( ++p == pe )
+ goto _test_eof5;
+case 5:
+ if ( (*p) == 110 )
+ goto st6;
+ goto st0;
+st6:
+ if ( ++p == pe )
+ goto _test_eof6;
+case 6:
+ if ( (*p) == 105 )
+ goto st7;
+ goto st0;
+st7:
+ if ( ++p == pe )
+ goto _test_eof7;
+case 7:
+ if ( (*p) == 116 )
+ goto st8;
+ goto st0;
+st8:
+ if ( ++p == pe )
+ goto _test_eof8;
+case 8:
+ if ( (*p) == 121 )
+ goto tr16;
+ goto st0;
+st9:
+ if ( ++p == pe )
+ goto _test_eof9;
+case 9:
+ if ( (*p) == 97 )
+ goto st10;
+ goto st0;
+st10:
+ if ( ++p == pe )
+ goto _test_eof10;
+case 10:
+ if ( (*p) == 78 )
+ goto tr18;
+ goto st0;
+st11:
+ if ( ++p == pe )
+ goto _test_eof11;
+case 11:
+ if ( (*p) == 97 )
+ goto st12;
+ goto st0;
+st12:
+ if ( ++p == pe )
+ goto _test_eof12;
+case 12:
+ if ( (*p) == 108 )
+ goto st13;
+ goto st0;
+st13:
+ if ( ++p == pe )
+ goto _test_eof13;
+case 13:
+ if ( (*p) == 115 )
+ goto st14;
+ goto st0;
+st14:
+ if ( ++p == pe )
+ goto _test_eof14;
+case 14:
+ if ( (*p) == 101 )
+ goto tr22;
+ goto st0;
+st15:
+ if ( ++p == pe )
+ goto _test_eof15;
+case 15:
+ if ( (*p) == 117 )
+ goto st16;
+ goto st0;
+st16:
+ if ( ++p == pe )
+ goto _test_eof16;
+case 16:
+ if ( (*p) == 108 )
+ goto st17;
+ goto st0;
+st17:
+ if ( ++p == pe )
+ goto _test_eof17;
+case 17:
+ if ( (*p) == 108 )
+ goto tr25;
+ goto st0;
+st18:
+ if ( ++p == pe )
+ goto _test_eof18;
+case 18:
+ if ( (*p) == 114 )
+ goto st19;
+ goto st0;
+st19:
+ if ( ++p == pe )
+ goto _test_eof19;
+case 19:
+ if ( (*p) == 117 )
+ goto st20;
+ goto st0;
+st20:
+ if ( ++p == pe )
+ goto _test_eof20;
+case 20:
+ if ( (*p) == 101 )
+ goto tr28;
+ goto st0;
+ }
+ _test_eof21: cs = 21; goto _test_eof;
+ _test_eof2: cs = 2; goto _test_eof;
+ _test_eof3: cs = 3; goto _test_eof;
+ _test_eof4: cs = 4; goto _test_eof;
+ _test_eof5: cs = 5; goto _test_eof;
+ _test_eof6: cs = 6; goto _test_eof;
+ _test_eof7: cs = 7; goto _test_eof;
+ _test_eof8: cs = 8; goto _test_eof;
+ _test_eof9: cs = 9; goto _test_eof;
+ _test_eof10: cs = 10; goto _test_eof;
+ _test_eof11: cs = 11; goto _test_eof;
+ _test_eof12: cs = 12; goto _test_eof;
+ _test_eof13: cs = 13; goto _test_eof;
+ _test_eof14: cs = 14; goto _test_eof;
+ _test_eof15: cs = 15; goto _test_eof;
+ _test_eof16: cs = 16; goto _test_eof;
+ _test_eof17: cs = 17; goto _test_eof;
+ _test_eof18: cs = 18; goto _test_eof;
+ _test_eof19: cs = 19; goto _test_eof;
+ _test_eof20: cs = 20; goto _test_eof;
+
+ _test_eof: {}
+ _out: {}
+ }
+
+#line 273 "parser.rl"
+
+ if (cs >= JSON_value_first_final) {
+ return p;
+ } else {
+ return NULL;
+ }
+}
+
+
+#line 773 "parser.c"
+static const int JSON_integer_start = 1;
+static const int JSON_integer_first_final = 3;
+static const int JSON_integer_error = 0;
+
+static const int JSON_integer_en_main = 1;
+
+
+#line 289 "parser.rl"
+
+
+static char *JSON_parse_integer(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+
+
+#line 789 "parser.c"
+ {
+ cs = JSON_integer_start;
+ }
+
+#line 296 "parser.rl"
+ json->memo = p;
+
+#line 797 "parser.c"
+ {
+ if ( p == pe )
+ goto _test_eof;
+ switch ( cs )
+ {
+case 1:
+ switch( (*p) ) {
+ case 45: goto st2;
+ case 48: goto st3;
+ }
+ if ( 49 <= (*p) && (*p) <= 57 )
+ goto st5;
+ goto st0;
+st0:
+cs = 0;
+ goto _out;
+st2:
+ if ( ++p == pe )
+ goto _test_eof2;
+case 2:
+ if ( (*p) == 48 )
+ goto st3;
+ if ( 49 <= (*p) && (*p) <= 57 )
+ goto st5;
+ goto st0;
+st3:
+ if ( ++p == pe )
+ goto _test_eof3;
+case 3:
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st0;
+ goto tr4;
+tr4:
+#line 286 "parser.rl"
+ { p--; {p++; cs = 4; goto _out;} }
+ goto st4;
+st4:
+ if ( ++p == pe )
+ goto _test_eof4;
+case 4:
+#line 838 "parser.c"
+ goto st0;
+st5:
+ if ( ++p == pe )
+ goto _test_eof5;
+case 5:
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st5;
+ goto tr4;
+ }
+ _test_eof2: cs = 2; goto _test_eof;
+ _test_eof3: cs = 3; goto _test_eof;
+ _test_eof4: cs = 4; goto _test_eof;
+ _test_eof5: cs = 5; goto _test_eof;
+
+ _test_eof: {}
+ _out: {}
+ }
+
+#line 298 "parser.rl"
+
+ if (cs >= JSON_integer_first_final) {
+ long len = p - json->memo;
+ *result = rb_Integer(rb_str_new(json->memo, len));
+ return p + 1;
+ } else {
+ return NULL;
+ }
+}
+
+
+#line 869 "parser.c"
+static const int JSON_float_start = 1;
+static const int JSON_float_first_final = 8;
+static const int JSON_float_error = 0;
+
+static const int JSON_float_en_main = 1;
+
+
+#line 320 "parser.rl"
+
+
+static char *JSON_parse_float(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+
+
+#line 885 "parser.c"
+ {
+ cs = JSON_float_start;
+ }
+
+#line 327 "parser.rl"
+ json->memo = p;
+
+#line 893 "parser.c"
+ {
+ if ( p == pe )
+ goto _test_eof;
+ switch ( cs )
+ {
+case 1:
+ switch( (*p) ) {
+ case 45: goto st2;
+ case 48: goto st3;
+ }
+ if ( 49 <= (*p) && (*p) <= 57 )
+ goto st7;
+ goto st0;
+st0:
+cs = 0;
+ goto _out;
+st2:
+ if ( ++p == pe )
+ goto _test_eof2;
+case 2:
+ if ( (*p) == 48 )
+ goto st3;
+ if ( 49 <= (*p) && (*p) <= 57 )
+ goto st7;
+ goto st0;
+st3:
+ if ( ++p == pe )
+ goto _test_eof3;
+case 3:
+ switch( (*p) ) {
+ case 46: goto st4;
+ case 69: goto st5;
+ case 101: goto st5;
+ }
+ goto st0;
+st4:
+ if ( ++p == pe )
+ goto _test_eof4;
+case 4:
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st8;
+ goto st0;
+st8:
+ if ( ++p == pe )
+ goto _test_eof8;
+case 8:
+ switch( (*p) ) {
+ case 69: goto st5;
+ case 101: goto st5;
+ }
+ if ( (*p) > 46 ) {
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st8;
+ } else if ( (*p) >= 45 )
+ goto st0;
+ goto tr9;
+tr9:
+#line 314 "parser.rl"
+ { p--; {p++; cs = 9; goto _out;} }
+ goto st9;
+st9:
+ if ( ++p == pe )
+ goto _test_eof9;
+case 9:
+#line 958 "parser.c"
+ goto st0;
+st5:
+ if ( ++p == pe )
+ goto _test_eof5;
+case 5:
+ switch( (*p) ) {
+ case 43: goto st6;
+ case 45: goto st6;
+ }
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st10;
+ goto st0;
+st6:
+ if ( ++p == pe )
+ goto _test_eof6;
+case 6:
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st10;
+ goto st0;
+st10:
+ if ( ++p == pe )
+ goto _test_eof10;
+case 10:
+ switch( (*p) ) {
+ case 69: goto st0;
+ case 101: goto st0;
+ }
+ if ( (*p) > 46 ) {
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st10;
+ } else if ( (*p) >= 45 )
+ goto st0;
+ goto tr9;
+st7:
+ if ( ++p == pe )
+ goto _test_eof7;
+case 7:
+ switch( (*p) ) {
+ case 46: goto st4;
+ case 69: goto st5;
+ case 101: goto st5;
+ }
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st7;
+ goto st0;
+ }
+ _test_eof2: cs = 2; goto _test_eof;
+ _test_eof3: cs = 3; goto _test_eof;
+ _test_eof4: cs = 4; goto _test_eof;
+ _test_eof8: cs = 8; goto _test_eof;
+ _test_eof9: cs = 9; goto _test_eof;
+ _test_eof5: cs = 5; goto _test_eof;
+ _test_eof6: cs = 6; goto _test_eof;
+ _test_eof10: cs = 10; goto _test_eof;
+ _test_eof7: cs = 7; goto _test_eof;
+
+ _test_eof: {}
+ _out: {}
+ }
+
+#line 329 "parser.rl"
+
+ if (cs >= JSON_float_first_final) {
+ long len = p - json->memo;
+ *result = rb_Float(rb_str_new(json->memo, len));
+ return p + 1;
+ } else {
+ return NULL;
+ }
+}
+
+
+
+#line 1032 "parser.c"
+static const int JSON_array_start = 1;
+static const int JSON_array_first_final = 17;
+static const int JSON_array_error = 0;
+
+static const int JSON_array_en_main = 1;
+
+
+#line 369 "parser.rl"
+
+
+static char *JSON_parse_array(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+ VALUE array_class = json->array_class;
+
+ if (json->max_nesting && json->current_nesting > json->max_nesting) {
+ rb_raise(eNestingError, "nesting of %d is too deep", json->current_nesting);
+ }
+ *result = NIL_P(array_class) ? rb_ary_new() : rb_class_new_instance(0, 0, array_class);
+
+
+#line 1054 "parser.c"
+ {
+ cs = JSON_array_start;
+ }
+
+#line 382 "parser.rl"
+
+#line 1061 "parser.c"
+ {
+ if ( p == pe )
+ goto _test_eof;
+ switch ( cs )
+ {
+case 1:
+ if ( (*p) == 91 )
+ goto st2;
+ goto st0;
+st0:
+cs = 0;
+ goto _out;
+st2:
+ if ( ++p == pe )
+ goto _test_eof2;
+case 2:
+ switch( (*p) ) {
+ case 13: goto st2;
+ case 32: goto st2;
+ case 34: goto tr2;
+ case 45: goto tr2;
+ case 47: goto st13;
+ case 73: goto tr2;
+ case 78: goto tr2;
+ case 91: goto tr2;
+ case 93: goto tr4;
+ case 102: goto tr2;
+ case 110: goto tr2;
+ case 116: goto tr2;
+ case 123: goto tr2;
+ }
+ if ( (*p) > 10 ) {
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto tr2;
+ } else if ( (*p) >= 9 )
+ goto st2;
+ goto st0;
+tr2:
+#line 346 "parser.rl"
+ {
+ VALUE v = Qnil;
+ char *np = JSON_parse_value(json, p, pe, &v);
+ if (np == NULL) {
+ p--; {p++; cs = 3; goto _out;}
+ } else {
+ if (NIL_P(json->array_class)) {
+ rb_ary_push(*result, v);
+ } else {
+ rb_funcall(*result, i_leftshift, 1, v);
+ }
+ {p = (( np))-1;}
+ }
+ }
+ goto st3;
+st3:
+ if ( ++p == pe )
+ goto _test_eof3;
+case 3:
+#line 1120 "parser.c"
+ switch( (*p) ) {
+ case 13: goto st3;
+ case 32: goto st3;
+ case 44: goto st4;
+ case 47: goto st9;
+ case 93: goto tr4;
+ }
+ if ( 9 <= (*p) && (*p) <= 10 )
+ goto st3;
+ goto st0;
+st4:
+ if ( ++p == pe )
+ goto _test_eof4;
+case 4:
+ switch( (*p) ) {
+ case 13: goto st4;
+ case 32: goto st4;
+ case 34: goto tr2;
+ case 45: goto tr2;
+ case 47: goto st5;
+ case 73: goto tr2;
+ case 78: goto tr2;
+ case 91: goto tr2;
+ case 102: goto tr2;
+ case 110: goto tr2;
+ case 116: goto tr2;
+ case 123: goto tr2;
+ }
+ if ( (*p) > 10 ) {
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto tr2;
+ } else if ( (*p) >= 9 )
+ goto st4;
+ goto st0;
+st5:
+ if ( ++p == pe )
+ goto _test_eof5;
+case 5:
+ switch( (*p) ) {
+ case 42: goto st6;
+ case 47: goto st8;
+ }
+ goto st0;
+st6:
+ if ( ++p == pe )
+ goto _test_eof6;
+case 6:
+ if ( (*p) == 42 )
+ goto st7;
+ goto st6;
+st7:
+ if ( ++p == pe )
+ goto _test_eof7;
+case 7:
+ switch( (*p) ) {
+ case 42: goto st7;
+ case 47: goto st4;
+ }
+ goto st6;
+st8:
+ if ( ++p == pe )
+ goto _test_eof8;
+case 8:
+ if ( (*p) == 10 )
+ goto st4;
+ goto st8;
+st9:
+ if ( ++p == pe )
+ goto _test_eof9;
+case 9:
+ switch( (*p) ) {
+ case 42: goto st10;
+ case 47: goto st12;
+ }
+ goto st0;
+st10:
+ if ( ++p == pe )
+ goto _test_eof10;
+case 10:
+ if ( (*p) == 42 )
+ goto st11;
+ goto st10;
+st11:
+ if ( ++p == pe )
+ goto _test_eof11;
+case 11:
+ switch( (*p) ) {
+ case 42: goto st11;
+ case 47: goto st3;
+ }
+ goto st10;
+st12:
+ if ( ++p == pe )
+ goto _test_eof12;
+case 12:
+ if ( (*p) == 10 )
+ goto st3;
+ goto st12;
+tr4:
+#line 361 "parser.rl"
+ { p--; {p++; cs = 17; goto _out;} }
+ goto st17;
+st17:
+ if ( ++p == pe )
+ goto _test_eof17;
+case 17:
+#line 1227 "parser.c"
+ goto st0;
+st13:
+ if ( ++p == pe )
+ goto _test_eof13;
+case 13:
+ switch( (*p) ) {
+ case 42: goto st14;
+ case 47: goto st16;
+ }
+ goto st0;
+st14:
+ if ( ++p == pe )
+ goto _test_eof14;
+case 14:
+ if ( (*p) == 42 )
+ goto st15;
+ goto st14;
+st15:
+ if ( ++p == pe )
+ goto _test_eof15;
+case 15:
+ switch( (*p) ) {
+ case 42: goto st15;
+ case 47: goto st2;
+ }
+ goto st14;
+st16:
+ if ( ++p == pe )
+ goto _test_eof16;
+case 16:
+ if ( (*p) == 10 )
+ goto st2;
+ goto st16;
+ }
+ _test_eof2: cs = 2; goto _test_eof;
+ _test_eof3: cs = 3; goto _test_eof;
+ _test_eof4: cs = 4; goto _test_eof;
+ _test_eof5: cs = 5; goto _test_eof;
+ _test_eof6: cs = 6; goto _test_eof;
+ _test_eof7: cs = 7; goto _test_eof;
+ _test_eof8: cs = 8; goto _test_eof;
+ _test_eof9: cs = 9; goto _test_eof;
+ _test_eof10: cs = 10; goto _test_eof;
+ _test_eof11: cs = 11; goto _test_eof;
+ _test_eof12: cs = 12; goto _test_eof;
+ _test_eof17: cs = 17; goto _test_eof;
+ _test_eof13: cs = 13; goto _test_eof;
+ _test_eof14: cs = 14; goto _test_eof;
+ _test_eof15: cs = 15; goto _test_eof;
+ _test_eof16: cs = 16; goto _test_eof;
+
+ _test_eof: {}
+ _out: {}
+ }
+
+#line 383 "parser.rl"
+
+ if(cs >= JSON_array_first_final) {
+ return p + 1;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p);
+ return NULL;
+ }
+}
+
+static VALUE json_string_unescape(VALUE result, char *string, char *stringEnd)
+{
+ char *p = string, *pe = string, *unescape;
+ int unescape_len;
+
+ while (pe < stringEnd) {
+ if (*pe == '\\') {
+ unescape = (char *) "?";
+ unescape_len = 1;
+ if (pe > p) rb_str_buf_cat(result, p, pe - p);
+ switch (*++pe) {
+ case 'n':
+ unescape = (char *) "\n";
+ break;
+ case 'r':
+ unescape = (char *) "\r";
+ break;
+ case 't':
+ unescape = (char *) "\t";
+ break;
+ case '"':
+ unescape = (char *) "\"";
+ break;
+ case '\\':
+ unescape = (char *) "\\";
+ break;
+ case 'b':
+ unescape = (char *) "\b";
+ break;
+ case 'f':
+ unescape = (char *) "\f";
+ break;
+ case 'u':
+ if (pe > stringEnd - 4) {
+ return Qnil;
+ } else {
+ char buf[4];
+ UTF32 ch = unescape_unicode((unsigned char *) ++pe);
+ pe += 3;
+ if (UNI_SUR_HIGH_START == (ch & 0xFC00)) {
+ pe++;
+ if (pe > stringEnd - 6) return Qnil;
+ if (pe[0] == '\\' && pe[1] == 'u') {
+ UTF32 sur = unescape_unicode((unsigned char *) pe + 2);
+ ch = (((ch & 0x3F) << 10) | ((((ch >> 6) & 0xF) + 1) << 16)
+ | (sur & 0x3FF));
+ pe += 5;
+ } else {
+ unescape = (char *) "?";
+ break;
+ }
+ }
+ unescape_len = convert_UTF32_to_UTF8(buf, ch);
+ unescape = buf;
+ }
+ break;
+ default:
+ p = pe;
+ continue;
+ }
+ rb_str_buf_cat(result, unescape, unescape_len);
+ p = ++pe;
+ } else {
+ pe++;
+ }
+ }
+ rb_str_buf_cat(result, p, pe - p);
+ return result;
+}
+
+
+#line 1364 "parser.c"
+static const int JSON_string_start = 1;
+static const int JSON_string_first_final = 8;
+static const int JSON_string_error = 0;
+
+static const int JSON_string_en_main = 1;
+
+
+#line 482 "parser.rl"
+
+
+static int
+match_i(VALUE regexp, VALUE klass, VALUE memo)
+{
+ if (regexp == Qundef) return ST_STOP;
+ if (RTEST(rb_funcall(klass, i_json_creatable_p, 0)) &&
+ RTEST(rb_funcall(regexp, i_match, 1, rb_ary_entry(memo, 0)))) {
+ rb_ary_push(memo, klass);
+ return ST_STOP;
+ }
+ return ST_CONTINUE;
+}
+
+static char *JSON_parse_string(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+ VALUE match_string;
+
+ *result = rb_str_buf_new(0);
+
+#line 1394 "parser.c"
+ {
+ cs = JSON_string_start;
+ }
+
+#line 503 "parser.rl"
+ json->memo = p;
+
+#line 1402 "parser.c"
+ {
+ if ( p == pe )
+ goto _test_eof;
+ switch ( cs )
+ {
+case 1:
+ if ( (*p) == 34 )
+ goto st2;
+ goto st0;
+st0:
+cs = 0;
+ goto _out;
+st2:
+ if ( ++p == pe )
+ goto _test_eof2;
+case 2:
+ switch( (*p) ) {
+ case 34: goto tr2;
+ case 92: goto st3;
+ }
+ if ( 0 <= (*p) && (*p) <= 31 )
+ goto st0;
+ goto st2;
+tr2:
+#line 468 "parser.rl"
+ {
+ *result = json_string_unescape(*result, json->memo + 1, p);
+ if (NIL_P(*result)) {
+ p--;
+ {p++; cs = 8; goto _out;}
+ } else {
+ FORCE_UTF8(*result);
+ {p = (( p + 1))-1;}
+ }
+ }
+#line 479 "parser.rl"
+ { p--; {p++; cs = 8; goto _out;} }
+ goto st8;
+st8:
+ if ( ++p == pe )
+ goto _test_eof8;
+case 8:
+#line 1445 "parser.c"
+ goto st0;
+st3:
+ if ( ++p == pe )
+ goto _test_eof3;
+case 3:
+ if ( (*p) == 117 )
+ goto st4;
+ if ( 0 <= (*p) && (*p) <= 31 )
+ goto st0;
+ goto st2;
+st4:
+ if ( ++p == pe )
+ goto _test_eof4;
+case 4:
+ if ( (*p) < 65 ) {
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st5;
+ } else if ( (*p) > 70 ) {
+ if ( 97 <= (*p) && (*p) <= 102 )
+ goto st5;
+ } else
+ goto st5;
+ goto st0;
+st5:
+ if ( ++p == pe )
+ goto _test_eof5;
+case 5:
+ if ( (*p) < 65 ) {
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st6;
+ } else if ( (*p) > 70 ) {
+ if ( 97 <= (*p) && (*p) <= 102 )
+ goto st6;
+ } else
+ goto st6;
+ goto st0;
+st6:
+ if ( ++p == pe )
+ goto _test_eof6;
+case 6:
+ if ( (*p) < 65 ) {
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st7;
+ } else if ( (*p) > 70 ) {
+ if ( 97 <= (*p) && (*p) <= 102 )
+ goto st7;
+ } else
+ goto st7;
+ goto st0;
+st7:
+ if ( ++p == pe )
+ goto _test_eof7;
+case 7:
+ if ( (*p) < 65 ) {
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto st2;
+ } else if ( (*p) > 70 ) {
+ if ( 97 <= (*p) && (*p) <= 102 )
+ goto st2;
+ } else
+ goto st2;
+ goto st0;
+ }
+ _test_eof2: cs = 2; goto _test_eof;
+ _test_eof8: cs = 8; goto _test_eof;
+ _test_eof3: cs = 3; goto _test_eof;
+ _test_eof4: cs = 4; goto _test_eof;
+ _test_eof5: cs = 5; goto _test_eof;
+ _test_eof6: cs = 6; goto _test_eof;
+ _test_eof7: cs = 7; goto _test_eof;
+
+ _test_eof: {}
+ _out: {}
+ }
+
+#line 505 "parser.rl"
+
+ if (json->create_additions && RTEST(match_string = json->match_string)) {
+ VALUE klass;
+ VALUE memo = rb_ary_new2(2);
+ rb_ary_push(memo, *result);
+ rb_hash_foreach(match_string, match_i, memo);
+ klass = rb_ary_entry(memo, 1);
+ if (RTEST(klass)) {
+ *result = rb_funcall(klass, i_json_create, 1, *result);
+ }
+ }
+
+ if (json->symbolize_names && json->parsing_name) {
+ *result = rb_str_intern(*result);
+ }
+ if (cs >= JSON_string_first_final) {
+ return p + 1;
+ } else {
+ return NULL;
+ }
+}
+
+/*
+ * Document-class: JSON::Ext::Parser
+ *
+ * This is the JSON parser implemented as a C extension. It can be configured
+ * to be used by setting
+ *
+ * JSON.parser = JSON::Ext::Parser
+ *
+ * with the method parser= in JSON.
+ *
+ */
+
+static VALUE convert_encoding(VALUE source)
+{
+ char *ptr = RSTRING_PTR(source);
+ long len = RSTRING_LEN(source);
+ if (len < 2) {
+ rb_raise(eParserError, "A JSON text must at least contain two octets!");
+ }
+#ifdef HAVE_RUBY_ENCODING_H
+ {
+ VALUE encoding = rb_funcall(source, i_encoding, 0);
+ if (encoding == CEncoding_ASCII_8BIT) {
+ if (len >= 4 && ptr[0] == 0 && ptr[1] == 0 && ptr[2] == 0) {
+ source = rb_funcall(source, i_encode, 2, CEncoding_UTF_8, CEncoding_UTF_32BE);
+ } else if (len >= 4 && ptr[0] == 0 && ptr[2] == 0) {
+ source = rb_funcall(source, i_encode, 2, CEncoding_UTF_8, CEncoding_UTF_16BE);
+ } else if (len >= 4 && ptr[1] == 0 && ptr[2] == 0 && ptr[3] == 0) {
+ source = rb_funcall(source, i_encode, 2, CEncoding_UTF_8, CEncoding_UTF_32LE);
+ } else if (len >= 4 && ptr[1] == 0 && ptr[3] == 0) {
+ source = rb_funcall(source, i_encode, 2, CEncoding_UTF_8, CEncoding_UTF_16LE);
+ } else {
+ source = rb_str_dup(source);
+ FORCE_UTF8(source);
+ }
+ } else {
+ source = rb_funcall(source, i_encode, 1, CEncoding_UTF_8);
+ }
+ }
+#else
+ if (len >= 4 && ptr[0] == 0 && ptr[1] == 0 && ptr[2] == 0) {
+ source = rb_funcall(mJSON, i_iconv, 3, rb_str_new2("utf-8"), rb_str_new2("utf-32be"), source);
+ } else if (len >= 4 && ptr[0] == 0 && ptr[2] == 0) {
+ source = rb_funcall(mJSON, i_iconv, 3, rb_str_new2("utf-8"), rb_str_new2("utf-16be"), source);
+ } else if (len >= 4 && ptr[1] == 0 && ptr[2] == 0 && ptr[3] == 0) {
+ source = rb_funcall(mJSON, i_iconv, 3, rb_str_new2("utf-8"), rb_str_new2("utf-32le"), source);
+ } else if (len >= 4 && ptr[1] == 0 && ptr[3] == 0) {
+ source = rb_funcall(mJSON, i_iconv, 3, rb_str_new2("utf-8"), rb_str_new2("utf-16le"), source);
+ }
+#endif
+ return source;
+}
+
+/*
+ * call-seq: new(source, opts => {})
+ *
+ * Creates a new JSON::Ext::Parser instance for the string _source_.
+ *
+ * Creates a new JSON::Ext::Parser instance for the string _source_.
+ *
+ * It will be configured by the _opts_ hash. _opts_ can have the following
+ * keys:
+ *
+ * _opts_ can have the following keys:
+ * * *max_nesting*: The maximum depth of nesting allowed in the parsed data
+ * structures. Disable depth checking with :max_nesting => false|nil|0, it
+ * defaults to 19.
+ * * *allow_nan*: If set to true, allow NaN, Infinity and -Infinity in
+ * defiance of RFC 4627 to be parsed by the Parser. This option defaults to
+ * false.
+ * * *symbolize_names*: If set to true, returns symbols for the names
+ * (keys) in a JSON object. Otherwise strings are returned, which is also
+ * the default.
+ * * *create_additions*: If set to false, the Parser doesn't create
+ * additions even if a matchin class and create_id was found. This option
+ * defaults to true.
+ * * *object_class*: Defaults to Hash
+ * * *array_class*: Defaults to Array
+ * * *quirks_mode*: Enables quirks_mode for parser, that is for example
+ * parsing single JSON values instead of documents is possible.
+ *
+ */
+static VALUE cParser_initialize(int argc, VALUE *argv, VALUE self)
+{
+ VALUE source, opts;
+ GET_PARSER_INIT;
+
+ if (json->Vsource) {
+ rb_raise(rb_eTypeError, "already initialized instance");
+ }
+ rb_scan_args(argc, argv, "11", &source, &opts);
+ if (!NIL_P(opts)) {
+ opts = rb_convert_type(opts, T_HASH, "Hash", "to_hash");
+ if (NIL_P(opts)) {
+ rb_raise(rb_eArgError, "opts needs to be like a hash");
+ } else {
+ VALUE tmp = ID2SYM(i_max_nesting);
+ if (option_given_p(opts, tmp)) {
+ VALUE max_nesting = rb_hash_aref(opts, tmp);
+ if (RTEST(max_nesting)) {
+ Check_Type(max_nesting, T_FIXNUM);
+ json->max_nesting = FIX2INT(max_nesting);
+ } else {
+ json->max_nesting = 0;
+ }
+ } else {
+ json->max_nesting = 19;
+ }
+ tmp = ID2SYM(i_allow_nan);
+ if (option_given_p(opts, tmp)) {
+ json->allow_nan = RTEST(rb_hash_aref(opts, tmp)) ? 1 : 0;
+ } else {
+ json->allow_nan = 0;
+ }
+ tmp = ID2SYM(i_symbolize_names);
+ if (option_given_p(opts, tmp)) {
+ json->symbolize_names = RTEST(rb_hash_aref(opts, tmp)) ? 1 : 0;
+ } else {
+ json->symbolize_names = 0;
+ }
+ tmp = ID2SYM(i_quirks_mode);
+ if (option_given_p(opts, tmp)) {
+ VALUE quirks_mode = rb_hash_aref(opts, tmp);
+ json->quirks_mode = RTEST(quirks_mode) ? 1 : 0;
+ } else {
+ json->quirks_mode = 0;
+ }
+ tmp = ID2SYM(i_create_additions);
+ if (option_given_p(opts, tmp)) {
+ json->create_additions = RTEST(rb_hash_aref(opts, tmp));
+ } else {
+ json->create_additions = 0;
+ }
+ tmp = ID2SYM(i_create_id);
+ if (option_given_p(opts, tmp)) {
+ json->create_id = rb_hash_aref(opts, tmp);
+ } else {
+ json->create_id = rb_funcall(mJSON, i_create_id, 0);
+ }
+ tmp = ID2SYM(i_object_class);
+ if (option_given_p(opts, tmp)) {
+ json->object_class = rb_hash_aref(opts, tmp);
+ } else {
+ json->object_class = Qnil;
+ }
+ tmp = ID2SYM(i_array_class);
+ if (option_given_p(opts, tmp)) {
+ json->array_class = rb_hash_aref(opts, tmp);
+ } else {
+ json->array_class = Qnil;
+ }
+ tmp = ID2SYM(i_match_string);
+ if (option_given_p(opts, tmp)) {
+ VALUE match_string = rb_hash_aref(opts, tmp);
+ json->match_string = RTEST(match_string) ? match_string : Qnil;
+ } else {
+ json->match_string = Qnil;
+ }
+ }
+ } else {
+ json->max_nesting = 19;
+ json->allow_nan = 0;
+ json->create_additions = 1;
+ json->create_id = rb_funcall(mJSON, i_create_id, 0);
+ json->object_class = Qnil;
+ json->array_class = Qnil;
+ }
+ if (!json->quirks_mode) {
+ source = convert_encoding(StringValue(source));
+ }
+ json->current_nesting = 0;
+ json->len = RSTRING_LEN(source);
+ json->source = RSTRING_PTR(source);;
+ json->Vsource = source;
+ return self;
+}
+
+
+#line 1722 "parser.c"
+static const int JSON_start = 1;
+static const int JSON_first_final = 10;
+static const int JSON_error = 0;
+
+static const int JSON_en_main = 1;
+
+
+#line 729 "parser.rl"
+
+
+static VALUE cParser_parse_strict(VALUE self)
+{
+ char *p, *pe;
+ int cs = EVIL;
+ VALUE result = Qnil;
+ GET_PARSER;
+
+
+#line 1741 "parser.c"
+ {
+ cs = JSON_start;
+ }
+
+#line 739 "parser.rl"
+ p = json->source;
+ pe = p + json->len;
+
+#line 1750 "parser.c"
+ {
+ if ( p == pe )
+ goto _test_eof;
+ switch ( cs )
+ {
+st1:
+ if ( ++p == pe )
+ goto _test_eof1;
+case 1:
+ switch( (*p) ) {
+ case 13: goto st1;
+ case 32: goto st1;
+ case 47: goto st2;
+ case 91: goto tr3;
+ case 123: goto tr4;
+ }
+ if ( 9 <= (*p) && (*p) <= 10 )
+ goto st1;
+ goto st0;
+st0:
+cs = 0;
+ goto _out;
+st2:
+ if ( ++p == pe )
+ goto _test_eof2;
+case 2:
+ switch( (*p) ) {
+ case 42: goto st3;
+ case 47: goto st5;
+ }
+ goto st0;
+st3:
+ if ( ++p == pe )
+ goto _test_eof3;
+case 3:
+ if ( (*p) == 42 )
+ goto st4;
+ goto st3;
+st4:
+ if ( ++p == pe )
+ goto _test_eof4;
+case 4:
+ switch( (*p) ) {
+ case 42: goto st4;
+ case 47: goto st1;
+ }
+ goto st3;
+st5:
+ if ( ++p == pe )
+ goto _test_eof5;
+case 5:
+ if ( (*p) == 10 )
+ goto st1;
+ goto st5;
+tr3:
+#line 718 "parser.rl"
+ {
+ char *np;
+ json->current_nesting = 1;
+ np = JSON_parse_array(json, p, pe, &result);
+ if (np == NULL) { p--; {p++; cs = 10; goto _out;} } else {p = (( np))-1;}
+ }
+ goto st10;
+tr4:
+#line 711 "parser.rl"
+ {
+ char *np;
+ json->current_nesting = 1;
+ np = JSON_parse_object(json, p, pe, &result);
+ if (np == NULL) { p--; {p++; cs = 10; goto _out;} } else {p = (( np))-1;}
+ }
+ goto st10;
+st10:
+ if ( ++p == pe )
+ goto _test_eof10;
+case 10:
+#line 1827 "parser.c"
+ switch( (*p) ) {
+ case 13: goto st10;
+ case 32: goto st10;
+ case 47: goto st6;
+ }
+ if ( 9 <= (*p) && (*p) <= 10 )
+ goto st10;
+ goto st0;
+st6:
+ if ( ++p == pe )
+ goto _test_eof6;
+case 6:
+ switch( (*p) ) {
+ case 42: goto st7;
+ case 47: goto st9;
+ }
+ goto st0;
+st7:
+ if ( ++p == pe )
+ goto _test_eof7;
+case 7:
+ if ( (*p) == 42 )
+ goto st8;
+ goto st7;
+st8:
+ if ( ++p == pe )
+ goto _test_eof8;
+case 8:
+ switch( (*p) ) {
+ case 42: goto st8;
+ case 47: goto st10;
+ }
+ goto st7;
+st9:
+ if ( ++p == pe )
+ goto _test_eof9;
+case 9:
+ if ( (*p) == 10 )
+ goto st10;
+ goto st9;
+ }
+ _test_eof1: cs = 1; goto _test_eof;
+ _test_eof2: cs = 2; goto _test_eof;
+ _test_eof3: cs = 3; goto _test_eof;
+ _test_eof4: cs = 4; goto _test_eof;
+ _test_eof5: cs = 5; goto _test_eof;
+ _test_eof10: cs = 10; goto _test_eof;
+ _test_eof6: cs = 6; goto _test_eof;
+ _test_eof7: cs = 7; goto _test_eof;
+ _test_eof8: cs = 8; goto _test_eof;
+ _test_eof9: cs = 9; goto _test_eof;
+
+ _test_eof: {}
+ _out: {}
+ }
+
+#line 742 "parser.rl"
+
+ if (cs >= JSON_first_final && p == pe) {
+ return result;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p);
+ return Qnil;
+ }
+}
+
+
+
+#line 1896 "parser.c"
+static const int JSON_quirks_mode_start = 1;
+static const int JSON_quirks_mode_first_final = 10;
+static const int JSON_quirks_mode_error = 0;
+
+static const int JSON_quirks_mode_en_main = 1;
+
+
+#line 767 "parser.rl"
+
+
+static VALUE cParser_parse_quirks_mode(VALUE self)
+{
+ char *p, *pe;
+ int cs = EVIL;
+ VALUE result = Qnil;
+ GET_PARSER;
+
+
+#line 1915 "parser.c"
+ {
+ cs = JSON_quirks_mode_start;
+ }
+
+#line 777 "parser.rl"
+ p = json->source;
+ pe = p + json->len;
+
+#line 1924 "parser.c"
+ {
+ if ( p == pe )
+ goto _test_eof;
+ switch ( cs )
+ {
+st1:
+ if ( ++p == pe )
+ goto _test_eof1;
+case 1:
+ switch( (*p) ) {
+ case 13: goto st1;
+ case 32: goto st1;
+ case 34: goto tr2;
+ case 45: goto tr2;
+ case 47: goto st6;
+ case 73: goto tr2;
+ case 78: goto tr2;
+ case 91: goto tr2;
+ case 102: goto tr2;
+ case 110: goto tr2;
+ case 116: goto tr2;
+ case 123: goto tr2;
+ }
+ if ( (*p) > 10 ) {
+ if ( 48 <= (*p) && (*p) <= 57 )
+ goto tr2;
+ } else if ( (*p) >= 9 )
+ goto st1;
+ goto st0;
+st0:
+cs = 0;
+ goto _out;
+tr2:
+#line 759 "parser.rl"
+ {
+ char *np = JSON_parse_value(json, p, pe, &result);
+ if (np == NULL) { p--; {p++; cs = 10; goto _out;} } else {p = (( np))-1;}
+ }
+ goto st10;
+st10:
+ if ( ++p == pe )
+ goto _test_eof10;
+case 10:
+#line 1968 "parser.c"
+ switch( (*p) ) {
+ case 13: goto st10;
+ case 32: goto st10;
+ case 47: goto st2;
+ }
+ if ( 9 <= (*p) && (*p) <= 10 )
+ goto st10;
+ goto st0;
+st2:
+ if ( ++p == pe )
+ goto _test_eof2;
+case 2:
+ switch( (*p) ) {
+ case 42: goto st3;
+ case 47: goto st5;
+ }
+ goto st0;
+st3:
+ if ( ++p == pe )
+ goto _test_eof3;
+case 3:
+ if ( (*p) == 42 )
+ goto st4;
+ goto st3;
+st4:
+ if ( ++p == pe )
+ goto _test_eof4;
+case 4:
+ switch( (*p) ) {
+ case 42: goto st4;
+ case 47: goto st10;
+ }
+ goto st3;
+st5:
+ if ( ++p == pe )
+ goto _test_eof5;
+case 5:
+ if ( (*p) == 10 )
+ goto st10;
+ goto st5;
+st6:
+ if ( ++p == pe )
+ goto _test_eof6;
+case 6:
+ switch( (*p) ) {
+ case 42: goto st7;
+ case 47: goto st9;
+ }
+ goto st0;
+st7:
+ if ( ++p == pe )
+ goto _test_eof7;
+case 7:
+ if ( (*p) == 42 )
+ goto st8;
+ goto st7;
+st8:
+ if ( ++p == pe )
+ goto _test_eof8;
+case 8:
+ switch( (*p) ) {
+ case 42: goto st8;
+ case 47: goto st1;
+ }
+ goto st7;
+st9:
+ if ( ++p == pe )
+ goto _test_eof9;
+case 9:
+ if ( (*p) == 10 )
+ goto st1;
+ goto st9;
+ }
+ _test_eof1: cs = 1; goto _test_eof;
+ _test_eof10: cs = 10; goto _test_eof;
+ _test_eof2: cs = 2; goto _test_eof;
+ _test_eof3: cs = 3; goto _test_eof;
+ _test_eof4: cs = 4; goto _test_eof;
+ _test_eof5: cs = 5; goto _test_eof;
+ _test_eof6: cs = 6; goto _test_eof;
+ _test_eof7: cs = 7; goto _test_eof;
+ _test_eof8: cs = 8; goto _test_eof;
+ _test_eof9: cs = 9; goto _test_eof;
+
+ _test_eof: {}
+ _out: {}
+ }
+
+#line 780 "parser.rl"
+
+ if (cs >= JSON_quirks_mode_first_final && p == pe) {
+ return result;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p);
+ return Qnil;
+ }
+}
+
+/*
+ * call-seq: parse()
+ *
+ * Parses the current JSON text _source_ and returns the complete data
+ * structure as a result.
+ */
+static VALUE cParser_parse(VALUE self)
+{
+ GET_PARSER;
+
+ if (json->quirks_mode) {
+ return cParser_parse_quirks_mode(self);
+ } else {
+ return cParser_parse_strict(self);
+ }
+}
+
+
+static JSON_Parser *JSON_allocate()
+{
+ JSON_Parser *json = ALLOC(JSON_Parser);
+ MEMZERO(json, JSON_Parser, 1);
+ return json;
+}
+
+static void JSON_mark(JSON_Parser *json)
+{
+ rb_gc_mark_maybe(json->Vsource);
+ rb_gc_mark_maybe(json->create_id);
+ rb_gc_mark_maybe(json->object_class);
+ rb_gc_mark_maybe(json->array_class);
+ rb_gc_mark_maybe(json->match_string);
+}
+
+static void JSON_free(JSON_Parser *json)
+{
+ ruby_xfree(json);
+}
+
+static VALUE cJSON_parser_s_allocate(VALUE klass)
+{
+ JSON_Parser *json = JSON_allocate();
+ return Data_Wrap_Struct(klass, JSON_mark, JSON_free, json);
+}
+
+/*
+ * call-seq: source()
+ *
+ * Returns a copy of the current _source_ string, that was used to construct
+ * this Parser.
+ */
+static VALUE cParser_source(VALUE self)
+{
+ GET_PARSER;
+ return rb_str_dup(json->Vsource);
+}
+
+/*
+ * call-seq: quirks_mode?()
+ *
+ * Returns a true, if this parser is in quirks_mode, false otherwise.
+ */
+static VALUE cParser_quirks_mode_p(VALUE self)
+{
+ GET_PARSER;
+ return json->quirks_mode ? Qtrue : Qfalse;
+}
+
+
+void Init_parser()
+{
+ rb_require("json/common");
+ mJSON = rb_define_module("JSON");
+ mExt = rb_define_module_under(mJSON, "Ext");
+ cParser = rb_define_class_under(mExt, "Parser", rb_cObject);
+ eParserError = rb_path2class("JSON::ParserError");
+ eNestingError = rb_path2class("JSON::NestingError");
+ rb_define_alloc_func(cParser, cJSON_parser_s_allocate);
+ rb_define_method(cParser, "initialize", cParser_initialize, -1);
+ rb_define_method(cParser, "parse", cParser_parse, 0);
+ rb_define_method(cParser, "source", cParser_source, 0);
+ rb_define_method(cParser, "quirks_mode?", cParser_quirks_mode_p, 0);
+
+ CNaN = rb_const_get(mJSON, rb_intern("NaN"));
+ CInfinity = rb_const_get(mJSON, rb_intern("Infinity"));
+ CMinusInfinity = rb_const_get(mJSON, rb_intern("MinusInfinity"));
+
+ i_json_creatable_p = rb_intern("json_creatable?");
+ i_json_create = rb_intern("json_create");
+ i_create_id = rb_intern("create_id");
+ i_create_additions = rb_intern("create_additions");
+ i_chr = rb_intern("chr");
+ i_max_nesting = rb_intern("max_nesting");
+ i_allow_nan = rb_intern("allow_nan");
+ i_symbolize_names = rb_intern("symbolize_names");
+ i_quirks_mode = rb_intern("quirks_mode");
+ i_object_class = rb_intern("object_class");
+ i_array_class = rb_intern("array_class");
+ i_match = rb_intern("match");
+ i_match_string = rb_intern("match_string");
+ i_key_p = rb_intern("key?");
+ i_deep_const_get = rb_intern("deep_const_get");
+ i_aset = rb_intern("[]=");
+ i_leftshift = rb_intern("<<");
+#ifdef HAVE_RUBY_ENCODING_H
+ CEncoding_UTF_8 = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-8"));
+ CEncoding_UTF_16BE = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-16be"));
+ CEncoding_UTF_16LE = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-16le"));
+ CEncoding_UTF_32BE = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-32be"));
+ CEncoding_UTF_32LE = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-32le"));
+ CEncoding_ASCII_8BIT = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("ascii-8bit"));
+ i_encoding = rb_intern("encoding");
+ i_encode = rb_intern("encode");
+#else
+ i_iconv = rb_intern("iconv");
+#endif
+}
+
+/*
+ * Local variables:
+ * mode: c
+ * c-file-style: ruby
+ * indent-tabs-mode: nil
+ * End:
+ */
--- /dev/null
+#ifndef _PARSER_H_
+#define _PARSER_H_
+
+#include "ruby.h"
+
+#if HAVE_RE_H
+#include "re.h"
+#endif
+
+#ifdef HAVE_RUBY_ENCODING_H
+#include "ruby/encoding.h"
+#define FORCE_UTF8(obj) ((obj) = rb_enc_associate(rb_str_dup(obj), rb_utf8_encoding()))
+#else
+#define FORCE_UTF8(obj)
+#endif
+#ifdef HAVE_RUBY_ST_H
+#include "ruby/st.h"
+#else
+#include "st.h"
+#endif
+
+#define option_given_p(opts, key) RTEST(rb_funcall(opts, i_key_p, 1, key))
+
+/* unicode */
+
+typedef unsigned long UTF32; /* at least 32 bits */
+typedef unsigned short UTF16; /* at least 16 bits */
+typedef unsigned char UTF8; /* typically 8 bits */
+
+#define UNI_REPLACEMENT_CHAR (UTF32)0x0000FFFD
+#define UNI_SUR_HIGH_START (UTF32)0xD800
+#define UNI_SUR_HIGH_END (UTF32)0xDBFF
+#define UNI_SUR_LOW_START (UTF32)0xDC00
+#define UNI_SUR_LOW_END (UTF32)0xDFFF
+
+typedef struct JSON_ParserStruct {
+ VALUE Vsource;
+ char *source;
+ long len;
+ char *memo;
+ VALUE create_id;
+ int max_nesting;
+ int current_nesting;
+ int allow_nan;
+ int parsing_name;
+ int symbolize_names;
+ int quirks_mode;
+ VALUE object_class;
+ VALUE array_class;
+ int create_additions;
+ VALUE match_string;
+} JSON_Parser;
+
+#define GET_PARSER \
+ GET_PARSER_INIT; \
+ if (!json->Vsource) rb_raise(rb_eTypeError, "uninitialized instance")
+#define GET_PARSER_INIT \
+ JSON_Parser *json; \
+ Data_Get_Struct(self, JSON_Parser, json)
+
+#define MinusInfinity "-Infinity"
+#define EVIL 0x666
+
+static UTF32 unescape_unicode(const unsigned char *p);
+static int convert_UTF32_to_UTF8(char *buf, UTF32 ch);
+static char *JSON_parse_object(JSON_Parser *json, char *p, char *pe, VALUE *result);
+static char *JSON_parse_value(JSON_Parser *json, char *p, char *pe, VALUE *result);
+static char *JSON_parse_integer(JSON_Parser *json, char *p, char *pe, VALUE *result);
+static char *JSON_parse_float(JSON_Parser *json, char *p, char *pe, VALUE *result);
+static char *JSON_parse_array(JSON_Parser *json, char *p, char *pe, VALUE *result);
+static VALUE json_string_unescape(VALUE result, char *string, char *stringEnd);
+static char *JSON_parse_string(JSON_Parser *json, char *p, char *pe, VALUE *result);
+static VALUE convert_encoding(VALUE source);
+static VALUE cParser_initialize(int argc, VALUE *argv, VALUE self);
+static VALUE cParser_parse(VALUE self);
+static JSON_Parser *JSON_allocate();
+static void JSON_mark(JSON_Parser *json);
+static void JSON_free(JSON_Parser *json);
+static VALUE cJSON_parser_s_allocate(VALUE klass);
+static VALUE cParser_source(VALUE self);
+
+#endif
--- /dev/null
+#include "parser.h"
+
+/* unicode */
+
+static const char digit_values[256] = {
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -1,
+ -1, -1, -1, -1, -1, -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
+ -1, -1, -1, -1, -1, -1, -1
+};
+
+static UTF32 unescape_unicode(const unsigned char *p)
+{
+ char b;
+ UTF32 result = 0;
+ b = digit_values[p[0]];
+ if (b < 0) return UNI_REPLACEMENT_CHAR;
+ result = (result << 4) | b;
+ b = digit_values[p[1]];
+ result = (result << 4) | b;
+ if (b < 0) return UNI_REPLACEMENT_CHAR;
+ b = digit_values[p[2]];
+ result = (result << 4) | b;
+ if (b < 0) return UNI_REPLACEMENT_CHAR;
+ b = digit_values[p[3]];
+ result = (result << 4) | b;
+ if (b < 0) return UNI_REPLACEMENT_CHAR;
+ return result;
+}
+
+static int convert_UTF32_to_UTF8(char *buf, UTF32 ch)
+{
+ int len = 1;
+ if (ch <= 0x7F) {
+ buf[0] = (char) ch;
+ } else if (ch <= 0x07FF) {
+ buf[0] = (char) ((ch >> 6) | 0xC0);
+ buf[1] = (char) ((ch & 0x3F) | 0x80);
+ len++;
+ } else if (ch <= 0xFFFF) {
+ buf[0] = (char) ((ch >> 12) | 0xE0);
+ buf[1] = (char) (((ch >> 6) & 0x3F) | 0x80);
+ buf[2] = (char) ((ch & 0x3F) | 0x80);
+ len += 2;
+ } else if (ch <= 0x1fffff) {
+ buf[0] =(char) ((ch >> 18) | 0xF0);
+ buf[1] =(char) (((ch >> 12) & 0x3F) | 0x80);
+ buf[2] =(char) (((ch >> 6) & 0x3F) | 0x80);
+ buf[3] =(char) ((ch & 0x3F) | 0x80);
+ len += 3;
+ } else {
+ buf[0] = '?';
+ }
+ return len;
+}
+
+#ifdef HAVE_RUBY_ENCODING_H
+static VALUE CEncoding_ASCII_8BIT, CEncoding_UTF_8, CEncoding_UTF_16BE,
+ CEncoding_UTF_16LE, CEncoding_UTF_32BE, CEncoding_UTF_32LE;
+static ID i_encoding, i_encode;
+#else
+static ID i_iconv;
+#endif
+
+static VALUE mJSON, mExt, cParser, eParserError, eNestingError;
+static VALUE CNaN, CInfinity, CMinusInfinity;
+
+static ID i_json_creatable_p, i_json_create, i_create_id, i_create_additions,
+ i_chr, i_max_nesting, i_allow_nan, i_symbolize_names, i_quirks_mode,
+ i_object_class, i_array_class, i_key_p, i_deep_const_get, i_match,
+ i_match_string, i_aset, i_leftshift;
+
+%%{
+ machine JSON_common;
+
+ cr = '\n';
+ cr_neg = [^\n];
+ ws = [ \t\r\n];
+ c_comment = '/*' ( any* - (any* '*/' any* ) ) '*/';
+ cpp_comment = '//' cr_neg* cr;
+ comment = c_comment | cpp_comment;
+ ignore = ws | comment;
+ name_separator = ':';
+ value_separator = ',';
+ Vnull = 'null';
+ Vfalse = 'false';
+ Vtrue = 'true';
+ VNaN = 'NaN';
+ VInfinity = 'Infinity';
+ VMinusInfinity = '-Infinity';
+ begin_value = [nft\"\-\[\{NI] | digit;
+ begin_object = '{';
+ end_object = '}';
+ begin_array = '[';
+ end_array = ']';
+ begin_string = '"';
+ begin_name = begin_string;
+ begin_number = digit | '-';
+}%%
+
+%%{
+ machine JSON_object;
+ include JSON_common;
+
+ write data;
+
+ action parse_value {
+ VALUE v = Qnil;
+ char *np = JSON_parse_value(json, fpc, pe, &v);
+ if (np == NULL) {
+ fhold; fbreak;
+ } else {
+ if (NIL_P(json->object_class)) {
+ rb_hash_aset(*result, last_name, v);
+ } else {
+ rb_funcall(*result, i_aset, 2, last_name, v);
+ }
+ fexec np;
+ }
+ }
+
+ action parse_name {
+ char *np;
+ json->parsing_name = 1;
+ np = JSON_parse_string(json, fpc, pe, &last_name);
+ json->parsing_name = 0;
+ if (np == NULL) { fhold; fbreak; } else fexec np;
+ }
+
+ action exit { fhold; fbreak; }
+
+ pair = ignore* begin_name >parse_name ignore* name_separator ignore* begin_value >parse_value;
+ next_pair = ignore* value_separator pair;
+
+ main := (
+ begin_object
+ (pair (next_pair)*)? ignore*
+ end_object
+ ) @exit;
+}%%
+
+static char *JSON_parse_object(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+ VALUE last_name = Qnil;
+ VALUE object_class = json->object_class;
+
+ if (json->max_nesting && json->current_nesting > json->max_nesting) {
+ rb_raise(eNestingError, "nesting of %d is too deep", json->current_nesting);
+ }
+
+ *result = NIL_P(object_class) ? rb_hash_new() : rb_class_new_instance(0, 0, object_class);
+
+ %% write init;
+ %% write exec;
+
+ if (cs >= JSON_object_first_final) {
+ if (json->create_additions) {
+ VALUE klassname = rb_hash_aref(*result, json->create_id);
+ if (!NIL_P(klassname)) {
+ VALUE klass = rb_funcall(mJSON, i_deep_const_get, 1, klassname);
+ if (RTEST(rb_funcall(klass, i_json_creatable_p, 0))) {
+ *result = rb_funcall(klass, i_json_create, 1, *result);
+ }
+ }
+ }
+ return p + 1;
+ } else {
+ return NULL;
+ }
+}
+
+
+%%{
+ machine JSON_value;
+ include JSON_common;
+
+ write data;
+
+ action parse_null {
+ *result = Qnil;
+ }
+ action parse_false {
+ *result = Qfalse;
+ }
+ action parse_true {
+ *result = Qtrue;
+ }
+ action parse_nan {
+ if (json->allow_nan) {
+ *result = CNaN;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p - 2);
+ }
+ }
+ action parse_infinity {
+ if (json->allow_nan) {
+ *result = CInfinity;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p - 8);
+ }
+ }
+ action parse_string {
+ char *np = JSON_parse_string(json, fpc, pe, result);
+ if (np == NULL) { fhold; fbreak; } else fexec np;
+ }
+
+ action parse_number {
+ char *np;
+ if(pe > fpc + 9 - json->quirks_mode && !strncmp(MinusInfinity, fpc, 9)) {
+ if (json->allow_nan) {
+ *result = CMinusInfinity;
+ fexec p + 10;
+ fhold; fbreak;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p);
+ }
+ }
+ np = JSON_parse_float(json, fpc, pe, result);
+ if (np != NULL) fexec np;
+ np = JSON_parse_integer(json, fpc, pe, result);
+ if (np != NULL) fexec np;
+ fhold; fbreak;
+ }
+
+ action parse_array {
+ char *np;
+ json->current_nesting++;
+ np = JSON_parse_array(json, fpc, pe, result);
+ json->current_nesting--;
+ if (np == NULL) { fhold; fbreak; } else fexec np;
+ }
+
+ action parse_object {
+ char *np;
+ json->current_nesting++;
+ np = JSON_parse_object(json, fpc, pe, result);
+ json->current_nesting--;
+ if (np == NULL) { fhold; fbreak; } else fexec np;
+ }
+
+ action exit { fhold; fbreak; }
+
+main := (
+ Vnull @parse_null |
+ Vfalse @parse_false |
+ Vtrue @parse_true |
+ VNaN @parse_nan |
+ VInfinity @parse_infinity |
+ begin_number >parse_number |
+ begin_string >parse_string |
+ begin_array >parse_array |
+ begin_object >parse_object
+ ) %*exit;
+}%%
+
+static char *JSON_parse_value(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+
+ %% write init;
+ %% write exec;
+
+ if (cs >= JSON_value_first_final) {
+ return p;
+ } else {
+ return NULL;
+ }
+}
+
+%%{
+ machine JSON_integer;
+
+ write data;
+
+ action exit { fhold; fbreak; }
+
+ main := '-'? ('0' | [1-9][0-9]*) (^[0-9]? @exit);
+}%%
+
+static char *JSON_parse_integer(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+
+ %% write init;
+ json->memo = p;
+ %% write exec;
+
+ if (cs >= JSON_integer_first_final) {
+ long len = p - json->memo;
+ *result = rb_Integer(rb_str_new(json->memo, len));
+ return p + 1;
+ } else {
+ return NULL;
+ }
+}
+
+%%{
+ machine JSON_float;
+ include JSON_common;
+
+ write data;
+
+ action exit { fhold; fbreak; }
+
+ main := '-'? (
+ (('0' | [1-9][0-9]*) '.' [0-9]+ ([Ee] [+\-]?[0-9]+)?)
+ | (('0' | [1-9][0-9]*) ([Ee] [+\-]?[0-9]+))
+ ) (^[0-9Ee.\-]? @exit );
+}%%
+
+static char *JSON_parse_float(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+
+ %% write init;
+ json->memo = p;
+ %% write exec;
+
+ if (cs >= JSON_float_first_final) {
+ long len = p - json->memo;
+ *result = rb_Float(rb_str_new(json->memo, len));
+ return p + 1;
+ } else {
+ return NULL;
+ }
+}
+
+
+%%{
+ machine JSON_array;
+ include JSON_common;
+
+ write data;
+
+ action parse_value {
+ VALUE v = Qnil;
+ char *np = JSON_parse_value(json, fpc, pe, &v);
+ if (np == NULL) {
+ fhold; fbreak;
+ } else {
+ if (NIL_P(json->array_class)) {
+ rb_ary_push(*result, v);
+ } else {
+ rb_funcall(*result, i_leftshift, 1, v);
+ }
+ fexec np;
+ }
+ }
+
+ action exit { fhold; fbreak; }
+
+ next_element = value_separator ignore* begin_value >parse_value;
+
+ main := begin_array ignore*
+ ((begin_value >parse_value ignore*)
+ (ignore* next_element ignore*)*)?
+ end_array @exit;
+}%%
+
+static char *JSON_parse_array(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+ VALUE array_class = json->array_class;
+
+ if (json->max_nesting && json->current_nesting > json->max_nesting) {
+ rb_raise(eNestingError, "nesting of %d is too deep", json->current_nesting);
+ }
+ *result = NIL_P(array_class) ? rb_ary_new() : rb_class_new_instance(0, 0, array_class);
+
+ %% write init;
+ %% write exec;
+
+ if(cs >= JSON_array_first_final) {
+ return p + 1;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p);
+ return NULL;
+ }
+}
+
+static VALUE json_string_unescape(VALUE result, char *string, char *stringEnd)
+{
+ char *p = string, *pe = string, *unescape;
+ int unescape_len;
+
+ while (pe < stringEnd) {
+ if (*pe == '\\') {
+ unescape = (char *) "?";
+ unescape_len = 1;
+ if (pe > p) rb_str_buf_cat(result, p, pe - p);
+ switch (*++pe) {
+ case 'n':
+ unescape = (char *) "\n";
+ break;
+ case 'r':
+ unescape = (char *) "\r";
+ break;
+ case 't':
+ unescape = (char *) "\t";
+ break;
+ case '"':
+ unescape = (char *) "\"";
+ break;
+ case '\\':
+ unescape = (char *) "\\";
+ break;
+ case 'b':
+ unescape = (char *) "\b";
+ break;
+ case 'f':
+ unescape = (char *) "\f";
+ break;
+ case 'u':
+ if (pe > stringEnd - 4) {
+ return Qnil;
+ } else {
+ char buf[4];
+ UTF32 ch = unescape_unicode((unsigned char *) ++pe);
+ pe += 3;
+ if (UNI_SUR_HIGH_START == (ch & 0xFC00)) {
+ pe++;
+ if (pe > stringEnd - 6) return Qnil;
+ if (pe[0] == '\\' && pe[1] == 'u') {
+ UTF32 sur = unescape_unicode((unsigned char *) pe + 2);
+ ch = (((ch & 0x3F) << 10) | ((((ch >> 6) & 0xF) + 1) << 16)
+ | (sur & 0x3FF));
+ pe += 5;
+ } else {
+ unescape = (char *) "?";
+ break;
+ }
+ }
+ unescape_len = convert_UTF32_to_UTF8(buf, ch);
+ unescape = buf;
+ }
+ break;
+ default:
+ p = pe;
+ continue;
+ }
+ rb_str_buf_cat(result, unescape, unescape_len);
+ p = ++pe;
+ } else {
+ pe++;
+ }
+ }
+ rb_str_buf_cat(result, p, pe - p);
+ return result;
+}
+
+%%{
+ machine JSON_string;
+ include JSON_common;
+
+ write data;
+
+ action parse_string {
+ *result = json_string_unescape(*result, json->memo + 1, p);
+ if (NIL_P(*result)) {
+ fhold;
+ fbreak;
+ } else {
+ FORCE_UTF8(*result);
+ fexec p + 1;
+ }
+ }
+
+ action exit { fhold; fbreak; }
+
+ main := '"' ((^([\"\\] | 0..0x1f) | '\\'[\"\\/bfnrt] | '\\u'[0-9a-fA-F]{4} | '\\'^([\"\\/bfnrtu]|0..0x1f))* %parse_string) '"' @exit;
+}%%
+
+static int
+match_i(VALUE regexp, VALUE klass, VALUE memo)
+{
+ if (regexp == Qundef) return ST_STOP;
+ if (RTEST(rb_funcall(klass, i_json_creatable_p, 0)) &&
+ RTEST(rb_funcall(regexp, i_match, 1, rb_ary_entry(memo, 0)))) {
+ rb_ary_push(memo, klass);
+ return ST_STOP;
+ }
+ return ST_CONTINUE;
+}
+
+static char *JSON_parse_string(JSON_Parser *json, char *p, char *pe, VALUE *result)
+{
+ int cs = EVIL;
+ VALUE match_string;
+
+ *result = rb_str_buf_new(0);
+ %% write init;
+ json->memo = p;
+ %% write exec;
+
+ if (json->create_additions && RTEST(match_string = json->match_string)) {
+ VALUE klass;
+ VALUE memo = rb_ary_new2(2);
+ rb_ary_push(memo, *result);
+ rb_hash_foreach(match_string, match_i, memo);
+ klass = rb_ary_entry(memo, 1);
+ if (RTEST(klass)) {
+ *result = rb_funcall(klass, i_json_create, 1, *result);
+ }
+ }
+
+ if (json->symbolize_names && json->parsing_name) {
+ *result = rb_str_intern(*result);
+ }
+ if (cs >= JSON_string_first_final) {
+ return p + 1;
+ } else {
+ return NULL;
+ }
+}
+
+/*
+ * Document-class: JSON::Ext::Parser
+ *
+ * This is the JSON parser implemented as a C extension. It can be configured
+ * to be used by setting
+ *
+ * JSON.parser = JSON::Ext::Parser
+ *
+ * with the method parser= in JSON.
+ *
+ */
+
+static VALUE convert_encoding(VALUE source)
+{
+ char *ptr = RSTRING_PTR(source);
+ long len = RSTRING_LEN(source);
+ if (len < 2) {
+ rb_raise(eParserError, "A JSON text must at least contain two octets!");
+ }
+#ifdef HAVE_RUBY_ENCODING_H
+ {
+ VALUE encoding = rb_funcall(source, i_encoding, 0);
+ if (encoding == CEncoding_ASCII_8BIT) {
+ if (len >= 4 && ptr[0] == 0 && ptr[1] == 0 && ptr[2] == 0) {
+ source = rb_funcall(source, i_encode, 2, CEncoding_UTF_8, CEncoding_UTF_32BE);
+ } else if (len >= 4 && ptr[0] == 0 && ptr[2] == 0) {
+ source = rb_funcall(source, i_encode, 2, CEncoding_UTF_8, CEncoding_UTF_16BE);
+ } else if (len >= 4 && ptr[1] == 0 && ptr[2] == 0 && ptr[3] == 0) {
+ source = rb_funcall(source, i_encode, 2, CEncoding_UTF_8, CEncoding_UTF_32LE);
+ } else if (len >= 4 && ptr[1] == 0 && ptr[3] == 0) {
+ source = rb_funcall(source, i_encode, 2, CEncoding_UTF_8, CEncoding_UTF_16LE);
+ } else {
+ source = rb_str_dup(source);
+ FORCE_UTF8(source);
+ }
+ } else {
+ source = rb_funcall(source, i_encode, 1, CEncoding_UTF_8);
+ }
+ }
+#else
+ if (len >= 4 && ptr[0] == 0 && ptr[1] == 0 && ptr[2] == 0) {
+ source = rb_funcall(mJSON, i_iconv, 3, rb_str_new2("utf-8"), rb_str_new2("utf-32be"), source);
+ } else if (len >= 4 && ptr[0] == 0 && ptr[2] == 0) {
+ source = rb_funcall(mJSON, i_iconv, 3, rb_str_new2("utf-8"), rb_str_new2("utf-16be"), source);
+ } else if (len >= 4 && ptr[1] == 0 && ptr[2] == 0 && ptr[3] == 0) {
+ source = rb_funcall(mJSON, i_iconv, 3, rb_str_new2("utf-8"), rb_str_new2("utf-32le"), source);
+ } else if (len >= 4 && ptr[1] == 0 && ptr[3] == 0) {
+ source = rb_funcall(mJSON, i_iconv, 3, rb_str_new2("utf-8"), rb_str_new2("utf-16le"), source);
+ }
+#endif
+ return source;
+}
+
+/*
+ * call-seq: new(source, opts => {})
+ *
+ * Creates a new JSON::Ext::Parser instance for the string _source_.
+ *
+ * Creates a new JSON::Ext::Parser instance for the string _source_.
+ *
+ * It will be configured by the _opts_ hash. _opts_ can have the following
+ * keys:
+ *
+ * _opts_ can have the following keys:
+ * * *max_nesting*: The maximum depth of nesting allowed in the parsed data
+ * structures. Disable depth checking with :max_nesting => false|nil|0, it
+ * defaults to 19.
+ * * *allow_nan*: If set to true, allow NaN, Infinity and -Infinity in
+ * defiance of RFC 4627 to be parsed by the Parser. This option defaults to
+ * false.
+ * * *symbolize_names*: If set to true, returns symbols for the names
+ * (keys) in a JSON object. Otherwise strings are returned, which is also
+ * the default.
+ * * *create_additions*: If set to false, the Parser doesn't create
+ * additions even if a matchin class and create_id was found. This option
+ * defaults to true.
+ * * *object_class*: Defaults to Hash
+ * * *array_class*: Defaults to Array
+ * * *quirks_mode*: Enables quirks_mode for parser, that is for example
+ * parsing single JSON values instead of documents is possible.
+ *
+ */
+static VALUE cParser_initialize(int argc, VALUE *argv, VALUE self)
+{
+ VALUE source, opts;
+ GET_PARSER_INIT;
+
+ if (json->Vsource) {
+ rb_raise(rb_eTypeError, "already initialized instance");
+ }
+ rb_scan_args(argc, argv, "11", &source, &opts);
+ if (!NIL_P(opts)) {
+ opts = rb_convert_type(opts, T_HASH, "Hash", "to_hash");
+ if (NIL_P(opts)) {
+ rb_raise(rb_eArgError, "opts needs to be like a hash");
+ } else {
+ VALUE tmp = ID2SYM(i_max_nesting);
+ if (option_given_p(opts, tmp)) {
+ VALUE max_nesting = rb_hash_aref(opts, tmp);
+ if (RTEST(max_nesting)) {
+ Check_Type(max_nesting, T_FIXNUM);
+ json->max_nesting = FIX2INT(max_nesting);
+ } else {
+ json->max_nesting = 0;
+ }
+ } else {
+ json->max_nesting = 19;
+ }
+ tmp = ID2SYM(i_allow_nan);
+ if (option_given_p(opts, tmp)) {
+ json->allow_nan = RTEST(rb_hash_aref(opts, tmp)) ? 1 : 0;
+ } else {
+ json->allow_nan = 0;
+ }
+ tmp = ID2SYM(i_symbolize_names);
+ if (option_given_p(opts, tmp)) {
+ json->symbolize_names = RTEST(rb_hash_aref(opts, tmp)) ? 1 : 0;
+ } else {
+ json->symbolize_names = 0;
+ }
+ tmp = ID2SYM(i_quirks_mode);
+ if (option_given_p(opts, tmp)) {
+ VALUE quirks_mode = rb_hash_aref(opts, tmp);
+ json->quirks_mode = RTEST(quirks_mode) ? 1 : 0;
+ } else {
+ json->quirks_mode = 0;
+ }
+ tmp = ID2SYM(i_create_additions);
+ if (option_given_p(opts, tmp)) {
+ json->create_additions = RTEST(rb_hash_aref(opts, tmp));
+ } else {
+ json->create_additions = 0;
+ }
+ tmp = ID2SYM(i_create_id);
+ if (option_given_p(opts, tmp)) {
+ json->create_id = rb_hash_aref(opts, tmp);
+ } else {
+ json->create_id = rb_funcall(mJSON, i_create_id, 0);
+ }
+ tmp = ID2SYM(i_object_class);
+ if (option_given_p(opts, tmp)) {
+ json->object_class = rb_hash_aref(opts, tmp);
+ } else {
+ json->object_class = Qnil;
+ }
+ tmp = ID2SYM(i_array_class);
+ if (option_given_p(opts, tmp)) {
+ json->array_class = rb_hash_aref(opts, tmp);
+ } else {
+ json->array_class = Qnil;
+ }
+ tmp = ID2SYM(i_match_string);
+ if (option_given_p(opts, tmp)) {
+ VALUE match_string = rb_hash_aref(opts, tmp);
+ json->match_string = RTEST(match_string) ? match_string : Qnil;
+ } else {
+ json->match_string = Qnil;
+ }
+ }
+ } else {
+ json->max_nesting = 19;
+ json->allow_nan = 0;
+ json->create_additions = 1;
+ json->create_id = rb_funcall(mJSON, i_create_id, 0);
+ json->object_class = Qnil;
+ json->array_class = Qnil;
+ }
+ if (!json->quirks_mode) {
+ source = convert_encoding(StringValue(source));
+ }
+ json->current_nesting = 0;
+ json->len = RSTRING_LEN(source);
+ json->source = RSTRING_PTR(source);;
+ json->Vsource = source;
+ return self;
+}
+
+%%{
+ machine JSON;
+
+ write data;
+
+ include JSON_common;
+
+ action parse_object {
+ char *np;
+ json->current_nesting = 1;
+ np = JSON_parse_object(json, fpc, pe, &result);
+ if (np == NULL) { fhold; fbreak; } else fexec np;
+ }
+
+ action parse_array {
+ char *np;
+ json->current_nesting = 1;
+ np = JSON_parse_array(json, fpc, pe, &result);
+ if (np == NULL) { fhold; fbreak; } else fexec np;
+ }
+
+ main := ignore* (
+ begin_object >parse_object |
+ begin_array >parse_array
+ ) ignore*;
+}%%
+
+static VALUE cParser_parse_strict(VALUE self)
+{
+ char *p, *pe;
+ int cs = EVIL;
+ VALUE result = Qnil;
+ GET_PARSER;
+
+ %% write init;
+ p = json->source;
+ pe = p + json->len;
+ %% write exec;
+
+ if (cs >= JSON_first_final && p == pe) {
+ return result;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p);
+ return Qnil;
+ }
+}
+
+
+%%{
+ machine JSON_quirks_mode;
+
+ write data;
+
+ include JSON_common;
+
+ action parse_value {
+ char *np = JSON_parse_value(json, fpc, pe, &result);
+ if (np == NULL) { fhold; fbreak; } else fexec np;
+ }
+
+ main := ignore* (
+ begin_value >parse_value
+ ) ignore*;
+}%%
+
+static VALUE cParser_parse_quirks_mode(VALUE self)
+{
+ char *p, *pe;
+ int cs = EVIL;
+ VALUE result = Qnil;
+ GET_PARSER;
+
+ %% write init;
+ p = json->source;
+ pe = p + json->len;
+ %% write exec;
+
+ if (cs >= JSON_quirks_mode_first_final && p == pe) {
+ return result;
+ } else {
+ rb_raise(eParserError, "%u: unexpected token at '%s'", __LINE__, p);
+ return Qnil;
+ }
+}
+
+/*
+ * call-seq: parse()
+ *
+ * Parses the current JSON text _source_ and returns the complete data
+ * structure as a result.
+ */
+static VALUE cParser_parse(VALUE self)
+{
+ GET_PARSER;
+
+ if (json->quirks_mode) {
+ return cParser_parse_quirks_mode(self);
+ } else {
+ return cParser_parse_strict(self);
+ }
+}
+
+
+static JSON_Parser *JSON_allocate()
+{
+ JSON_Parser *json = ALLOC(JSON_Parser);
+ MEMZERO(json, JSON_Parser, 1);
+ return json;
+}
+
+static void JSON_mark(JSON_Parser *json)
+{
+ rb_gc_mark_maybe(json->Vsource);
+ rb_gc_mark_maybe(json->create_id);
+ rb_gc_mark_maybe(json->object_class);
+ rb_gc_mark_maybe(json->array_class);
+ rb_gc_mark_maybe(json->match_string);
+}
+
+static void JSON_free(JSON_Parser *json)
+{
+ ruby_xfree(json);
+}
+
+static VALUE cJSON_parser_s_allocate(VALUE klass)
+{
+ JSON_Parser *json = JSON_allocate();
+ return Data_Wrap_Struct(klass, JSON_mark, JSON_free, json);
+}
+
+/*
+ * call-seq: source()
+ *
+ * Returns a copy of the current _source_ string, that was used to construct
+ * this Parser.
+ */
+static VALUE cParser_source(VALUE self)
+{
+ GET_PARSER;
+ return rb_str_dup(json->Vsource);
+}
+
+/*
+ * call-seq: quirks_mode?()
+ *
+ * Returns a true, if this parser is in quirks_mode, false otherwise.
+ */
+static VALUE cParser_quirks_mode_p(VALUE self)
+{
+ GET_PARSER;
+ return json->quirks_mode ? Qtrue : Qfalse;
+}
+
+
+void Init_parser()
+{
+ rb_require("json/common");
+ mJSON = rb_define_module("JSON");
+ mExt = rb_define_module_under(mJSON, "Ext");
+ cParser = rb_define_class_under(mExt, "Parser", rb_cObject);
+ eParserError = rb_path2class("JSON::ParserError");
+ eNestingError = rb_path2class("JSON::NestingError");
+ rb_define_alloc_func(cParser, cJSON_parser_s_allocate);
+ rb_define_method(cParser, "initialize", cParser_initialize, -1);
+ rb_define_method(cParser, "parse", cParser_parse, 0);
+ rb_define_method(cParser, "source", cParser_source, 0);
+ rb_define_method(cParser, "quirks_mode?", cParser_quirks_mode_p, 0);
+
+ CNaN = rb_const_get(mJSON, rb_intern("NaN"));
+ CInfinity = rb_const_get(mJSON, rb_intern("Infinity"));
+ CMinusInfinity = rb_const_get(mJSON, rb_intern("MinusInfinity"));
+
+ i_json_creatable_p = rb_intern("json_creatable?");
+ i_json_create = rb_intern("json_create");
+ i_create_id = rb_intern("create_id");
+ i_create_additions = rb_intern("create_additions");
+ i_chr = rb_intern("chr");
+ i_max_nesting = rb_intern("max_nesting");
+ i_allow_nan = rb_intern("allow_nan");
+ i_symbolize_names = rb_intern("symbolize_names");
+ i_quirks_mode = rb_intern("quirks_mode");
+ i_object_class = rb_intern("object_class");
+ i_array_class = rb_intern("array_class");
+ i_match = rb_intern("match");
+ i_match_string = rb_intern("match_string");
+ i_key_p = rb_intern("key?");
+ i_deep_const_get = rb_intern("deep_const_get");
+ i_aset = rb_intern("[]=");
+ i_leftshift = rb_intern("<<");
+#ifdef HAVE_RUBY_ENCODING_H
+ CEncoding_UTF_8 = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-8"));
+ CEncoding_UTF_16BE = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-16be"));
+ CEncoding_UTF_16LE = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-16le"));
+ CEncoding_UTF_32BE = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-32be"));
+ CEncoding_UTF_32LE = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("utf-32le"));
+ CEncoding_ASCII_8BIT = rb_funcall(rb_path2class("Encoding"), rb_intern("find"), 1, rb_str_new2("ascii-8bit"));
+ i_encoding = rb_intern("encoding");
+ i_encode = rb_intern("encode");
+#else
+ i_iconv = rb_intern("iconv");
+#endif
+}
+
+/*
+ * Local variables:
+ * mode: c
+ * c-file-style: ruby
+ * indent-tabs-mode: nil
+ * End:
+ */
--- /dev/null
+#!/usr/bin/env ruby
+
+require 'rbconfig'
+require 'fileutils'
+include FileUtils::Verbose
+
+include Config
+
+bindir = CONFIG["bindir"]
+cd 'bin' do
+ filename = 'edit_json.rb'
+ #install(filename, bindir)
+end
+sitelibdir = CONFIG["sitelibdir"]
+cd 'lib' do
+ install('json.rb', sitelibdir)
+ mkdir_p File.join(sitelibdir, 'json')
+ for file in Dir['json/**/*.{rb,xpm}']
+ d = File.join(sitelibdir, file)
+ mkdir_p File.dirname(d)
+ install(file, d)
+ end
+ install(File.join('json', 'editor.rb'), File.join(sitelibdir,'json'))
+ install(File.join('json', 'json.xpm'), File.join(sitelibdir,'json'))
+end
+warn " *** Installed PURE ruby library."
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import org.jruby.exceptions.RaiseException;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.util.ByteList;
+
+/**
+ * A class specialized in transcoding a certain String format into another,
+ * using UTF-8 ByteLists as both input and output.
+ */
+abstract class ByteListTranscoder {
+ protected final ThreadContext context;
+
+ protected ByteList src;
+ protected int srcEnd;
+ /** Position where the last read character started */
+ protected int charStart;
+ /** Position of the next character to read */
+ protected int pos;
+
+ private ByteList out;
+ /**
+ * When a character that can be copied straight into the output is found,
+ * its index is stored on this variable, and copying is delayed until
+ * the sequence of characters that can be copied ends.
+ *
+ * <p>The variable stores -1 when not in a plain sequence.
+ */
+ private int quoteStart = -1;
+
+ protected ByteListTranscoder(ThreadContext context) {
+ this.context = context;
+ }
+
+ protected void init(ByteList src, ByteList out) {
+ this.init(src, 0, src.length(), out);
+ }
+
+ protected void init(ByteList src, int start, int end, ByteList out) {
+ this.src = src;
+ this.pos = start;
+ this.charStart = start;
+ this.srcEnd = end;
+ this.out = out;
+ }
+
+ /**
+ * Returns whether there are any characters left to be read.
+ */
+ protected boolean hasNext() {
+ return pos < srcEnd;
+ }
+
+ /**
+ * Returns the next character in the buffer.
+ */
+ private char next() {
+ return src.charAt(pos++);
+ }
+
+ /**
+ * Reads an UTF-8 character from the input and returns its code point,
+ * while advancing the input position.
+ *
+ * <p>Raises an {@link #invalidUtf8()} exception if an invalid byte
+ * is found.
+ */
+ protected int readUtf8Char() {
+ charStart = pos;
+ char head = next();
+ if (head <= 0x7f) { // 0b0xxxxxxx (ASCII)
+ return head;
+ }
+ if (head <= 0xbf) { // 0b10xxxxxx
+ throw invalidUtf8(); // tail byte with no head
+ }
+ if (head <= 0xdf) { // 0b110xxxxx
+ ensureMin(1);
+ int cp = ((head & 0x1f) << 6)
+ | nextPart();
+ if (cp < 0x0080) throw invalidUtf8();
+ return cp;
+ }
+ if (head <= 0xef) { // 0b1110xxxx
+ ensureMin(2);
+ int cp = ((head & 0x0f) << 12)
+ | (nextPart() << 6)
+ | nextPart();
+ if (cp < 0x0800) throw invalidUtf8();
+ return cp;
+ }
+ if (head <= 0xf7) { // 0b11110xxx
+ ensureMin(3);
+ int cp = ((head & 0x07) << 18)
+ | (nextPart() << 12)
+ | (nextPart() << 6)
+ | nextPart();
+ if (!Character.isValidCodePoint(cp)) throw invalidUtf8();
+ return cp;
+ }
+ // 0b11111xxx?
+ throw invalidUtf8();
+ }
+
+ /**
+ * Throws a GeneratorError if the input list doesn't have at least this
+ * many bytes left.
+ */
+ protected void ensureMin(int n) {
+ if (pos + n > srcEnd) throw incompleteUtf8();
+ }
+
+ /**
+ * Reads the next byte of a multi-byte UTF-8 character and returns its
+ * contents (lower 6 bits).
+ *
+ * <p>Throws a GeneratorError if the byte is not a valid tail.
+ */
+ private int nextPart() {
+ char c = next();
+ // tail bytes must be 0b10xxxxxx
+ if ((c & 0xc0) != 0x80) throw invalidUtf8();
+ return c & 0x3f;
+ }
+
+
+ protected void quoteStart() {
+ if (quoteStart == -1) quoteStart = charStart;
+ }
+
+ /**
+ * When in a sequence of characters that can be copied directly,
+ * interrupts the sequence and copies it to the output buffer.
+ *
+ * @param endPos The offset until which the direct character quoting should
+ * occur. You may pass {@link #pos} to quote until the most
+ * recently read character, or {@link #charStart} to quote
+ * until the character before it.
+ */
+ protected void quoteStop(int endPos) {
+ if (quoteStart != -1) {
+ out.append(src, quoteStart, endPos - quoteStart);
+ quoteStart = -1;
+ }
+ }
+
+ protected void append(int b) {
+ out.append(b);
+ }
+
+ protected void append(byte[] origin, int start, int length) {
+ out.append(origin, start, length);
+ }
+
+
+ protected abstract RaiseException invalidUtf8();
+
+ protected RaiseException incompleteUtf8() {
+ return invalidUtf8();
+ }
+}
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import org.jruby.Ruby;
+import org.jruby.RubyArray;
+import org.jruby.RubyBignum;
+import org.jruby.RubyBoolean;
+import org.jruby.RubyClass;
+import org.jruby.RubyFixnum;
+import org.jruby.RubyFloat;
+import org.jruby.RubyHash;
+import org.jruby.RubyNumeric;
+import org.jruby.RubyString;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.runtime.builtin.IRubyObject;
+import org.jruby.util.ByteList;
+
+public final class Generator {
+ private Generator() {
+ throw new RuntimeException();
+ }
+
+ /**
+ * Encodes the given object as a JSON string, using the given handler.
+ */
+ static <T extends IRubyObject> RubyString
+ generateJson(ThreadContext context, T object,
+ Handler<? super T> handler, IRubyObject[] args) {
+ Session session = new Session(context, args.length > 0 ? args[0]
+ : null);
+ return session.infect(handler.generateNew(session, object));
+ }
+
+ /**
+ * Encodes the given object as a JSON string, detecting the appropriate handler
+ * for the given object.
+ */
+ static <T extends IRubyObject> RubyString
+ generateJson(ThreadContext context, T object, IRubyObject[] args) {
+ Handler<? super T> handler = getHandlerFor(context.getRuntime(), object);
+ return generateJson(context, object, handler, args);
+ }
+
+ /**
+ * Encodes the given object as a JSON string, using the appropriate
+ * handler if one is found or calling #to_json if not.
+ */
+ public static <T extends IRubyObject> RubyString
+ generateJson(ThreadContext context, T object,
+ GeneratorState config) {
+ Session session = new Session(context, config);
+ Handler<? super T> handler = getHandlerFor(context.getRuntime(), object);
+ return handler.generateNew(session, object);
+ }
+
+ /**
+ * Returns the best serialization handler for the given object.
+ */
+ // Java's generics can't handle this satisfactorily, so I'll just leave
+ // the best I could get and ignore the warnings
+ @SuppressWarnings("unchecked")
+ private static <T extends IRubyObject>
+ Handler<? super T> getHandlerFor(Ruby runtime, T object) {
+ RubyClass metaClass = object.getMetaClass();
+ if (metaClass == runtime.getString()) return (Handler)STRING_HANDLER;
+ if (metaClass == runtime.getFixnum()) return (Handler)FIXNUM_HANDLER;
+ if (metaClass == runtime.getHash()) return (Handler)HASH_HANDLER;
+ if (metaClass == runtime.getArray()) return (Handler)ARRAY_HANDLER;
+ if (object.isNil()) return (Handler)NIL_HANDLER;
+ if (object == runtime.getTrue()) return (Handler)TRUE_HANDLER;
+ if (object == runtime.getFalse()) return (Handler)FALSE_HANDLER;
+ if (metaClass == runtime.getFloat()) return (Handler)FLOAT_HANDLER;
+ if (metaClass == runtime.getBignum()) return (Handler)BIGNUM_HANDLER;
+ return GENERIC_HANDLER;
+ }
+
+
+ /* Generator context */
+
+ /**
+ * A class that concentrates all the information that is shared by
+ * generators working on a single session.
+ *
+ * <p>A session is defined as the process of serializing a single root
+ * object; any handler directly called by container handlers (arrays and
+ * hashes/objects) shares this object with its caller.
+ *
+ * <p>Note that anything called indirectly (via {@link GENERIC_HANDLER})
+ * won't be part of the session.
+ */
+ static class Session {
+ private final ThreadContext context;
+ private GeneratorState state;
+ private IRubyObject possibleState;
+ private RuntimeInfo info;
+ private StringEncoder stringEncoder;
+
+ private boolean tainted = false;
+ private boolean untrusted = false;
+
+ Session(ThreadContext context, GeneratorState state) {
+ this.context = context;
+ this.state = state;
+ }
+
+ Session(ThreadContext context, IRubyObject possibleState) {
+ this.context = context;
+ this.possibleState = possibleState == null || possibleState.isNil()
+ ? null : possibleState;
+ }
+
+ public ThreadContext getContext() {
+ return context;
+ }
+
+ public Ruby getRuntime() {
+ return context.getRuntime();
+ }
+
+ public GeneratorState getState() {
+ if (state == null) {
+ state = GeneratorState.fromState(context, getInfo(), possibleState);
+ }
+ return state;
+ }
+
+ public RuntimeInfo getInfo() {
+ if (info == null) info = RuntimeInfo.forRuntime(getRuntime());
+ return info;
+ }
+
+ public StringEncoder getStringEncoder() {
+ if (stringEncoder == null) {
+ stringEncoder = new StringEncoder(context, getState().asciiOnly());
+ }
+ return stringEncoder;
+ }
+
+ public void infectBy(IRubyObject object) {
+ if (object.isTaint()) tainted = true;
+ if (object.isUntrusted()) untrusted = true;
+ }
+
+ public <T extends IRubyObject> T infect(T object) {
+ if (tainted) object.setTaint(true);
+ if (untrusted) object.setUntrusted(true);
+ return object;
+ }
+ }
+
+
+ /* Handler base classes */
+
+ private static abstract class Handler<T extends IRubyObject> {
+ /**
+ * Returns an estimative of how much space the serialization of the
+ * given object will take. Used for allocating enough buffer space
+ * before invoking other methods.
+ */
+ int guessSize(Session session, T object) {
+ return 4;
+ }
+
+ RubyString generateNew(Session session, T object) {
+ ByteList buffer = new ByteList(guessSize(session, object));
+ generate(session, object, buffer);
+ return RubyString.newString(session.getRuntime(), buffer);
+ }
+
+ abstract void generate(Session session, T object, ByteList buffer);
+ }
+
+ /**
+ * A handler that returns a fixed keyword regardless of the passed object.
+ */
+ private static class KeywordHandler<T extends IRubyObject>
+ extends Handler<T> {
+ private final ByteList keyword;
+
+ private KeywordHandler(String keyword) {
+ this.keyword = new ByteList(ByteList.plain(keyword), false);
+ }
+
+ @Override
+ int guessSize(Session session, T object) {
+ return keyword.length();
+ }
+
+ @Override
+ RubyString generateNew(Session session, T object) {
+ return RubyString.newStringShared(session.getRuntime(), keyword);
+ }
+
+ @Override
+ void generate(Session session, T object, ByteList buffer) {
+ buffer.append(keyword);
+ }
+ }
+
+
+ /* Handlers */
+
+ static final Handler<RubyBignum> BIGNUM_HANDLER =
+ new Handler<RubyBignum>() {
+ @Override
+ void generate(Session session, RubyBignum object, ByteList buffer) {
+ // JRUBY-4751: RubyBignum.to_s() returns generic object
+ // representation (fixed in 1.5, but we maintain backwards
+ // compatibility; call to_s(IRubyObject[]) then
+ buffer.append(((RubyString)object.to_s(IRubyObject.NULL_ARRAY)).getByteList());
+ }
+ };
+
+ static final Handler<RubyFixnum> FIXNUM_HANDLER =
+ new Handler<RubyFixnum>() {
+ @Override
+ void generate(Session session, RubyFixnum object, ByteList buffer) {
+ buffer.append(object.to_s().getByteList());
+ }
+ };
+
+ static final Handler<RubyFloat> FLOAT_HANDLER =
+ new Handler<RubyFloat>() {
+ @Override
+ void generate(Session session, RubyFloat object, ByteList buffer) {
+ double value = RubyFloat.num2dbl(object);
+
+ if (Double.isInfinite(value) || Double.isNaN(value)) {
+ if (!session.getState().allowNaN()) {
+ throw Utils.newException(session.getContext(),
+ Utils.M_GENERATOR_ERROR,
+ object + " not allowed in JSON");
+ }
+ }
+ buffer.append(((RubyString)object.to_s()).getByteList());
+ }
+ };
+
+ static final Handler<RubyArray> ARRAY_HANDLER =
+ new Handler<RubyArray>() {
+ @Override
+ int guessSize(Session session, RubyArray object) {
+ GeneratorState state = session.getState();
+ int depth = state.getDepth();
+ int perItem =
+ 4 // prealloc
+ + (depth + 1) * state.getIndent().length() // indent
+ + 1 + state.getArrayNl().length(); // ',' arrayNl
+ return 2 + object.size() * perItem;
+ }
+
+ @Override
+ void generate(Session session, RubyArray object, ByteList buffer) {
+ ThreadContext context = session.getContext();
+ Ruby runtime = context.getRuntime();
+ GeneratorState state = session.getState();
+ int depth = state.increaseDepth();
+
+ ByteList indentUnit = state.getIndent();
+ byte[] shift = Utils.repeat(indentUnit, depth);
+
+ ByteList arrayNl = state.getArrayNl();
+ byte[] delim = new byte[1 + arrayNl.length()];
+ delim[0] = ',';
+ System.arraycopy(arrayNl.unsafeBytes(), arrayNl.begin(), delim, 1,
+ arrayNl.length());
+
+ session.infectBy(object);
+
+ buffer.append((byte)'[');
+ buffer.append(arrayNl);
+ boolean firstItem = true;
+ for (int i = 0, t = object.getLength(); i < t; i++) {
+ IRubyObject element = object.eltInternal(i);
+ session.infectBy(element);
+ if (firstItem) {
+ firstItem = false;
+ } else {
+ buffer.append(delim);
+ }
+ buffer.append(shift);
+ Handler<IRubyObject> handler = getHandlerFor(runtime, element);
+ handler.generate(session, element, buffer);
+ }
+
+ state.decreaseDepth();
+ if (arrayNl.length() != 0) {
+ buffer.append(arrayNl);
+ buffer.append(shift, 0, state.getDepth() * indentUnit.length());
+ }
+
+ buffer.append((byte)']');
+ }
+ };
+
+ static final Handler<RubyHash> HASH_HANDLER =
+ new Handler<RubyHash>() {
+ @Override
+ int guessSize(Session session, RubyHash object) {
+ GeneratorState state = session.getState();
+ int perItem =
+ 12 // key, colon, comma
+ + (state.getDepth() + 1) * state.getIndent().length()
+ + state.getSpaceBefore().length()
+ + state.getSpace().length();
+ return 2 + object.size() * perItem;
+ }
+
+ @Override
+ void generate(final Session session, RubyHash object,
+ final ByteList buffer) {
+ ThreadContext context = session.getContext();
+ final Ruby runtime = context.getRuntime();
+ final GeneratorState state = session.getState();
+ final int depth = state.increaseDepth();
+
+ final ByteList objectNl = state.getObjectNl();
+ final byte[] indent = Utils.repeat(state.getIndent(), depth);
+ final ByteList spaceBefore = state.getSpaceBefore();
+ final ByteList space = state.getSpace();
+
+ buffer.append((byte)'{');
+ buffer.append(objectNl);
+ object.visitAll(new RubyHash.Visitor() {
+ private boolean firstPair = true;
+
+ @Override
+ public void visit(IRubyObject key, IRubyObject value) {
+ if (firstPair) {
+ firstPair = false;
+ } else {
+ buffer.append((byte)',');
+ buffer.append(objectNl);
+ }
+ if (objectNl.length() != 0) buffer.append(indent);
+
+ STRING_HANDLER.generate(session, key.asString(), buffer);
+ session.infectBy(key);
+
+ buffer.append(spaceBefore);
+ buffer.append((byte)':');
+ buffer.append(space);
+
+ Handler<IRubyObject> valueHandler = getHandlerFor(runtime, value);
+ valueHandler.generate(session, value, buffer);
+ session.infectBy(value);
+ }
+ });
+ state.decreaseDepth();
+ if (objectNl.length() != 0) {
+ buffer.append(objectNl);
+ buffer.append(Utils.repeat(state.getIndent(), state.getDepth()));
+ }
+ buffer.append((byte)'}');
+ }
+ };
+
+ static final Handler<RubyString> STRING_HANDLER =
+ new Handler<RubyString>() {
+ @Override
+ int guessSize(Session session, RubyString object) {
+ // for most applications, most strings will be just a set of
+ // printable ASCII characters without any escaping, so let's
+ // just allocate enough space for that + the quotes
+ return 2 + object.getByteList().length();
+ }
+
+ @Override
+ void generate(Session session, RubyString object, ByteList buffer) {
+ RuntimeInfo info = session.getInfo();
+ RubyString src;
+
+ if (info.encodingsSupported() &&
+ object.encoding(session.getContext()) != info.utf8.get()) {
+ src = (RubyString)object.encode(session.getContext(),
+ info.utf8.get());
+ } else {
+ src = object;
+ }
+
+ session.getStringEncoder().encode(src.getByteList(), buffer);
+ }
+ };
+
+ static final Handler<RubyBoolean> TRUE_HANDLER =
+ new KeywordHandler<RubyBoolean>("true");
+ static final Handler<RubyBoolean> FALSE_HANDLER =
+ new KeywordHandler<RubyBoolean>("false");
+ static final Handler<IRubyObject> NIL_HANDLER =
+ new KeywordHandler<IRubyObject>("null");
+
+ /**
+ * The default handler (<code>Object#to_json</code>): coerces the object
+ * to string using <code>#to_s</code>, and serializes that string.
+ */
+ static final Handler<IRubyObject> OBJECT_HANDLER =
+ new Handler<IRubyObject>() {
+ @Override
+ RubyString generateNew(Session session, IRubyObject object) {
+ RubyString str = object.asString();
+ return STRING_HANDLER.generateNew(session, str);
+ }
+
+ @Override
+ void generate(Session session, IRubyObject object, ByteList buffer) {
+ RubyString str = object.asString();
+ STRING_HANDLER.generate(session, str, buffer);
+ }
+ };
+
+ /**
+ * A handler that simply calls <code>#to_json(state)</code> on the
+ * given object.
+ */
+ static final Handler<IRubyObject> GENERIC_HANDLER =
+ new Handler<IRubyObject>() {
+ @Override
+ RubyString generateNew(Session session, IRubyObject object) {
+ IRubyObject result =
+ object.callMethod(session.getContext(), "to_json",
+ new IRubyObject[] {session.getState()});
+ if (result instanceof RubyString) return (RubyString)result;
+ throw session.getRuntime().newTypeError("to_json must return a String");
+ }
+
+ @Override
+ void generate(Session session, IRubyObject object, ByteList buffer) {
+ RubyString result = generateNew(session, object);
+ buffer.append(result.getByteList());
+ }
+ };
+}
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import java.lang.ref.WeakReference;
+import org.jruby.Ruby;
+import org.jruby.RubyArray;
+import org.jruby.RubyBoolean;
+import org.jruby.RubyFixnum;
+import org.jruby.RubyFloat;
+import org.jruby.RubyHash;
+import org.jruby.RubyInteger;
+import org.jruby.RubyModule;
+import org.jruby.RubyNumeric;
+import org.jruby.RubyString;
+import org.jruby.anno.JRubyMethod;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.runtime.builtin.IRubyObject;
+import org.jruby.util.ByteList;
+
+/**
+ * A class that populates the
+ * <code>Json::Ext::Generator::GeneratorMethods</code> module.
+ *
+ * @author mernen
+ */
+class GeneratorMethods {
+ /**
+ * Populates the given module with all modules and their methods
+ * @param info
+ * @param generatorMethodsModule The module to populate
+ * (normally <code>JSON::Generator::GeneratorMethods</code>)
+ */
+ static void populate(RuntimeInfo info, RubyModule module) {
+ defineMethods(module, "Array", RbArray.class);
+ defineMethods(module, "FalseClass", RbFalse.class);
+ defineMethods(module, "Float", RbFloat.class);
+ defineMethods(module, "Hash", RbHash.class);
+ defineMethods(module, "Integer", RbInteger.class);
+ defineMethods(module, "NilClass", RbNil.class);
+ defineMethods(module, "Object", RbObject.class);
+ defineMethods(module, "String", RbString.class);
+ defineMethods(module, "TrueClass", RbTrue.class);
+
+ info.stringExtendModule = new WeakReference<RubyModule>(module.defineModuleUnder("String")
+ .defineModuleUnder("Extend"));
+ info.stringExtendModule.get().defineAnnotatedMethods(StringExtend.class);
+ }
+
+ /**
+ * Convenience method for defining methods on a submodule.
+ * @param parentModule
+ * @param submoduleName
+ * @param klass
+ */
+ private static void defineMethods(RubyModule parentModule,
+ String submoduleName, Class klass) {
+ RubyModule submodule = parentModule.defineModuleUnder(submoduleName);
+ submodule.defineAnnotatedMethods(klass);
+ }
+
+
+ public static class RbHash {
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json(ThreadContext context,
+ IRubyObject vSelf, IRubyObject[] args) {
+ return Generator.generateJson(context, (RubyHash)vSelf,
+ Generator.HASH_HANDLER, args);
+ }
+ }
+
+ public static class RbArray {
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json(ThreadContext context,
+ IRubyObject vSelf, IRubyObject[] args) {
+ return Generator.generateJson(context, (RubyArray)vSelf,
+ Generator.ARRAY_HANDLER, args);
+ }
+ }
+
+ public static class RbInteger {
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json(ThreadContext context,
+ IRubyObject vSelf, IRubyObject[] args) {
+ return Generator.generateJson(context, vSelf, args);
+ }
+ }
+
+ public static class RbFloat {
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json(ThreadContext context,
+ IRubyObject vSelf, IRubyObject[] args) {
+ return Generator.generateJson(context, (RubyFloat)vSelf,
+ Generator.FLOAT_HANDLER, args);
+ }
+ }
+
+ public static class RbString {
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json(ThreadContext context,
+ IRubyObject vSelf, IRubyObject[] args) {
+ return Generator.generateJson(context, (RubyString)vSelf,
+ Generator.STRING_HANDLER, args);
+ }
+
+ /**
+ * <code>{@link RubyString String}#to_json_raw(*)</code>
+ *
+ * <p>This method creates a JSON text from the result of a call to
+ * {@link #to_json_raw_object} of this String.
+ */
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json_raw(ThreadContext context,
+ IRubyObject vSelf, IRubyObject[] args) {
+ RubyHash obj = toJsonRawObject(context, Utils.ensureString(vSelf));
+ return Generator.generateJson(context, obj,
+ Generator.HASH_HANDLER, args);
+ }
+
+ /**
+ * <code>{@link RubyString String}#to_json_raw_object(*)</code>
+ *
+ * <p>This method creates a raw object Hash, that can be nested into
+ * other data structures and will be unparsed as a raw string. This
+ * method should be used if you want to convert raw strings to JSON
+ * instead of UTF-8 strings, e.g. binary data.
+ */
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json_raw_object(ThreadContext context,
+ IRubyObject vSelf, IRubyObject[] args) {
+ return toJsonRawObject(context, Utils.ensureString(vSelf));
+ }
+
+ private static RubyHash toJsonRawObject(ThreadContext context,
+ RubyString self) {
+ Ruby runtime = context.getRuntime();
+ RubyHash result = RubyHash.newHash(runtime);
+
+ IRubyObject createId = RuntimeInfo.forRuntime(runtime)
+ .jsonModule.get().callMethod(context, "create_id");
+ result.op_aset(context, createId, self.getMetaClass().to_s());
+
+ ByteList bl = self.getByteList();
+ byte[] uBytes = bl.unsafeBytes();
+ RubyArray array = runtime.newArray(bl.length());
+ for (int i = bl.begin(), t = bl.begin() + bl.length(); i < t; i++) {
+ array.store(i, runtime.newFixnum(uBytes[i] & 0xff));
+ }
+
+ result.op_aset(context, runtime.newString("raw"), array);
+ return result;
+ }
+
+ @JRubyMethod(required=1, module=true)
+ public static IRubyObject included(ThreadContext context,
+ IRubyObject vSelf, IRubyObject module) {
+ RuntimeInfo info = RuntimeInfo.forRuntime(context.getRuntime());
+ return module.callMethod(context, "extend", info.stringExtendModule.get());
+ }
+ }
+
+ public static class StringExtend {
+ /**
+ * <code>{@link RubyString String}#json_create(o)</code>
+ *
+ * <p>Raw Strings are JSON Objects (the raw bytes are stored in an
+ * array for the key "raw"). The Ruby String can be created by this
+ * module method.
+ */
+ @JRubyMethod(required=1)
+ public static IRubyObject json_create(ThreadContext context,
+ IRubyObject vSelf, IRubyObject vHash) {
+ Ruby runtime = context.getRuntime();
+ RubyHash o = vHash.convertToHash();
+ IRubyObject rawData = o.fastARef(runtime.newString("raw"));
+ if (rawData == null) {
+ throw runtime.newArgumentError("\"raw\" value not defined "
+ + "for encoded String");
+ }
+ RubyArray ary = Utils.ensureArray(rawData);
+ byte[] bytes = new byte[ary.getLength()];
+ for (int i = 0, t = ary.getLength(); i < t; i++) {
+ IRubyObject element = ary.eltInternal(i);
+ if (element instanceof RubyFixnum) {
+ bytes[i] = (byte)RubyNumeric.fix2long(element);
+ } else {
+ throw runtime.newTypeError(element, runtime.getFixnum());
+ }
+ }
+ return runtime.newString(new ByteList(bytes, false));
+ }
+ }
+
+ public static class RbTrue {
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json(ThreadContext context,
+ IRubyObject vSelf, IRubyObject[] args) {
+ return Generator.generateJson(context, (RubyBoolean)vSelf,
+ Generator.TRUE_HANDLER, args);
+ }
+ }
+
+ public static class RbFalse {
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json(ThreadContext context,
+ IRubyObject vSelf, IRubyObject[] args) {
+ return Generator.generateJson(context, (RubyBoolean)vSelf,
+ Generator.FALSE_HANDLER, args);
+ }
+ }
+
+ public static class RbNil {
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json(ThreadContext context,
+ IRubyObject vSelf, IRubyObject[] args) {
+ return Generator.generateJson(context, vSelf,
+ Generator.NIL_HANDLER, args);
+ }
+ }
+
+ public static class RbObject {
+ @JRubyMethod(rest=true)
+ public static IRubyObject to_json(ThreadContext context,
+ IRubyObject self, IRubyObject[] args) {
+ return RbString.to_json(context, self.asString(), args);
+ }
+ }
+}
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import java.io.IOException;
+import java.lang.ref.WeakReference;
+
+import org.jruby.Ruby;
+import org.jruby.RubyClass;
+import org.jruby.RubyModule;
+import org.jruby.runtime.load.BasicLibraryService;
+
+/**
+ * The service invoked by JRuby's {@link org.jruby.runtime.load.LoadService LoadService}.
+ * Defines the <code>JSON::Ext::Generator</code> module.
+ * @author mernen
+ */
+public class GeneratorService implements BasicLibraryService {
+ public boolean basicLoad(Ruby runtime) throws IOException {
+ runtime.getLoadService().require("json/common");
+ RuntimeInfo info = RuntimeInfo.initRuntime(runtime);
+
+ info.jsonModule = new WeakReference<RubyModule>(runtime.defineModule("JSON"));
+ RubyModule jsonExtModule = info.jsonModule.get().defineModuleUnder("Ext");
+ RubyModule generatorModule = jsonExtModule.defineModuleUnder("Generator");
+
+ RubyClass stateClass =
+ generatorModule.defineClassUnder("State", runtime.getObject(),
+ GeneratorState.ALLOCATOR);
+ stateClass.defineAnnotatedMethods(GeneratorState.class);
+ info.generatorStateClass = new WeakReference<RubyClass>(stateClass);
+
+ RubyModule generatorMethods =
+ generatorModule.defineModuleUnder("GeneratorMethods");
+ GeneratorMethods.populate(info, generatorMethods);
+
+ return true;
+ }
+}
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import org.jruby.Ruby;
+import org.jruby.RubyBoolean;
+import org.jruby.RubyClass;
+import org.jruby.RubyHash;
+import org.jruby.RubyInteger;
+import org.jruby.RubyNumeric;
+import org.jruby.RubyObject;
+import org.jruby.RubyString;
+import org.jruby.anno.JRubyMethod;
+import org.jruby.runtime.Block;
+import org.jruby.runtime.ObjectAllocator;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.runtime.Visibility;
+import org.jruby.runtime.builtin.IRubyObject;
+import org.jruby.util.ByteList;
+
+/**
+ * The <code>JSON::Ext::Generator::State</code> class.
+ *
+ * <p>This class is used to create State instances, that are use to hold data
+ * while generating a JSON text from a a Ruby data structure.
+ *
+ * @author mernen
+ */
+public class GeneratorState extends RubyObject {
+ /**
+ * The indenting unit string. Will be repeated several times for larger
+ * indenting levels.
+ */
+ private ByteList indent = ByteList.EMPTY_BYTELIST;
+ /**
+ * The spacing to be added after a semicolon on a JSON object.
+ * @see #spaceBefore
+ */
+ private ByteList space = ByteList.EMPTY_BYTELIST;
+ /**
+ * The spacing to be added before a semicolon on a JSON object.
+ * @see #space
+ */
+ private ByteList spaceBefore = ByteList.EMPTY_BYTELIST;
+ /**
+ * Any suffix to be added after the comma for each element on a JSON object.
+ * It is assumed to be a newline, if set.
+ */
+ private ByteList objectNl = ByteList.EMPTY_BYTELIST;
+ /**
+ * Any suffix to be added after the comma for each element on a JSON Array.
+ * It is assumed to be a newline, if set.
+ */
+ private ByteList arrayNl = ByteList.EMPTY_BYTELIST;
+
+ /**
+ * The maximum level of nesting of structures allowed.
+ * <code>0</code> means disabled.
+ */
+ private int maxNesting = DEFAULT_MAX_NESTING;
+ static final int DEFAULT_MAX_NESTING = 19;
+ /**
+ * Whether special float values (<code>NaN</code>, <code>Infinity</code>,
+ * <code>-Infinity</code>) are accepted.
+ * If set to <code>false</code>, an exception will be thrown upon
+ * encountering one.
+ */
+ private boolean allowNaN = DEFAULT_ALLOW_NAN;
+ static final boolean DEFAULT_ALLOW_NAN = false;
+ /**
+ * If set to <code>true</code> all JSON documents generated do not contain
+ * any other characters than ASCII characters.
+ */
+ private boolean asciiOnly = DEFAULT_ASCII_ONLY;
+ static final boolean DEFAULT_ASCII_ONLY = false;
+ /**
+ * If set to <code>true</code> all JSON values generated might not be
+ * RFC-conform JSON documents.
+ */
+ private boolean quirksMode = DEFAULT_QUIRKS_MODE;
+ static final boolean DEFAULT_QUIRKS_MODE = false;
+
+ /**
+ * The current depth (inside a #to_json call)
+ */
+ private int depth = 0;
+
+ static final ObjectAllocator ALLOCATOR = new ObjectAllocator() {
+ public IRubyObject allocate(Ruby runtime, RubyClass klazz) {
+ return new GeneratorState(runtime, klazz);
+ }
+ };
+
+ public GeneratorState(Ruby runtime, RubyClass metaClass) {
+ super(runtime, metaClass);
+ }
+
+ /**
+ * <code>State.from_state(opts)</code>
+ *
+ * <p>Creates a State object from <code>opts</code>, which ought to be
+ * {@link RubyHash Hash} to create a new <code>State</code> instance
+ * configured by <codes>opts</code>, something else to create an
+ * unconfigured instance. If <code>opts</code> is a <code>State</code>
+ * object, it is just returned.
+ * @param clazzParam The receiver of the method call
+ * ({@link RubyClass} <code>State</code>)
+ * @param opts The object to use as a base for the new <code>State</code>
+ * @param block The block passed to the method
+ * @return A <code>GeneratorState</code> as determined above
+ */
+ @JRubyMethod(meta=true)
+ public static IRubyObject from_state(ThreadContext context,
+ IRubyObject klass, IRubyObject opts) {
+ return fromState(context, opts);
+ }
+
+ static GeneratorState fromState(ThreadContext context, IRubyObject opts) {
+ return fromState(context, RuntimeInfo.forRuntime(context.getRuntime()), opts);
+ }
+
+ static GeneratorState fromState(ThreadContext context, RuntimeInfo info,
+ IRubyObject opts) {
+ RubyClass klass = info.generatorStateClass.get();
+ if (opts != null) {
+ // if the given parameter is a Generator::State, return itself
+ if (klass.isInstance(opts)) return (GeneratorState)opts;
+
+ // if the given parameter is a Hash, pass it to the instantiator
+ if (context.getRuntime().getHash().isInstance(opts)) {
+ return (GeneratorState)klass.newInstance(context,
+ new IRubyObject[] {opts}, Block.NULL_BLOCK);
+ }
+ }
+
+ // for other values, return the safe prototype
+ return (GeneratorState)info.getSafeStatePrototype(context).dup();
+ }
+
+ /**
+ * <code>State#initialize(opts = {})</code>
+ *
+ * Instantiates a new <code>State</code> object, configured by <code>opts</code>.
+ *
+ * <code>opts</code> can have the following keys:
+ *
+ * <dl>
+ * <dt><code>:indent</code>
+ * <dd>a {@link RubyString String} used to indent levels (default: <code>""</code>)
+ * <dt><code>:space</code>
+ * <dd>a String that is put after a <code>':'</code> or <code>','</code>
+ * delimiter (default: <code>""</code>)
+ * <dt><code>:space_before</code>
+ * <dd>a String that is put before a <code>":"</code> pair delimiter
+ * (default: <code>""</code>)
+ * <dt><code>:object_nl</code>
+ * <dd>a String that is put at the end of a JSON object (default: <code>""</code>)
+ * <dt><code>:array_nl</code>
+ * <dd>a String that is put at the end of a JSON array (default: <code>""</code>)
+ * <dt><code>:allow_nan</code>
+ * <dd><code>true</code> if <code>NaN</code>, <code>Infinity</code>, and
+ * <code>-Infinity</code> should be generated, otherwise an exception is
+ * thrown if these values are encountered.
+ * This options defaults to <code>false</code>.
+ */
+ @JRubyMethod(optional=1, visibility=Visibility.PRIVATE)
+ public IRubyObject initialize(ThreadContext context, IRubyObject[] args) {
+ configure(context, args.length > 0 ? args[0] : null);
+ return this;
+ }
+
+ @JRubyMethod
+ public IRubyObject initialize_copy(ThreadContext context, IRubyObject vOrig) {
+ Ruby runtime = context.getRuntime();
+ if (!(vOrig instanceof GeneratorState)) {
+ throw runtime.newTypeError(vOrig, getType());
+ }
+ GeneratorState orig = (GeneratorState)vOrig;
+ this.indent = orig.indent;
+ this.space = orig.space;
+ this.spaceBefore = orig.spaceBefore;
+ this.objectNl = orig.objectNl;
+ this.arrayNl = orig.arrayNl;
+ this.maxNesting = orig.maxNesting;
+ this.allowNaN = orig.allowNaN;
+ this.asciiOnly = orig.asciiOnly;
+ this.quirksMode = orig.quirksMode;
+ this.depth = orig.depth;
+ return this;
+ }
+
+ /**
+ * Generates a valid JSON document from object <code>obj</code> and returns
+ * the result. If no valid JSON document can be created this method raises
+ * a GeneratorError exception.
+ */
+ @JRubyMethod
+ public IRubyObject generate(ThreadContext context, IRubyObject obj) {
+ RubyString result = Generator.generateJson(context, obj, this);
+ if (!quirksMode && !objectOrArrayLiteral(result)) {
+ throw Utils.newException(context, Utils.M_GENERATOR_ERROR,
+ "only generation of JSON objects or arrays allowed");
+ }
+ return result;
+ }
+
+ /**
+ * Ensures the given string is in the form "[...]" or "{...}", being
+ * possibly surrounded by white space.
+ * The string's encoding must be ASCII-compatible.
+ * @param value
+ * @return
+ */
+ private static boolean objectOrArrayLiteral(RubyString value) {
+ ByteList bl = value.getByteList();
+ int len = bl.length();
+
+ for (int pos = 0; pos < len - 1; pos++) {
+ int b = bl.get(pos);
+ if (Character.isWhitespace(b)) continue;
+
+ // match the opening brace
+ switch (b) {
+ case '[':
+ return matchClosingBrace(bl, pos, len, ']');
+ case '{':
+ return matchClosingBrace(bl, pos, len, '}');
+ default:
+ return false;
+ }
+ }
+ return false;
+ }
+
+ private static boolean matchClosingBrace(ByteList bl, int pos, int len,
+ int brace) {
+ for (int endPos = len - 1; endPos > pos; endPos--) {
+ int b = bl.get(endPos);
+ if (Character.isWhitespace(b)) continue;
+ return b == brace;
+ }
+ return false;
+ }
+
+ @JRubyMethod(name="[]", required=1)
+ public IRubyObject op_aref(ThreadContext context, IRubyObject vName) {
+ String name = vName.asJavaString();
+ if (getMetaClass().isMethodBound(name, true)) {
+ return send(context, vName, Block.NULL_BLOCK);
+ }
+ return context.getRuntime().getNil();
+ }
+
+ public ByteList getIndent() {
+ return indent;
+ }
+
+ @JRubyMethod(name="indent")
+ public RubyString indent_get(ThreadContext context) {
+ return context.getRuntime().newString(indent);
+ }
+
+ @JRubyMethod(name="indent=")
+ public IRubyObject indent_set(ThreadContext context, IRubyObject indent) {
+ this.indent = prepareByteList(context, indent);
+ return indent;
+ }
+
+ public ByteList getSpace() {
+ return space;
+ }
+
+ @JRubyMethod(name="space")
+ public RubyString space_get(ThreadContext context) {
+ return context.getRuntime().newString(space);
+ }
+
+ @JRubyMethod(name="space=")
+ public IRubyObject space_set(ThreadContext context, IRubyObject space) {
+ this.space = prepareByteList(context, space);
+ return space;
+ }
+
+ public ByteList getSpaceBefore() {
+ return spaceBefore;
+ }
+
+ @JRubyMethod(name="space_before")
+ public RubyString space_before_get(ThreadContext context) {
+ return context.getRuntime().newString(spaceBefore);
+ }
+
+ @JRubyMethod(name="space_before=")
+ public IRubyObject space_before_set(ThreadContext context,
+ IRubyObject spaceBefore) {
+ this.spaceBefore = prepareByteList(context, spaceBefore);
+ return spaceBefore;
+ }
+
+ public ByteList getObjectNl() {
+ return objectNl;
+ }
+
+ @JRubyMethod(name="object_nl")
+ public RubyString object_nl_get(ThreadContext context) {
+ return context.getRuntime().newString(objectNl);
+ }
+
+ @JRubyMethod(name="object_nl=")
+ public IRubyObject object_nl_set(ThreadContext context,
+ IRubyObject objectNl) {
+ this.objectNl = prepareByteList(context, objectNl);
+ return objectNl;
+ }
+
+ public ByteList getArrayNl() {
+ return arrayNl;
+ }
+
+ @JRubyMethod(name="array_nl")
+ public RubyString array_nl_get(ThreadContext context) {
+ return context.getRuntime().newString(arrayNl);
+ }
+
+ @JRubyMethod(name="array_nl=")
+ public IRubyObject array_nl_set(ThreadContext context,
+ IRubyObject arrayNl) {
+ this.arrayNl = prepareByteList(context, arrayNl);
+ return arrayNl;
+ }
+
+ @JRubyMethod(name="check_circular?")
+ public RubyBoolean check_circular_p(ThreadContext context) {
+ return context.getRuntime().newBoolean(maxNesting != 0);
+ }
+
+ /**
+ * Returns the maximum level of nesting configured for this state.
+ */
+ public int getMaxNesting() {
+ return maxNesting;
+ }
+
+ @JRubyMethod(name="max_nesting")
+ public RubyInteger max_nesting_get(ThreadContext context) {
+ return context.getRuntime().newFixnum(maxNesting);
+ }
+
+ @JRubyMethod(name="max_nesting=")
+ public IRubyObject max_nesting_set(IRubyObject max_nesting) {
+ maxNesting = RubyNumeric.fix2int(max_nesting);
+ return max_nesting;
+ }
+
+ public boolean allowNaN() {
+ return allowNaN;
+ }
+
+ @JRubyMethod(name="allow_nan?")
+ public RubyBoolean allow_nan_p(ThreadContext context) {
+ return context.getRuntime().newBoolean(allowNaN);
+ }
+
+ public boolean asciiOnly() {
+ return asciiOnly;
+ }
+
+ @JRubyMethod(name="ascii_only?")
+ public RubyBoolean ascii_only_p(ThreadContext context) {
+ return context.getRuntime().newBoolean(asciiOnly);
+ }
+
+ @JRubyMethod(name="quirks_mode")
+ public RubyBoolean quirks_mode_get(ThreadContext context) {
+ return context.getRuntime().newBoolean(quirksMode);
+ }
+
+ @JRubyMethod(name="quirks_mode=")
+ public IRubyObject quirks_mode_set(IRubyObject quirks_mode) {
+ quirksMode = quirks_mode.isTrue();
+ return quirks_mode.getRuntime().newBoolean(quirksMode);
+ }
+
+ @JRubyMethod(name="quirks_mode?")
+ public RubyBoolean quirks_mode_p(ThreadContext context) {
+ return context.getRuntime().newBoolean(quirksMode);
+ }
+
+ public int getDepth() {
+ return depth;
+ }
+
+ @JRubyMethod(name="depth")
+ public RubyInteger depth_get(ThreadContext context) {
+ return context.getRuntime().newFixnum(depth);
+ }
+
+ @JRubyMethod(name="depth=")
+ public IRubyObject depth_set(IRubyObject vDepth) {
+ depth = RubyNumeric.fix2int(vDepth);
+ return vDepth;
+ }
+
+ private ByteList prepareByteList(ThreadContext context, IRubyObject value) {
+ RubyString str = value.convertToString();
+ RuntimeInfo info = RuntimeInfo.forRuntime(context.getRuntime());
+ if (info.encodingsSupported() && str.encoding(context) != info.utf8.get()) {
+ str = (RubyString)str.encode(context, info.utf8.get());
+ }
+ return str.getByteList().dup();
+ }
+
+ /**
+ * <code>State#configure(opts)</code>
+ *
+ * <p>Configures this State instance with the {@link RubyHash Hash}
+ * <code>opts</code>, and returns itself.
+ * @param vOpts The options hash
+ * @return The receiver
+ */
+ @JRubyMethod
+ public IRubyObject configure(ThreadContext context, IRubyObject vOpts) {
+ OptionsReader opts = new OptionsReader(context, vOpts);
+
+ ByteList indent = opts.getString("indent");
+ if (indent != null) this.indent = indent;
+
+ ByteList space = opts.getString("space");
+ if (space != null) this.space = space;
+
+ ByteList spaceBefore = opts.getString("space_before");
+ if (spaceBefore != null) this.spaceBefore = spaceBefore;
+
+ ByteList arrayNl = opts.getString("array_nl");
+ if (arrayNl != null) this.arrayNl = arrayNl;
+
+ ByteList objectNl = opts.getString("object_nl");
+ if (objectNl != null) this.objectNl = objectNl;
+
+ maxNesting = opts.getInt("max_nesting", DEFAULT_MAX_NESTING);
+ allowNaN = opts.getBool("allow_nan", DEFAULT_ALLOW_NAN);
+ asciiOnly = opts.getBool("ascii_only", DEFAULT_ASCII_ONLY);
+ quirksMode = opts.getBool("quirks_mode", DEFAULT_QUIRKS_MODE);
+
+ depth = opts.getInt("depth", 0);
+
+ return this;
+ }
+
+ /**
+ * <code>State#to_h()</code>
+ *
+ * <p>Returns the configuration instance variables as a hash, that can be
+ * passed to the configure method.
+ * @return
+ */
+ @JRubyMethod
+ public RubyHash to_h(ThreadContext context) {
+ Ruby runtime = context.getRuntime();
+ RubyHash result = RubyHash.newHash(runtime);
+
+ result.op_aset(context, runtime.newSymbol("indent"), indent_get(context));
+ result.op_aset(context, runtime.newSymbol("space"), space_get(context));
+ result.op_aset(context, runtime.newSymbol("space_before"), space_before_get(context));
+ result.op_aset(context, runtime.newSymbol("object_nl"), object_nl_get(context));
+ result.op_aset(context, runtime.newSymbol("array_nl"), array_nl_get(context));
+ result.op_aset(context, runtime.newSymbol("allow_nan"), allow_nan_p(context));
+ result.op_aset(context, runtime.newSymbol("ascii_only"), ascii_only_p(context));
+ result.op_aset(context, runtime.newSymbol("quirks_mode"), quirks_mode_p(context));
+ result.op_aset(context, runtime.newSymbol("max_nesting"), max_nesting_get(context));
+ result.op_aset(context, runtime.newSymbol("depth"), depth_get(context));
+ return result;
+ }
+
+ public int increaseDepth() {
+ depth++;
+ checkMaxNesting();
+ return depth;
+ }
+
+ public int decreaseDepth() {
+ return --depth;
+ }
+
+ /**
+ * Checks if the current depth is allowed as per this state's options.
+ * @param context
+ * @param depth The corrent depth
+ */
+ private void checkMaxNesting() {
+ if (maxNesting != 0 && depth > maxNesting) {
+ depth--;
+ throw Utils.newException(getRuntime().getCurrentContext(),
+ Utils.M_NESTING_ERROR, "nesting of " + depth + " is too deep");
+ }
+ }
+}
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import org.jruby.Ruby;
+import org.jruby.RubyClass;
+import org.jruby.RubyHash;
+import org.jruby.RubyNumeric;
+import org.jruby.RubyString;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.runtime.builtin.IRubyObject;
+import org.jruby.util.ByteList;
+
+final class OptionsReader {
+ private final ThreadContext context;
+ private final Ruby runtime;
+ private final RubyHash opts;
+ private RuntimeInfo info;
+
+ OptionsReader(ThreadContext context, IRubyObject vOpts) {
+ this.context = context;
+ this.runtime = context.getRuntime();
+
+ if (vOpts == null || vOpts.isNil()) {
+ opts = null;
+ } else if (vOpts.respondsTo("to_hash")) {
+ opts = vOpts.convertToHash();
+ } else {
+ opts = vOpts.callMethod(context, "to_h").convertToHash();
+ }
+ }
+
+ private RuntimeInfo getRuntimeInfo() {
+ if (info != null) return info;
+ info = RuntimeInfo.forRuntime(runtime);
+ return info;
+ }
+
+ /**
+ * Efficiently looks up items with a {@link RubySymbol Symbol} key
+ * @param key The Symbol name to look up for
+ * @return The item in the {@link RubyHash Hash}, or <code>null</code>
+ * if not found
+ */
+ IRubyObject get(String key) {
+ return opts == null ? null : opts.fastARef(runtime.newSymbol(key));
+ }
+
+ boolean getBool(String key, boolean defaultValue) {
+ IRubyObject value = get(key);
+ return value == null ? defaultValue : value.isTrue();
+ }
+
+ int getInt(String key, int defaultValue) {
+ IRubyObject value = get(key);
+ if (value == null) return defaultValue;
+ if (!value.isTrue()) return 0;
+ return RubyNumeric.fix2int(value);
+ }
+
+ /**
+ * Reads the setting from the options hash. If no entry is set for this
+ * key or if it evaluates to <code>false</code>, returns null; attempts to
+ * coerce the value to {@link RubyString String} otherwise.
+ * @param key The Symbol name to look up for
+ * @return <code>null</code> if the key is not in the Hash or if
+ * its value evaluates to <code>false</code>
+ * @throws RaiseException <code>TypeError</code> if the value does not
+ * evaluate to <code>false</code> and can't be
+ * converted to string
+ */
+ ByteList getString(String key) {
+ RubyString str = getString(key, null);
+ return str == null ? null : str.getByteList().dup();
+ }
+
+ RubyString getString(String key, RubyString defaultValue) {
+ IRubyObject value = get(key);
+ if (value == null || !value.isTrue()) return defaultValue;
+
+ RubyString str = value.convertToString();
+ RuntimeInfo info = getRuntimeInfo();
+ if (info.encodingsSupported() && str.encoding(context) != info.utf8.get()) {
+ str = (RubyString)str.encode(context, info.utf8.get());
+ }
+ return str;
+ }
+
+ /**
+ * Reads the setting from the options hash. If it is <code>nil</code> or
+ * undefined, returns the default value given.
+ * If not, ensures it is a RubyClass instance and shares the same
+ * allocator as the default value (i.e. for the basic types which have
+ * their specific allocators, this ensures the passed value is
+ * a subclass of them).
+ */
+ RubyClass getClass(String key, RubyClass defaultValue) {
+ IRubyObject value = get(key);
+
+ if (value == null || value.isNil()) return defaultValue;
+
+ if (value instanceof RubyClass &&
+ ((RubyClass)value).getAllocator() == defaultValue.getAllocator()) {
+ return (RubyClass)value;
+ }
+ throw runtime.newTypeError(key + " option must be a subclass of "
+ + defaultValue);
+ }
+
+ public RubyHash getHash(String key) {
+ IRubyObject value = get(key);
+ if (value == null || value.isNil()) return new RubyHash(runtime);
+ return (RubyHash) value;
+ }
+}
--- /dev/null
+
+// line 1 "Parser.rl"
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import org.jruby.Ruby;
+import org.jruby.RubyArray;
+import org.jruby.RubyClass;
+import org.jruby.RubyEncoding;
+import org.jruby.RubyFloat;
+import org.jruby.RubyHash;
+import org.jruby.RubyInteger;
+import org.jruby.RubyModule;
+import org.jruby.RubyNumeric;
+import org.jruby.RubyObject;
+import org.jruby.RubyString;
+import org.jruby.anno.JRubyMethod;
+import org.jruby.exceptions.JumpException;
+import org.jruby.exceptions.RaiseException;
+import org.jruby.runtime.Block;
+import org.jruby.runtime.ObjectAllocator;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.runtime.Visibility;
+import org.jruby.runtime.builtin.IRubyObject;
+import org.jruby.util.ByteList;
+
+/**
+ * The <code>JSON::Ext::Parser</code> class.
+ *
+ * <p>This is the JSON parser implemented as a Java class. To use it as the
+ * standard parser, set
+ * <pre>JSON.parser = JSON::Ext::Parser</pre>
+ * This is performed for you when you <code>include "json/ext"</code>.
+ *
+ * <p>This class does not perform the actual parsing, just acts as an interface
+ * to Ruby code. When the {@link #parse()} method is invoked, a
+ * Parser.ParserSession object is instantiated, which handles the process.
+ *
+ * @author mernen
+ */
+public class Parser extends RubyObject {
+ private final RuntimeInfo info;
+ private RubyString vSource;
+ private RubyString createId;
+ private boolean createAdditions;
+ private int maxNesting;
+ private boolean allowNaN;
+ private boolean symbolizeNames;
+ private boolean quirksMode;
+ private RubyClass objectClass;
+ private RubyClass arrayClass;
+ private RubyHash match_string;
+
+ private static final int DEFAULT_MAX_NESTING = 19;
+
+ private static final String JSON_MINUS_INFINITY = "-Infinity";
+ // constant names in the JSON module containing those values
+ private static final String CONST_NAN = "NaN";
+ private static final String CONST_INFINITY = "Infinity";
+ private static final String CONST_MINUS_INFINITY = "MinusInfinity";
+
+ static final ObjectAllocator ALLOCATOR = new ObjectAllocator() {
+ public IRubyObject allocate(Ruby runtime, RubyClass klazz) {
+ return new Parser(runtime, klazz);
+ }
+ };
+
+ /**
+ * Multiple-value return for internal parser methods.
+ *
+ * <p>All the <code>parse<var>Stuff</var></code> methods return instances of
+ * <code>ParserResult</code> when successful, or <code>null</code> when
+ * there's a problem with the input data.
+ */
+ static final class ParserResult {
+ /**
+ * The result of the successful parsing. Should never be
+ * <code>null</code>.
+ */
+ final IRubyObject result;
+ /**
+ * The point where the parser returned.
+ */
+ final int p;
+
+ ParserResult(IRubyObject result, int p) {
+ this.result = result;
+ this.p = p;
+ }
+ }
+
+ public Parser(Ruby runtime, RubyClass metaClass) {
+ super(runtime, metaClass);
+ info = RuntimeInfo.forRuntime(runtime);
+ }
+
+ /**
+ * <code>Parser.new(source, opts = {})</code>
+ *
+ * <p>Creates a new <code>JSON::Ext::Parser</code> instance for the string
+ * <code>source</code>.
+ * It will be configured by the <code>opts</code> Hash.
+ * <code>opts</code> can have the following keys:
+ *
+ * <dl>
+ * <dt><code>:max_nesting</code>
+ * <dd>The maximum depth of nesting allowed in the parsed data
+ * structures. Disable depth checking with <code>:max_nesting => false|nil|0</code>,
+ * it defaults to 19.
+ *
+ * <dt><code>:allow_nan</code>
+ * <dd>If set to <code>true</code>, allow <code>NaN</code>,
+ * <code>Infinity</code> and <code>-Infinity</code> in defiance of RFC 4627
+ * to be parsed by the Parser. This option defaults to <code>false</code>.
+ *
+ * <dt><code>:symbolize_names</code>
+ * <dd>If set to <code>true</code>, returns symbols for the names (keys) in
+ * a JSON object. Otherwise strings are returned, which is also the default.
+ *
+ * <dt><code>:quirks_mode?</code>
+ * <dd>If set to <code>true</code>, if the parse is in quirks_mode, false
+ * otherwise.
+ *
+ * <dt><code>:create_additions</code>
+ * <dd>If set to <code>false</code>, the Parser doesn't create additions
+ * even if a matchin class and <code>create_id</code> was found. This option
+ * defaults to <code>true</code>.
+ *
+ * <dt><code>:object_class</code>
+ * <dd>Defaults to Hash.
+ *
+ * <dt><code>:array_class</code>
+ * <dd>Defaults to Array.
+ * </dl>
+ */
+ @JRubyMethod(name = "new", required = 1, optional = 1, meta = true)
+ public static IRubyObject newInstance(IRubyObject clazz, IRubyObject[] args, Block block) {
+ Parser parser = (Parser)((RubyClass)clazz).allocate();
+
+ parser.callInit(args, block);
+
+ return parser;
+ }
+
+ @JRubyMethod(required = 1, optional = 1, visibility = Visibility.PRIVATE)
+ public IRubyObject initialize(ThreadContext context, IRubyObject[] args) {
+ Ruby runtime = context.getRuntime();
+ if (this.vSource != null) {
+ throw runtime.newTypeError("already initialized instance");
+ }
+
+ OptionsReader opts = new OptionsReader(context, args.length > 1 ? args[1] : null);
+ this.maxNesting = opts.getInt("max_nesting", DEFAULT_MAX_NESTING);
+ this.allowNaN = opts.getBool("allow_nan", false);
+ this.symbolizeNames = opts.getBool("symbolize_names", false);
+ this.quirksMode = opts.getBool("quirks_mode", false);
+ this.createId = opts.getString("create_id", getCreateId(context));
+ this.createAdditions = opts.getBool("create_additions", false);
+ this.objectClass = opts.getClass("object_class", runtime.getHash());
+ this.arrayClass = opts.getClass("array_class", runtime.getArray());
+ this.match_string = opts.getHash("match_string");
+
+ this.vSource = args[0].convertToString();
+ if (!quirksMode) this.vSource = convertEncoding(context, vSource);
+
+ return this;
+ }
+
+ /**
+ * Checks the given string's encoding. If a non-UTF-8 encoding is detected,
+ * a converted copy is returned.
+ * Returns the source string if no conversion is needed.
+ */
+ private RubyString convertEncoding(ThreadContext context, RubyString source) {
+ ByteList bl = source.getByteList();
+ int len = bl.length();
+ if (len < 2) {
+ throw Utils.newException(context, Utils.M_PARSER_ERROR,
+ "A JSON text must at least contain two octets!");
+ }
+
+ if (info.encodingsSupported()) {
+ RubyEncoding encoding = (RubyEncoding)source.encoding(context);
+ if (encoding != info.ascii8bit.get()) {
+ return (RubyString)source.encode(context, info.utf8.get());
+ }
+
+ String sniffedEncoding = sniffByteList(bl);
+ if (sniffedEncoding == null) return source; // assume UTF-8
+ return reinterpretEncoding(context, source, sniffedEncoding);
+ }
+
+ String sniffedEncoding = sniffByteList(bl);
+ if (sniffedEncoding == null) return source; // assume UTF-8
+ Ruby runtime = context.getRuntime();
+ return (RubyString)info.jsonModule.get().
+ callMethod(context, "iconv",
+ new IRubyObject[] {
+ runtime.newString("utf-8"),
+ runtime.newString(sniffedEncoding),
+ source});
+ }
+
+ /**
+ * Checks the first four bytes of the given ByteList to infer its encoding,
+ * using the principle demonstrated on section 3 of RFC 4627 (JSON).
+ */
+ private static String sniffByteList(ByteList bl) {
+ if (bl.length() < 4) return null;
+ if (bl.get(0) == 0 && bl.get(2) == 0) {
+ return bl.get(1) == 0 ? "utf-32be" : "utf-16be";
+ }
+ if (bl.get(1) == 0 && bl.get(3) == 0) {
+ return bl.get(2) == 0 ? "utf-32le" : "utf-16le";
+ }
+ return null;
+ }
+
+ /**
+ * Assumes the given (binary) RubyString to be in the given encoding, then
+ * converts it to UTF-8.
+ */
+ private RubyString reinterpretEncoding(ThreadContext context,
+ RubyString str, String sniffedEncoding) {
+ RubyEncoding actualEncoding = info.getEncoding(context, sniffedEncoding);
+ RubyEncoding targetEncoding = info.utf8.get();
+ RubyString dup = (RubyString)str.dup();
+ dup.force_encoding(context, actualEncoding);
+ return (RubyString)dup.encode_bang(context, targetEncoding);
+ }
+
+ /**
+ * <code>Parser#parse()</code>
+ *
+ * <p>Parses the current JSON text <code>source</code> and returns the
+ * complete data structure as a result.
+ */
+ @JRubyMethod
+ public IRubyObject parse(ThreadContext context) {
+ return new ParserSession(this, context).parse();
+ }
+
+ /**
+ * <code>Parser#source()</code>
+ *
+ * <p>Returns a copy of the current <code>source</code> string, that was
+ * used to construct this Parser.
+ */
+ @JRubyMethod(name = "source")
+ public IRubyObject source_get() {
+ return checkAndGetSource().dup();
+ }
+
+ /**
+ * <code>Parser#quirks_mode?()</code>
+ *
+ * <p>If set to <code>true</code>, if the parse is in quirks_mode, false
+ * otherwise.
+ */
+ @JRubyMethod(name = "quirks_mode?")
+ public IRubyObject quirks_mode_p(ThreadContext context) {
+ return context.getRuntime().newBoolean(quirksMode);
+ }
+
+ public RubyString checkAndGetSource() {
+ if (vSource != null) {
+ return vSource;
+ } else {
+ throw getRuntime().newTypeError("uninitialized instance");
+ }
+ }
+
+ /**
+ * Queries <code>JSON.create_id</code>. Returns <code>null</code> if it is
+ * set to <code>nil</code> or <code>false</code>, and a String if not.
+ */
+ private RubyString getCreateId(ThreadContext context) {
+ IRubyObject v = info.jsonModule.get().callMethod(context, "create_id");
+ return v.isTrue() ? v.convertToString() : null;
+ }
+
+ /**
+ * A string parsing session.
+ *
+ * <p>Once a ParserSession is instantiated, the source string should not
+ * change until the parsing is complete. The ParserSession object assumes
+ * the source {@link RubyString} is still associated to its original
+ * {@link ByteList}, which in turn must still be bound to the same
+ * <code>byte[]</code> value (and on the same offset).
+ */
+ // Ragel uses lots of fall-through
+ @SuppressWarnings("fallthrough")
+ private static class ParserSession {
+ private final Parser parser;
+ private final ThreadContext context;
+ private final ByteList byteList;
+ private final byte[] data;
+ private final StringDecoder decoder;
+ private int currentNesting = 0;
+
+ // initialization value for all state variables.
+ // no idea about the origins of this value, ask Flori ;)
+ private static final int EVIL = 0x666;
+
+ private ParserSession(Parser parser, ThreadContext context) {
+ this.parser = parser;
+ this.context = context;
+ this.byteList = parser.checkAndGetSource().getByteList();
+ this.data = byteList.unsafeBytes();
+ this.decoder = new StringDecoder(context);
+ }
+
+ private RaiseException unexpectedToken(int absStart, int absEnd) {
+ RubyString msg = getRuntime().newString("unexpected token at '")
+ .cat(data, absStart, absEnd - absStart)
+ .cat((byte)'\'');
+ return newException(Utils.M_PARSER_ERROR, msg);
+ }
+
+ private Ruby getRuntime() {
+ return context.getRuntime();
+ }
+
+
+// line 353 "Parser.rl"
+
+
+
+// line 335 "Parser.java"
+private static byte[] init__JSON_value_actions_0()
+{
+ return new byte [] {
+ 0, 1, 0, 1, 1, 1, 2, 1, 3, 1, 4, 1,
+ 5, 1, 6, 1, 7, 1, 8, 1, 9
+ };
+}
+
+private static final byte _JSON_value_actions[] = init__JSON_value_actions_0();
+
+
+private static byte[] init__JSON_value_key_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25, 26, 27, 28, 29, 30
+ };
+}
+
+private static final byte _JSON_value_key_offsets[] = init__JSON_value_key_offsets_0();
+
+
+private static char[] init__JSON_value_trans_keys_0()
+{
+ return new char [] {
+ 34, 45, 73, 78, 91, 102, 110, 116, 123, 48, 57, 110,
+ 102, 105, 110, 105, 116, 121, 97, 78, 97, 108, 115, 101,
+ 117, 108, 108, 114, 117, 101, 0
+ };
+}
+
+private static final char _JSON_value_trans_keys[] = init__JSON_value_trans_keys_0();
+
+
+private static byte[] init__JSON_value_single_lengths_0()
+{
+ return new byte [] {
+ 0, 9, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1, 1, 0
+ };
+}
+
+private static final byte _JSON_value_single_lengths[] = init__JSON_value_single_lengths_0();
+
+
+private static byte[] init__JSON_value_range_lengths_0()
+{
+ return new byte [] {
+ 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+ };
+}
+
+private static final byte _JSON_value_range_lengths[] = init__JSON_value_range_lengths_0();
+
+
+private static byte[] init__JSON_value_index_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29,
+ 31, 33, 35, 37, 39, 41, 43, 45, 47, 49
+ };
+}
+
+private static final byte _JSON_value_index_offsets[] = init__JSON_value_index_offsets_0();
+
+
+private static byte[] init__JSON_value_trans_targs_0()
+{
+ return new byte [] {
+ 21, 21, 2, 9, 21, 11, 15, 18, 21, 21, 0, 3,
+ 0, 4, 0, 5, 0, 6, 0, 7, 0, 8, 0, 21,
+ 0, 10, 0, 21, 0, 12, 0, 13, 0, 14, 0, 21,
+ 0, 16, 0, 17, 0, 21, 0, 19, 0, 20, 0, 21,
+ 0, 0, 0
+ };
+}
+
+private static final byte _JSON_value_trans_targs[] = init__JSON_value_trans_targs_0();
+
+
+private static byte[] init__JSON_value_trans_actions_0()
+{
+ return new byte [] {
+ 13, 11, 0, 0, 15, 0, 0, 0, 17, 11, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 9,
+ 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 3,
+ 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 5,
+ 0, 0, 0
+ };
+}
+
+private static final byte _JSON_value_trans_actions[] = init__JSON_value_trans_actions_0();
+
+
+private static byte[] init__JSON_value_from_state_actions_0()
+{
+ return new byte [] {
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 19
+ };
+}
+
+private static final byte _JSON_value_from_state_actions[] = init__JSON_value_from_state_actions_0();
+
+
+static final int JSON_value_start = 1;
+static final int JSON_value_first_final = 21;
+static final int JSON_value_error = 0;
+
+static final int JSON_value_en_main = 1;
+
+
+// line 459 "Parser.rl"
+
+
+ ParserResult parseValue(int p, int pe) {
+ int cs = EVIL;
+ IRubyObject result = null;
+
+
+// line 457 "Parser.java"
+ {
+ cs = JSON_value_start;
+ }
+
+// line 466 "Parser.rl"
+
+// line 464 "Parser.java"
+ {
+ int _klen;
+ int _trans = 0;
+ int _acts;
+ int _nacts;
+ int _keys;
+ int _goto_targ = 0;
+
+ _goto: while (true) {
+ switch ( _goto_targ ) {
+ case 0:
+ if ( p == pe ) {
+ _goto_targ = 4;
+ continue _goto;
+ }
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+case 1:
+ _acts = _JSON_value_from_state_actions[cs];
+ _nacts = (int) _JSON_value_actions[_acts++];
+ while ( _nacts-- > 0 ) {
+ switch ( _JSON_value_actions[_acts++] ) {
+ case 9:
+// line 444 "Parser.rl"
+ {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ }
+ break;
+// line 496 "Parser.java"
+ }
+ }
+
+ _match: do {
+ _keys = _JSON_value_key_offsets[cs];
+ _trans = _JSON_value_index_offsets[cs];
+ _klen = _JSON_value_single_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + _klen - 1;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + ((_upper-_lower) >> 1);
+ if ( data[p] < _JSON_value_trans_keys[_mid] )
+ _upper = _mid - 1;
+ else if ( data[p] > _JSON_value_trans_keys[_mid] )
+ _lower = _mid + 1;
+ else {
+ _trans += (_mid - _keys);
+ break _match;
+ }
+ }
+ _keys += _klen;
+ _trans += _klen;
+ }
+
+ _klen = _JSON_value_range_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + (_klen<<1) - 2;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + (((_upper-_lower) >> 1) & ~1);
+ if ( data[p] < _JSON_value_trans_keys[_mid] )
+ _upper = _mid - 2;
+ else if ( data[p] > _JSON_value_trans_keys[_mid+1] )
+ _lower = _mid + 2;
+ else {
+ _trans += ((_mid - _keys)>>1);
+ break _match;
+ }
+ }
+ _trans += _klen;
+ }
+ } while (false);
+
+ cs = _JSON_value_trans_targs[_trans];
+
+ if ( _JSON_value_trans_actions[_trans] != 0 ) {
+ _acts = _JSON_value_trans_actions[_trans];
+ _nacts = (int) _JSON_value_actions[_acts++];
+ while ( _nacts-- > 0 )
+ {
+ switch ( _JSON_value_actions[_acts++] )
+ {
+ case 0:
+// line 361 "Parser.rl"
+ {
+ result = getRuntime().getNil();
+ }
+ break;
+ case 1:
+// line 364 "Parser.rl"
+ {
+ result = getRuntime().getFalse();
+ }
+ break;
+ case 2:
+// line 367 "Parser.rl"
+ {
+ result = getRuntime().getTrue();
+ }
+ break;
+ case 3:
+// line 370 "Parser.rl"
+ {
+ if (parser.allowNaN) {
+ result = getConstant(CONST_NAN);
+ } else {
+ throw unexpectedToken(p - 2, pe);
+ }
+ }
+ break;
+ case 4:
+// line 377 "Parser.rl"
+ {
+ if (parser.allowNaN) {
+ result = getConstant(CONST_INFINITY);
+ } else {
+ throw unexpectedToken(p - 7, pe);
+ }
+ }
+ break;
+ case 5:
+// line 384 "Parser.rl"
+ {
+ if (pe > p + 9 - (parser.quirksMode ? 1 : 0) &&
+ absSubSequence(p, p + 9).toString().equals(JSON_MINUS_INFINITY)) {
+
+ if (parser.allowNaN) {
+ result = getConstant(CONST_MINUS_INFINITY);
+ {p = (( p + 10))-1;}
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ throw unexpectedToken(p, pe);
+ }
+ }
+ ParserResult res = parseFloat(p, pe);
+ if (res != null) {
+ result = res.result;
+ {p = (( res.p))-1;}
+ }
+ res = parseInteger(p, pe);
+ if (res != null) {
+ result = res.result;
+ {p = (( res.p))-1;}
+ }
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ }
+ break;
+ case 6:
+// line 410 "Parser.rl"
+ {
+ ParserResult res = parseString(p, pe);
+ if (res == null) {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ result = res.result;
+ {p = (( res.p))-1;}
+ }
+ }
+ break;
+ case 7:
+// line 420 "Parser.rl"
+ {
+ currentNesting++;
+ ParserResult res = parseArray(p, pe);
+ currentNesting--;
+ if (res == null) {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ result = res.result;
+ {p = (( res.p))-1;}
+ }
+ }
+ break;
+ case 8:
+// line 432 "Parser.rl"
+ {
+ currentNesting++;
+ ParserResult res = parseObject(p, pe);
+ currentNesting--;
+ if (res == null) {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ result = res.result;
+ {p = (( res.p))-1;}
+ }
+ }
+ break;
+// line 668 "Parser.java"
+ }
+ }
+ }
+
+case 2:
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+ if ( ++p != pe ) {
+ _goto_targ = 1;
+ continue _goto;
+ }
+case 4:
+case 5:
+ }
+ break; }
+ }
+
+// line 467 "Parser.rl"
+
+ if (cs >= JSON_value_first_final && result != null) {
+ return new ParserResult(result, p);
+ } else {
+ return null;
+ }
+ }
+
+
+// line 698 "Parser.java"
+private static byte[] init__JSON_integer_actions_0()
+{
+ return new byte [] {
+ 0, 1, 0
+ };
+}
+
+private static final byte _JSON_integer_actions[] = init__JSON_integer_actions_0();
+
+
+private static byte[] init__JSON_integer_key_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 4, 7, 9, 9
+ };
+}
+
+private static final byte _JSON_integer_key_offsets[] = init__JSON_integer_key_offsets_0();
+
+
+private static char[] init__JSON_integer_trans_keys_0()
+{
+ return new char [] {
+ 45, 48, 49, 57, 48, 49, 57, 48, 57, 48, 57, 0
+ };
+}
+
+private static final char _JSON_integer_trans_keys[] = init__JSON_integer_trans_keys_0();
+
+
+private static byte[] init__JSON_integer_single_lengths_0()
+{
+ return new byte [] {
+ 0, 2, 1, 0, 0, 0
+ };
+}
+
+private static final byte _JSON_integer_single_lengths[] = init__JSON_integer_single_lengths_0();
+
+
+private static byte[] init__JSON_integer_range_lengths_0()
+{
+ return new byte [] {
+ 0, 1, 1, 1, 0, 1
+ };
+}
+
+private static final byte _JSON_integer_range_lengths[] = init__JSON_integer_range_lengths_0();
+
+
+private static byte[] init__JSON_integer_index_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 4, 7, 9, 10
+ };
+}
+
+private static final byte _JSON_integer_index_offsets[] = init__JSON_integer_index_offsets_0();
+
+
+private static byte[] init__JSON_integer_indicies_0()
+{
+ return new byte [] {
+ 0, 2, 3, 1, 2, 3, 1, 1, 4, 1, 3, 4,
+ 0
+ };
+}
+
+private static final byte _JSON_integer_indicies[] = init__JSON_integer_indicies_0();
+
+
+private static byte[] init__JSON_integer_trans_targs_0()
+{
+ return new byte [] {
+ 2, 0, 3, 5, 4
+ };
+}
+
+private static final byte _JSON_integer_trans_targs[] = init__JSON_integer_trans_targs_0();
+
+
+private static byte[] init__JSON_integer_trans_actions_0()
+{
+ return new byte [] {
+ 0, 0, 0, 0, 1
+ };
+}
+
+private static final byte _JSON_integer_trans_actions[] = init__JSON_integer_trans_actions_0();
+
+
+static final int JSON_integer_start = 1;
+static final int JSON_integer_first_final = 3;
+static final int JSON_integer_error = 0;
+
+static final int JSON_integer_en_main = 1;
+
+
+// line 486 "Parser.rl"
+
+
+ ParserResult parseInteger(int p, int pe) {
+ int cs = EVIL;
+
+
+// line 804 "Parser.java"
+ {
+ cs = JSON_integer_start;
+ }
+
+// line 492 "Parser.rl"
+ int memo = p;
+
+// line 812 "Parser.java"
+ {
+ int _klen;
+ int _trans = 0;
+ int _acts;
+ int _nacts;
+ int _keys;
+ int _goto_targ = 0;
+
+ _goto: while (true) {
+ switch ( _goto_targ ) {
+ case 0:
+ if ( p == pe ) {
+ _goto_targ = 4;
+ continue _goto;
+ }
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+case 1:
+ _match: do {
+ _keys = _JSON_integer_key_offsets[cs];
+ _trans = _JSON_integer_index_offsets[cs];
+ _klen = _JSON_integer_single_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + _klen - 1;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + ((_upper-_lower) >> 1);
+ if ( data[p] < _JSON_integer_trans_keys[_mid] )
+ _upper = _mid - 1;
+ else if ( data[p] > _JSON_integer_trans_keys[_mid] )
+ _lower = _mid + 1;
+ else {
+ _trans += (_mid - _keys);
+ break _match;
+ }
+ }
+ _keys += _klen;
+ _trans += _klen;
+ }
+
+ _klen = _JSON_integer_range_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + (_klen<<1) - 2;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + (((_upper-_lower) >> 1) & ~1);
+ if ( data[p] < _JSON_integer_trans_keys[_mid] )
+ _upper = _mid - 2;
+ else if ( data[p] > _JSON_integer_trans_keys[_mid+1] )
+ _lower = _mid + 2;
+ else {
+ _trans += ((_mid - _keys)>>1);
+ break _match;
+ }
+ }
+ _trans += _klen;
+ }
+ } while (false);
+
+ _trans = _JSON_integer_indicies[_trans];
+ cs = _JSON_integer_trans_targs[_trans];
+
+ if ( _JSON_integer_trans_actions[_trans] != 0 ) {
+ _acts = _JSON_integer_trans_actions[_trans];
+ _nacts = (int) _JSON_integer_actions[_acts++];
+ while ( _nacts-- > 0 )
+ {
+ switch ( _JSON_integer_actions[_acts++] )
+ {
+ case 0:
+// line 480 "Parser.rl"
+ {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ }
+ break;
+// line 899 "Parser.java"
+ }
+ }
+ }
+
+case 2:
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+ if ( ++p != pe ) {
+ _goto_targ = 1;
+ continue _goto;
+ }
+case 4:
+case 5:
+ }
+ break; }
+ }
+
+// line 494 "Parser.rl"
+
+ if (cs < JSON_integer_first_final) {
+ return null;
+ }
+
+ ByteList num = absSubSequence(memo, p);
+ // note: this is actually a shared string, but since it is temporary and
+ // read-only, it doesn't really matter
+ RubyString expr = RubyString.newStringLight(getRuntime(), num);
+ RubyInteger number = RubyNumeric.str2inum(getRuntime(), expr, 10, true);
+ return new ParserResult(number, p + 1);
+ }
+
+
+// line 934 "Parser.java"
+private static byte[] init__JSON_float_actions_0()
+{
+ return new byte [] {
+ 0, 1, 0
+ };
+}
+
+private static final byte _JSON_float_actions[] = init__JSON_float_actions_0();
+
+
+private static byte[] init__JSON_float_key_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 4, 7, 10, 12, 16, 18, 23, 29, 29
+ };
+}
+
+private static final byte _JSON_float_key_offsets[] = init__JSON_float_key_offsets_0();
+
+
+private static char[] init__JSON_float_trans_keys_0()
+{
+ return new char [] {
+ 45, 48, 49, 57, 48, 49, 57, 46, 69, 101, 48, 57,
+ 43, 45, 48, 57, 48, 57, 46, 69, 101, 48, 57, 69,
+ 101, 45, 46, 48, 57, 69, 101, 45, 46, 48, 57, 0
+ };
+}
+
+private static final char _JSON_float_trans_keys[] = init__JSON_float_trans_keys_0();
+
+
+private static byte[] init__JSON_float_single_lengths_0()
+{
+ return new byte [] {
+ 0, 2, 1, 3, 0, 2, 0, 3, 2, 0, 2
+ };
+}
+
+private static final byte _JSON_float_single_lengths[] = init__JSON_float_single_lengths_0();
+
+
+private static byte[] init__JSON_float_range_lengths_0()
+{
+ return new byte [] {
+ 0, 1, 1, 0, 1, 1, 1, 1, 2, 0, 2
+ };
+}
+
+private static final byte _JSON_float_range_lengths[] = init__JSON_float_range_lengths_0();
+
+
+private static byte[] init__JSON_float_index_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 4, 7, 11, 13, 17, 19, 24, 29, 30
+ };
+}
+
+private static final byte _JSON_float_index_offsets[] = init__JSON_float_index_offsets_0();
+
+
+private static byte[] init__JSON_float_indicies_0()
+{
+ return new byte [] {
+ 0, 2, 3, 1, 2, 3, 1, 4, 5, 5, 1, 6,
+ 1, 7, 7, 8, 1, 8, 1, 4, 5, 5, 3, 1,
+ 5, 5, 1, 6, 9, 1, 1, 1, 1, 8, 9, 0
+ };
+}
+
+private static final byte _JSON_float_indicies[] = init__JSON_float_indicies_0();
+
+
+private static byte[] init__JSON_float_trans_targs_0()
+{
+ return new byte [] {
+ 2, 0, 3, 7, 4, 5, 8, 6, 10, 9
+ };
+}
+
+private static final byte _JSON_float_trans_targs[] = init__JSON_float_trans_targs_0();
+
+
+private static byte[] init__JSON_float_trans_actions_0()
+{
+ return new byte [] {
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 1
+ };
+}
+
+private static final byte _JSON_float_trans_actions[] = init__JSON_float_trans_actions_0();
+
+
+static final int JSON_float_start = 1;
+static final int JSON_float_first_final = 8;
+static final int JSON_float_error = 0;
+
+static final int JSON_float_en_main = 1;
+
+
+// line 522 "Parser.rl"
+
+
+ ParserResult parseFloat(int p, int pe) {
+ int cs = EVIL;
+
+
+// line 1043 "Parser.java"
+ {
+ cs = JSON_float_start;
+ }
+
+// line 528 "Parser.rl"
+ int memo = p;
+
+// line 1051 "Parser.java"
+ {
+ int _klen;
+ int _trans = 0;
+ int _acts;
+ int _nacts;
+ int _keys;
+ int _goto_targ = 0;
+
+ _goto: while (true) {
+ switch ( _goto_targ ) {
+ case 0:
+ if ( p == pe ) {
+ _goto_targ = 4;
+ continue _goto;
+ }
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+case 1:
+ _match: do {
+ _keys = _JSON_float_key_offsets[cs];
+ _trans = _JSON_float_index_offsets[cs];
+ _klen = _JSON_float_single_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + _klen - 1;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + ((_upper-_lower) >> 1);
+ if ( data[p] < _JSON_float_trans_keys[_mid] )
+ _upper = _mid - 1;
+ else if ( data[p] > _JSON_float_trans_keys[_mid] )
+ _lower = _mid + 1;
+ else {
+ _trans += (_mid - _keys);
+ break _match;
+ }
+ }
+ _keys += _klen;
+ _trans += _klen;
+ }
+
+ _klen = _JSON_float_range_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + (_klen<<1) - 2;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + (((_upper-_lower) >> 1) & ~1);
+ if ( data[p] < _JSON_float_trans_keys[_mid] )
+ _upper = _mid - 2;
+ else if ( data[p] > _JSON_float_trans_keys[_mid+1] )
+ _lower = _mid + 2;
+ else {
+ _trans += ((_mid - _keys)>>1);
+ break _match;
+ }
+ }
+ _trans += _klen;
+ }
+ } while (false);
+
+ _trans = _JSON_float_indicies[_trans];
+ cs = _JSON_float_trans_targs[_trans];
+
+ if ( _JSON_float_trans_actions[_trans] != 0 ) {
+ _acts = _JSON_float_trans_actions[_trans];
+ _nacts = (int) _JSON_float_actions[_acts++];
+ while ( _nacts-- > 0 )
+ {
+ switch ( _JSON_float_actions[_acts++] )
+ {
+ case 0:
+// line 513 "Parser.rl"
+ {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ }
+ break;
+// line 1138 "Parser.java"
+ }
+ }
+ }
+
+case 2:
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+ if ( ++p != pe ) {
+ _goto_targ = 1;
+ continue _goto;
+ }
+case 4:
+case 5:
+ }
+ break; }
+ }
+
+// line 530 "Parser.rl"
+
+ if (cs < JSON_float_first_final) {
+ return null;
+ }
+
+ ByteList num = absSubSequence(memo, p);
+ // note: this is actually a shared string, but since it is temporary and
+ // read-only, it doesn't really matter
+ RubyString expr = RubyString.newStringLight(getRuntime(), num);
+ RubyFloat number = RubyNumeric.str2fnum(getRuntime(), expr, true);
+ return new ParserResult(number, p + 1);
+ }
+
+
+// line 1173 "Parser.java"
+private static byte[] init__JSON_string_actions_0()
+{
+ return new byte [] {
+ 0, 2, 0, 1
+ };
+}
+
+private static final byte _JSON_string_actions[] = init__JSON_string_actions_0();
+
+
+private static byte[] init__JSON_string_key_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 1, 5, 8, 14, 20, 26, 32
+ };
+}
+
+private static final byte _JSON_string_key_offsets[] = init__JSON_string_key_offsets_0();
+
+
+private static char[] init__JSON_string_trans_keys_0()
+{
+ return new char [] {
+ 34, 34, 92, 0, 31, 117, 0, 31, 48, 57, 65, 70,
+ 97, 102, 48, 57, 65, 70, 97, 102, 48, 57, 65, 70,
+ 97, 102, 48, 57, 65, 70, 97, 102, 0
+ };
+}
+
+private static final char _JSON_string_trans_keys[] = init__JSON_string_trans_keys_0();
+
+
+private static byte[] init__JSON_string_single_lengths_0()
+{
+ return new byte [] {
+ 0, 1, 2, 1, 0, 0, 0, 0, 0
+ };
+}
+
+private static final byte _JSON_string_single_lengths[] = init__JSON_string_single_lengths_0();
+
+
+private static byte[] init__JSON_string_range_lengths_0()
+{
+ return new byte [] {
+ 0, 0, 1, 1, 3, 3, 3, 3, 0
+ };
+}
+
+private static final byte _JSON_string_range_lengths[] = init__JSON_string_range_lengths_0();
+
+
+private static byte[] init__JSON_string_index_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 2, 6, 9, 13, 17, 21, 25
+ };
+}
+
+private static final byte _JSON_string_index_offsets[] = init__JSON_string_index_offsets_0();
+
+
+private static byte[] init__JSON_string_indicies_0()
+{
+ return new byte [] {
+ 0, 1, 2, 3, 1, 0, 4, 1, 0, 5, 5, 5,
+ 1, 6, 6, 6, 1, 7, 7, 7, 1, 0, 0, 0,
+ 1, 1, 0
+ };
+}
+
+private static final byte _JSON_string_indicies[] = init__JSON_string_indicies_0();
+
+
+private static byte[] init__JSON_string_trans_targs_0()
+{
+ return new byte [] {
+ 2, 0, 8, 3, 4, 5, 6, 7
+ };
+}
+
+private static final byte _JSON_string_trans_targs[] = init__JSON_string_trans_targs_0();
+
+
+private static byte[] init__JSON_string_trans_actions_0()
+{
+ return new byte [] {
+ 0, 0, 1, 0, 0, 0, 0, 0
+ };
+}
+
+private static final byte _JSON_string_trans_actions[] = init__JSON_string_trans_actions_0();
+
+
+static final int JSON_string_start = 1;
+static final int JSON_string_first_final = 8;
+static final int JSON_string_error = 0;
+
+static final int JSON_string_en_main = 1;
+
+
+// line 574 "Parser.rl"
+
+
+ ParserResult parseString(int p, int pe) {
+ int cs = EVIL;
+ IRubyObject result = null;
+
+
+// line 1283 "Parser.java"
+ {
+ cs = JSON_string_start;
+ }
+
+// line 581 "Parser.rl"
+ int memo = p;
+
+// line 1291 "Parser.java"
+ {
+ int _klen;
+ int _trans = 0;
+ int _acts;
+ int _nacts;
+ int _keys;
+ int _goto_targ = 0;
+
+ _goto: while (true) {
+ switch ( _goto_targ ) {
+ case 0:
+ if ( p == pe ) {
+ _goto_targ = 4;
+ continue _goto;
+ }
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+case 1:
+ _match: do {
+ _keys = _JSON_string_key_offsets[cs];
+ _trans = _JSON_string_index_offsets[cs];
+ _klen = _JSON_string_single_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + _klen - 1;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + ((_upper-_lower) >> 1);
+ if ( data[p] < _JSON_string_trans_keys[_mid] )
+ _upper = _mid - 1;
+ else if ( data[p] > _JSON_string_trans_keys[_mid] )
+ _lower = _mid + 1;
+ else {
+ _trans += (_mid - _keys);
+ break _match;
+ }
+ }
+ _keys += _klen;
+ _trans += _klen;
+ }
+
+ _klen = _JSON_string_range_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + (_klen<<1) - 2;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + (((_upper-_lower) >> 1) & ~1);
+ if ( data[p] < _JSON_string_trans_keys[_mid] )
+ _upper = _mid - 2;
+ else if ( data[p] > _JSON_string_trans_keys[_mid+1] )
+ _lower = _mid + 2;
+ else {
+ _trans += ((_mid - _keys)>>1);
+ break _match;
+ }
+ }
+ _trans += _klen;
+ }
+ } while (false);
+
+ _trans = _JSON_string_indicies[_trans];
+ cs = _JSON_string_trans_targs[_trans];
+
+ if ( _JSON_string_trans_actions[_trans] != 0 ) {
+ _acts = _JSON_string_trans_actions[_trans];
+ _nacts = (int) _JSON_string_actions[_acts++];
+ while ( _nacts-- > 0 )
+ {
+ switch ( _JSON_string_actions[_acts++] )
+ {
+ case 0:
+// line 549 "Parser.rl"
+ {
+ int offset = byteList.begin();
+ ByteList decoded = decoder.decode(byteList, memo + 1 - offset,
+ p - offset);
+ result = getRuntime().newString(decoded);
+ if (result == null) {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ {p = (( p + 1))-1;}
+ }
+ }
+ break;
+ case 1:
+// line 562 "Parser.rl"
+ {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ }
+ break;
+// line 1393 "Parser.java"
+ }
+ }
+ }
+
+case 2:
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+ if ( ++p != pe ) {
+ _goto_targ = 1;
+ continue _goto;
+ }
+case 4:
+case 5:
+ }
+ break; }
+ }
+
+// line 583 "Parser.rl"
+
+ if (parser.createAdditions) {
+ RubyHash match_string = parser.match_string;
+ if (match_string != null) {
+ final IRubyObject[] memoArray = { result, null };
+ try {
+ match_string.visitAll(new RubyHash.Visitor() {
+ @Override
+ public void visit(IRubyObject pattern, IRubyObject klass) {
+ if (pattern.callMethod(context, "===", memoArray[0]).isTrue()) {
+ memoArray[1] = klass;
+ throw JumpException.SPECIAL_JUMP;
+ }
+ }
+ });
+ } catch (JumpException e) { }
+ if (memoArray[1] != null) {
+ RubyClass klass = (RubyClass) memoArray[1];
+ if (klass.respondsTo("json_creatable?") &&
+ klass.callMethod(context, "json_creatable?").isTrue()) {
+ result = klass.callMethod(context, "json_create", result);
+ }
+ }
+ }
+ }
+
+ if (cs >= JSON_string_first_final && result != null) {
+ return new ParserResult(result, p + 1);
+ } else {
+ return null;
+ }
+ }
+
+
+// line 1448 "Parser.java"
+private static byte[] init__JSON_array_actions_0()
+{
+ return new byte [] {
+ 0, 1, 0, 1, 1
+ };
+}
+
+private static final byte _JSON_array_actions[] = init__JSON_array_actions_0();
+
+
+private static byte[] init__JSON_array_key_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 1, 18, 25, 41, 43, 44, 46, 47, 49, 50,
+ 52, 53, 55, 56, 58, 59
+ };
+}
+
+private static final byte _JSON_array_key_offsets[] = init__JSON_array_key_offsets_0();
+
+
+private static char[] init__JSON_array_trans_keys_0()
+{
+ return new char [] {
+ 91, 13, 32, 34, 45, 47, 73, 78, 91, 93, 102, 110,
+ 116, 123, 9, 10, 48, 57, 13, 32, 44, 47, 93, 9,
+ 10, 13, 32, 34, 45, 47, 73, 78, 91, 102, 110, 116,
+ 123, 9, 10, 48, 57, 42, 47, 42, 42, 47, 10, 42,
+ 47, 42, 42, 47, 10, 42, 47, 42, 42, 47, 10, 0
+ };
+}
+
+private static final char _JSON_array_trans_keys[] = init__JSON_array_trans_keys_0();
+
+
+private static byte[] init__JSON_array_single_lengths_0()
+{
+ return new byte [] {
+ 0, 1, 13, 5, 12, 2, 1, 2, 1, 2, 1, 2,
+ 1, 2, 1, 2, 1, 0
+ };
+}
+
+private static final byte _JSON_array_single_lengths[] = init__JSON_array_single_lengths_0();
+
+
+private static byte[] init__JSON_array_range_lengths_0()
+{
+ return new byte [] {
+ 0, 0, 2, 1, 2, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0
+ };
+}
+
+private static final byte _JSON_array_range_lengths[] = init__JSON_array_range_lengths_0();
+
+
+private static byte[] init__JSON_array_index_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 2, 18, 25, 40, 43, 45, 48, 50, 53, 55,
+ 58, 60, 63, 65, 68, 70
+ };
+}
+
+private static final byte _JSON_array_index_offsets[] = init__JSON_array_index_offsets_0();
+
+
+private static byte[] init__JSON_array_indicies_0()
+{
+ return new byte [] {
+ 0, 1, 0, 0, 2, 2, 3, 2, 2, 2, 4, 2,
+ 2, 2, 2, 0, 2, 1, 5, 5, 6, 7, 4, 5,
+ 1, 6, 6, 2, 2, 8, 2, 2, 2, 2, 2, 2,
+ 2, 6, 2, 1, 9, 10, 1, 11, 9, 11, 6, 9,
+ 6, 10, 12, 13, 1, 14, 12, 14, 5, 12, 5, 13,
+ 15, 16, 1, 17, 15, 17, 0, 15, 0, 16, 1, 0
+ };
+}
+
+private static final byte _JSON_array_indicies[] = init__JSON_array_indicies_0();
+
+
+private static byte[] init__JSON_array_trans_targs_0()
+{
+ return new byte [] {
+ 2, 0, 3, 13, 17, 3, 4, 9, 5, 6, 8, 7,
+ 10, 12, 11, 14, 16, 15
+ };
+}
+
+private static final byte _JSON_array_trans_targs[] = init__JSON_array_trans_targs_0();
+
+
+private static byte[] init__JSON_array_trans_actions_0()
+{
+ return new byte [] {
+ 0, 0, 1, 0, 3, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0
+ };
+}
+
+private static final byte _JSON_array_trans_actions[] = init__JSON_array_trans_actions_0();
+
+
+static final int JSON_array_start = 1;
+static final int JSON_array_first_final = 17;
+static final int JSON_array_error = 0;
+
+static final int JSON_array_en_main = 1;
+
+
+// line 653 "Parser.rl"
+
+
+ ParserResult parseArray(int p, int pe) {
+ int cs = EVIL;
+
+ if (parser.maxNesting > 0 && currentNesting > parser.maxNesting) {
+ throw newException(Utils.M_NESTING_ERROR,
+ "nesting of " + currentNesting + " is too deep");
+ }
+
+ // this is guaranteed to be a RubyArray due to the earlier
+ // allocator test at OptionsReader#getClass
+ RubyArray result =
+ (RubyArray)parser.arrayClass.newInstance(context,
+ IRubyObject.NULL_ARRAY, Block.NULL_BLOCK);
+
+
+// line 1579 "Parser.java"
+ {
+ cs = JSON_array_start;
+ }
+
+// line 670 "Parser.rl"
+
+// line 1586 "Parser.java"
+ {
+ int _klen;
+ int _trans = 0;
+ int _acts;
+ int _nacts;
+ int _keys;
+ int _goto_targ = 0;
+
+ _goto: while (true) {
+ switch ( _goto_targ ) {
+ case 0:
+ if ( p == pe ) {
+ _goto_targ = 4;
+ continue _goto;
+ }
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+case 1:
+ _match: do {
+ _keys = _JSON_array_key_offsets[cs];
+ _trans = _JSON_array_index_offsets[cs];
+ _klen = _JSON_array_single_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + _klen - 1;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + ((_upper-_lower) >> 1);
+ if ( data[p] < _JSON_array_trans_keys[_mid] )
+ _upper = _mid - 1;
+ else if ( data[p] > _JSON_array_trans_keys[_mid] )
+ _lower = _mid + 1;
+ else {
+ _trans += (_mid - _keys);
+ break _match;
+ }
+ }
+ _keys += _klen;
+ _trans += _klen;
+ }
+
+ _klen = _JSON_array_range_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + (_klen<<1) - 2;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + (((_upper-_lower) >> 1) & ~1);
+ if ( data[p] < _JSON_array_trans_keys[_mid] )
+ _upper = _mid - 2;
+ else if ( data[p] > _JSON_array_trans_keys[_mid+1] )
+ _lower = _mid + 2;
+ else {
+ _trans += ((_mid - _keys)>>1);
+ break _match;
+ }
+ }
+ _trans += _klen;
+ }
+ } while (false);
+
+ _trans = _JSON_array_indicies[_trans];
+ cs = _JSON_array_trans_targs[_trans];
+
+ if ( _JSON_array_trans_actions[_trans] != 0 ) {
+ _acts = _JSON_array_trans_actions[_trans];
+ _nacts = (int) _JSON_array_actions[_acts++];
+ while ( _nacts-- > 0 )
+ {
+ switch ( _JSON_array_actions[_acts++] )
+ {
+ case 0:
+// line 622 "Parser.rl"
+ {
+ ParserResult res = parseValue(p, pe);
+ if (res == null) {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ if (!parser.arrayClass.getName().equals("Array")) {
+ result.callMethod(context, "<<", res.result);
+ } else {
+ result.append(res.result);
+ }
+ {p = (( res.p))-1;}
+ }
+ }
+ break;
+ case 1:
+// line 637 "Parser.rl"
+ {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ }
+ break;
+// line 1690 "Parser.java"
+ }
+ }
+ }
+
+case 2:
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+ if ( ++p != pe ) {
+ _goto_targ = 1;
+ continue _goto;
+ }
+case 4:
+case 5:
+ }
+ break; }
+ }
+
+// line 671 "Parser.rl"
+
+ if (cs >= JSON_array_first_final) {
+ return new ParserResult(result, p + 1);
+ } else {
+ throw unexpectedToken(p, pe);
+ }
+ }
+
+
+// line 1720 "Parser.java"
+private static byte[] init__JSON_object_actions_0()
+{
+ return new byte [] {
+ 0, 1, 0, 1, 1, 1, 2
+ };
+}
+
+private static final byte _JSON_object_actions[] = init__JSON_object_actions_0();
+
+
+private static byte[] init__JSON_object_key_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 1, 8, 14, 16, 17, 19, 20, 36, 43, 49,
+ 51, 52, 54, 55, 57, 58, 60, 61, 63, 64, 66, 67,
+ 69, 70, 72, 73
+ };
+}
+
+private static final byte _JSON_object_key_offsets[] = init__JSON_object_key_offsets_0();
+
+
+private static char[] init__JSON_object_trans_keys_0()
+{
+ return new char [] {
+ 123, 13, 32, 34, 47, 125, 9, 10, 13, 32, 47, 58,
+ 9, 10, 42, 47, 42, 42, 47, 10, 13, 32, 34, 45,
+ 47, 73, 78, 91, 102, 110, 116, 123, 9, 10, 48, 57,
+ 13, 32, 44, 47, 125, 9, 10, 13, 32, 34, 47, 9,
+ 10, 42, 47, 42, 42, 47, 10, 42, 47, 42, 42, 47,
+ 10, 42, 47, 42, 42, 47, 10, 42, 47, 42, 42, 47,
+ 10, 0
+ };
+}
+
+private static final char _JSON_object_trans_keys[] = init__JSON_object_trans_keys_0();
+
+
+private static byte[] init__JSON_object_single_lengths_0()
+{
+ return new byte [] {
+ 0, 1, 5, 4, 2, 1, 2, 1, 12, 5, 4, 2,
+ 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2,
+ 1, 2, 1, 0
+ };
+}
+
+private static final byte _JSON_object_single_lengths[] = init__JSON_object_single_lengths_0();
+
+
+private static byte[] init__JSON_object_range_lengths_0()
+{
+ return new byte [] {
+ 0, 0, 1, 1, 0, 0, 0, 0, 2, 1, 1, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0
+ };
+}
+
+private static final byte _JSON_object_range_lengths[] = init__JSON_object_range_lengths_0();
+
+
+private static byte[] init__JSON_object_index_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 2, 9, 15, 18, 20, 23, 25, 40, 47, 53,
+ 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83,
+ 86, 88, 91, 93
+ };
+}
+
+private static final byte _JSON_object_index_offsets[] = init__JSON_object_index_offsets_0();
+
+
+private static byte[] init__JSON_object_indicies_0()
+{
+ return new byte [] {
+ 0, 1, 0, 0, 2, 3, 4, 0, 1, 5, 5, 6,
+ 7, 5, 1, 8, 9, 1, 10, 8, 10, 5, 8, 5,
+ 9, 7, 7, 11, 11, 12, 11, 11, 11, 11, 11, 11,
+ 11, 7, 11, 1, 13, 13, 14, 15, 4, 13, 1, 14,
+ 14, 2, 16, 14, 1, 17, 18, 1, 19, 17, 19, 14,
+ 17, 14, 18, 20, 21, 1, 22, 20, 22, 13, 20, 13,
+ 21, 23, 24, 1, 25, 23, 25, 7, 23, 7, 24, 26,
+ 27, 1, 28, 26, 28, 0, 26, 0, 27, 1, 0
+ };
+}
+
+private static final byte _JSON_object_indicies[] = init__JSON_object_indicies_0();
+
+
+private static byte[] init__JSON_object_trans_targs_0()
+{
+ return new byte [] {
+ 2, 0, 3, 23, 27, 3, 4, 8, 5, 7, 6, 9,
+ 19, 9, 10, 15, 11, 12, 14, 13, 16, 18, 17, 20,
+ 22, 21, 24, 26, 25
+ };
+}
+
+private static final byte _JSON_object_trans_targs[] = init__JSON_object_trans_targs_0();
+
+
+private static byte[] init__JSON_object_trans_actions_0()
+{
+ return new byte [] {
+ 0, 0, 3, 0, 5, 0, 0, 0, 0, 0, 0, 1,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0
+ };
+}
+
+private static final byte _JSON_object_trans_actions[] = init__JSON_object_trans_actions_0();
+
+
+static final int JSON_object_start = 1;
+static final int JSON_object_first_final = 27;
+static final int JSON_object_error = 0;
+
+static final int JSON_object_en_main = 1;
+
+
+// line 730 "Parser.rl"
+
+
+ ParserResult parseObject(int p, int pe) {
+ int cs = EVIL;
+ IRubyObject lastName = null;
+
+ if (parser.maxNesting > 0 && currentNesting > parser.maxNesting) {
+ throw newException(Utils.M_NESTING_ERROR,
+ "nesting of " + currentNesting + " is too deep");
+ }
+
+ // this is guaranteed to be a RubyHash due to the earlier
+ // allocator test at OptionsReader#getClass
+ RubyHash result =
+ (RubyHash)parser.objectClass.newInstance(context,
+ IRubyObject.NULL_ARRAY, Block.NULL_BLOCK);
+
+
+// line 1862 "Parser.java"
+ {
+ cs = JSON_object_start;
+ }
+
+// line 748 "Parser.rl"
+
+// line 1869 "Parser.java"
+ {
+ int _klen;
+ int _trans = 0;
+ int _acts;
+ int _nacts;
+ int _keys;
+ int _goto_targ = 0;
+
+ _goto: while (true) {
+ switch ( _goto_targ ) {
+ case 0:
+ if ( p == pe ) {
+ _goto_targ = 4;
+ continue _goto;
+ }
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+case 1:
+ _match: do {
+ _keys = _JSON_object_key_offsets[cs];
+ _trans = _JSON_object_index_offsets[cs];
+ _klen = _JSON_object_single_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + _klen - 1;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + ((_upper-_lower) >> 1);
+ if ( data[p] < _JSON_object_trans_keys[_mid] )
+ _upper = _mid - 1;
+ else if ( data[p] > _JSON_object_trans_keys[_mid] )
+ _lower = _mid + 1;
+ else {
+ _trans += (_mid - _keys);
+ break _match;
+ }
+ }
+ _keys += _klen;
+ _trans += _klen;
+ }
+
+ _klen = _JSON_object_range_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + (_klen<<1) - 2;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + (((_upper-_lower) >> 1) & ~1);
+ if ( data[p] < _JSON_object_trans_keys[_mid] )
+ _upper = _mid - 2;
+ else if ( data[p] > _JSON_object_trans_keys[_mid+1] )
+ _lower = _mid + 2;
+ else {
+ _trans += ((_mid - _keys)>>1);
+ break _match;
+ }
+ }
+ _trans += _klen;
+ }
+ } while (false);
+
+ _trans = _JSON_object_indicies[_trans];
+ cs = _JSON_object_trans_targs[_trans];
+
+ if ( _JSON_object_trans_actions[_trans] != 0 ) {
+ _acts = _JSON_object_trans_actions[_trans];
+ _nacts = (int) _JSON_object_actions[_acts++];
+ while ( _nacts-- > 0 )
+ {
+ switch ( _JSON_object_actions[_acts++] )
+ {
+ case 0:
+// line 685 "Parser.rl"
+ {
+ ParserResult res = parseValue(p, pe);
+ if (res == null) {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ if (!parser.objectClass.getName().equals("Hash")) {
+ result.callMethod(context, "[]=", new IRubyObject[] { lastName, res.result });
+ } else {
+ result.op_aset(context, lastName, res.result);
+ }
+ {p = (( res.p))-1;}
+ }
+ }
+ break;
+ case 1:
+// line 700 "Parser.rl"
+ {
+ ParserResult res = parseString(p, pe);
+ if (res == null) {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ RubyString name = (RubyString)res.result;
+ if (parser.symbolizeNames) {
+ lastName = context.getRuntime().is1_9()
+ ? name.intern19()
+ : name.intern();
+ } else {
+ lastName = name;
+ }
+ {p = (( res.p))-1;}
+ }
+ }
+ break;
+ case 2:
+// line 718 "Parser.rl"
+ {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ }
+ break;
+// line 1993 "Parser.java"
+ }
+ }
+ }
+
+case 2:
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+ if ( ++p != pe ) {
+ _goto_targ = 1;
+ continue _goto;
+ }
+case 4:
+case 5:
+ }
+ break; }
+ }
+
+// line 749 "Parser.rl"
+
+ if (cs < JSON_object_first_final) {
+ return null;
+ }
+
+ IRubyObject returnedResult = result;
+
+ // attempt to de-serialize object
+ if (parser.createAdditions) {
+ IRubyObject vKlassName = result.op_aref(context, parser.createId);
+ if (!vKlassName.isNil()) {
+ // might throw ArgumentError, we let it propagate
+ IRubyObject klass = parser.info.jsonModule.get().
+ callMethod(context, "deep_const_get", vKlassName);
+ if (klass.respondsTo("json_creatable?") &&
+ klass.callMethod(context, "json_creatable?").isTrue()) {
+
+ returnedResult = klass.callMethod(context, "json_create", result);
+ }
+ }
+ }
+ return new ParserResult(returnedResult, p + 1);
+ }
+
+
+// line 2039 "Parser.java"
+private static byte[] init__JSON_actions_0()
+{
+ return new byte [] {
+ 0, 1, 0, 1, 1
+ };
+}
+
+private static final byte _JSON_actions[] = init__JSON_actions_0();
+
+
+private static byte[] init__JSON_key_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 7, 9, 10, 12, 13, 15, 16, 18, 19
+ };
+}
+
+private static final byte _JSON_key_offsets[] = init__JSON_key_offsets_0();
+
+
+private static char[] init__JSON_trans_keys_0()
+{
+ return new char [] {
+ 13, 32, 47, 91, 123, 9, 10, 42, 47, 42, 42, 47,
+ 10, 42, 47, 42, 42, 47, 10, 13, 32, 47, 9, 10,
+ 0
+ };
+}
+
+private static final char _JSON_trans_keys[] = init__JSON_trans_keys_0();
+
+
+private static byte[] init__JSON_single_lengths_0()
+{
+ return new byte [] {
+ 0, 5, 2, 1, 2, 1, 2, 1, 2, 1, 3
+ };
+}
+
+private static final byte _JSON_single_lengths[] = init__JSON_single_lengths_0();
+
+
+private static byte[] init__JSON_range_lengths_0()
+{
+ return new byte [] {
+ 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1
+ };
+}
+
+private static final byte _JSON_range_lengths[] = init__JSON_range_lengths_0();
+
+
+private static byte[] init__JSON_index_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 7, 10, 12, 15, 17, 20, 22, 25, 27
+ };
+}
+
+private static final byte _JSON_index_offsets[] = init__JSON_index_offsets_0();
+
+
+private static byte[] init__JSON_indicies_0()
+{
+ return new byte [] {
+ 0, 0, 2, 3, 4, 0, 1, 5, 6, 1, 7, 5,
+ 7, 0, 5, 0, 6, 8, 9, 1, 10, 8, 10, 11,
+ 8, 11, 9, 11, 11, 12, 11, 1, 0
+ };
+}
+
+private static final byte _JSON_indicies[] = init__JSON_indicies_0();
+
+
+private static byte[] init__JSON_trans_targs_0()
+{
+ return new byte [] {
+ 1, 0, 2, 10, 10, 3, 5, 4, 7, 9, 8, 10,
+ 6
+ };
+}
+
+private static final byte _JSON_trans_targs[] = init__JSON_trans_targs_0();
+
+
+private static byte[] init__JSON_trans_actions_0()
+{
+ return new byte [] {
+ 0, 0, 0, 3, 1, 0, 0, 0, 0, 0, 0, 0,
+ 0
+ };
+}
+
+private static final byte _JSON_trans_actions[] = init__JSON_trans_actions_0();
+
+
+static final int JSON_start = 1;
+static final int JSON_first_final = 10;
+static final int JSON_error = 0;
+
+static final int JSON_en_main = 1;
+
+
+// line 807 "Parser.rl"
+
+
+ public IRubyObject parseStrict() {
+ int cs = EVIL;
+ int p, pe;
+ IRubyObject result = null;
+
+
+// line 2152 "Parser.java"
+ {
+ cs = JSON_start;
+ }
+
+// line 815 "Parser.rl"
+ p = byteList.begin();
+ pe = p + byteList.length();
+
+// line 2161 "Parser.java"
+ {
+ int _klen;
+ int _trans = 0;
+ int _acts;
+ int _nacts;
+ int _keys;
+ int _goto_targ = 0;
+
+ _goto: while (true) {
+ switch ( _goto_targ ) {
+ case 0:
+ if ( p == pe ) {
+ _goto_targ = 4;
+ continue _goto;
+ }
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+case 1:
+ _match: do {
+ _keys = _JSON_key_offsets[cs];
+ _trans = _JSON_index_offsets[cs];
+ _klen = _JSON_single_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + _klen - 1;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + ((_upper-_lower) >> 1);
+ if ( data[p] < _JSON_trans_keys[_mid] )
+ _upper = _mid - 1;
+ else if ( data[p] > _JSON_trans_keys[_mid] )
+ _lower = _mid + 1;
+ else {
+ _trans += (_mid - _keys);
+ break _match;
+ }
+ }
+ _keys += _klen;
+ _trans += _klen;
+ }
+
+ _klen = _JSON_range_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + (_klen<<1) - 2;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + (((_upper-_lower) >> 1) & ~1);
+ if ( data[p] < _JSON_trans_keys[_mid] )
+ _upper = _mid - 2;
+ else if ( data[p] > _JSON_trans_keys[_mid+1] )
+ _lower = _mid + 2;
+ else {
+ _trans += ((_mid - _keys)>>1);
+ break _match;
+ }
+ }
+ _trans += _klen;
+ }
+ } while (false);
+
+ _trans = _JSON_indicies[_trans];
+ cs = _JSON_trans_targs[_trans];
+
+ if ( _JSON_trans_actions[_trans] != 0 ) {
+ _acts = _JSON_trans_actions[_trans];
+ _nacts = (int) _JSON_actions[_acts++];
+ while ( _nacts-- > 0 )
+ {
+ switch ( _JSON_actions[_acts++] )
+ {
+ case 0:
+// line 779 "Parser.rl"
+ {
+ currentNesting = 1;
+ ParserResult res = parseObject(p, pe);
+ if (res == null) {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ result = res.result;
+ {p = (( res.p))-1;}
+ }
+ }
+ break;
+ case 1:
+// line 791 "Parser.rl"
+ {
+ currentNesting = 1;
+ ParserResult res = parseArray(p, pe);
+ if (res == null) {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ result = res.result;
+ {p = (( res.p))-1;}
+ }
+ }
+ break;
+// line 2269 "Parser.java"
+ }
+ }
+ }
+
+case 2:
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+ if ( ++p != pe ) {
+ _goto_targ = 1;
+ continue _goto;
+ }
+case 4:
+case 5:
+ }
+ break; }
+ }
+
+// line 818 "Parser.rl"
+
+ if (cs >= JSON_first_final && p == pe) {
+ return result;
+ } else {
+ throw unexpectedToken(p, pe);
+ }
+ }
+
+
+// line 2299 "Parser.java"
+private static byte[] init__JSON_quirks_mode_actions_0()
+{
+ return new byte [] {
+ 0, 1, 0
+ };
+}
+
+private static final byte _JSON_quirks_mode_actions[] = init__JSON_quirks_mode_actions_0();
+
+
+private static byte[] init__JSON_quirks_mode_key_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 16, 18, 19, 21, 22, 24, 25, 27, 28
+ };
+}
+
+private static final byte _JSON_quirks_mode_key_offsets[] = init__JSON_quirks_mode_key_offsets_0();
+
+
+private static char[] init__JSON_quirks_mode_trans_keys_0()
+{
+ return new char [] {
+ 13, 32, 34, 45, 47, 73, 78, 91, 102, 110, 116, 123,
+ 9, 10, 48, 57, 42, 47, 42, 42, 47, 10, 42, 47,
+ 42, 42, 47, 10, 13, 32, 47, 9, 10, 0
+ };
+}
+
+private static final char _JSON_quirks_mode_trans_keys[] = init__JSON_quirks_mode_trans_keys_0();
+
+
+private static byte[] init__JSON_quirks_mode_single_lengths_0()
+{
+ return new byte [] {
+ 0, 12, 2, 1, 2, 1, 2, 1, 2, 1, 3
+ };
+}
+
+private static final byte _JSON_quirks_mode_single_lengths[] = init__JSON_quirks_mode_single_lengths_0();
+
+
+private static byte[] init__JSON_quirks_mode_range_lengths_0()
+{
+ return new byte [] {
+ 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 1
+ };
+}
+
+private static final byte _JSON_quirks_mode_range_lengths[] = init__JSON_quirks_mode_range_lengths_0();
+
+
+private static byte[] init__JSON_quirks_mode_index_offsets_0()
+{
+ return new byte [] {
+ 0, 0, 15, 18, 20, 23, 25, 28, 30, 33, 35
+ };
+}
+
+private static final byte _JSON_quirks_mode_index_offsets[] = init__JSON_quirks_mode_index_offsets_0();
+
+
+private static byte[] init__JSON_quirks_mode_indicies_0()
+{
+ return new byte [] {
+ 0, 0, 2, 2, 3, 2, 2, 2, 2, 2, 2, 2,
+ 0, 2, 1, 4, 5, 1, 6, 4, 6, 7, 4, 7,
+ 5, 8, 9, 1, 10, 8, 10, 0, 8, 0, 9, 7,
+ 7, 11, 7, 1, 0
+ };
+}
+
+private static final byte _JSON_quirks_mode_indicies[] = init__JSON_quirks_mode_indicies_0();
+
+
+private static byte[] init__JSON_quirks_mode_trans_targs_0()
+{
+ return new byte [] {
+ 1, 0, 10, 6, 3, 5, 4, 10, 7, 9, 8, 2
+ };
+}
+
+private static final byte _JSON_quirks_mode_trans_targs[] = init__JSON_quirks_mode_trans_targs_0();
+
+
+private static byte[] init__JSON_quirks_mode_trans_actions_0()
+{
+ return new byte [] {
+ 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0
+ };
+}
+
+private static final byte _JSON_quirks_mode_trans_actions[] = init__JSON_quirks_mode_trans_actions_0();
+
+
+static final int JSON_quirks_mode_start = 1;
+static final int JSON_quirks_mode_first_final = 10;
+static final int JSON_quirks_mode_error = 0;
+
+static final int JSON_quirks_mode_en_main = 1;
+
+
+// line 846 "Parser.rl"
+
+
+ public IRubyObject parseQuirksMode() {
+ int cs = EVIL;
+ int p, pe;
+ IRubyObject result = null;
+
+
+// line 2411 "Parser.java"
+ {
+ cs = JSON_quirks_mode_start;
+ }
+
+// line 854 "Parser.rl"
+ p = byteList.begin();
+ pe = p + byteList.length();
+
+// line 2420 "Parser.java"
+ {
+ int _klen;
+ int _trans = 0;
+ int _acts;
+ int _nacts;
+ int _keys;
+ int _goto_targ = 0;
+
+ _goto: while (true) {
+ switch ( _goto_targ ) {
+ case 0:
+ if ( p == pe ) {
+ _goto_targ = 4;
+ continue _goto;
+ }
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+case 1:
+ _match: do {
+ _keys = _JSON_quirks_mode_key_offsets[cs];
+ _trans = _JSON_quirks_mode_index_offsets[cs];
+ _klen = _JSON_quirks_mode_single_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + _klen - 1;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + ((_upper-_lower) >> 1);
+ if ( data[p] < _JSON_quirks_mode_trans_keys[_mid] )
+ _upper = _mid - 1;
+ else if ( data[p] > _JSON_quirks_mode_trans_keys[_mid] )
+ _lower = _mid + 1;
+ else {
+ _trans += (_mid - _keys);
+ break _match;
+ }
+ }
+ _keys += _klen;
+ _trans += _klen;
+ }
+
+ _klen = _JSON_quirks_mode_range_lengths[cs];
+ if ( _klen > 0 ) {
+ int _lower = _keys;
+ int _mid;
+ int _upper = _keys + (_klen<<1) - 2;
+ while (true) {
+ if ( _upper < _lower )
+ break;
+
+ _mid = _lower + (((_upper-_lower) >> 1) & ~1);
+ if ( data[p] < _JSON_quirks_mode_trans_keys[_mid] )
+ _upper = _mid - 2;
+ else if ( data[p] > _JSON_quirks_mode_trans_keys[_mid+1] )
+ _lower = _mid + 2;
+ else {
+ _trans += ((_mid - _keys)>>1);
+ break _match;
+ }
+ }
+ _trans += _klen;
+ }
+ } while (false);
+
+ _trans = _JSON_quirks_mode_indicies[_trans];
+ cs = _JSON_quirks_mode_trans_targs[_trans];
+
+ if ( _JSON_quirks_mode_trans_actions[_trans] != 0 ) {
+ _acts = _JSON_quirks_mode_trans_actions[_trans];
+ _nacts = (int) _JSON_quirks_mode_actions[_acts++];
+ while ( _nacts-- > 0 )
+ {
+ switch ( _JSON_quirks_mode_actions[_acts++] )
+ {
+ case 0:
+// line 832 "Parser.rl"
+ {
+ ParserResult res = parseValue(p, pe);
+ if (res == null) {
+ p--;
+ { p += 1; _goto_targ = 5; if (true) continue _goto;}
+ } else {
+ result = res.result;
+ {p = (( res.p))-1;}
+ }
+ }
+ break;
+// line 2513 "Parser.java"
+ }
+ }
+ }
+
+case 2:
+ if ( cs == 0 ) {
+ _goto_targ = 5;
+ continue _goto;
+ }
+ if ( ++p != pe ) {
+ _goto_targ = 1;
+ continue _goto;
+ }
+case 4:
+case 5:
+ }
+ break; }
+ }
+
+// line 857 "Parser.rl"
+
+ if (cs >= JSON_quirks_mode_first_final && p == pe) {
+ return result;
+ } else {
+ throw unexpectedToken(p, pe);
+ }
+ }
+
+ public IRubyObject parse() {
+ if (parser.quirksMode) {
+ return parseQuirksMode();
+ } else {
+ return parseStrict();
+ }
+
+ }
+
+ /**
+ * Returns a subsequence of the source ByteList, based on source
+ * array byte offsets (i.e., the ByteList's own begin offset is not
+ * automatically added).
+ * @param start
+ * @param end
+ */
+ private ByteList absSubSequence(int absStart, int absEnd) {
+ int offset = byteList.begin();
+ return (ByteList)byteList.subSequence(absStart - offset,
+ absEnd - offset);
+ }
+
+ /**
+ * Retrieves a constant directly descended from the <code>JSON</code> module.
+ * @param name The constant name
+ */
+ private IRubyObject getConstant(String name) {
+ return parser.info.jsonModule.get().getConstant(name);
+ }
+
+ private RaiseException newException(String className, String message) {
+ return Utils.newException(context, className, message);
+ }
+
+ private RaiseException newException(String className, RubyString message) {
+ return Utils.newException(context, className, message);
+ }
+
+ private RaiseException newException(String className,
+ String messageBegin, ByteList messageEnd) {
+ return newException(className,
+ getRuntime().newString(messageBegin).cat(messageEnd));
+ }
+ }
+}
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import org.jruby.Ruby;
+import org.jruby.RubyArray;
+import org.jruby.RubyClass;
+import org.jruby.RubyEncoding;
+import org.jruby.RubyFloat;
+import org.jruby.RubyHash;
+import org.jruby.RubyInteger;
+import org.jruby.RubyModule;
+import org.jruby.RubyNumeric;
+import org.jruby.RubyObject;
+import org.jruby.RubyString;
+import org.jruby.anno.JRubyMethod;
+import org.jruby.exceptions.JumpException;
+import org.jruby.exceptions.RaiseException;
+import org.jruby.runtime.Block;
+import org.jruby.runtime.ObjectAllocator;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.runtime.Visibility;
+import org.jruby.runtime.builtin.IRubyObject;
+import org.jruby.util.ByteList;
+
+/**
+ * The <code>JSON::Ext::Parser</code> class.
+ *
+ * <p>This is the JSON parser implemented as a Java class. To use it as the
+ * standard parser, set
+ * <pre>JSON.parser = JSON::Ext::Parser</pre>
+ * This is performed for you when you <code>include "json/ext"</code>.
+ *
+ * <p>This class does not perform the actual parsing, just acts as an interface
+ * to Ruby code. When the {@link #parse()} method is invoked, a
+ * Parser.ParserSession object is instantiated, which handles the process.
+ *
+ * @author mernen
+ */
+public class Parser extends RubyObject {
+ private final RuntimeInfo info;
+ private RubyString vSource;
+ private RubyString createId;
+ private boolean createAdditions;
+ private int maxNesting;
+ private boolean allowNaN;
+ private boolean symbolizeNames;
+ private boolean quirksMode;
+ private RubyClass objectClass;
+ private RubyClass arrayClass;
+ private RubyHash match_string;
+
+ private static final int DEFAULT_MAX_NESTING = 19;
+
+ private static final String JSON_MINUS_INFINITY = "-Infinity";
+ // constant names in the JSON module containing those values
+ private static final String CONST_NAN = "NaN";
+ private static final String CONST_INFINITY = "Infinity";
+ private static final String CONST_MINUS_INFINITY = "MinusInfinity";
+
+ static final ObjectAllocator ALLOCATOR = new ObjectAllocator() {
+ public IRubyObject allocate(Ruby runtime, RubyClass klazz) {
+ return new Parser(runtime, klazz);
+ }
+ };
+
+ /**
+ * Multiple-value return for internal parser methods.
+ *
+ * <p>All the <code>parse<var>Stuff</var></code> methods return instances of
+ * <code>ParserResult</code> when successful, or <code>null</code> when
+ * there's a problem with the input data.
+ */
+ static final class ParserResult {
+ /**
+ * The result of the successful parsing. Should never be
+ * <code>null</code>.
+ */
+ final IRubyObject result;
+ /**
+ * The point where the parser returned.
+ */
+ final int p;
+
+ ParserResult(IRubyObject result, int p) {
+ this.result = result;
+ this.p = p;
+ }
+ }
+
+ public Parser(Ruby runtime, RubyClass metaClass) {
+ super(runtime, metaClass);
+ info = RuntimeInfo.forRuntime(runtime);
+ }
+
+ /**
+ * <code>Parser.new(source, opts = {})</code>
+ *
+ * <p>Creates a new <code>JSON::Ext::Parser</code> instance for the string
+ * <code>source</code>.
+ * It will be configured by the <code>opts</code> Hash.
+ * <code>opts</code> can have the following keys:
+ *
+ * <dl>
+ * <dt><code>:max_nesting</code>
+ * <dd>The maximum depth of nesting allowed in the parsed data
+ * structures. Disable depth checking with <code>:max_nesting => false|nil|0</code>,
+ * it defaults to 19.
+ *
+ * <dt><code>:allow_nan</code>
+ * <dd>If set to <code>true</code>, allow <code>NaN</code>,
+ * <code>Infinity</code> and <code>-Infinity</code> in defiance of RFC 4627
+ * to be parsed by the Parser. This option defaults to <code>false</code>.
+ *
+ * <dt><code>:symbolize_names</code>
+ * <dd>If set to <code>true</code>, returns symbols for the names (keys) in
+ * a JSON object. Otherwise strings are returned, which is also the default.
+ *
+ * <dt><code>:quirks_mode?</code>
+ * <dd>If set to <code>true</code>, if the parse is in quirks_mode, false
+ * otherwise.
+ *
+ * <dt><code>:create_additions</code>
+ * <dd>If set to <code>false</code>, the Parser doesn't create additions
+ * even if a matchin class and <code>create_id</code> was found. This option
+ * defaults to <code>true</code>.
+ *
+ * <dt><code>:object_class</code>
+ * <dd>Defaults to Hash.
+ *
+ * <dt><code>:array_class</code>
+ * <dd>Defaults to Array.
+ *
+ * <dt><code>:quirks_mode</code>
+ * <dd>Enables quirks_mode for parser, that is for example parsing single
+ * JSON values instead of documents is possible.
+ * </dl>
+ */
+ @JRubyMethod(name = "new", required = 1, optional = 1, meta = true)
+ public static IRubyObject newInstance(IRubyObject clazz, IRubyObject[] args, Block block) {
+ Parser parser = (Parser)((RubyClass)clazz).allocate();
+
+ parser.callInit(args, block);
+
+ return parser;
+ }
+
+ @JRubyMethod(required = 1, optional = 1, visibility = Visibility.PRIVATE)
+ public IRubyObject initialize(ThreadContext context, IRubyObject[] args) {
+ Ruby runtime = context.getRuntime();
+ if (this.vSource != null) {
+ throw runtime.newTypeError("already initialized instance");
+ }
+
+ OptionsReader opts = new OptionsReader(context, args.length > 1 ? args[1] : null);
+ this.maxNesting = opts.getInt("max_nesting", DEFAULT_MAX_NESTING);
+ this.allowNaN = opts.getBool("allow_nan", false);
+ this.symbolizeNames = opts.getBool("symbolize_names", false);
+ this.quirksMode = opts.getBool("quirks_mode", false);
+ this.createId = opts.getString("create_id", getCreateId(context));
+ this.createAdditions = opts.getBool("create_additions", false);
+ this.objectClass = opts.getClass("object_class", runtime.getHash());
+ this.arrayClass = opts.getClass("array_class", runtime.getArray());
+ this.match_string = opts.getHash("match_string");
+
+ this.vSource = args[0].convertToString();
+ if (!quirksMode) this.vSource = convertEncoding(context, vSource);
+
+ return this;
+ }
+
+ /**
+ * Checks the given string's encoding. If a non-UTF-8 encoding is detected,
+ * a converted copy is returned.
+ * Returns the source string if no conversion is needed.
+ */
+ private RubyString convertEncoding(ThreadContext context, RubyString source) {
+ ByteList bl = source.getByteList();
+ int len = bl.length();
+ if (len < 2) {
+ throw Utils.newException(context, Utils.M_PARSER_ERROR,
+ "A JSON text must at least contain two octets!");
+ }
+
+ if (info.encodingsSupported()) {
+ RubyEncoding encoding = (RubyEncoding)source.encoding(context);
+ if (encoding != info.ascii8bit.get()) {
+ return (RubyString)source.encode(context, info.utf8.get());
+ }
+
+ String sniffedEncoding = sniffByteList(bl);
+ if (sniffedEncoding == null) return source; // assume UTF-8
+ return reinterpretEncoding(context, source, sniffedEncoding);
+ }
+
+ String sniffedEncoding = sniffByteList(bl);
+ if (sniffedEncoding == null) return source; // assume UTF-8
+ Ruby runtime = context.getRuntime();
+ return (RubyString)info.jsonModule.get().
+ callMethod(context, "iconv",
+ new IRubyObject[] {
+ runtime.newString("utf-8"),
+ runtime.newString(sniffedEncoding),
+ source});
+ }
+
+ /**
+ * Checks the first four bytes of the given ByteList to infer its encoding,
+ * using the principle demonstrated on section 3 of RFC 4627 (JSON).
+ */
+ private static String sniffByteList(ByteList bl) {
+ if (bl.length() < 4) return null;
+ if (bl.get(0) == 0 && bl.get(2) == 0) {
+ return bl.get(1) == 0 ? "utf-32be" : "utf-16be";
+ }
+ if (bl.get(1) == 0 && bl.get(3) == 0) {
+ return bl.get(2) == 0 ? "utf-32le" : "utf-16le";
+ }
+ return null;
+ }
+
+ /**
+ * Assumes the given (binary) RubyString to be in the given encoding, then
+ * converts it to UTF-8.
+ */
+ private RubyString reinterpretEncoding(ThreadContext context,
+ RubyString str, String sniffedEncoding) {
+ RubyEncoding actualEncoding = info.getEncoding(context, sniffedEncoding);
+ RubyEncoding targetEncoding = info.utf8.get();
+ RubyString dup = (RubyString)str.dup();
+ dup.force_encoding(context, actualEncoding);
+ return (RubyString)dup.encode_bang(context, targetEncoding);
+ }
+
+ /**
+ * <code>Parser#parse()</code>
+ *
+ * <p>Parses the current JSON text <code>source</code> and returns the
+ * complete data structure as a result.
+ */
+ @JRubyMethod
+ public IRubyObject parse(ThreadContext context) {
+ return new ParserSession(this, context).parse();
+ }
+
+ /**
+ * <code>Parser#source()</code>
+ *
+ * <p>Returns a copy of the current <code>source</code> string, that was
+ * used to construct this Parser.
+ */
+ @JRubyMethod(name = "source")
+ public IRubyObject source_get() {
+ return checkAndGetSource().dup();
+ }
+
+ /**
+ * <code>Parser#quirks_mode?()</code>
+ *
+ * <p>If set to <code>true</code>, if the parse is in quirks_mode, false
+ * otherwise.
+ */
+ @JRubyMethod(name = "quirks_mode?")
+ public IRubyObject quirks_mode_p(ThreadContext context) {
+ return context.getRuntime().newBoolean(quirksMode);
+ }
+
+ public RubyString checkAndGetSource() {
+ if (vSource != null) {
+ return vSource;
+ } else {
+ throw getRuntime().newTypeError("uninitialized instance");
+ }
+ }
+
+ /**
+ * Queries <code>JSON.create_id</code>. Returns <code>null</code> if it is
+ * set to <code>nil</code> or <code>false</code>, and a String if not.
+ */
+ private RubyString getCreateId(ThreadContext context) {
+ IRubyObject v = info.jsonModule.get().callMethod(context, "create_id");
+ return v.isTrue() ? v.convertToString() : null;
+ }
+
+ /**
+ * A string parsing session.
+ *
+ * <p>Once a ParserSession is instantiated, the source string should not
+ * change until the parsing is complete. The ParserSession object assumes
+ * the source {@link RubyString} is still associated to its original
+ * {@link ByteList}, which in turn must still be bound to the same
+ * <code>byte[]</code> value (and on the same offset).
+ */
+ // Ragel uses lots of fall-through
+ @SuppressWarnings("fallthrough")
+ private static class ParserSession {
+ private final Parser parser;
+ private final ThreadContext context;
+ private final ByteList byteList;
+ private final byte[] data;
+ private final StringDecoder decoder;
+ private int currentNesting = 0;
+
+ // initialization value for all state variables.
+ // no idea about the origins of this value, ask Flori ;)
+ private static final int EVIL = 0x666;
+
+ private ParserSession(Parser parser, ThreadContext context) {
+ this.parser = parser;
+ this.context = context;
+ this.byteList = parser.checkAndGetSource().getByteList();
+ this.data = byteList.unsafeBytes();
+ this.decoder = new StringDecoder(context);
+ }
+
+ private RaiseException unexpectedToken(int absStart, int absEnd) {
+ RubyString msg = getRuntime().newString("unexpected token at '")
+ .cat(data, absStart, absEnd - absStart)
+ .cat((byte)'\'');
+ return newException(Utils.M_PARSER_ERROR, msg);
+ }
+
+ private Ruby getRuntime() {
+ return context.getRuntime();
+ }
+
+ %%{
+ machine JSON_common;
+
+ cr = '\n';
+ cr_neg = [^\n];
+ ws = [ \t\r\n];
+ c_comment = '/*' ( any* - (any* '*/' any* ) ) '*/';
+ cpp_comment = '//' cr_neg* cr;
+ comment = c_comment | cpp_comment;
+ ignore = ws | comment;
+ name_separator = ':';
+ value_separator = ',';
+ Vnull = 'null';
+ Vfalse = 'false';
+ Vtrue = 'true';
+ VNaN = 'NaN';
+ VInfinity = 'Infinity';
+ VMinusInfinity = '-Infinity';
+ begin_value = [nft"\-[{NI] | digit;
+ begin_object = '{';
+ end_object = '}';
+ begin_array = '[';
+ end_array = ']';
+ begin_string = '"';
+ begin_name = begin_string;
+ begin_number = digit | '-';
+ }%%
+
+ %%{
+ machine JSON_value;
+ include JSON_common;
+
+ write data;
+
+ action parse_null {
+ result = getRuntime().getNil();
+ }
+ action parse_false {
+ result = getRuntime().getFalse();
+ }
+ action parse_true {
+ result = getRuntime().getTrue();
+ }
+ action parse_nan {
+ if (parser.allowNaN) {
+ result = getConstant(CONST_NAN);
+ } else {
+ throw unexpectedToken(p - 2, pe);
+ }
+ }
+ action parse_infinity {
+ if (parser.allowNaN) {
+ result = getConstant(CONST_INFINITY);
+ } else {
+ throw unexpectedToken(p - 7, pe);
+ }
+ }
+ action parse_number {
+ if (pe > fpc + 9 - (parser.quirksMode ? 1 : 0) &&
+ absSubSequence(fpc, fpc + 9).toString().equals(JSON_MINUS_INFINITY)) {
+
+ if (parser.allowNaN) {
+ result = getConstant(CONST_MINUS_INFINITY);
+ fexec p + 10;
+ fhold;
+ fbreak;
+ } else {
+ throw unexpectedToken(p, pe);
+ }
+ }
+ ParserResult res = parseFloat(fpc, pe);
+ if (res != null) {
+ result = res.result;
+ fexec res.p;
+ }
+ res = parseInteger(fpc, pe);
+ if (res != null) {
+ result = res.result;
+ fexec res.p;
+ }
+ fhold;
+ fbreak;
+ }
+ action parse_string {
+ ParserResult res = parseString(fpc, pe);
+ if (res == null) {
+ fhold;
+ fbreak;
+ } else {
+ result = res.result;
+ fexec res.p;
+ }
+ }
+ action parse_array {
+ currentNesting++;
+ ParserResult res = parseArray(fpc, pe);
+ currentNesting--;
+ if (res == null) {
+ fhold;
+ fbreak;
+ } else {
+ result = res.result;
+ fexec res.p;
+ }
+ }
+ action parse_object {
+ currentNesting++;
+ ParserResult res = parseObject(fpc, pe);
+ currentNesting--;
+ if (res == null) {
+ fhold;
+ fbreak;
+ } else {
+ result = res.result;
+ fexec res.p;
+ }
+ }
+ action exit {
+ fhold;
+ fbreak;
+ }
+
+ main := ( Vnull @parse_null |
+ Vfalse @parse_false |
+ Vtrue @parse_true |
+ VNaN @parse_nan |
+ VInfinity @parse_infinity |
+ begin_number >parse_number |
+ begin_string >parse_string |
+ begin_array >parse_array |
+ begin_object >parse_object
+ ) %*exit;
+ }%%
+
+ ParserResult parseValue(int p, int pe) {
+ int cs = EVIL;
+ IRubyObject result = null;
+
+ %% write init;
+ %% write exec;
+
+ if (cs >= JSON_value_first_final && result != null) {
+ return new ParserResult(result, p);
+ } else {
+ return null;
+ }
+ }
+
+ %%{
+ machine JSON_integer;
+
+ write data;
+
+ action exit {
+ fhold;
+ fbreak;
+ }
+
+ main := '-'? ( '0' | [1-9][0-9]* ) ( ^[0-9]? @exit );
+ }%%
+
+ ParserResult parseInteger(int p, int pe) {
+ int cs = EVIL;
+
+ %% write init;
+ int memo = p;
+ %% write exec;
+
+ if (cs < JSON_integer_first_final) {
+ return null;
+ }
+
+ ByteList num = absSubSequence(memo, p);
+ // note: this is actually a shared string, but since it is temporary and
+ // read-only, it doesn't really matter
+ RubyString expr = RubyString.newStringLight(getRuntime(), num);
+ RubyInteger number = RubyNumeric.str2inum(getRuntime(), expr, 10, true);
+ return new ParserResult(number, p + 1);
+ }
+
+ %%{
+ machine JSON_float;
+ include JSON_common;
+
+ write data;
+
+ action exit {
+ fhold;
+ fbreak;
+ }
+
+ main := '-'?
+ ( ( ( '0' | [1-9][0-9]* ) '.' [0-9]+ ( [Ee] [+\-]?[0-9]+ )? )
+ | ( ( '0' | [1-9][0-9]* ) ( [Ee] [+\-]? [0-9]+ ) ) )
+ ( ^[0-9Ee.\-]? @exit );
+ }%%
+
+ ParserResult parseFloat(int p, int pe) {
+ int cs = EVIL;
+
+ %% write init;
+ int memo = p;
+ %% write exec;
+
+ if (cs < JSON_float_first_final) {
+ return null;
+ }
+
+ ByteList num = absSubSequence(memo, p);
+ // note: this is actually a shared string, but since it is temporary and
+ // read-only, it doesn't really matter
+ RubyString expr = RubyString.newStringLight(getRuntime(), num);
+ RubyFloat number = RubyNumeric.str2fnum(getRuntime(), expr, true);
+ return new ParserResult(number, p + 1);
+ }
+
+ %%{
+ machine JSON_string;
+ include JSON_common;
+
+ write data;
+
+ action parse_string {
+ int offset = byteList.begin();
+ ByteList decoded = decoder.decode(byteList, memo + 1 - offset,
+ p - offset);
+ result = getRuntime().newString(decoded);
+ if (result == null) {
+ fhold;
+ fbreak;
+ } else {
+ fexec p + 1;
+ }
+ }
+
+ action exit {
+ fhold;
+ fbreak;
+ }
+
+ main := '"'
+ ( ( ^(["\\]|0..0x1f)
+ | '\\'["\\/bfnrt]
+ | '\\u'[0-9a-fA-F]{4}
+ | '\\'^(["\\/bfnrtu]|0..0x1f)
+ )* %parse_string
+ ) '"' @exit;
+ }%%
+
+ ParserResult parseString(int p, int pe) {
+ int cs = EVIL;
+ IRubyObject result = null;
+
+ %% write init;
+ int memo = p;
+ %% write exec;
+
+ if (parser.createAdditions) {
+ RubyHash match_string = parser.match_string;
+ if (match_string != null) {
+ final IRubyObject[] memoArray = { result, null };
+ try {
+ match_string.visitAll(new RubyHash.Visitor() {
+ @Override
+ public void visit(IRubyObject pattern, IRubyObject klass) {
+ if (pattern.callMethod(context, "===", memoArray[0]).isTrue()) {
+ memoArray[1] = klass;
+ throw JumpException.SPECIAL_JUMP;
+ }
+ }
+ });
+ } catch (JumpException e) { }
+ if (memoArray[1] != null) {
+ RubyClass klass = (RubyClass) memoArray[1];
+ if (klass.respondsTo("json_creatable?") &&
+ klass.callMethod(context, "json_creatable?").isTrue()) {
+ result = klass.callMethod(context, "json_create", result);
+ }
+ }
+ }
+ }
+
+ if (cs >= JSON_string_first_final && result != null) {
+ return new ParserResult(result, p + 1);
+ } else {
+ return null;
+ }
+ }
+
+ %%{
+ machine JSON_array;
+ include JSON_common;
+
+ write data;
+
+ action parse_value {
+ ParserResult res = parseValue(fpc, pe);
+ if (res == null) {
+ fhold;
+ fbreak;
+ } else {
+ if (!parser.arrayClass.getName().equals("Array")) {
+ result.callMethod(context, "<<", res.result);
+ } else {
+ result.append(res.result);
+ }
+ fexec res.p;
+ }
+ }
+
+ action exit {
+ fhold;
+ fbreak;
+ }
+
+ next_element = value_separator ignore* begin_value >parse_value;
+
+ main := begin_array
+ ignore*
+ ( ( begin_value >parse_value
+ ignore* )
+ ( ignore*
+ next_element
+ ignore* )* )?
+ ignore*
+ end_array @exit;
+ }%%
+
+ ParserResult parseArray(int p, int pe) {
+ int cs = EVIL;
+
+ if (parser.maxNesting > 0 && currentNesting > parser.maxNesting) {
+ throw newException(Utils.M_NESTING_ERROR,
+ "nesting of " + currentNesting + " is too deep");
+ }
+
+ // this is guaranteed to be a RubyArray due to the earlier
+ // allocator test at OptionsReader#getClass
+ RubyArray result =
+ (RubyArray)parser.arrayClass.newInstance(context,
+ IRubyObject.NULL_ARRAY, Block.NULL_BLOCK);
+
+ %% write init;
+ %% write exec;
+
+ if (cs >= JSON_array_first_final) {
+ return new ParserResult(result, p + 1);
+ } else {
+ throw unexpectedToken(p, pe);
+ }
+ }
+
+ %%{
+ machine JSON_object;
+ include JSON_common;
+
+ write data;
+
+ action parse_value {
+ ParserResult res = parseValue(fpc, pe);
+ if (res == null) {
+ fhold;
+ fbreak;
+ } else {
+ if (!parser.objectClass.getName().equals("Hash")) {
+ result.callMethod(context, "[]=", new IRubyObject[] { lastName, res.result });
+ } else {
+ result.op_aset(context, lastName, res.result);
+ }
+ fexec res.p;
+ }
+ }
+
+ action parse_name {
+ ParserResult res = parseString(fpc, pe);
+ if (res == null) {
+ fhold;
+ fbreak;
+ } else {
+ RubyString name = (RubyString)res.result;
+ if (parser.symbolizeNames) {
+ lastName = context.getRuntime().is1_9()
+ ? name.intern19()
+ : name.intern();
+ } else {
+ lastName = name;
+ }
+ fexec res.p;
+ }
+ }
+
+ action exit {
+ fhold;
+ fbreak;
+ }
+
+ pair = ignore* begin_name >parse_name ignore* name_separator
+ ignore* begin_value >parse_value;
+ next_pair = ignore* value_separator pair;
+
+ main := (
+ begin_object (pair (next_pair)*)? ignore* end_object
+ ) @exit;
+ }%%
+
+ ParserResult parseObject(int p, int pe) {
+ int cs = EVIL;
+ IRubyObject lastName = null;
+
+ if (parser.maxNesting > 0 && currentNesting > parser.maxNesting) {
+ throw newException(Utils.M_NESTING_ERROR,
+ "nesting of " + currentNesting + " is too deep");
+ }
+
+ // this is guaranteed to be a RubyHash due to the earlier
+ // allocator test at OptionsReader#getClass
+ RubyHash result =
+ (RubyHash)parser.objectClass.newInstance(context,
+ IRubyObject.NULL_ARRAY, Block.NULL_BLOCK);
+
+ %% write init;
+ %% write exec;
+
+ if (cs < JSON_object_first_final) {
+ return null;
+ }
+
+ IRubyObject returnedResult = result;
+
+ // attempt to de-serialize object
+ if (parser.createAdditions) {
+ IRubyObject vKlassName = result.op_aref(context, parser.createId);
+ if (!vKlassName.isNil()) {
+ // might throw ArgumentError, we let it propagate
+ IRubyObject klass = parser.info.jsonModule.get().
+ callMethod(context, "deep_const_get", vKlassName);
+ if (klass.respondsTo("json_creatable?") &&
+ klass.callMethod(context, "json_creatable?").isTrue()) {
+
+ returnedResult = klass.callMethod(context, "json_create", result);
+ }
+ }
+ }
+ return new ParserResult(returnedResult, p + 1);
+ }
+
+ %%{
+ machine JSON;
+ include JSON_common;
+
+ write data;
+
+ action parse_object {
+ currentNesting = 1;
+ ParserResult res = parseObject(fpc, pe);
+ if (res == null) {
+ fhold;
+ fbreak;
+ } else {
+ result = res.result;
+ fexec res.p;
+ }
+ }
+
+ action parse_array {
+ currentNesting = 1;
+ ParserResult res = parseArray(fpc, pe);
+ if (res == null) {
+ fhold;
+ fbreak;
+ } else {
+ result = res.result;
+ fexec res.p;
+ }
+ }
+
+ main := ignore*
+ ( begin_object >parse_object
+ | begin_array >parse_array )
+ ignore*;
+ }%%
+
+ public IRubyObject parseStrict() {
+ int cs = EVIL;
+ int p, pe;
+ IRubyObject result = null;
+
+ %% write init;
+ p = byteList.begin();
+ pe = p + byteList.length();
+ %% write exec;
+
+ if (cs >= JSON_first_final && p == pe) {
+ return result;
+ } else {
+ throw unexpectedToken(p, pe);
+ }
+ }
+
+ %%{
+ machine JSON_quirks_mode;
+ include JSON_common;
+
+ write data;
+
+ action parse_value {
+ ParserResult res = parseValue(fpc, pe);
+ if (res == null) {
+ fhold;
+ fbreak;
+ } else {
+ result = res.result;
+ fexec res.p;
+ }
+ }
+
+ main := ignore*
+ ( begin_value >parse_value)
+ ignore*;
+ }%%
+
+ public IRubyObject parseQuirksMode() {
+ int cs = EVIL;
+ int p, pe;
+ IRubyObject result = null;
+
+ %% write init;
+ p = byteList.begin();
+ pe = p + byteList.length();
+ %% write exec;
+
+ if (cs >= JSON_quirks_mode_first_final && p == pe) {
+ return result;
+ } else {
+ throw unexpectedToken(p, pe);
+ }
+ }
+
+ public IRubyObject parse() {
+ if (parser.quirksMode) {
+ return parseQuirksMode();
+ } else {
+ return parseStrict();
+ }
+
+ }
+
+ /**
+ * Returns a subsequence of the source ByteList, based on source
+ * array byte offsets (i.e., the ByteList's own begin offset is not
+ * automatically added).
+ * @param start
+ * @param end
+ */
+ private ByteList absSubSequence(int absStart, int absEnd) {
+ int offset = byteList.begin();
+ return (ByteList)byteList.subSequence(absStart - offset,
+ absEnd - offset);
+ }
+
+ /**
+ * Retrieves a constant directly descended from the <code>JSON</code> module.
+ * @param name The constant name
+ */
+ private IRubyObject getConstant(String name) {
+ return parser.info.jsonModule.get().getConstant(name);
+ }
+
+ private RaiseException newException(String className, String message) {
+ return Utils.newException(context, className, message);
+ }
+
+ private RaiseException newException(String className, RubyString message) {
+ return Utils.newException(context, className, message);
+ }
+
+ private RaiseException newException(String className,
+ String messageBegin, ByteList messageEnd) {
+ return newException(className,
+ getRuntime().newString(messageBegin).cat(messageEnd));
+ }
+ }
+}
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import java.io.IOException;
+import java.lang.ref.WeakReference;
+
+import org.jruby.Ruby;
+import org.jruby.RubyClass;
+import org.jruby.RubyModule;
+import org.jruby.runtime.load.BasicLibraryService;
+
+/**
+ * The service invoked by JRuby's {@link org.jruby.runtime.load.LoadService LoadService}.
+ * Defines the <code>JSON::Ext::Parser</code> class.
+ * @author mernen
+ */
+public class ParserService implements BasicLibraryService {
+ public boolean basicLoad(Ruby runtime) throws IOException {
+ runtime.getLoadService().require("json/common");
+ RuntimeInfo info = RuntimeInfo.initRuntime(runtime);
+
+ info.jsonModule = new WeakReference<RubyModule>(runtime.defineModule("JSON"));
+ RubyModule jsonExtModule = info.jsonModule.get().defineModuleUnder("Ext");
+ RubyClass parserClass =
+ jsonExtModule.defineClassUnder("Parser", runtime.getObject(),
+ Parser.ALLOCATOR);
+ parserClass.defineAnnotatedMethods(Parser.class);
+ return true;
+ }
+}
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import java.lang.ref.WeakReference;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.WeakHashMap;
+import org.jruby.Ruby;
+import org.jruby.RubyClass;
+import org.jruby.RubyEncoding;
+import org.jruby.RubyModule;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.runtime.builtin.IRubyObject;
+
+
+final class RuntimeInfo {
+ // since the vast majority of cases runs just one runtime,
+ // we optimize for that
+ private static WeakReference<Ruby> runtime1 = new WeakReference<Ruby>(null);
+ private static RuntimeInfo info1;
+ // store remaining runtimes here (does not include runtime1)
+ private static Map<Ruby, RuntimeInfo> runtimes;
+
+ // these fields are filled by the service loaders
+ // Use WeakReferences so that RuntimeInfo doesn't indirectly hold a hard reference to
+ // the Ruby runtime object, which would cause memory leaks in the runtimes map above.
+ /** JSON */
+ WeakReference<RubyModule> jsonModule;
+ /** JSON::Ext::Generator::GeneratorMethods::String::Extend */
+ WeakReference<RubyModule> stringExtendModule;
+ /** JSON::Ext::Generator::State */
+ WeakReference<RubyClass> generatorStateClass;
+ /** JSON::SAFE_STATE_PROTOTYPE */
+ WeakReference<GeneratorState> safeStatePrototype;
+
+ final WeakReference<RubyEncoding> utf8;
+ final WeakReference<RubyEncoding> ascii8bit;
+ // other encodings
+ private final Map<String, WeakReference<RubyEncoding>> encodings;
+
+ private RuntimeInfo(Ruby runtime) {
+ RubyClass encodingClass = runtime.getEncoding();
+ if (encodingClass == null) { // 1.8 mode
+ utf8 = ascii8bit = null;
+ encodings = null;
+ } else {
+ ThreadContext context = runtime.getCurrentContext();
+
+ utf8 = new WeakReference<RubyEncoding>((RubyEncoding)RubyEncoding.find(context,
+ encodingClass, runtime.newString("utf-8")));
+ ascii8bit = new WeakReference<RubyEncoding>((RubyEncoding)RubyEncoding.find(context,
+ encodingClass, runtime.newString("ascii-8bit")));
+ encodings = new HashMap<String, WeakReference<RubyEncoding>>();
+ }
+ }
+
+ static RuntimeInfo initRuntime(Ruby runtime) {
+ synchronized (RuntimeInfo.class) {
+ if (runtime1.get() == runtime) {
+ return info1;
+ } else if (runtime1.get() == null) {
+ runtime1 = new WeakReference<Ruby>(runtime);
+ info1 = new RuntimeInfo(runtime);
+ return info1;
+ } else {
+ if (runtimes == null) {
+ runtimes = new WeakHashMap<Ruby, RuntimeInfo>(1);
+ }
+ RuntimeInfo cache = runtimes.get(runtime);
+ if (cache == null) {
+ cache = new RuntimeInfo(runtime);
+ runtimes.put(runtime, cache);
+ }
+ return cache;
+ }
+ }
+ }
+
+ public static RuntimeInfo forRuntime(Ruby runtime) {
+ synchronized (RuntimeInfo.class) {
+ if (runtime1.get() == runtime) return info1;
+ RuntimeInfo cache = null;
+ if (runtimes != null) cache = runtimes.get(runtime);
+ assert cache != null : "Runtime given has not initialized JSON::Ext";
+ return cache;
+ }
+ }
+
+ public boolean encodingsSupported() {
+ return utf8 != null && utf8.get() != null;
+ }
+
+ public RubyEncoding getEncoding(ThreadContext context, String name) {
+ synchronized (encodings) {
+ WeakReference<RubyEncoding> encoding = encodings.get(name);
+ if (encoding == null) {
+ Ruby runtime = context.getRuntime();
+ encoding = new WeakReference<RubyEncoding>((RubyEncoding)RubyEncoding.find(context,
+ runtime.getEncoding(), runtime.newString(name)));
+ encodings.put(name, encoding);
+ }
+ return encoding.get();
+ }
+ }
+
+ public GeneratorState getSafeStatePrototype(ThreadContext context) {
+ if (safeStatePrototype == null) {
+ IRubyObject value = jsonModule.get().getConstant("SAFE_STATE_PROTOTYPE");
+ if (!(value instanceof GeneratorState)) {
+ throw context.getRuntime().newTypeError(value, generatorStateClass.get());
+ }
+ safeStatePrototype = new WeakReference<GeneratorState>((GeneratorState)value);
+ }
+ return safeStatePrototype.get();
+ }
+}
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import org.jruby.exceptions.RaiseException;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.util.ByteList;
+
+/**
+ * A decoder that reads a JSON-encoded string from the given sources and
+ * returns its decoded form on a new ByteList. Escaped Unicode characters
+ * are encoded as UTF-8.
+ */
+final class StringDecoder extends ByteListTranscoder {
+ /**
+ * Stores the offset of the high surrogate when reading a surrogate pair,
+ * or -1 when not.
+ */
+ private int surrogatePairStart = -1;
+
+ // Array used for writing multi-byte characters into the buffer at once
+ private final byte[] aux = new byte[4];
+
+ StringDecoder(ThreadContext context) {
+ super(context);
+ }
+
+ ByteList decode(ByteList src, int start, int end) {
+ ByteList out = new ByteList(end - start);
+ init(src, start, end, out);
+ while (hasNext()) {
+ handleChar(readUtf8Char());
+ }
+ quoteStop(pos);
+ return out;
+ }
+
+ private void handleChar(int c) {
+ if (c == '\\') {
+ quoteStop(charStart);
+ handleEscapeSequence();
+ } else {
+ quoteStart();
+ }
+ }
+
+ private void handleEscapeSequence() {
+ ensureMin(1);
+ switch (readUtf8Char()) {
+ case 'b':
+ append('\b');
+ break;
+ case 'f':
+ append('\f');
+ break;
+ case 'n':
+ append('\n');
+ break;
+ case 'r':
+ append('\r');
+ break;
+ case 't':
+ append('\t');
+ break;
+ case 'u':
+ ensureMin(4);
+ int cp = readHex();
+ if (Character.isHighSurrogate((char)cp)) {
+ handleLowSurrogate((char)cp);
+ } else if (Character.isLowSurrogate((char)cp)) {
+ // low surrogate with no high surrogate
+ throw invalidUtf8();
+ } else {
+ writeUtf8Char(cp);
+ }
+ break;
+ default: // '\\', '"', '/'...
+ quoteStart();
+ }
+ }
+
+ private void handleLowSurrogate(char highSurrogate) {
+ surrogatePairStart = charStart;
+ ensureMin(1);
+ int lowSurrogate = readUtf8Char();
+
+ if (lowSurrogate == '\\') {
+ ensureMin(5);
+ if (readUtf8Char() != 'u') throw invalidUtf8();
+ lowSurrogate = readHex();
+ }
+
+ if (Character.isLowSurrogate((char)lowSurrogate)) {
+ writeUtf8Char(Character.toCodePoint(highSurrogate,
+ (char)lowSurrogate));
+ surrogatePairStart = -1;
+ } else {
+ throw invalidUtf8();
+ }
+ }
+
+ private void writeUtf8Char(int codePoint) {
+ if (codePoint < 0x80) {
+ append(codePoint);
+ } else if (codePoint < 0x800) {
+ aux[0] = (byte)(0xc0 | (codePoint >>> 6));
+ aux[1] = tailByte(codePoint & 0x3f);
+ append(aux, 0, 2);
+ } else if (codePoint < 0x10000) {
+ aux[0] = (byte)(0xe0 | (codePoint >>> 12));
+ aux[1] = tailByte(codePoint >>> 6);
+ aux[2] = tailByte(codePoint);
+ append(aux, 0, 3);
+ } else {
+ aux[0] = (byte)(0xf0 | codePoint >>> 18);
+ aux[1] = tailByte(codePoint >>> 12);
+ aux[2] = tailByte(codePoint >>> 6);
+ aux[3] = tailByte(codePoint);
+ append(aux, 0, 4);
+ }
+ }
+
+ private byte tailByte(int value) {
+ return (byte)(0x80 | (value & 0x3f));
+ }
+
+ /**
+ * Reads a 4-digit unsigned hexadecimal number from the source.
+ */
+ private int readHex() {
+ int numberStart = pos;
+ int result = 0;
+ int length = 4;
+ for (int i = 0; i < length; i++) {
+ int digit = readUtf8Char();
+ int digitValue;
+ if (digit >= '0' && digit <= '9') {
+ digitValue = digit - '0';
+ } else if (digit >= 'a' && digit <= 'f') {
+ digitValue = 10 + digit - 'a';
+ } else if (digit >= 'A' && digit <= 'F') {
+ digitValue = 10 + digit - 'A';
+ } else {
+ throw new NumberFormatException("Invalid base 16 number "
+ + src.subSequence(numberStart, numberStart + length));
+ }
+ result = result * 16 + digitValue;
+ }
+ return result;
+ }
+
+ @Override
+ protected RaiseException invalidUtf8() {
+ ByteList message = new ByteList(
+ ByteList.plain("partial character in source, " +
+ "but hit end near "));
+ int start = surrogatePairStart != -1 ? surrogatePairStart : charStart;
+ message.append(src, start, srcEnd - start);
+ return Utils.newException(context, Utils.M_PARSER_ERROR,
+ context.getRuntime().newString(message));
+ }
+}
--- /dev/null
+package json.ext;
+
+import org.jruby.exceptions.RaiseException;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.util.ByteList;
+
+/**
+ * An encoder that reads from the given source and outputs its representation
+ * to another ByteList. The source string is fully checked for UTF-8 validity,
+ * and throws a GeneratorError if any problem is found.
+ */
+final class StringEncoder extends ByteListTranscoder {
+ private final boolean asciiOnly;
+
+ // Escaped characters will reuse this array, to avoid new allocations
+ // or appending them byte-by-byte
+ private final byte[] aux =
+ new byte[] {/* First unicode character */
+ '\\', 'u', 0, 0, 0, 0,
+ /* Second unicode character (for surrogate pairs) */
+ '\\', 'u', 0, 0, 0, 0,
+ /* "\X" characters */
+ '\\', 0};
+ // offsets on the array above
+ private static final int ESCAPE_UNI1_OFFSET = 0;
+ private static final int ESCAPE_UNI2_OFFSET = ESCAPE_UNI1_OFFSET + 6;
+ private static final int ESCAPE_CHAR_OFFSET = ESCAPE_UNI2_OFFSET + 6;
+ /** Array used for code point decomposition in surrogates */
+ private final char[] utf16 = new char[2];
+
+ private static final byte[] HEX =
+ new byte[] {'0', '1', '2', '3', '4', '5', '6', '7',
+ '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'};
+
+ StringEncoder(ThreadContext context, boolean asciiOnly) {
+ super(context);
+ this.asciiOnly = asciiOnly;
+ }
+
+ void encode(ByteList src, ByteList out) {
+ init(src, out);
+ append('"');
+ while (hasNext()) {
+ handleChar(readUtf8Char());
+ }
+ quoteStop(pos);
+ append('"');
+ }
+
+ private void handleChar(int c) {
+ switch (c) {
+ case '"':
+ case '\\':
+ escapeChar((char)c);
+ break;
+ case '\n':
+ escapeChar('n');
+ break;
+ case '\r':
+ escapeChar('r');
+ break;
+ case '\t':
+ escapeChar('t');
+ break;
+ case '\f':
+ escapeChar('f');
+ break;
+ case '\b':
+ escapeChar('b');
+ break;
+ default:
+ if (c >= 0x20 && c <= 0x7f ||
+ (c >= 0x80 && !asciiOnly)) {
+ quoteStart();
+ } else {
+ quoteStop(charStart);
+ escapeUtf8Char(c);
+ }
+ }
+ }
+
+ private void escapeChar(char c) {
+ quoteStop(charStart);
+ aux[ESCAPE_CHAR_OFFSET + 1] = (byte)c;
+ append(aux, ESCAPE_CHAR_OFFSET, 2);
+ }
+
+ private void escapeUtf8Char(int codePoint) {
+ int numChars = Character.toChars(codePoint, utf16, 0);
+ escapeCodeUnit(utf16[0], ESCAPE_UNI1_OFFSET + 2);
+ if (numChars > 1) escapeCodeUnit(utf16[1], ESCAPE_UNI2_OFFSET + 2);
+ append(aux, ESCAPE_UNI1_OFFSET, 6 * numChars);
+ }
+
+ private void escapeCodeUnit(char c, int auxOffset) {
+ for (int i = 0; i < 4; i++) {
+ aux[auxOffset + i] = HEX[(c >>> (12 - 4 * i)) & 0xf];
+ }
+ }
+
+ @Override
+ protected RaiseException invalidUtf8() {
+ return Utils.newException(context, Utils.M_GENERATOR_ERROR,
+ "source sequence is illegal/malformed utf-8");
+ }
+}
--- /dev/null
+/*
+ * This code is copyrighted work by Daniel Luz <dev at mernen dot com>.
+ *
+ * Distributed under the Ruby and GPLv2 licenses; see COPYING and GPL files
+ * for details.
+ */
+package json.ext;
+
+import org.jruby.Ruby;
+import org.jruby.RubyArray;
+import org.jruby.RubyClass;
+import org.jruby.RubyException;
+import org.jruby.RubyHash;
+import org.jruby.RubyString;
+import org.jruby.exceptions.RaiseException;
+import org.jruby.runtime.Block;
+import org.jruby.runtime.ThreadContext;
+import org.jruby.runtime.builtin.IRubyObject;
+import org.jruby.util.ByteList;
+
+/**
+ * Library of miscellaneous utility functions
+ */
+final class Utils {
+ public static final String M_GENERATOR_ERROR = "GeneratorError";
+ public static final String M_NESTING_ERROR = "NestingError";
+ public static final String M_PARSER_ERROR = "ParserError";
+
+ private Utils() {
+ throw new RuntimeException();
+ }
+
+ /**
+ * Safe {@link RubyArray} type-checking.
+ * Returns the given object if it is an <code>Array</code>,
+ * or throws an exception if not.
+ * @param object The object to test
+ * @return The given object if it is an <code>Array</code>
+ * @throws RaiseException <code>TypeError</code> if the object is not
+ * of the expected type
+ */
+ static RubyArray ensureArray(IRubyObject object) throws RaiseException {
+ if (object instanceof RubyArray) return (RubyArray)object;
+ Ruby runtime = object.getRuntime();
+ throw runtime.newTypeError(object, runtime.getArray());
+ }
+
+ static RubyHash ensureHash(IRubyObject object) throws RaiseException {
+ if (object instanceof RubyHash) return (RubyHash)object;
+ Ruby runtime = object.getRuntime();
+ throw runtime.newTypeError(object, runtime.getHash());
+ }
+
+ static RubyString ensureString(IRubyObject object) throws RaiseException {
+ if (object instanceof RubyString) return (RubyString)object;
+ Ruby runtime = object.getRuntime();
+ throw runtime.newTypeError(object, runtime.getString());
+ }
+
+ static RaiseException newException(ThreadContext context,
+ String className, String message) {
+ return newException(context, className,
+ context.getRuntime().newString(message));
+ }
+
+ static RaiseException newException(ThreadContext context,
+ String className, RubyString message) {
+ RuntimeInfo info = RuntimeInfo.forRuntime(context.getRuntime());
+ RubyClass klazz = info.jsonModule.get().getClass(className);
+ RubyException excptn =
+ (RubyException)klazz.newInstance(context,
+ new IRubyObject[] {message}, Block.NULL_BLOCK);
+ return new RaiseException(excptn);
+ }
+
+ static byte[] repeat(ByteList a, int n) {
+ return repeat(a.unsafeBytes(), a.begin(), a.length(), n);
+ }
+
+ static byte[] repeat(byte[] a, int begin, int length, int n) {
+ if (length == 0) return ByteList.NULL_ARRAY;
+ int resultLen = length * n;
+ byte[] result = new byte[resultLen];
+ for (int pos = 0; pos < resultLen; pos += length) {
+ System.arraycopy(a, begin, result, pos, length);
+ }
+ return result;
+ }
+}
--- /dev/null
+#!/usr/bin/env jruby
+require "rubygems"
+
+spec = Gem::Specification.new do |s|
+ s.name = "json"
+ s.version = File.read("VERSION").chomp
+ s.summary = "JSON implementation for JRuby"
+ s.description = "A JSON implementation as a JRuby extension."
+ s.author = "Daniel Luz"
+ s.email = "dev+ruby@mernen.com"
+ s.homepage = "http://json-jruby.rubyforge.org/"
+ s.platform = 'java'
+ s.rubyforge_project = "json-jruby"
+
+ s.files = Dir["{docs,lib,tests}/**/*"]
+end
+
+if $0 == __FILE__
+ Gem::Builder.new(spec).build
+else
+ spec
+end
--- /dev/null
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+ s.name = "json"
+ s.version = "1.5.5"
+
+ s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+ s.authors = ["Florian Frank"]
+ s.date = "2013-02-10"
+ s.description = "This is a JSON implementation as a Ruby extension in C."
+ s.email = "flori@ping.de"
+ s.executables = ["edit_json.rb", "prettify_json.rb"]
+ s.extensions = ["ext/json/ext/generator/extconf.rb", "ext/json/ext/parser/extconf.rb"]
+ s.extra_rdoc_files = ["README.rdoc"]
+ s.files = ["0001-Security-fix-create_additons-JSON-GenericObject.patch", "0001-Security-fix-create_additons-problem-1.5.5.patch", "0001-Security-fix-for-create_additions-problem-1.6.8.patch", "CHANGES", "COPYING", "COPYING-json-jruby", "GPL", "Gemfile", "Gemfile.lock", "README-json-jruby.markdown", "README.rdoc", "Rakefile", "TODO", "VERSION", "benchmarks", "benchmarks/data", "benchmarks/data-p4-3GHz-ruby18", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkComparison.log", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_fast-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_fast.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_pretty-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_pretty.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_safe-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_safe.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt.log", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_fast-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_fast.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_pretty-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_pretty.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_safe-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_safe.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure.log", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkRails#generator-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkRails#generator.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkRails.log", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkComparison.log", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkExt#parser-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkExt#parser.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkExt.log", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkPure#parser-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkPure#parser.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkPure.log", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkRails#parser-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkRails#parser.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkRails.log", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkYAML#parser-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkYAML#parser.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkYAML.log", "benchmarks/generator2_benchmark.rb", "benchmarks/generator_benchmark.rb", "benchmarks/ohai.json", "benchmarks/ohai.ruby", "benchmarks/parser2_benchmark.rb", "benchmarks/parser_benchmark.rb", "bin", "bin/edit_json.rb", "bin/prettify_json.rb", "data", "data/example.json", "data/index.html", "data/prototype.js", "diagrams", "ext", "ext/json", "ext/json/ext", "ext/json/ext/generator", "ext/json/ext/generator/extconf.rb", "ext/json/ext/generator/generator.c", "ext/json/ext/generator/generator.h", "ext/json/ext/parser", "ext/json/ext/parser/extconf.rb", "ext/json/ext/parser/parser.c", "ext/json/ext/parser/parser.h", "ext/json/ext/parser/parser.rl", "install.rb", "java", "java/lib", "java/lib/bytelist-1.0.6.jar", "java/lib/jcodings.jar", "java/src", "java/src/json", "java/src/json/ext", "java/src/json/ext/ByteListTranscoder.java", "java/src/json/ext/Generator.java", "java/src/json/ext/GeneratorMethods.java", "java/src/json/ext/GeneratorService.java", "java/src/json/ext/GeneratorState.java", "java/src/json/ext/OptionsReader.java", "java/src/json/ext/Parser.java", "java/src/json/ext/Parser.rl", "java/src/json/ext/ParserService.java", "java/src/json/ext/RuntimeInfo.java", "java/src/json/ext/StringDecoder.java", "java/src/json/ext/StringEncoder.java", "java/src/json/ext/Utils.java", "json-java.gemspec", "json.gemspec", "json_pure.gemspec", "lib", "lib/json", "lib/json.rb", "lib/json/Array.xpm", "lib/json/FalseClass.xpm", "lib/json/Hash.xpm", "lib/json/Key.xpm", "lib/json/NilClass.xpm", "lib/json/Numeric.xpm", "lib/json/String.xpm", "lib/json/TrueClass.xpm", "lib/json/add", "lib/json/add/complex.rb", "lib/json/add/core.rb", "lib/json/add/rational.rb", "lib/json/common.rb", "lib/json/editor.rb", "lib/json/ext", "lib/json/ext.rb", "lib/json/json.xpm", "lib/json/pure", "lib/json/pure.rb", "lib/json/pure/generator.rb", "lib/json/pure/parser.rb", "lib/json/version.rb", "tests", "tests/fixtures", "tests/fixtures/fail1.json", "tests/fixtures/fail10.json", "tests/fixtures/fail11.json", "tests/fixtures/fail12.json", "tests/fixtures/fail13.json", "tests/fixtures/fail14.json", "tests/fixtures/fail18.json", "tests/fixtures/fail19.json", "tests/fixtures/fail2.json", "tests/fixtures/fail20.json", "tests/fixtures/fail21.json", "tests/fixtures/fail22.json", "tests/fixtures/fail23.json", "tests/fixtures/fail24.json", "tests/fixtures/fail25.json", "tests/fixtures/fail27.json", "tests/fixtures/fail28.json", "tests/fixtures/fail3.json", "tests/fixtures/fail4.json", "tests/fixtures/fail5.json", "tests/fixtures/fail6.json", "tests/fixtures/fail7.json", "tests/fixtures/fail8.json", "tests/fixtures/fail9.json", "tests/fixtures/pass1.json", "tests/fixtures/pass15.json", "tests/fixtures/pass16.json", "tests/fixtures/pass17.json", "tests/fixtures/pass2.json", "tests/fixtures/pass26.json", "tests/fixtures/pass3.json", "tests/setup_variant.rb", "tests/test_json.rb", "tests/test_json_addition.rb", "tests/test_json_encoding.rb", "tests/test_json_fixtures.rb", "tests/test_json_generate.rb", "tests/test_json_string_matching.rb", "tests/test_json_unicode.rb", "tools", "tools/fuzz.rb", "tools/server.rb", "./tests/test_json_string_matching.rb", "./tests/test_json_fixtures.rb", "./tests/test_json_unicode.rb", "./tests/test_json_addition.rb", "./tests/test_json_generate.rb", "./tests/test_json_encoding.rb", "./tests/test_json.rb"]
+ s.homepage = "http://flori.github.com/json"
+ s.rdoc_options = ["--title", "JSON implemention for Ruby", "--main", "README.rdoc"]
+ s.require_paths = ["ext/json/ext", "ext", "lib"]
+ s.rubyforge_project = "json"
+ s.rubygems_version = "1.8.25"
+ s.summary = "JSON Implementation for Ruby"
+ s.test_files = ["./tests/test_json_string_matching.rb", "./tests/test_json_fixtures.rb", "./tests/test_json_unicode.rb", "./tests/test_json_addition.rb", "./tests/test_json_generate.rb", "./tests/test_json_encoding.rb", "./tests/test_json.rb"]
+
+ if s.respond_to? :specification_version then
+ s.specification_version = 3
+
+ if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+ s.add_development_dependency(%q<permutation>, [">= 0"])
+ s.add_development_dependency(%q<bullshit>, [">= 0"])
+ s.add_development_dependency(%q<sdoc>, [">= 0"])
+ else
+ s.add_dependency(%q<permutation>, [">= 0"])
+ s.add_dependency(%q<bullshit>, [">= 0"])
+ s.add_dependency(%q<sdoc>, [">= 0"])
+ end
+ else
+ s.add_dependency(%q<permutation>, [">= 0"])
+ s.add_dependency(%q<bullshit>, [">= 0"])
+ s.add_dependency(%q<sdoc>, [">= 0"])
+ end
+end
--- /dev/null
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+ s.name = "json_pure"
+ s.version = "1.5.5"
+
+ s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+ s.authors = ["Florian Frank"]
+ s.date = "2013-02-10"
+ s.description = "This is a JSON implementation in pure Ruby."
+ s.email = "flori@ping.de"
+ s.executables = ["edit_json.rb", "prettify_json.rb"]
+ s.extra_rdoc_files = ["README.rdoc"]
+ s.files = ["0001-Security-fix-create_additons-JSON-GenericObject.patch", "0001-Security-fix-create_additons-problem-1.5.5.patch", "0001-Security-fix-for-create_additions-problem-1.6.8.patch", "CHANGES", "COPYING", "COPYING-json-jruby", "GPL", "Gemfile", "Gemfile.lock", "README-json-jruby.markdown", "README.rdoc", "Rakefile", "TODO", "VERSION", "benchmarks", "benchmarks/data", "benchmarks/data-p4-3GHz-ruby18", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkComparison.log", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_fast-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_fast.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_pretty-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_pretty.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_safe-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt#generator_safe.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkExt.log", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_fast-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_fast.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_pretty-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_pretty.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_safe-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure#generator_safe.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkPure.log", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkRails#generator-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkRails#generator.dat", "benchmarks/data-p4-3GHz-ruby18/GeneratorBenchmarkRails.log", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkComparison.log", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkExt#parser-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkExt#parser.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkExt.log", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkPure#parser-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkPure#parser.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkPure.log", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkRails#parser-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkRails#parser.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkRails.log", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkYAML#parser-autocorrelation.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkYAML#parser.dat", "benchmarks/data-p4-3GHz-ruby18/ParserBenchmarkYAML.log", "benchmarks/generator2_benchmark.rb", "benchmarks/generator_benchmark.rb", "benchmarks/ohai.json", "benchmarks/ohai.ruby", "benchmarks/parser2_benchmark.rb", "benchmarks/parser_benchmark.rb", "bin", "bin/edit_json.rb", "bin/prettify_json.rb", "data", "data/example.json", "data/index.html", "data/prototype.js", "diagrams", "ext", "ext/json", "ext/json/ext", "ext/json/ext/generator", "ext/json/ext/generator/extconf.rb", "ext/json/ext/generator/generator.c", "ext/json/ext/generator/generator.h", "ext/json/ext/parser", "ext/json/ext/parser/extconf.rb", "ext/json/ext/parser/parser.c", "ext/json/ext/parser/parser.h", "ext/json/ext/parser/parser.rl", "install.rb", "java", "java/lib", "java/lib/bytelist-1.0.6.jar", "java/lib/jcodings.jar", "java/src", "java/src/json", "java/src/json/ext", "java/src/json/ext/ByteListTranscoder.java", "java/src/json/ext/Generator.java", "java/src/json/ext/GeneratorMethods.java", "java/src/json/ext/GeneratorService.java", "java/src/json/ext/GeneratorState.java", "java/src/json/ext/OptionsReader.java", "java/src/json/ext/Parser.java", "java/src/json/ext/Parser.rl", "java/src/json/ext/ParserService.java", "java/src/json/ext/RuntimeInfo.java", "java/src/json/ext/StringDecoder.java", "java/src/json/ext/StringEncoder.java", "java/src/json/ext/Utils.java", "json-java.gemspec", "json.gemspec", "json_pure.gemspec", "lib", "lib/json", "lib/json.rb", "lib/json/Array.xpm", "lib/json/FalseClass.xpm", "lib/json/Hash.xpm", "lib/json/Key.xpm", "lib/json/NilClass.xpm", "lib/json/Numeric.xpm", "lib/json/String.xpm", "lib/json/TrueClass.xpm", "lib/json/add", "lib/json/add/complex.rb", "lib/json/add/core.rb", "lib/json/add/rational.rb", "lib/json/common.rb", "lib/json/editor.rb", "lib/json/ext", "lib/json/ext.rb", "lib/json/json.xpm", "lib/json/pure", "lib/json/pure.rb", "lib/json/pure/generator.rb", "lib/json/pure/parser.rb", "lib/json/version.rb", "tests", "tests/fixtures", "tests/fixtures/fail1.json", "tests/fixtures/fail10.json", "tests/fixtures/fail11.json", "tests/fixtures/fail12.json", "tests/fixtures/fail13.json", "tests/fixtures/fail14.json", "tests/fixtures/fail18.json", "tests/fixtures/fail19.json", "tests/fixtures/fail2.json", "tests/fixtures/fail20.json", "tests/fixtures/fail21.json", "tests/fixtures/fail22.json", "tests/fixtures/fail23.json", "tests/fixtures/fail24.json", "tests/fixtures/fail25.json", "tests/fixtures/fail27.json", "tests/fixtures/fail28.json", "tests/fixtures/fail3.json", "tests/fixtures/fail4.json", "tests/fixtures/fail5.json", "tests/fixtures/fail6.json", "tests/fixtures/fail7.json", "tests/fixtures/fail8.json", "tests/fixtures/fail9.json", "tests/fixtures/pass1.json", "tests/fixtures/pass15.json", "tests/fixtures/pass16.json", "tests/fixtures/pass17.json", "tests/fixtures/pass2.json", "tests/fixtures/pass26.json", "tests/fixtures/pass3.json", "tests/setup_variant.rb", "tests/test_json.rb", "tests/test_json_addition.rb", "tests/test_json_encoding.rb", "tests/test_json_fixtures.rb", "tests/test_json_generate.rb", "tests/test_json_string_matching.rb", "tests/test_json_unicode.rb", "tools", "tools/fuzz.rb", "tools/server.rb", "./tests/test_json_string_matching.rb", "./tests/test_json_fixtures.rb", "./tests/test_json_unicode.rb", "./tests/test_json_addition.rb", "./tests/test_json_generate.rb", "./tests/test_json_encoding.rb", "./tests/test_json.rb"]
+ s.homepage = "http://flori.github.com/json"
+ s.rdoc_options = ["--title", "JSON implemention for ruby", "--main", "README.rdoc"]
+ s.require_paths = ["lib"]
+ s.rubyforge_project = "json"
+ s.rubygems_version = "1.8.25"
+ s.summary = "JSON Implementation for Ruby"
+ s.test_files = ["./tests/test_json_string_matching.rb", "./tests/test_json_fixtures.rb", "./tests/test_json_unicode.rb", "./tests/test_json_addition.rb", "./tests/test_json_generate.rb", "./tests/test_json_encoding.rb", "./tests/test_json.rb"]
+
+ if s.respond_to? :specification_version then
+ s.specification_version = 3
+
+ if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+ s.add_development_dependency(%q<permutation>, [">= 0"])
+ s.add_development_dependency(%q<bullshit>, [">= 0"])
+ s.add_development_dependency(%q<sdoc>, [">= 0"])
+ s.add_development_dependency(%q<rake>, ["~> 0.9.2"])
+ s.add_runtime_dependency(%q<spruz>, ["~> 0.2.8"])
+ else
+ s.add_dependency(%q<permutation>, [">= 0"])
+ s.add_dependency(%q<bullshit>, [">= 0"])
+ s.add_dependency(%q<sdoc>, [">= 0"])
+ s.add_dependency(%q<rake>, ["~> 0.9.2"])
+ s.add_dependency(%q<spruz>, ["~> 0.2.8"])
+ end
+ else
+ s.add_dependency(%q<permutation>, [">= 0"])
+ s.add_dependency(%q<bullshit>, [">= 0"])
+ s.add_dependency(%q<sdoc>, [">= 0"])
+ s.add_dependency(%q<rake>, ["~> 0.9.2"])
+ s.add_dependency(%q<spruz>, ["~> 0.2.8"])
+ end
+end
--- /dev/null
+##
+# = JavaScript Object Notation (JSON)
+#
+# JSON is a lightweight data-interchange format. It is easy for us
+# humans to read and write. Plus, equally simple for machines to generate or parse.
+# JSON is completely language agnostic, making it the ideal interchange format.
+#
+# Built on two universally available structures:
+# 1. A collection of name/value pairs. Often referred to as an _object_, hash table, record, struct, keyed list, or associative array.
+# 2. An ordered list of values. More commonly called an _array_, vector, sequence or list.
+#
+# To read more about JSON visit: http://json.org
+#
+# == Parsing JSON
+#
+# To parse a JSON string received by another application or generated within
+# your existing application:
+#
+# require 'json'
+#
+# my_hash = JSON.parse('{"hello": "goodbye"}')
+# puts my_hash["hello"] => "goodbye"
+#
+# Notice the extra quotes <tt>''</tt> around the hash notation. Ruby expects
+# the argument to be a string and can't convert objects like a hash or array.
+#
+# Ruby converts your string into a hash
+#
+# == Generating JSON
+#
+# Creating a JSON string for communication or serialization is
+# just as simple.
+#
+# require 'json'
+#
+# my_hash = {:hello => "goodbye"}
+# puts JSON.generate(my_hash) => "{\"hello\":\"goodbye\"}"
+#
+# Or an alternative way:
+#
+# require 'json'
+# puts {:hello => "goodbye"}.to_json => "{\"hello\":\"goodbye\"}"
+#
+# <tt>JSON.generate</tt> only allows objects or arrays to be converted
+# to JSON syntax. <tt>to_json</tt>, however, accepts many Ruby classes
+# even though it acts only as a method for serialization:
+#
+# require 'json'
+#
+# 1.to_json => "1"
+#
+
+require 'json/common'
+module JSON
+ require 'json/version'
+
+ begin
+ require 'json/ext'
+ rescue LoadError
+ require 'json/pure'
+ end
+end
--- /dev/null
+/* XPM */
+static char * Array_xpm[] = {
+"16 16 2 1",
+" c None",
+". c #000000",
+" ",
+" ",
+" ",
+" .......... ",
+" . . ",
+" . . ",
+" . . ",
+" . . ",
+" . . ",
+" . . ",
+" . . ",
+" . . ",
+" .......... ",
+" ",
+" ",
+" "};
--- /dev/null
+/* XPM */
+static char * False_xpm[] = {
+"16 16 2 1",
+" c None",
+". c #FF0000",
+" ",
+" ",
+" ",
+" ...... ",
+" . ",
+" . ",
+" . ",
+" ...... ",
+" . ",
+" . ",
+" . ",
+" . ",
+" . ",
+" ",
+" ",
+" "};
--- /dev/null
+/* XPM */
+static char * Hash_xpm[] = {
+"16 16 2 1",
+" c None",
+". c #000000",
+" ",
+" ",
+" ",
+" . . ",
+" . . ",
+" . . ",
+" ......... ",
+" . . ",
+" . . ",
+" ......... ",
+" . . ",
+" . . ",
+" . . ",
+" ",
+" ",
+" "};
--- /dev/null
+/* XPM */
+static char * Key_xpm[] = {
+"16 16 54 1",
+" c None",
+". c #110007",
+"+ c #0E0900",
+"@ c #000013",
+"# c #070600",
+"$ c #F6F006",
+"% c #ECE711",
+"& c #E5EE00",
+"* c #16021E",
+"= c #120900",
+"- c #EDF12B",
+"; c #000033",
+"> c #0F0000",
+", c #FFFE03",
+"' c #E6E500",
+") c #16021B",
+"! c #F7F502",
+"~ c #000E00",
+"{ c #130000",
+"] c #FFF000",
+"^ c #FFE711",
+"/ c #140005",
+"( c #190025",
+"_ c #E9DD27",
+": c #E7DC04",
+"< c #FFEC09",
+"[ c #FFE707",
+"} c #FFDE10",
+"| c #150021",
+"1 c #160700",
+"2 c #FAF60E",
+"3 c #EFE301",
+"4 c #FEF300",
+"5 c #E7E000",
+"6 c #FFFF08",
+"7 c #0E0206",
+"8 c #040000",
+"9 c #03052E",
+"0 c #041212",
+"a c #070300",
+"b c #F2E713",
+"c c #F9DE13",
+"d c #36091E",
+"e c #00001C",
+"f c #1F0010",
+"g c #FFF500",
+"h c #DEDE00",
+"i c #050A00",
+"j c #FAF14A",
+"k c #F5F200",
+"l c #040404",
+"m c #1A0D00",
+"n c #EDE43D",
+"o c #ECE007",
+" ",
+" ",
+" .+@ ",
+" #$%&* ",
+" =-;>,') ",
+" >!~{]^/ ",
+" (_:<[}| ",
+" 1234567 ",
+" 890abcd ",
+" efghi ",
+" >jkl ",
+" mnol ",
+" >kl ",
+" ll ",
+" ",
+" "};
--- /dev/null
+/* XPM */
+static char * False_xpm[] = {
+"16 16 2 1",
+" c None",
+". c #000000",
+" ",
+" ",
+" ",
+" ... ",
+" . . ",
+" . . ",
+" . . ",
+" . . ",
+" . . ",
+" . . ",
+" . . ",
+" . . ",
+" ... ",
+" ",
+" ",
+" "};
--- /dev/null
+/* XPM */
+static char * Numeric_xpm[] = {
+"16 16 9 1",
+" c None",
+". c #FF0000",
+"+ c #0000FF",
+"@ c #0023DB",
+"# c #00EA14",
+"$ c #00FF00",
+"% c #004FAF",
+"& c #0028D6",
+"* c #00F20C",
+" ",
+" ",
+" ",
+" ... +++@#$$$$ ",
+" .+ %& $$ ",
+" . + $ ",
+" . + $$ ",
+" . ++$$$$ ",
+" . + $$ ",
+" . + $ ",
+" . + $ ",
+" . + $ $$ ",
+" .....++++*$$ ",
+" ",
+" ",
+" "};
--- /dev/null
+/* XPM */
+static char * String_xpm[] = {
+"16 16 77 1",
+" c None",
+". c #000000",
+"+ c #040404",
+"@ c #080806",
+"# c #090606",
+"$ c #EEEAE1",
+"% c #E7E3DA",
+"& c #E0DBD1",
+"* c #D4B46F",
+"= c #0C0906",
+"- c #E3C072",
+"; c #E4C072",
+"> c #060505",
+", c #0B0A08",
+"' c #D5B264",
+") c #D3AF5A",
+"! c #080602",
+"~ c #E1B863",
+"{ c #DDB151",
+"] c #DBAE4A",
+"^ c #DDB152",
+"/ c #DDB252",
+"( c #070705",
+"_ c #0C0A07",
+": c #D3A33B",
+"< c #020201",
+"[ c #DAAA41",
+"} c #040302",
+"| c #E4D9BF",
+"1 c #0B0907",
+"2 c #030201",
+"3 c #020200",
+"4 c #C99115",
+"5 c #080704",
+"6 c #DBC8A2",
+"7 c #E7D7B4",
+"8 c #E0CD9E",
+"9 c #080601",
+"0 c #040400",
+"a c #010100",
+"b c #0B0B08",
+"c c #DCBF83",
+"d c #DCBC75",
+"e c #DEB559",
+"f c #040301",
+"g c #BC8815",
+"h c #120E07",
+"i c #060402",
+"j c #0A0804",
+"k c #D4A747",
+"l c #D6A12F",
+"m c #0E0C05",
+"n c #C8C1B0",
+"o c #1D1B15",
+"p c #D7AD51",
+"q c #070502",
+"r c #080804",
+"s c #BC953B",
+"t c #C4BDAD",
+"u c #0B0807",
+"v c #DBAC47",
+"w c #1B150A",
+"x c #B78A2C",
+"y c #D8A83C",
+"z c #D4A338",
+"A c #0F0B03",
+"B c #181105",
+"C c #C59325",
+"D c #C18E1F",
+"E c #060600",
+"F c #CC992D",
+"G c #B98B25",
+"H c #B3831F",
+"I c #C08C1C",
+"J c #060500",
+"K c #0E0C03",
+"L c #0D0A00",
+" ",
+" .+@# ",
+" .$%&*= ",
+" .-;>,')! ",
+" .~. .{]. ",
+" .^/. (_:< ",
+" .[.}|$12 ",
+" 345678}90 ",
+" a2bcdefgh ",
+" ijkl.mno ",
+" <pq. rstu ",
+" .]v. wx= ",
+" .yzABCDE ",
+" .FGHIJ ",
+" 0KL0 ",
+" "};
--- /dev/null
+/* XPM */
+static char * TrueClass_xpm[] = {
+"16 16 2 1",
+" c None",
+". c #0BF311",
+" ",
+" ",
+" ",
+" ......... ",
+" . ",
+" . ",
+" . ",
+" . ",
+" . ",
+" . ",
+" . ",
+" . ",
+" . ",
+" ",
+" ",
+" "};
--- /dev/null
+unless defined?(::JSON::JSON_LOADED) and ::JSON::JSON_LOADED
+ require 'json'
+end
+defined?(::Complex) or require 'complex'
+
+class Complex
+ def self.json_create(object)
+ Complex(object['r'], object['i'])
+ end
+
+ def as_json(*)
+ {
+ JSON.create_id => self.class.name,
+ 'r' => real,
+ 'i' => imag,
+ }
+ end
+
+ def to_json(*)
+ as_json.to_json
+ end
+end
--- /dev/null
+# This file contains implementations of ruby core's custom objects for
+# serialisation/deserialisation.
+
+unless defined?(::JSON::JSON_LOADED) and ::JSON::JSON_LOADED
+ require 'json'
+end
+require 'date'
+
+# Symbol serialization/deserialization
+class Symbol
+ # Returns a hash, that will be turned into a JSON object and represent this
+ # object.
+ def as_json(*)
+ {
+ JSON.create_id => self.class.name,
+ 's' => to_s,
+ }
+ end
+
+ # Stores class name (Symbol) with String representation of Symbol as a JSON string.
+ def to_json(*a)
+ as_json.to_json(*a)
+ end
+
+ # Deserializes JSON string by converting the <tt>string</tt> value stored in the object to a Symbol
+ def self.json_create(o)
+ o['s'].to_sym
+ end
+end
+
+# Time serialization/deserialization
+class Time
+
+ # Deserializes JSON string by converting time since epoch to Time
+ def self.json_create(object)
+ if usec = object.delete('u') # used to be tv_usec -> tv_nsec
+ object['n'] = usec * 1000
+ end
+ if instance_methods.include?(:tv_nsec)
+ at(object['s'], Rational(object['n'], 1000))
+ else
+ at(object['s'], object['n'] / 1000)
+ end
+ end
+
+ # Returns a hash, that will be turned into a JSON object and represent this
+ # object.
+ def as_json(*)
+ nanoseconds = [ tv_usec * 1000 ]
+ respond_to?(:tv_nsec) and nanoseconds << tv_nsec
+ nanoseconds = nanoseconds.max
+ {
+ JSON.create_id => self.class.name,
+ 's' => tv_sec,
+ 'n' => nanoseconds,
+ }
+ end
+
+ # Stores class name (Time) with number of seconds since epoch and number of
+ # microseconds for Time as JSON string
+ def to_json(*args)
+ as_json.to_json(*args)
+ end
+end
+
+# Date serialization/deserialization
+class Date
+
+ # Deserializes JSON string by converting Julian year <tt>y</tt>, month
+ # <tt>m</tt>, day <tt>d</tt> and Day of Calendar Reform <tt>sg</tt> to Date.
+ def self.json_create(object)
+ civil(*object.values_at('y', 'm', 'd', 'sg'))
+ end
+
+ alias start sg unless method_defined?(:start)
+
+ # Returns a hash, that will be turned into a JSON object and represent this
+ # object.
+ def as_json(*)
+ {
+ JSON.create_id => self.class.name,
+ 'y' => year,
+ 'm' => month,
+ 'd' => day,
+ 'sg' => start,
+ }
+ end
+
+ # Stores class name (Date) with Julian year <tt>y</tt>, month <tt>m</tt>, day
+ # <tt>d</tt> and Day of Calendar Reform <tt>sg</tt> as JSON string
+ def to_json(*args)
+ as_json.to_json(*args)
+ end
+end
+
+# DateTime serialization/deserialization
+class DateTime
+
+ # Deserializes JSON string by converting year <tt>y</tt>, month <tt>m</tt>,
+ # day <tt>d</tt>, hour <tt>H</tt>, minute <tt>M</tt>, second <tt>S</tt>,
+ # offset <tt>of</tt> and Day of Calendar Reform <tt>sg</tt> to DateTime.
+ def self.json_create(object)
+ args = object.values_at('y', 'm', 'd', 'H', 'M', 'S')
+ of_a, of_b = object['of'].split('/')
+ if of_b and of_b != '0'
+ args << Rational(of_a.to_i, of_b.to_i)
+ else
+ args << of_a
+ end
+ args << object['sg']
+ civil(*args)
+ end
+
+ alias start sg unless method_defined?(:start)
+
+ # Returns a hash, that will be turned into a JSON object and represent this
+ # object.
+ def as_json(*)
+ {
+ JSON.create_id => self.class.name,
+ 'y' => year,
+ 'm' => month,
+ 'd' => day,
+ 'H' => hour,
+ 'M' => min,
+ 'S' => sec,
+ 'of' => offset.to_s,
+ 'sg' => start,
+ }
+ end
+
+ # Stores class name (DateTime) with Julian year <tt>y</tt>, month <tt>m</tt>,
+ # day <tt>d</tt>, hour <tt>H</tt>, minute <tt>M</tt>, second <tt>S</tt>,
+ # offset <tt>of</tt> and Day of Calendar Reform <tt>sg</tt> as JSON string
+ def to_json(*args)
+ as_json.to_json(*args)
+ end
+end
+
+# Range serialization/deserialization
+class Range
+
+ # Deserializes JSON string by constructing new Range object with arguments
+ # <tt>a</tt> serialized by <tt>to_json</tt>.
+ def self.json_create(object)
+ new(*object['a'])
+ end
+
+ # Returns a hash, that will be turned into a JSON object and represent this
+ # object.
+ def as_json(*)
+ {
+ JSON.create_id => self.class.name,
+ 'a' => [ first, last, exclude_end? ]
+ }
+ end
+
+ # Stores class name (Range) with JSON array of arguments <tt>a</tt> which
+ # include <tt>first</tt> (integer), <tt>last</tt> (integer), and
+ # <tt>exclude_end?</tt> (boolean) as JSON string.
+ def to_json(*args)
+ as_json.to_json(*args)
+ end
+end
+
+# Struct serialization/deserialization
+class Struct
+
+ # Deserializes JSON string by constructing new Struct object with values
+ # <tt>v</tt> serialized by <tt>to_json</tt>.
+ def self.json_create(object)
+ new(*object['v'])
+ end
+
+ # Returns a hash, that will be turned into a JSON object and represent this
+ # object.
+ def as_json(*)
+ klass = self.class.name
+ klass.to_s.empty? and raise JSON::JSONError, "Only named structs are supported!"
+ {
+ JSON.create_id => klass,
+ 'v' => values,
+ }
+ end
+
+ # Stores class name (Struct) with Struct values <tt>v</tt> as a JSON string.
+ # Only named structs are supported.
+ def to_json(*args)
+ as_json.to_json(*args)
+ end
+end
+
+# Exception serialization/deserialization
+class Exception
+
+ # Deserializes JSON string by constructing new Exception object with message
+ # <tt>m</tt> and backtrace <tt>b</tt> serialized with <tt>to_json</tt>
+ def self.json_create(object)
+ result = new(object['m'])
+ result.set_backtrace object['b']
+ result
+ end
+
+ # Returns a hash, that will be turned into a JSON object and represent this
+ # object.
+ def as_json(*)
+ {
+ JSON.create_id => self.class.name,
+ 'm' => message,
+ 'b' => backtrace,
+ }
+ end
+
+ # Stores class name (Exception) with message <tt>m</tt> and backtrace array
+ # <tt>b</tt> as JSON string
+ def to_json(*args)
+ as_json.to_json(*args)
+ end
+end
+
+# Regexp serialization/deserialization
+class Regexp
+
+ # Deserializes JSON string by constructing new Regexp object with source
+ # <tt>s</tt> (Regexp or String) and options <tt>o</tt> serialized by
+ # <tt>to_json</tt>
+ def self.json_create(object)
+ new(object['s'], object['o'])
+ end
+
+ # Returns a hash, that will be turned into a JSON object and represent this
+ # object.
+ def as_json(*)
+ {
+ JSON.create_id => self.class.name,
+ 'o' => options,
+ 's' => source,
+ }
+ end
+
+ # Stores class name (Regexp) with options <tt>o</tt> and source <tt>s</tt>
+ # (Regexp or String) as JSON string
+ def to_json(*)
+ as_json.to_json
+ end
+end
--- /dev/null
+unless defined?(::JSON::JSON_LOADED) and ::JSON::JSON_LOADED
+ require 'json'
+end
+defined?(::Rational) or require 'rational'
+
+class Rational
+ def self.json_create(object)
+ Rational(object['n'], object['d'])
+ end
+
+ def as_json(*)
+ {
+ JSON.create_id => self.class.name,
+ 'n' => numerator,
+ 'd' => denominator,
+ }
+ end
+
+ def to_json(*)
+ as_json.to_json
+ end
+end
--- /dev/null
+require 'json/version'
+
+module JSON
+ class << self
+ # If _object_ is string-like, parse the string and return the parsed result
+ # as a Ruby data structure. Otherwise generate a JSON text from the Ruby
+ # data structure object and return it.
+ #
+ # The _opts_ argument is passed through to generate/parse respectively. See
+ # generate and parse for their documentation.
+ def [](object, opts = {})
+ if object.respond_to? :to_str
+ JSON.parse(object.to_str, opts)
+ else
+ JSON.generate(object, opts)
+ end
+ end
+
+ # Returns the JSON parser class that is used by JSON. This is either
+ # JSON::Ext::Parser or JSON::Pure::Parser.
+ attr_reader :parser
+
+ # Set the JSON parser class _parser_ to be used by JSON.
+ def parser=(parser) # :nodoc:
+ @parser = parser
+ remove_const :Parser if JSON.const_defined_in?(self, :Parser)
+ const_set :Parser, parser
+ end
+
+ # Return the constant located at _path_. The format of _path_ has to be
+ # either ::A::B::C or A::B::C. In any case, A has to be located at the top
+ # level (absolute namespace path?). If there doesn't exist a constant at
+ # the given path, an ArgumentError is raised.
+ def deep_const_get(path) # :nodoc:
+ path.to_s.split(/::/).inject(Object) do |p, c|
+ case
+ when c.empty? then p
+ when JSON.const_defined_in?(p, c) then p.const_get(c)
+ else
+ begin
+ p.const_missing(c)
+ rescue NameError => e
+ raise ArgumentError, "can't get const #{path}: #{e}"
+ end
+ end
+ end
+ end
+
+ # Set the module _generator_ to be used by JSON.
+ def generator=(generator) # :nodoc:
+ old, $VERBOSE = $VERBOSE, nil
+ @generator = generator
+ generator_methods = generator::GeneratorMethods
+ for const in generator_methods.constants
+ klass = deep_const_get(const)
+ modul = generator_methods.const_get(const)
+ klass.class_eval do
+ instance_methods(false).each do |m|
+ m.to_s == 'to_json' and remove_method m
+ end
+ include modul
+ end
+ end
+ self.state = generator::State
+ const_set :State, self.state
+ const_set :SAFE_STATE_PROTOTYPE, State.new
+ const_set :FAST_STATE_PROTOTYPE, State.new(
+ :indent => '',
+ :space => '',
+ :object_nl => "",
+ :array_nl => "",
+ :max_nesting => false
+ )
+ const_set :PRETTY_STATE_PROTOTYPE, State.new(
+ :indent => ' ',
+ :space => ' ',
+ :object_nl => "\n",
+ :array_nl => "\n"
+ )
+ ensure
+ $VERBOSE = old
+ end
+
+ # Returns the JSON generator module that is used by JSON. This is
+ # either JSON::Ext::Generator or JSON::Pure::Generator.
+ attr_reader :generator
+
+ # Returns the JSON generator state class that is used by JSON. This is
+ # either JSON::Ext::Generator::State or JSON::Pure::Generator::State.
+ attr_accessor :state
+
+ # This is create identifier, which is used to decide if the _json_create_
+ # hook of a class should be called. It defaults to 'json_class'.
+ attr_accessor :create_id
+ end
+ self.create_id = 'json_class'
+
+ NaN = 0.0/0
+
+ Infinity = 1.0/0
+
+ MinusInfinity = -Infinity
+
+ # The base exception for JSON errors.
+ class JSONError < StandardError; end
+
+ # This exception is raised if a parser error occurs.
+ class ParserError < JSONError; end
+
+ # This exception is raised if the nesting of parsed data structures is too
+ # deep.
+ class NestingError < ParserError; end
+
+ # :stopdoc:
+ class CircularDatastructure < NestingError; end
+ # :startdoc:
+
+ # This exception is raised if a generator or unparser error occurs.
+ class GeneratorError < JSONError; end
+ # For backwards compatibility
+ UnparserError = GeneratorError
+
+ # This exception is raised if the required unicode support is missing on the
+ # system. Usually this means that the iconv library is not installed.
+ class MissingUnicodeSupport < JSONError; end
+
+ module_function
+
+ # Parse the JSON document _source_ into a Ruby data structure and return it.
+ #
+ # _opts_ can have the following
+ # keys:
+ # * *max_nesting*: The maximum depth of nesting allowed in the parsed data
+ # structures. Disable depth checking with :max_nesting => false. It defaults
+ # to 19.
+ # * *allow_nan*: If set to true, allow NaN, Infinity and -Infinity in
+ # defiance of RFC 4627 to be parsed by the Parser. This option defaults
+ # to false.
+ # * *symbolize_names*: If set to true, returns symbols for the names
+ # (keys) in a JSON object. Otherwise strings are returned. Strings are
+ # the default.
+ # * *create_additions*: If set to false, the Parser doesn't create
+ # additions even if a matching class and create_id was found. This option
+ # defaults to false.
+ # * *object_class*: Defaults to Hash
+ # * *array_class*: Defaults to Array
+ def parse(source, opts = {})
+ Parser.new(source, opts).parse
+ end
+
+ # Parse the JSON document _source_ into a Ruby data structure and return it.
+ # The bang version of the parse method defaults to the more dangerous values
+ # for the _opts_ hash, so be sure only to parse trusted _source_ documents.
+ #
+ # _opts_ can have the following keys:
+ # * *max_nesting*: The maximum depth of nesting allowed in the parsed data
+ # structures. Enable depth checking with :max_nesting => anInteger. The parse!
+ # methods defaults to not doing max depth checking: This can be dangerous
+ # if someone wants to fill up your stack.
+ # * *allow_nan*: If set to true, allow NaN, Infinity, and -Infinity in
+ # defiance of RFC 4627 to be parsed by the Parser. This option defaults
+ # to true.
+ # * *create_additions*: If set to false, the Parser doesn't create
+ # additions even if a matching class and create_id was found. This option
+ # defaults to false.
+ def parse!(source, opts = {})
+ opts = {
+ :max_nesting => false,
+ :allow_nan => true
+ }.update(opts)
+ Parser.new(source, opts).parse
+ end
+
+ # Generate a JSON document from the Ruby data structure _obj_ and return
+ # it. _state_ is * a JSON::State object,
+ # * or a Hash like object (responding to to_hash),
+ # * an object convertible into a hash by a to_h method,
+ # that is used as or to configure a State object.
+ #
+ # It defaults to a state object, that creates the shortest possible JSON text
+ # in one line, checks for circular data structures and doesn't allow NaN,
+ # Infinity, and -Infinity.
+ #
+ # A _state_ hash can have the following keys:
+ # * *indent*: a string used to indent levels (default: ''),
+ # * *space*: a string that is put after, a : or , delimiter (default: ''),
+ # * *space_before*: a string that is put before a : pair delimiter (default: ''),
+ # * *object_nl*: a string that is put at the end of a JSON object (default: ''),
+ # * *array_nl*: a string that is put at the end of a JSON array (default: ''),
+ # * *allow_nan*: true if NaN, Infinity, and -Infinity should be
+ # generated, otherwise an exception is thrown if these values are
+ # encountered. This options defaults to false.
+ # * *max_nesting*: The maximum depth of nesting allowed in the data
+ # structures from which JSON is to be generated. Disable depth checking
+ # with :max_nesting => false, it defaults to 19.
+ #
+ # See also the fast_generate for the fastest creation method with the least
+ # amount of sanity checks, and the pretty_generate method for some
+ # defaults for pretty output.
+ def generate(obj, opts = nil)
+ if State === opts
+ state, opts = opts, nil
+ else
+ state = SAFE_STATE_PROTOTYPE.dup
+ end
+ if opts
+ if opts.respond_to? :to_hash
+ opts = opts.to_hash
+ elsif opts.respond_to? :to_h
+ opts = opts.to_h
+ else
+ raise TypeError, "can't convert #{opts.class} into Hash"
+ end
+ state = state.configure(opts)
+ end
+ state.generate(obj)
+ end
+
+ # :stopdoc:
+ # I want to deprecate these later, so I'll first be silent about them, and
+ # later delete them.
+ alias unparse generate
+ module_function :unparse
+ # :startdoc:
+
+ # Generate a JSON document from the Ruby data structure _obj_ and return it.
+ # This method disables the checks for circles in Ruby objects.
+ #
+ # *WARNING*: Be careful not to pass any Ruby data structures with circles as
+ # _obj_ argument because this will cause JSON to go into an infinite loop.
+ def fast_generate(obj, opts = nil)
+ if State === opts
+ state, opts = opts, nil
+ else
+ state = FAST_STATE_PROTOTYPE.dup
+ end
+ if opts
+ if opts.respond_to? :to_hash
+ opts = opts.to_hash
+ elsif opts.respond_to? :to_h
+ opts = opts.to_h
+ else
+ raise TypeError, "can't convert #{opts.class} into Hash"
+ end
+ state.configure(opts)
+ end
+ state.generate(obj)
+ end
+
+ # :stopdoc:
+ # I want to deprecate these later, so I'll first be silent about them, and later delete them.
+ alias fast_unparse fast_generate
+ module_function :fast_unparse
+ # :startdoc:
+
+ # Generate a JSON document from the Ruby data structure _obj_ and return it.
+ # The returned document is a prettier form of the document returned by
+ # #unparse.
+ #
+ # The _opts_ argument can be used to configure the generator. See the
+ # generate method for a more detailed explanation.
+ def pretty_generate(obj, opts = nil)
+ if State === opts
+ state, opts = opts, nil
+ else
+ state = PRETTY_STATE_PROTOTYPE.dup
+ end
+ if opts
+ if opts.respond_to? :to_hash
+ opts = opts.to_hash
+ elsif opts.respond_to? :to_h
+ opts = opts.to_h
+ else
+ raise TypeError, "can't convert #{opts.class} into Hash"
+ end
+ state.configure(opts)
+ end
+ state.generate(obj)
+ end
+
+ # :stopdoc:
+ # I want to deprecate these later, so I'll first be silent about them, and later delete them.
+ alias pretty_unparse pretty_generate
+ module_function :pretty_unparse
+ # :startdoc:
+
+ # Load a ruby data structure from a JSON _source_ and return it. A source can
+ # either be a string-like object, an IO-like object, or an object responding
+ # to the read method. If _proc_ was given, it will be called with any nested
+ # Ruby object as an argument recursively in depth first order. To modify the
+ # default options pass in the optional _options_ argument as well.
+ #
+ # This method is part of the implementation of the load/dump interface of
+ # Marshal and YAML.
+ def load(source, proc = nil, options = {})
+ load_default_options = {
+ :max_nesting => false,
+ :allow_nan => true,
+ :create_additions => false
+ }
+ opts = load_default_options.merge options
+ if source.respond_to? :to_str
+ source = source.to_str
+ elsif source.respond_to? :to_io
+ source = source.to_io.read
+ else
+ source = source.read
+ end
+ result = parse(source, opts)
+ recurse_proc(result, &proc) if proc
+ result
+ end
+
+ # Recursively calls passed _Proc_ if the parsed data structure is an _Array_ or _Hash_
+ def recurse_proc(result, &proc)
+ case result
+ when Array
+ result.each { |x| recurse_proc x, &proc }
+ proc.call result
+ when Hash
+ result.each { |x, y| recurse_proc x, &proc; recurse_proc y, &proc }
+ proc.call result
+ else
+ proc.call result
+ end
+ end
+
+ alias restore load
+ module_function :restore
+
+ # Dumps _obj_ as a JSON string, i.e. calls generate on the object and returns
+ # the result.
+ #
+ # If anIO (an IO-like object or an object that responds to the write method)
+ # was given, the resulting JSON is written to it.
+ #
+ # If the number of nested arrays or objects exceeds _limit_, an ArgumentError
+ # exception is raised. This argument is similar (but not exactly the
+ # same!) to the _limit_ argument in Marshal.dump.
+ #
+ # This method is part of the implementation of the load/dump interface of
+ # Marshal and YAML.
+ def dump(obj, anIO = nil, limit = nil)
+ if anIO and limit.nil?
+ anIO = anIO.to_io if anIO.respond_to?(:to_io)
+ unless anIO.respond_to?(:write)
+ limit = anIO
+ anIO = nil
+ end
+ end
+ limit ||= 0
+ result = generate(obj, :allow_nan => true, :max_nesting => limit)
+ if anIO
+ anIO.write result
+ anIO
+ else
+ result
+ end
+ rescue JSON::NestingError
+ raise ArgumentError, "exceed depth limit"
+ end
+
+ # Swap consecutive bytes of _string_ in place.
+ def self.swap!(string) # :nodoc:
+ 0.upto(string.size / 2) do |i|
+ break unless string[2 * i + 1]
+ string[2 * i], string[2 * i + 1] = string[2 * i + 1], string[2 * i]
+ end
+ string
+ end
+
+ # Shortuct for iconv.
+ if ::String.method_defined?(:encode)
+ # Encodes string using Ruby's _String.encode_
+ def self.iconv(to, from, string)
+ string.encode(to, from)
+ end
+ else
+ require 'iconv'
+ # Encodes string using _iconv_ library
+ def self.iconv(to, from, string)
+ Iconv.conv(to, from, string)
+ end
+ end
+
+ if ::Object.method(:const_defined?).arity == 1
+ def self.const_defined_in?(modul, constant)
+ modul.const_defined?(constant)
+ end
+ else
+ def self.const_defined_in?(modul, constant)
+ modul.const_defined?(constant, false)
+ end
+ end
+end
+
+module ::Kernel
+ private
+
+ # Outputs _objs_ to STDOUT as JSON strings in the shortest form, that is in
+ # one line.
+ def j(*objs)
+ objs.each do |obj|
+ puts JSON::generate(obj, :allow_nan => true, :max_nesting => false)
+ end
+ nil
+ end
+
+ # Ouputs _objs_ to STDOUT as JSON strings in a pretty format, with
+ # indentation and over many lines.
+ def jj(*objs)
+ objs.each do |obj|
+ puts JSON::pretty_generate(obj, :allow_nan => true, :max_nesting => false)
+ end
+ nil
+ end
+
+ # If _object_ is string-like, parse the string and return the parsed result as
+ # a Ruby data structure. Otherwise, generate a JSON text from the Ruby data
+ # structure object and return it.
+ #
+ # The _opts_ argument is passed through to generate/parse respectively. See
+ # generate and parse for their documentation.
+ def JSON(object, *args)
+ if object.respond_to? :to_str
+ JSON.parse(object.to_str, args.first)
+ else
+ JSON.generate(object, args.first)
+ end
+ end
+end
+
+# Extends any Class to include _json_creatable?_ method.
+class ::Class
+ # Returns true if this class can be used to create an instance
+ # from a serialised JSON string. The class has to implement a class
+ # method _json_create_ that expects a hash as first parameter. The hash
+ # should include the required data.
+ def json_creatable?
+ respond_to?(:json_create)
+ end
+end
--- /dev/null
+# To use the GUI JSON editor, start the edit_json.rb executable script. It
+# requires ruby-gtk to be installed.
+
+require 'gtk2'
+require 'json'
+require 'rbconfig'
+require 'open-uri'
+
+module JSON
+ module Editor
+ include Gtk
+
+ # Beginning of the editor window title
+ TITLE = 'JSON Editor'.freeze
+
+ # Columns constants
+ ICON_COL, TYPE_COL, CONTENT_COL = 0, 1, 2
+
+ # JSON primitive types (Containers)
+ CONTAINER_TYPES = %w[Array Hash].sort
+ # All JSON primitive types
+ ALL_TYPES = (%w[TrueClass FalseClass Numeric String NilClass] +
+ CONTAINER_TYPES).sort
+
+ # The Nodes necessary for the tree representation of a JSON document
+ ALL_NODES = (ALL_TYPES + %w[Key]).sort
+
+ DEFAULT_DIALOG_KEY_PRESS_HANDLER = lambda do |dialog, event|
+ case event.keyval
+ when Gdk::Keyval::GDK_Return
+ dialog.response Dialog::RESPONSE_ACCEPT
+ when Gdk::Keyval::GDK_Escape
+ dialog.response Dialog::RESPONSE_REJECT
+ end
+ end
+
+ # Returns the Gdk::Pixbuf of the icon named _name_ from the icon cache.
+ def Editor.fetch_icon(name)
+ @icon_cache ||= {}
+ unless @icon_cache.key?(name)
+ path = File.dirname(__FILE__)
+ @icon_cache[name] = Gdk::Pixbuf.new(File.join(path, name + '.xpm'))
+ end
+ @icon_cache[name]
+ end
+
+ # Opens an error dialog on top of _window_ showing the error message
+ # _text_.
+ def Editor.error_dialog(window, text)
+ dialog = MessageDialog.new(window, Dialog::MODAL,
+ MessageDialog::ERROR,
+ MessageDialog::BUTTONS_CLOSE, text)
+ dialog.show_all
+ dialog.run
+ rescue TypeError
+ dialog = MessageDialog.new(Editor.window, Dialog::MODAL,
+ MessageDialog::ERROR,
+ MessageDialog::BUTTONS_CLOSE, text)
+ dialog.show_all
+ dialog.run
+ ensure
+ dialog.destroy if dialog
+ end
+
+ # Opens a yes/no question dialog on top of _window_ showing the error
+ # message _text_. If yes was answered _true_ is returned, otherwise
+ # _false_.
+ def Editor.question_dialog(window, text)
+ dialog = MessageDialog.new(window, Dialog::MODAL,
+ MessageDialog::QUESTION,
+ MessageDialog::BUTTONS_YES_NO, text)
+ dialog.show_all
+ dialog.run do |response|
+ return Gtk::Dialog::RESPONSE_YES === response
+ end
+ ensure
+ dialog.destroy if dialog
+ end
+
+ # Convert the tree model starting from Gtk::TreeIter _iter_ into a Ruby
+ # data structure and return it.
+ def Editor.model2data(iter)
+ return nil if iter.nil?
+ case iter.type
+ when 'Hash'
+ hash = {}
+ iter.each { |c| hash[c.content] = Editor.model2data(c.first_child) }
+ hash
+ when 'Array'
+ array = Array.new(iter.n_children)
+ iter.each_with_index { |c, i| array[i] = Editor.model2data(c) }
+ array
+ when 'Key'
+ iter.content
+ when 'String'
+ iter.content
+ when 'Numeric'
+ content = iter.content
+ if /\./.match(content)
+ content.to_f
+ else
+ content.to_i
+ end
+ when 'TrueClass'
+ true
+ when 'FalseClass'
+ false
+ when 'NilClass'
+ nil
+ else
+ fail "Unknown type found in model: #{iter.type}"
+ end
+ end
+
+ # Convert the Ruby data structure _data_ into tree model data for Gtk and
+ # returns the whole model. If the parameter _model_ wasn't given a new
+ # Gtk::TreeStore is created as the model. The _parent_ parameter specifies
+ # the parent node (iter, Gtk:TreeIter instance) to which the data is
+ # appended, alternativeley the result of the yielded block is used as iter.
+ def Editor.data2model(data, model = nil, parent = nil)
+ model ||= TreeStore.new(Gdk::Pixbuf, String, String)
+ iter = if block_given?
+ yield model
+ else
+ model.append(parent)
+ end
+ case data
+ when Hash
+ iter.type = 'Hash'
+ data.sort.each do |key, value|
+ pair_iter = model.append(iter)
+ pair_iter.type = 'Key'
+ pair_iter.content = key.to_s
+ Editor.data2model(value, model, pair_iter)
+ end
+ when Array
+ iter.type = 'Array'
+ data.each do |value|
+ Editor.data2model(value, model, iter)
+ end
+ when Numeric
+ iter.type = 'Numeric'
+ iter.content = data.to_s
+ when String, true, false, nil
+ iter.type = data.class.name
+ iter.content = data.nil? ? 'null' : data.to_s
+ else
+ iter.type = 'String'
+ iter.content = data.to_s
+ end
+ model
+ end
+
+ # The Gtk::TreeIter class is reopened and some auxiliary methods are added.
+ class Gtk::TreeIter
+ include Enumerable
+
+ # Traverse each of this Gtk::TreeIter instance's children
+ # and yield to them.
+ def each
+ n_children.times { |i| yield nth_child(i) }
+ end
+
+ # Recursively traverse all nodes of this Gtk::TreeIter's subtree
+ # (including self) and yield to them.
+ def recursive_each(&block)
+ yield self
+ each do |i|
+ i.recursive_each(&block)
+ end
+ end
+
+ # Remove the subtree of this Gtk::TreeIter instance from the
+ # model _model_.
+ def remove_subtree(model)
+ while current = first_child
+ model.remove(current)
+ end
+ end
+
+ # Returns the type of this node.
+ def type
+ self[TYPE_COL]
+ end
+
+ # Sets the type of this node to _value_. This implies setting
+ # the respective icon accordingly.
+ def type=(value)
+ self[TYPE_COL] = value
+ self[ICON_COL] = Editor.fetch_icon(value)
+ end
+
+ # Returns the content of this node.
+ def content
+ self[CONTENT_COL]
+ end
+
+ # Sets the content of this node to _value_.
+ def content=(value)
+ self[CONTENT_COL] = value
+ end
+ end
+
+ # This module bundles some method, that can be used to create a menu. It
+ # should be included into the class in question.
+ module MenuExtension
+ include Gtk
+
+ # Creates a Menu, that includes MenuExtension. _treeview_ is the
+ # Gtk::TreeView, on which it operates.
+ def initialize(treeview)
+ @treeview = treeview
+ @menu = Menu.new
+ end
+
+ # Returns the Gtk::TreeView of this menu.
+ attr_reader :treeview
+
+ # Returns the menu.
+ attr_reader :menu
+
+ # Adds a Gtk::SeparatorMenuItem to this instance's #menu.
+ def add_separator
+ menu.append SeparatorMenuItem.new
+ end
+
+ # Adds a Gtk::MenuItem to this instance's #menu. _label_ is the label
+ # string, _klass_ is the item type, and _callback_ is the procedure, that
+ # is called if the _item_ is activated.
+ def add_item(label, keyval = nil, klass = MenuItem, &callback)
+ label = "#{label} (C-#{keyval.chr})" if keyval
+ item = klass.new(label)
+ item.signal_connect(:activate, &callback)
+ if keyval
+ self.signal_connect(:'key-press-event') do |item, event|
+ if event.state & Gdk::Window::ModifierType::CONTROL_MASK != 0 and
+ event.keyval == keyval
+ callback.call item
+ end
+ end
+ end
+ menu.append item
+ item
+ end
+
+ # This method should be implemented in subclasses to create the #menu of
+ # this instance. It has to be called after an instance of this class is
+ # created, to build the menu.
+ def create
+ raise NotImplementedError
+ end
+
+ def method_missing(*a, &b)
+ treeview.__send__(*a, &b)
+ end
+ end
+
+ # This class creates the popup menu, that opens when clicking onto the
+ # treeview.
+ class PopUpMenu
+ include MenuExtension
+
+ # Change the type or content of the selected node.
+ def change_node(item)
+ if current = selection.selected
+ parent = current.parent
+ old_type, old_content = current.type, current.content
+ if ALL_TYPES.include?(old_type)
+ @clipboard_data = Editor.model2data(current)
+ type, content = ask_for_element(parent, current.type,
+ current.content)
+ if type
+ current.type, current.content = type, content
+ current.remove_subtree(model)
+ toplevel.display_status("Changed a node in tree.")
+ window.change
+ end
+ else
+ toplevel.display_status(
+ "Cannot change node of type #{old_type} in tree!")
+ end
+ end
+ end
+
+ # Cut the selected node and its subtree, and save it into the
+ # clipboard.
+ def cut_node(item)
+ if current = selection.selected
+ if current and current.type == 'Key'
+ @clipboard_data = {
+ current.content => Editor.model2data(current.first_child)
+ }
+ else
+ @clipboard_data = Editor.model2data(current)
+ end
+ model.remove(current)
+ window.change
+ toplevel.display_status("Cut a node from tree.")
+ end
+ end
+
+ # Copy the selected node and its subtree, and save it into the
+ # clipboard.
+ def copy_node(item)
+ if current = selection.selected
+ if current and current.type == 'Key'
+ @clipboard_data = {
+ current.content => Editor.model2data(current.first_child)
+ }
+ else
+ @clipboard_data = Editor.model2data(current)
+ end
+ window.change
+ toplevel.display_status("Copied a node from tree.")
+ end
+ end
+
+ # Paste the data in the clipboard into the selected Array or Hash by
+ # appending it.
+ def paste_node_appending(item)
+ if current = selection.selected
+ if @clipboard_data
+ case current.type
+ when 'Array'
+ Editor.data2model(@clipboard_data, model, current)
+ expand_collapse(current)
+ when 'Hash'
+ if @clipboard_data.is_a? Hash
+ parent = current.parent
+ hash = Editor.model2data(current)
+ model.remove(current)
+ hash.update(@clipboard_data)
+ Editor.data2model(hash, model, parent)
+ if parent
+ expand_collapse(parent)
+ elsif @expanded
+ expand_all
+ end
+ window.change
+ else
+ toplevel.display_status(
+ "Cannot paste non-#{current.type} data into '#{current.type}'!")
+ end
+ else
+ toplevel.display_status(
+ "Cannot paste node below '#{current.type}'!")
+ end
+ else
+ toplevel.display_status("Nothing to paste in clipboard!")
+ end
+ else
+ toplevel.display_status("Append a node into the root first!")
+ end
+ end
+
+ # Paste the data in the clipboard into the selected Array inserting it
+ # before the selected element.
+ def paste_node_inserting_before(item)
+ if current = selection.selected
+ if @clipboard_data
+ parent = current.parent or return
+ parent_type = parent.type
+ if parent_type == 'Array'
+ selected_index = parent.each_with_index do |c, i|
+ break i if c == current
+ end
+ Editor.data2model(@clipboard_data, model, parent) do |m|
+ m.insert_before(parent, current)
+ end
+ expand_collapse(current)
+ toplevel.display_status("Inserted an element to " +
+ "'#{parent_type}' before index #{selected_index}.")
+ window.change
+ else
+ toplevel.display_status(
+ "Cannot insert node below '#{parent_type}'!")
+ end
+ else
+ toplevel.display_status("Nothing to paste in clipboard!")
+ end
+ else
+ toplevel.display_status("Append a node into the root first!")
+ end
+ end
+
+ # Append a new node to the selected Hash or Array.
+ def append_new_node(item)
+ if parent = selection.selected
+ parent_type = parent.type
+ case parent_type
+ when 'Hash'
+ key, type, content = ask_for_hash_pair(parent)
+ key or return
+ iter = create_node(parent, 'Key', key)
+ iter = create_node(iter, type, content)
+ toplevel.display_status(
+ "Added a (key, value)-pair to '#{parent_type}'.")
+ window.change
+ when 'Array'
+ type, content = ask_for_element(parent)
+ type or return
+ iter = create_node(parent, type, content)
+ window.change
+ toplevel.display_status("Appendend an element to '#{parent_type}'.")
+ else
+ toplevel.display_status("Cannot append to '#{parent_type}'!")
+ end
+ else
+ type, content = ask_for_element
+ type or return
+ iter = create_node(nil, type, content)
+ window.change
+ end
+ end
+
+ # Insert a new node into an Array before the selected element.
+ def insert_new_node(item)
+ if current = selection.selected
+ parent = current.parent or return
+ parent_parent = parent.parent
+ parent_type = parent.type
+ if parent_type == 'Array'
+ selected_index = parent.each_with_index do |c, i|
+ break i if c == current
+ end
+ type, content = ask_for_element(parent)
+ type or return
+ iter = model.insert_before(parent, current)
+ iter.type, iter.content = type, content
+ toplevel.display_status("Inserted an element to " +
+ "'#{parent_type}' before index #{selected_index}.")
+ window.change
+ else
+ toplevel.display_status(
+ "Cannot insert node below '#{parent_type}'!")
+ end
+ else
+ toplevel.display_status("Append a node into the root first!")
+ end
+ end
+
+ # Recursively collapse/expand a subtree starting from the selected node.
+ def collapse_expand(item)
+ if current = selection.selected
+ if row_expanded?(current.path)
+ collapse_row(current.path)
+ else
+ expand_row(current.path, true)
+ end
+ else
+ toplevel.display_status("Append a node into the root first!")
+ end
+ end
+
+ # Create the menu.
+ def create
+ add_item("Change node", ?n, &method(:change_node))
+ add_separator
+ add_item("Cut node", ?X, &method(:cut_node))
+ add_item("Copy node", ?C, &method(:copy_node))
+ add_item("Paste node (appending)", ?A, &method(:paste_node_appending))
+ add_item("Paste node (inserting before)", ?I,
+ &method(:paste_node_inserting_before))
+ add_separator
+ add_item("Append new node", ?a, &method(:append_new_node))
+ add_item("Insert new node before", ?i, &method(:insert_new_node))
+ add_separator
+ add_item("Collapse/Expand node (recursively)", ?e,
+ &method(:collapse_expand))
+
+ menu.show_all
+ signal_connect(:button_press_event) do |widget, event|
+ if event.kind_of? Gdk::EventButton and event.button == 3
+ menu.popup(nil, nil, event.button, event.time)
+ end
+ end
+ signal_connect(:popup_menu) do
+ menu.popup(nil, nil, 0, Gdk::Event::CURRENT_TIME)
+ end
+ end
+ end
+
+ # This class creates the File pulldown menu.
+ class FileMenu
+ include MenuExtension
+
+ # Clear the model and filename, but ask to save the JSON document, if
+ # unsaved changes have occured.
+ def new(item)
+ window.clear
+ end
+
+ # Open a file and load it into the editor. Ask to save the JSON document
+ # first, if unsaved changes have occured.
+ def open(item)
+ window.file_open
+ end
+
+ def open_location(item)
+ window.location_open
+ end
+
+ # Revert the current JSON document in the editor to the saved version.
+ def revert(item)
+ window.instance_eval do
+ @filename and file_open(@filename)
+ end
+ end
+
+ # Save the current JSON document.
+ def save(item)
+ window.file_save
+ end
+
+ # Save the current JSON document under the given filename.
+ def save_as(item)
+ window.file_save_as
+ end
+
+ # Quit the editor, after asking to save any unsaved changes first.
+ def quit(item)
+ window.quit
+ end
+
+ # Create the menu.
+ def create
+ title = MenuItem.new('File')
+ title.submenu = menu
+ add_item('New', &method(:new))
+ add_item('Open', ?o, &method(:open))
+ add_item('Open location', ?l, &method(:open_location))
+ add_item('Revert', &method(:revert))
+ add_separator
+ add_item('Save', ?s, &method(:save))
+ add_item('Save As', ?S, &method(:save_as))
+ add_separator
+ add_item('Quit', ?q, &method(:quit))
+ title
+ end
+ end
+
+ # This class creates the Edit pulldown menu.
+ class EditMenu
+ include MenuExtension
+
+ # Copy data from model into primary clipboard.
+ def copy(item)
+ data = Editor.model2data(model.iter_first)
+ json = JSON.pretty_generate(data, :max_nesting => false)
+ c = Gtk::Clipboard.get(Gdk::Selection::PRIMARY)
+ c.text = json
+ end
+
+ # Copy json text from primary clipboard into model.
+ def paste(item)
+ c = Gtk::Clipboard.get(Gdk::Selection::PRIMARY)
+ if json = c.wait_for_text
+ window.ask_save if @changed
+ begin
+ window.edit json
+ rescue JSON::ParserError
+ window.clear
+ end
+ end
+ end
+
+ # Find a string in all nodes' contents and select the found node in the
+ # treeview.
+ def find(item)
+ @search = ask_for_find_term(@search) or return
+ iter = model.get_iter('0') or return
+ iter.recursive_each do |i|
+ if @iter
+ if @iter != i
+ next
+ else
+ @iter = nil
+ next
+ end
+ elsif @search.match(i[CONTENT_COL])
+ set_cursor(i.path, nil, false)
+ @iter = i
+ break
+ end
+ end
+ end
+
+ # Repeat the last search given by #find.
+ def find_again(item)
+ @search or return
+ iter = model.get_iter('0')
+ iter.recursive_each do |i|
+ if @iter
+ if @iter != i
+ next
+ else
+ @iter = nil
+ next
+ end
+ elsif @search.match(i[CONTENT_COL])
+ set_cursor(i.path, nil, false)
+ @iter = i
+ break
+ end
+ end
+ end
+
+ # Sort (Reverse sort) all elements of the selected array by the given
+ # expression. _x_ is the element in question.
+ def sort(item)
+ if current = selection.selected
+ if current.type == 'Array'
+ parent = current.parent
+ ary = Editor.model2data(current)
+ order, reverse = ask_for_order
+ order or return
+ begin
+ block = eval "lambda { |x| #{order} }"
+ if reverse
+ ary.sort! { |a,b| block[b] <=> block[a] }
+ else
+ ary.sort! { |a,b| block[a] <=> block[b] }
+ end
+ rescue => e
+ Editor.error_dialog(self, "Failed to sort Array with #{order}: #{e}!")
+ else
+ Editor.data2model(ary, model, parent) do |m|
+ m.insert_before(parent, current)
+ end
+ model.remove(current)
+ expand_collapse(parent)
+ window.change
+ toplevel.display_status("Array has been sorted.")
+ end
+ else
+ toplevel.display_status("Only Array nodes can be sorted!")
+ end
+ else
+ toplevel.display_status("Select an Array to sort first!")
+ end
+ end
+
+ # Create the menu.
+ def create
+ title = MenuItem.new('Edit')
+ title.submenu = menu
+ add_item('Copy', ?c, &method(:copy))
+ add_item('Paste', ?v, &method(:paste))
+ add_separator
+ add_item('Find', ?f, &method(:find))
+ add_item('Find Again', ?g, &method(:find_again))
+ add_separator
+ add_item('Sort', ?S, &method(:sort))
+ title
+ end
+ end
+
+ class OptionsMenu
+ include MenuExtension
+
+ # Collapse/Expand all nodes by default.
+ def collapsed_nodes(item)
+ if expanded
+ self.expanded = false
+ collapse_all
+ else
+ self.expanded = true
+ expand_all
+ end
+ end
+
+ # Toggle pretty saving mode on/off.
+ def pretty_saving(item)
+ @pretty_item.toggled
+ window.change
+ end
+
+ attr_reader :pretty_item
+
+ # Create the menu.
+ def create
+ title = MenuItem.new('Options')
+ title.submenu = menu
+ add_item('Collapsed nodes', nil, CheckMenuItem, &method(:collapsed_nodes))
+ @pretty_item = add_item('Pretty saving', nil, CheckMenuItem,
+ &method(:pretty_saving))
+ @pretty_item.active = true
+ window.unchange
+ title
+ end
+ end
+
+ # This class inherits from Gtk::TreeView, to configure it and to add a lot
+ # of behaviour to it.
+ class JSONTreeView < Gtk::TreeView
+ include Gtk
+
+ # Creates a JSONTreeView instance, the parameter _window_ is
+ # a MainWindow instance and used for self delegation.
+ def initialize(window)
+ @window = window
+ super(TreeStore.new(Gdk::Pixbuf, String, String))
+ self.selection.mode = SELECTION_BROWSE
+
+ @expanded = false
+ self.headers_visible = false
+ add_columns
+ add_popup_menu
+ end
+
+ # Returns the MainWindow instance of this JSONTreeView.
+ attr_reader :window
+
+ # Returns true, if nodes are autoexpanding, false otherwise.
+ attr_accessor :expanded
+
+ private
+
+ def add_columns
+ cell = CellRendererPixbuf.new
+ column = TreeViewColumn.new('Icon', cell,
+ 'pixbuf' => ICON_COL
+ )
+ append_column(column)
+
+ cell = CellRendererText.new
+ column = TreeViewColumn.new('Type', cell,
+ 'text' => TYPE_COL
+ )
+ append_column(column)
+
+ cell = CellRendererText.new
+ cell.editable = true
+ column = TreeViewColumn.new('Content', cell,
+ 'text' => CONTENT_COL
+ )
+ cell.signal_connect(:edited, &method(:cell_edited))
+ append_column(column)
+ end
+
+ def unify_key(iter, key)
+ return unless iter.type == 'Key'
+ parent = iter.parent
+ if parent.any? { |c| c != iter and c.content == key }
+ old_key = key
+ i = 0
+ begin
+ key = sprintf("%s.%d", old_key, i += 1)
+ end while parent.any? { |c| c != iter and c.content == key }
+ end
+ iter.content = key
+ end
+
+ def cell_edited(cell, path, value)
+ iter = model.get_iter(path)
+ case iter.type
+ when 'Key'
+ unify_key(iter, value)
+ toplevel.display_status('Key has been changed.')
+ when 'FalseClass'
+ value.downcase!
+ if value == 'true'
+ iter.type, iter.content = 'TrueClass', 'true'
+ end
+ when 'TrueClass'
+ value.downcase!
+ if value == 'false'
+ iter.type, iter.content = 'FalseClass', 'false'
+ end
+ when 'Numeric'
+ iter.content =
+ if value == 'Infinity'
+ value
+ else
+ (Integer(value) rescue Float(value) rescue 0).to_s
+ end
+ when 'String'
+ iter.content = value
+ when 'Hash', 'Array'
+ return
+ else
+ fail "Unknown type found in model: #{iter.type}"
+ end
+ window.change
+ end
+
+ def configure_value(value, type)
+ value.editable = false
+ case type
+ when 'Array', 'Hash'
+ value.text = ''
+ when 'TrueClass'
+ value.text = 'true'
+ when 'FalseClass'
+ value.text = 'false'
+ when 'NilClass'
+ value.text = 'null'
+ when 'Numeric', 'String'
+ value.text ||= ''
+ value.editable = true
+ else
+ raise ArgumentError, "unknown type '#{type}' encountered"
+ end
+ end
+
+ def add_popup_menu
+ menu = PopUpMenu.new(self)
+ menu.create
+ end
+
+ public
+
+ # Create a _type_ node with content _content_, and add it to _parent_
+ # in the model. If _parent_ is nil, create a new model and put it into
+ # the editor treeview.
+ def create_node(parent, type, content)
+ iter = if parent
+ model.append(parent)
+ else
+ new_model = Editor.data2model(nil)
+ toplevel.view_new_model(new_model)
+ new_model.iter_first
+ end
+ iter.type, iter.content = type, content
+ expand_collapse(parent) if parent
+ iter
+ end
+
+ # Ask for a hash key, value pair to be added to the Hash node _parent_.
+ def ask_for_hash_pair(parent)
+ key_input = type_input = value_input = nil
+
+ dialog = Dialog.new("New (key, value) pair for Hash", nil, nil,
+ [ Stock::OK, Dialog::RESPONSE_ACCEPT ],
+ [ Stock::CANCEL, Dialog::RESPONSE_REJECT ]
+ )
+ dialog.width_request = 640
+
+ hbox = HBox.new(false, 5)
+ hbox.pack_start(Label.new("Key:"), false)
+ hbox.pack_start(key_input = Entry.new)
+ key_input.text = @key || ''
+ dialog.vbox.pack_start(hbox, false)
+ key_input.signal_connect(:activate) do
+ if parent.any? { |c| c.content == key_input.text }
+ toplevel.display_status('Key already exists in Hash!')
+ key_input.text = ''
+ else
+ toplevel.display_status('Key has been changed.')
+ end
+ end
+
+ hbox = HBox.new(false, 5)
+ hbox.pack_start(Label.new("Type:"), false)
+ hbox.pack_start(type_input = ComboBox.new(true))
+ ALL_TYPES.each { |t| type_input.append_text(t) }
+ type_input.active = @type || 0
+ dialog.vbox.pack_start(hbox, false)
+
+ type_input.signal_connect(:changed) do
+ value_input.editable = false
+ case ALL_TYPES[type_input.active]
+ when 'Array', 'Hash'
+ value_input.text = ''
+ when 'TrueClass'
+ value_input.text = 'true'
+ when 'FalseClass'
+ value_input.text = 'false'
+ when 'NilClass'
+ value_input.text = 'null'
+ else
+ value_input.text = ''
+ value_input.editable = true
+ end
+ end
+
+ hbox = HBox.new(false, 5)
+ hbox.pack_start(Label.new("Value:"), false)
+ hbox.pack_start(value_input = Entry.new)
+ value_input.width_chars = 60
+ value_input.text = @value || ''
+ dialog.vbox.pack_start(hbox, false)
+
+ dialog.signal_connect(:'key-press-event', &DEFAULT_DIALOG_KEY_PRESS_HANDLER)
+ dialog.show_all
+ self.focus = dialog
+ dialog.run do |response|
+ if response == Dialog::RESPONSE_ACCEPT
+ @key = key_input.text
+ type = ALL_TYPES[@type = type_input.active]
+ content = value_input.text
+ return @key, type, content
+ end
+ end
+ return
+ ensure
+ dialog.destroy
+ end
+
+ # Ask for an element to be appended _parent_.
+ def ask_for_element(parent = nil, default_type = nil, value_text = @content)
+ type_input = value_input = nil
+
+ dialog = Dialog.new(
+ "New element into #{parent ? parent.type : 'root'}",
+ nil, nil,
+ [ Stock::OK, Dialog::RESPONSE_ACCEPT ],
+ [ Stock::CANCEL, Dialog::RESPONSE_REJECT ]
+ )
+ hbox = HBox.new(false, 5)
+ hbox.pack_start(Label.new("Type:"), false)
+ hbox.pack_start(type_input = ComboBox.new(true))
+ default_active = 0
+ types = parent ? ALL_TYPES : CONTAINER_TYPES
+ types.each_with_index do |t, i|
+ type_input.append_text(t)
+ if t == default_type
+ default_active = i
+ end
+ end
+ type_input.active = default_active
+ dialog.vbox.pack_start(hbox, false)
+ type_input.signal_connect(:changed) do
+ configure_value(value_input, types[type_input.active])
+ end
+
+ hbox = HBox.new(false, 5)
+ hbox.pack_start(Label.new("Value:"), false)
+ hbox.pack_start(value_input = Entry.new)
+ value_input.width_chars = 60
+ value_input.text = value_text if value_text
+ configure_value(value_input, types[type_input.active])
+
+ dialog.vbox.pack_start(hbox, false)
+
+ dialog.signal_connect(:'key-press-event', &DEFAULT_DIALOG_KEY_PRESS_HANDLER)
+ dialog.show_all
+ self.focus = dialog
+ dialog.run do |response|
+ if response == Dialog::RESPONSE_ACCEPT
+ type = types[type_input.active]
+ @content = case type
+ when 'Numeric'
+ if (t = value_input.text) == 'Infinity'
+ 1 / 0.0
+ else
+ Integer(t) rescue Float(t) rescue 0
+ end
+ else
+ value_input.text
+ end.to_s
+ return type, @content
+ end
+ end
+ return
+ ensure
+ dialog.destroy if dialog
+ end
+
+ # Ask for an order criteria for sorting, using _x_ for the element in
+ # question. Returns the order criterium, and true/false for reverse
+ # sorting.
+ def ask_for_order
+ dialog = Dialog.new(
+ "Give an order criterium for 'x'.",
+ nil, nil,
+ [ Stock::OK, Dialog::RESPONSE_ACCEPT ],
+ [ Stock::CANCEL, Dialog::RESPONSE_REJECT ]
+ )
+ hbox = HBox.new(false, 5)
+
+ hbox.pack_start(Label.new("Order:"), false)
+ hbox.pack_start(order_input = Entry.new)
+ order_input.text = @order || 'x'
+ order_input.width_chars = 60
+
+ hbox.pack_start(reverse_checkbox = CheckButton.new('Reverse'), false)
+
+ dialog.vbox.pack_start(hbox, false)
+
+ dialog.signal_connect(:'key-press-event', &DEFAULT_DIALOG_KEY_PRESS_HANDLER)
+ dialog.show_all
+ self.focus = dialog
+ dialog.run do |response|
+ if response == Dialog::RESPONSE_ACCEPT
+ return @order = order_input.text, reverse_checkbox.active?
+ end
+ end
+ return
+ ensure
+ dialog.destroy if dialog
+ end
+
+ # Ask for a find term to search for in the tree. Returns the term as a
+ # string.
+ def ask_for_find_term(search = nil)
+ dialog = Dialog.new(
+ "Find a node matching regex in tree.",
+ nil, nil,
+ [ Stock::OK, Dialog::RESPONSE_ACCEPT ],
+ [ Stock::CANCEL, Dialog::RESPONSE_REJECT ]
+ )
+ hbox = HBox.new(false, 5)
+
+ hbox.pack_start(Label.new("Regex:"), false)
+ hbox.pack_start(regex_input = Entry.new)
+ hbox.pack_start(icase_checkbox = CheckButton.new('Icase'), false)
+ regex_input.width_chars = 60
+ if search
+ regex_input.text = search.source
+ icase_checkbox.active = search.casefold?
+ end
+
+ dialog.vbox.pack_start(hbox, false)
+
+ dialog.signal_connect(:'key-press-event', &DEFAULT_DIALOG_KEY_PRESS_HANDLER)
+ dialog.show_all
+ self.focus = dialog
+ dialog.run do |response|
+ if response == Dialog::RESPONSE_ACCEPT
+ begin
+ return Regexp.new(regex_input.text, icase_checkbox.active? ? Regexp::IGNORECASE : 0)
+ rescue => e
+ Editor.error_dialog(self, "Evaluation of regex /#{regex_input.text}/ failed: #{e}!")
+ return
+ end
+ end
+ end
+ return
+ ensure
+ dialog.destroy if dialog
+ end
+
+ # Expand or collapse row pointed to by _iter_ according
+ # to the #expanded attribute.
+ def expand_collapse(iter)
+ if expanded
+ expand_row(iter.path, true)
+ else
+ collapse_row(iter.path)
+ end
+ end
+ end
+
+ # The editor main window
+ class MainWindow < Gtk::Window
+ include Gtk
+
+ def initialize(encoding)
+ @changed = false
+ @encoding = encoding
+ super(TOPLEVEL)
+ display_title
+ set_default_size(800, 600)
+ signal_connect(:delete_event) { quit }
+
+ vbox = VBox.new(false, 0)
+ add(vbox)
+ #vbox.border_width = 0
+
+ @treeview = JSONTreeView.new(self)
+ @treeview.signal_connect(:'cursor-changed') do
+ display_status('')
+ end
+
+ menu_bar = create_menu_bar
+ vbox.pack_start(menu_bar, false, false, 0)
+
+ sw = ScrolledWindow.new(nil, nil)
+ sw.shadow_type = SHADOW_ETCHED_IN
+ sw.set_policy(POLICY_AUTOMATIC, POLICY_AUTOMATIC)
+ vbox.pack_start(sw, true, true, 0)
+ sw.add(@treeview)
+
+ @status_bar = Statusbar.new
+ vbox.pack_start(@status_bar, false, false, 0)
+
+ @filename ||= nil
+ if @filename
+ data = read_data(@filename)
+ view_new_model Editor.data2model(data)
+ end
+
+ signal_connect(:button_release_event) do |_,event|
+ if event.button == 2
+ c = Gtk::Clipboard.get(Gdk::Selection::PRIMARY)
+ if url = c.wait_for_text
+ location_open url
+ end
+ false
+ else
+ true
+ end
+ end
+ end
+
+ # Creates the menu bar with the pulldown menus and returns it.
+ def create_menu_bar
+ menu_bar = MenuBar.new
+ @file_menu = FileMenu.new(@treeview)
+ menu_bar.append @file_menu.create
+ @edit_menu = EditMenu.new(@treeview)
+ menu_bar.append @edit_menu.create
+ @options_menu = OptionsMenu.new(@treeview)
+ menu_bar.append @options_menu.create
+ menu_bar
+ end
+
+ # Sets editor status to changed, to indicate that the edited data
+ # containts unsaved changes.
+ def change
+ @changed = true
+ display_title
+ end
+
+ # Sets editor status to unchanged, to indicate that the edited data
+ # doesn't containt unsaved changes.
+ def unchange
+ @changed = false
+ display_title
+ end
+
+ # Puts a new model _model_ into the Gtk::TreeView to be edited.
+ def view_new_model(model)
+ @treeview.model = model
+ @treeview.expanded = true
+ @treeview.expand_all
+ unchange
+ end
+
+ # Displays _text_ in the status bar.
+ def display_status(text)
+ @cid ||= nil
+ @status_bar.pop(@cid) if @cid
+ @cid = @status_bar.get_context_id('dummy')
+ @status_bar.push(@cid, text)
+ end
+
+ # Opens a dialog, asking, if changes should be saved to a file.
+ def ask_save
+ if Editor.question_dialog(self,
+ "Unsaved changes to JSON model. Save?")
+ if @filename
+ file_save
+ else
+ file_save_as
+ end
+ end
+ end
+
+ # Quit this editor, that is, leave this editor's main loop.
+ def quit
+ ask_save if @changed
+ if Gtk.main_level > 0
+ destroy
+ Gtk.main_quit
+ end
+ nil
+ end
+
+ # Display the new title according to the editor's current state.
+ def display_title
+ title = TITLE.dup
+ title << ": #@filename" if @filename
+ title << " *" if @changed
+ self.title = title
+ end
+
+ # Clear the current model, after asking to save all unsaved changes.
+ def clear
+ ask_save if @changed
+ @filename = nil
+ self.view_new_model nil
+ end
+
+ def check_pretty_printed(json)
+ pretty = !!((nl_index = json.index("\n")) && nl_index != json.size - 1)
+ @options_menu.pretty_item.active = pretty
+ end
+ private :check_pretty_printed
+
+ # Open the data at the location _uri_, if given. Otherwise open a dialog
+ # to ask for the _uri_.
+ def location_open(uri = nil)
+ uri = ask_for_location unless uri
+ uri or return
+ ask_save if @changed
+ data = load_location(uri) or return
+ view_new_model Editor.data2model(data)
+ end
+
+ # Open the file _filename_ or call the #select_file method to ask for a
+ # filename.
+ def file_open(filename = nil)
+ filename = select_file('Open as a JSON file') unless filename
+ data = load_file(filename) or return
+ view_new_model Editor.data2model(data)
+ end
+
+ # Edit the string _json_ in the editor.
+ def edit(json)
+ if json.respond_to? :read
+ json = json.read
+ end
+ data = parse_json json
+ view_new_model Editor.data2model(data)
+ end
+
+ # Save the current file.
+ def file_save
+ if @filename
+ store_file(@filename)
+ else
+ file_save_as
+ end
+ end
+
+ # Save the current file as the filename
+ def file_save_as
+ filename = select_file('Save as a JSON file')
+ store_file(filename)
+ end
+
+ # Store the current JSON document to _path_.
+ def store_file(path)
+ if path
+ data = Editor.model2data(@treeview.model.iter_first)
+ File.open(path + '.tmp', 'wb') do |output|
+ data or break
+ if @options_menu.pretty_item.active?
+ output.puts JSON.pretty_generate(data, :max_nesting => false)
+ else
+ output.write JSON.generate(data, :max_nesting => false)
+ end
+ end
+ File.rename path + '.tmp', path
+ @filename = path
+ toplevel.display_status("Saved data to '#@filename'.")
+ unchange
+ end
+ rescue SystemCallError => e
+ Editor.error_dialog(self, "Failed to store JSON file: #{e}!")
+ end
+
+ # Load the file named _filename_ into the editor as a JSON document.
+ def load_file(filename)
+ if filename
+ if File.directory?(filename)
+ Editor.error_dialog(self, "Try to select a JSON file!")
+ nil
+ else
+ @filename = filename
+ if data = read_data(filename)
+ toplevel.display_status("Loaded data from '#@filename'.")
+ end
+ display_title
+ data
+ end
+ end
+ end
+
+ # Load the data at location _uri_ into the editor as a JSON document.
+ def load_location(uri)
+ data = read_data(uri) or return
+ @filename = nil
+ toplevel.display_status("Loaded data from '#{uri}'.")
+ display_title
+ data
+ end
+
+ def parse_json(json)
+ check_pretty_printed(json)
+ if @encoding && !/^utf8$/i.match(@encoding)
+ json = JSON.iconv 'utf-8', @encoding, json
+ end
+ JSON::parse(json, :max_nesting => false, :create_additions => false)
+ end
+ private :parse_json
+
+ # Read a JSON document from the file named _filename_, parse it into a
+ # ruby data structure, and return the data.
+ def read_data(filename)
+ open(filename) do |f|
+ json = f.read
+ return parse_json(json)
+ end
+ rescue => e
+ Editor.error_dialog(self, "Failed to parse JSON file: #{e}!")
+ return
+ end
+
+ # Open a file selecton dialog, displaying _message_, and return the
+ # selected filename or nil, if no file was selected.
+ def select_file(message)
+ filename = nil
+ fs = FileSelection.new(message)
+ fs.set_modal(true)
+ @default_dir = File.join(Dir.pwd, '') unless @default_dir
+ fs.set_filename(@default_dir)
+ fs.set_transient_for(self)
+ fs.signal_connect(:destroy) { Gtk.main_quit }
+ fs.ok_button.signal_connect(:clicked) do
+ filename = fs.filename
+ @default_dir = File.join(File.dirname(filename), '')
+ fs.destroy
+ Gtk.main_quit
+ end
+ fs.cancel_button.signal_connect(:clicked) do
+ fs.destroy
+ Gtk.main_quit
+ end
+ fs.show_all
+ Gtk.main
+ filename
+ end
+
+ # Ask for location URI a to load data from. Returns the URI as a string.
+ def ask_for_location
+ dialog = Dialog.new(
+ "Load data from location...",
+ nil, nil,
+ [ Stock::OK, Dialog::RESPONSE_ACCEPT ],
+ [ Stock::CANCEL, Dialog::RESPONSE_REJECT ]
+ )
+ hbox = HBox.new(false, 5)
+
+ hbox.pack_start(Label.new("Location:"), false)
+ hbox.pack_start(location_input = Entry.new)
+ location_input.width_chars = 60
+ location_input.text = @location || ''
+
+ dialog.vbox.pack_start(hbox, false)
+
+ dialog.signal_connect(:'key-press-event', &DEFAULT_DIALOG_KEY_PRESS_HANDLER)
+ dialog.show_all
+ dialog.run do |response|
+ if response == Dialog::RESPONSE_ACCEPT
+ return @location = location_input.text
+ end
+ end
+ return
+ ensure
+ dialog.destroy if dialog
+ end
+ end
+
+ class << self
+ # Starts a JSON Editor. If a block was given, it yields
+ # to the JSON::Editor::MainWindow instance.
+ def start(encoding = 'utf8') # :yield: window
+ Gtk.init
+ @window = Editor::MainWindow.new(encoding)
+ @window.icon_list = [ Editor.fetch_icon('json') ]
+ yield @window if block_given?
+ @window.show_all
+ Gtk.main
+ end
+
+ # Edit the string _json_ with encoding _encoding_ in the editor.
+ def edit(json, encoding = 'utf8')
+ start(encoding) do |window|
+ window.edit json
+ end
+ end
+
+ attr_reader :window
+ end
+ end
+end
--- /dev/null
+require 'json/common'
+
+module JSON
+ # This module holds all the modules/classes that implement JSON's
+ # functionality as C extensions.
+ module Ext
+ require 'json/ext/parser'
+ require 'json/ext/generator'
+ $DEBUG and warn "Using Ext extension for JSON."
+ JSON.parser = Parser
+ JSON.generator = Generator
+ end
+
+ JSON_LOADED = true unless defined?(::JSON::JSON_LOADED)
+end
--- /dev/null
+/* XPM */
+static char * json_xpm[] = {
+"64 64 1432 2",
+" c None",
+". c #641839",
+"+ c #CF163C",
+"@ c #D31C3B",
+"# c #E11A38",
+"$ c #5F242D",
+"% c #320C22",
+"& c #9B532D",
+"* c #F32E34",
+"= c #820F33",
+"- c #4B0F34",
+"; c #8E1237",
+"> c #944029",
+", c #961325",
+"' c #A00C24",
+") c #872C23",
+"! c #694021",
+"~ c #590D1F",
+"{ c #420528",
+"] c #D85A2D",
+"^ c #7E092B",
+"/ c #0E0925",
+"( c #0D081F",
+"_ c #0F081E",
+": c #12071F",
+"< c #360620",
+"[ c #682A21",
+"} c #673F21",
+"| c #780E21",
+"1 c #A82320",
+"2 c #8D1D1F",
+"3 c #970127",
+"4 c #0D0123",
+"5 c #0D0324",
+"6 c #3B1E28",
+"7 c #C28429",
+"8 c #0C0523",
+"9 c #0C041E",
+"0 c #0E031A",
+"a c #11031A",
+"b c #13031B",
+"c c #13031C",
+"d c #11031D",
+"e c #19051E",
+"f c #390E20",
+"g c #9C0C20",
+"h c #C00721",
+"i c #980320",
+"j c #14031E",
+"k c #CD9F32",
+"l c #C29F2E",
+"m c #0F0325",
+"n c #0D0321",
+"o c #0E0324",
+"p c #D08329",
+"q c #9D1B27",
+"r c #1C0320",
+"s c #0D011A",
+"t c #120117",
+"u c #130017",
+"v c #150018",
+"w c #160119",
+"x c #17021A",
+"y c #15021B",
+"z c #11021E",
+"A c #0F021F",
+"B c #8C1821",
+"C c #CF4522",
+"D c #831821",
+"E c #BA7033",
+"F c #EDB339",
+"G c #C89733",
+"H c #280727",
+"I c #0F051F",
+"J c #0E0420",
+"K c #591F27",
+"L c #E47129",
+"M c #612224",
+"N c #0C021D",
+"O c #120018",
+"P c #140017",
+"Q c #170017",
+"R c #190018",
+"S c #1B0019",
+"T c #1B011A",
+"U c #18011B",
+"V c #15011C",
+"W c #12031E",
+"X c #460A21",
+"Y c #A13823",
+"Z c #784323",
+"` c #5A0C21",
+" . c #BC4530",
+".. c #EB5B38",
+"+. c #CE4E3B",
+"@. c #DD9334",
+"#. c #751A27",
+"$. c #11071E",
+"%. c #0F041C",
+"&. c #1E0824",
+"*. c #955A28",
+"=. c #9A5027",
+"-. c #1E0321",
+";. c #11011A",
+">. c #140018",
+",. c #180018",
+"'. c #1F001A",
+"). c #20001B",
+"!. c #1E001A",
+"~. c #1B001A",
+"{. c #16021B",
+"]. c #16041E",
+"^. c #220622",
+"/. c #5F3525",
+"(. c #DE5724",
+"_. c #611021",
+":. c #0F0925",
+"<. c #D1892E",
+"[. c #F27036",
+"}. c #EC633B",
+"|. c #DA293C",
+"1. c #E64833",
+"2. c #912226",
+"3. c #11081C",
+"4. c #110419",
+"5. c #0F041E",
+"6. c #451425",
+"7. c #BF6F28",
+"8. c #332225",
+"9. c #0E021E",
+"0. c #13001B",
+"a. c #17001A",
+"b. c #1C001B",
+"c. c #21001C",
+"d. c #23001C",
+"e. c #21001B",
+"f. c #19021A",
+"g. c #17041E",
+"h. c #150721",
+"i. c #602424",
+"j. c #D51223",
+"k. c #540820",
+"l. c #D04D2D",
+"m. c #EA8933",
+"n. c #875637",
+"o. c #88543A",
+"p. c #E5923A",
+"q. c #891931",
+"r. c #130B25",
+"s. c #10051B",
+"t. c #110217",
+"u. c #12021A",
+"v. c #761826",
+"w. c #E2A728",
+"x. c #300224",
+"y. c #10011E",
+"z. c #16001B",
+"A. c #1B001B",
+"B. c #21001A",
+"C. c #1E0019",
+"D. c #1D0019",
+"E. c #1A011A",
+"F. c #17031C",
+"G. c #120720",
+"H. c #4E0822",
+"I. c #670721",
+"J. c #C07630",
+"K. c #F59734",
+"L. c #BE1B35",
+"M. c #0E1435",
+"N. c #522037",
+"O. c #DB8039",
+"P. c #D45933",
+"Q. c #420927",
+"R. c #0F041D",
+"S. c #140118",
+"T. c #13021D",
+"U. c #100423",
+"V. c #7B6227",
+"W. c #C04326",
+"X. c #0E0020",
+"Y. c #13001D",
+"Z. c #18001B",
+"`. c #1E001B",
+" + c #22001C",
+".+ c #22001B",
+"++ c #1B011B",
+"@+ c #16041D",
+"#+ c #130520",
+"$+ c #860521",
+"%+ c #710520",
+"&+ c #670A2A",
+"*+ c #A66431",
+"=+ c #E97536",
+"-+ c #F8833A",
+";+ c #F77A3A",
+">+ c #C45337",
+",+ c #0A1C35",
+"'+ c #993638",
+")+ c #F7863B",
+"!+ c #F49736",
+"~+ c #94462B",
+"{+ c #0E031F",
+"]+ c #130119",
+"^+ c #160018",
+"/+ c #16011B",
+"(+ c #15021F",
+"_+ c #120123",
+":+ c #A65C28",
+"<+ c #5C4D23",
+"[+ c #0F001F",
+"}+ c #14001D",
+"|+ c #1A001B",
+"1+ c #1F001B",
+"2+ c #24001D",
+"3+ c #25001D",
+"4+ c #24001C",
+"5+ c #1F001C",
+"6+ c #1A011C",
+"7+ c #16021E",
+"8+ c #3F0421",
+"9+ c #BC0522",
+"0+ c #1C041E",
+"a+ c #7F5531",
+"b+ c #E68A38",
+"c+ c #F8933E",
+"d+ c #FA7942",
+"e+ c #FB7543",
+"f+ c #FA6F41",
+"g+ c #F1793D",
+"h+ c #7D3B3A",
+"i+ c #28263B",
+"j+ c #D45441",
+"k+ c #F8A238",
+"l+ c #996B2D",
+"m+ c #0E0421",
+"n+ c #12011A",
+"o+ c #180019",
+"p+ c #17001C",
+"q+ c #12001F",
+"r+ c #4C2B2A",
+"s+ c #DB8130",
+"t+ c #540023",
+"u+ c #0F0120",
+"v+ c #16011C",
+"w+ c #22001D",
+"x+ c #25001F",
+"y+ c #26001F",
+"z+ c #25001E",
+"A+ c #24001E",
+"B+ c #1D001C",
+"C+ c #18011D",
+"D+ c #16031F",
+"E+ c #3C0522",
+"F+ c #9B0821",
+"G+ c #13041E",
+"H+ c #F6462E",
+"I+ c #E6AB37",
+"J+ c #E7A03E",
+"K+ c #FA9F44",
+"L+ c #FB8A48",
+"M+ c #FD7A4A",
+"N+ c #FD794A",
+"O+ c #FD7748",
+"P+ c #FD7E45",
+"Q+ c #FD8343",
+"R+ c #FB5D42",
+"S+ c #6E3A40",
+"T+ c #EE8A37",
+"U+ c #7E252B",
+"V+ c #100520",
+"W+ c #13011A",
+"X+ c #170019",
+"Y+ c #15001C",
+"Z+ c #0F0020",
+"`+ c #564427",
+" @ c #E0BA29",
+".@ c #5E2B25",
+"+@ c #10011F",
+"@@ c #17011C",
+"#@ c #1E001D",
+"$@ c #23001F",
+"%@ c #250020",
+"&@ c #24001F",
+"*@ c #23001E",
+"=@ c #21001E",
+"-@ c #1B001C",
+";@ c #17021D",
+">@ c #14041E",
+",@ c #AC0B25",
+"'@ c #5E1420",
+")@ c #F28635",
+"!@ c #C2733E",
+"~@ c #984C44",
+"{@ c #EA9148",
+"]@ c #FB844B",
+"^@ c #FD7E4C",
+"/@ c #FE7E4C",
+"(@ c #FE7E4B",
+"_@ c #FE7749",
+":@ c #FD7148",
+"<@ c #FB7D46",
+"[@ c #F89641",
+"}@ c #B95634",
+"|@ c #0D0927",
+"1@ c #11041D",
+"2@ c #150119",
+"3@ c #180017",
+"4@ c #16001A",
+"5@ c #13001E",
+"6@ c #110023",
+"7@ c #944C29",
+"8@ c #EE6229",
+"9@ c #3D0324",
+"0@ c #12021F",
+"a@ c #19011D",
+"b@ c #21001F",
+"c@ c #22001F",
+"d@ c #20001E",
+"e@ c #1F001D",
+"f@ c #1C001C",
+"g@ c #19011C",
+"h@ c #3D1621",
+"i@ c #B53622",
+"j@ c #31061F",
+"k@ c #841D34",
+"l@ c #F2703F",
+"m@ c #C14445",
+"n@ c #E67349",
+"o@ c #FB8E4B",
+"p@ c #FD834C",
+"q@ c #FE834D",
+"r@ c #FE834C",
+"s@ c #FE804C",
+"t@ c #FD814B",
+"u@ c #FB7D49",
+"v@ c #F79B43",
+"w@ c #AF1234",
+"x@ c #0D0625",
+"y@ c #13021C",
+"z@ c #1A0019",
+"A@ c #190019",
+"B@ c #410225",
+"C@ c #D39729",
+"D@ c #AA5927",
+"E@ c #0E0422",
+"F@ c #15021E",
+"G@ c #1A011D",
+"H@ c #1D001D",
+"I@ c #15031D",
+"J@ c #240820",
+"K@ c #A01023",
+"L@ c #670B21",
+"M@ c #3D0D33",
+"N@ c #E63C3E",
+"O@ c #EF7C45",
+"P@ c #F59048",
+"Q@ c #FB944A",
+"R@ c #FD904A",
+"S@ c #FE8E4B",
+"T@ c #FE854A",
+"U@ c #FE854B",
+"V@ c #FE884C",
+"W@ c #FC954B",
+"X@ c #F8AB45",
+"Y@ c #C37A35",
+"Z@ c #0D0425",
+"`@ c #13011B",
+" # c #170018",
+".# c #1A0018",
+"+# c #1C0019",
+"@# c #15001B",
+"## c #100120",
+"$# c #311F25",
+"%# c #E68E28",
+"&# c #7A1425",
+"*# c #130321",
+"=# c #17011E",
+"-# c #1A001D",
+";# c #19001B",
+"># c #16021C",
+",# c #130521",
+"'# c #6F3123",
+")# c #6D3022",
+"!# c #C89433",
+"~# c #EA7E3E",
+"{# c #DB2943",
+"]# c #EF7745",
+"^# c #FB8544",
+"/# c #FD9A43",
+"(# c #FE9941",
+"_# c #FE9D43",
+":# c #FEA548",
+"<# c #FEAE49",
+"[# c #FCB944",
+"}# c #CA9F35",
+"|# c #0E0225",
+"1# c #11001B",
+"2# c #160019",
+"3# c #12011B",
+"4# c #0F0220",
+"5# c #351D26",
+"6# c #D85B28",
+"7# c #6C0F26",
+"8# c #190121",
+"9# c #1B001E",
+"0# c #1A001C",
+"a# c #1D001B",
+"b# c #130220",
+"c# c #703A23",
+"d# c #713A23",
+"e# c #140327",
+"f# c #411B36",
+"g# c #C8713E",
+"h# c #7A3A3F",
+"i# c #CE2C3C",
+"j# c #E77338",
+"k# c #9C6535",
+"l# c #9C6233",
+"m# c #9C6332",
+"n# c #9C6A35",
+"o# c #C37D3C",
+"p# c #FEAC41",
+"q# c #FEC23E",
+"r# c #826330",
+"s# c #100122",
+"t# c #120019",
+"u# c #150017",
+"v# c #190017",
+"w# c #1B0018",
+"x# c #12001A",
+"y# c #10021F",
+"z# c #1A0326",
+"A# c #5F292A",
+"B# c #7B4E29",
+"C# c #3C0E25",
+"D# c #1A0020",
+"E# c #14021F",
+"F# c #723B23",
+"G# c #14001A",
+"H# c #58042A",
+"I# c #A28337",
+"J# c #C8813B",
+"K# c #B14B38",
+"L# c #761231",
+"M# c #5A132A",
+"N# c #0D0726",
+"O# c #0C0623",
+"P# c #0B0723",
+"Q# c #0B0A26",
+"R# c #321C2D",
+"S# c #C45B33",
+"T# c #FEBB33",
+"U# c #13052A",
+"V# c #13011F",
+"W# c #160017",
+"X# c #15001A",
+"Y# c #12001D",
+"Z# c #94062A",
+"`# c #630D2C",
+" $ c #85292B",
+".$ c #AA5E29",
+"+$ c #1F0123",
+"@$ c #19011F",
+"#$ c #1E001C",
+"$$ c #15031F",
+"%$ c #712122",
+"&$ c #712223",
+"*$ c #14011B",
+"=$ c #110321",
+"-$ c #AF0C2B",
+";$ c #E7D534",
+">$ c #EAC934",
+",$ c #84582D",
+"'$ c #1B0824",
+")$ c #11041E",
+"!$ c #10021B",
+"~$ c #100119",
+"{$ c #100218",
+"]$ c #0F041A",
+"^$ c #0E0720",
+"/$ c #2C1026",
+"($ c #D8A328",
+"_$ c #140322",
+":$ c #160016",
+"<$ c #14001F",
+"[$ c #120024",
+"}$ c #100128",
+"|$ c #3C032F",
+"1$ c #2C062E",
+"2$ c #29022B",
+"3$ c #A31D29",
+"4$ c #976A25",
+"5$ c #1A0321",
+"6$ c #17031E",
+"7$ c #1B021D",
+"8$ c #20001C",
+"9$ c #14041F",
+"0$ c #703422",
+"a$ c #6F3522",
+"b$ c #8D0328",
+"c$ c #920329",
+"d$ c #0F0326",
+"e$ c #100321",
+"f$ c #11021B",
+"g$ c #130117",
+"h$ c #140016",
+"i$ c #150015",
+"j$ c #140015",
+"k$ c #130116",
+"l$ c #120219",
+"m$ c #11031C",
+"n$ c #12031D",
+"o$ c #170016",
+"p$ c #160020",
+"q$ c #250029",
+"r$ c #670033",
+"s$ c #DCA238",
+"t$ c #F5C736",
+"u$ c #9A732E",
+"v$ c #110227",
+"w$ c #110324",
+"x$ c #811924",
+"y$ c #A04323",
+"z$ c #250721",
+"A$ c #1A041F",
+"B$ c #1E011D",
+"C$ c #1C011C",
+"D$ c #18031D",
+"E$ c #130721",
+"F$ c #6F3623",
+"G$ c #6B3622",
+"H$ c #1A001A",
+"I$ c #14011F",
+"J$ c #12011E",
+"K$ c #11011C",
+"L$ c #140117",
+"M$ c #170015",
+"N$ c #150016",
+"O$ c #120119",
+"P$ c #11011B",
+"Q$ c #11001A",
+"R$ c #130018",
+"S$ c #170118",
+"T$ c #170119",
+"U$ c #18021E",
+"V$ c #1A0126",
+"W$ c #6F2332",
+"X$ c #E5563B",
+"Y$ c #F1B83F",
+"Z$ c #F6CC38",
+"`$ c #9D7A2D",
+" % c #130123",
+".% c #130320",
+"+% c #2A0721",
+"@% c #B00E24",
+"#% c #7D0B23",
+"$% c #1F0522",
+"%% c #1E0220",
+"&% c #1D011E",
+"*% c #1A031E",
+"=% c #15051F",
+"-% c #241322",
+";% c #A32F23",
+">% c #670E21",
+",% c #1C001A",
+"'% c #19001A",
+")% c #180016",
+"!% c #160118",
+"~% c #140219",
+"{% c #11021C",
+"]% c #10021E",
+"^% c #0F011D",
+"/% c #170117",
+"(% c #160219",
+"_% c #17041D",
+":% c #190523",
+"<% c #8C042E",
+"[% c #B65838",
+"}% c #E9D73F",
+"|% c #EED43E",
+"1% c #D85538",
+"2% c #493129",
+"3% c #130120",
+"4% c #15021D",
+"5% c #330822",
+"6% c #8A0825",
+"7% c #3C0424",
+"8% c #1E0322",
+"9% c #1C0321",
+"0% c #180421",
+"a% c #130822",
+"b% c #AF2D24",
+"c% c #BC5623",
+"d% c #2F071F",
+"e% c #1A041C",
+"f% c #1C031C",
+"g% c #1D011C",
+"h% c #160117",
+"i% c #150419",
+"j% c #12081D",
+"k% c #0F0923",
+"l% c #A77027",
+"m% c #A60525",
+"n% c #11021A",
+"o% c #130218",
+"p% c #150319",
+"q% c #16061D",
+"r% c #180923",
+"s% c #9C1D2B",
+"t% c #A32636",
+"u% c #A66E3B",
+"v% c #4B2E3C",
+"w% c #412C36",
+"x% c #36012D",
+"y% c #140123",
+"z% c #17001E",
+"A% c #19011B",
+"B% c #1A0421",
+"C% c #340425",
+"D% c #9E0326",
+"E% c #1F0424",
+"F% c #1C0524",
+"G% c #180724",
+"H% c #A91024",
+"I% c #D55D24",
+"J% c #90071E",
+"K% c #3C051D",
+"L% c #1C021C",
+"M% c #1C011A",
+"N% c #1D001A",
+"O% c #160116",
+"P% c #150216",
+"Q% c #140217",
+"R% c #140618",
+"S% c #120D1D",
+"T% c #231925",
+"U% c #B16A2E",
+"V% c #FDAC34",
+"W% c #D58631",
+"X% c #280E2A",
+"Y% c #0D0A23",
+"Z% c #0F0920",
+"`% c #120C21",
+" & c #1F1026",
+".& c #A3352E",
+"+& c #EE9F36",
+"@& c #5D2A3C",
+"#& c #960D3C",
+"$& c #970638",
+"%& c #A00330",
+"&& c #4D0126",
+"*& c #1C001F",
+"=& c #280120",
+"-& c #290223",
+";& c #1F0425",
+">& c #260726",
+",& c #340A26",
+"'& c #850925",
+")& c #3A0823",
+"!& c #82071D",
+"~& c #5E071D",
+"{& c #18051C",
+"]& c #18021A",
+"^& c #190118",
+"/& c #160217",
+"(& c #150418",
+"_& c #130618",
+":& c #110718",
+"<& c #10081A",
+"[& c #110D1D",
+"}& c #291C24",
+"|& c #A73B2D",
+"1& c #FD6B36",
+"2& c #FD853C",
+"3& c #FD863B",
+"4& c #C24A35",
+"5& c #6B442F",
+"6& c #6D302D",
+"7& c #6E252E",
+"8& c #8E3B32",
+"9& c #DE7739",
+"0& c #F48E3F",
+"a& c #DD8D41",
+"b& c #854F3D",
+"c& c #7E2D35",
+"d& c #33082B",
+"e& c #1C0222",
+"f& c #20001F",
+"g& c #1F0222",
+"h& c #1A0524",
+"i& c #440C27",
+"j& c #BC1427",
+"k& c #20041B",
+"l& c #53061C",
+"m& c #25071B",
+"n& c #11061A",
+"o& c #130418",
+"p& c #140317",
+"q& c #150217",
+"r& c #160318",
+"s& c #12051B",
+"t& c #100C1D",
+"u& c #0E101E",
+"v& c #0C121F",
+"w& c #0C1321",
+"x& c #781725",
+"y& c #B25D2C",
+"z& c #FA6335",
+"A& c #FD633C",
+"B& c #FE6D42",
+"C& c #FE7C42",
+"D& c #FE813F",
+"E& c #FE873C",
+"F& c #FD743B",
+"G& c #FB683B",
+"H& c #FA7A3E",
+"I& c #F98242",
+"J& c #F97844",
+"K& c #F98943",
+"L& c #F79C3D",
+"M& c #A25133",
+"N& c #280B28",
+"O& c #1D021F",
+"P& c #1F011C",
+"Q& c #280321",
+"R& c #1C0724",
+"S& c #3F1C27",
+"T& c #D33C27",
+"U& c #0E061B",
+"V& c #0C091C",
+"W& c #0C0A1B",
+"X& c #0E091A",
+"Y& c #11081B",
+"Z& c #100A20",
+"`& c #0E0D23",
+" * c #551227",
+".* c #B21829",
+"+* c #C42329",
+"@* c #C62C29",
+"#* c #C55429",
+"$* c #E76F2B",
+"%* c #F14232",
+"&* c #F95E3A",
+"** c #FC6740",
+"=* c #FE6E45",
+"-* c #FE7246",
+";* c #FE7545",
+">* c #FE7744",
+",* c #FD7745",
+"'* c #FD7845",
+")* c #FD7847",
+"!* c #FD7948",
+"~* c #FD7B44",
+"{* c #FC7C3B",
+"]* c #6F3130",
+"^* c #140B24",
+"/* c #19031D",
+"(* c #1C011B",
+"_* c #5A011F",
+":* c #B70421",
+"<* c #380824",
+"[* c #3E2626",
+"}* c #9F5626",
+"|* c #13051E",
+"1* c #360A21",
+"2* c #361223",
+"3* c #371724",
+"4* c #381824",
+"5* c #3B1524",
+"6* c #3E1E26",
+"7* c #471A29",
+"8* c #DB252E",
+"9* c #ED2733",
+"0* c #EE5436",
+"a* c #F04237",
+"b* c #F33934",
+"c* c #F53D2F",
+"d* c #D7312B",
+"e* c #AF212B",
+"f* c #3A2C31",
+"g* c #F65F39",
+"h* c #FB6F41",
+"i* c #FD6D45",
+"j* c #FE7047",
+"k* c #FE7647",
+"l* c #FE7847",
+"m* c #FE7848",
+"n* c #FE7748",
+"o* c #FE7948",
+"p* c #FE7C48",
+"q* c #FE7C47",
+"r* c #FE7642",
+"s* c #FE7439",
+"t* c #6D332C",
+"u* c #100B21",
+"v* c #16031B",
+"w* c #2B001B",
+"x* c #22011F",
+"y* c #220521",
+"z* c #1B0A23",
+"A* c #421425",
+"B* c #951924",
+"C* c #381023",
+"D* c #E94028",
+"E* c #E7302B",
+"F* c #EF432D",
+"G* c #F4302E",
+"H* c #F32C30",
+"I* c #CB4432",
+"J* c #DD3235",
+"K* c #EF4B3A",
+"L* c #F0333E",
+"M* c #CC3D3F",
+"N* c #E4313C",
+"O* c #F34834",
+"P* c #D13E2C",
+"Q* c #431825",
+"R* c #0E1424",
+"S* c #3C202C",
+"T* c #F15537",
+"U* c #F97140",
+"V* c #FC6E45",
+"W* c #FE7547",
+"X* c #FE7947",
+"Y* c #FE7B48",
+"Z* c #FE7D48",
+"`* c #FE8047",
+" = c #FE7A42",
+".= c #FE7A38",
+"+= c #6D442B",
+"@= c #0F0B21",
+"#= c #15031A",
+"$= c #49001B",
+"%= c #2F001C",
+"&= c #21021E",
+"*= c #220620",
+"== c #1B0D23",
+"-= c #641625",
+";= c #951823",
+">= c #390F25",
+",= c #AC3A2A",
+"'= c #B6492E",
+")= c #ED7531",
+"!= c #F45A34",
+"~= c #F54C36",
+"{= c #C72D39",
+"]= c #DE283C",
+"^= c #F33B40",
+"/= c #F34142",
+"(= c #D0393F",
+"_= c #E72E39",
+":= c #DB3C2E",
+"<= c #461724",
+"[= c #0F0D1E",
+"}= c #140B1E",
+"|= c #341427",
+"1= c #CB4834",
+"2= c #F7743F",
+"3= c #FB7145",
+"4= c #FE7747",
+"5= c #FE7A47",
+"6= c #FF7B48",
+"7= c #FF7C48",
+"8= c #FE7F47",
+"9= c #FE8247",
+"0= c #FE8642",
+"a= c #FE8439",
+"b= c #6D442D",
+"c= c #0F0A21",
+"d= c #14031A",
+"e= c #20031D",
+"f= c #210821",
+"g= c #191024",
+"h= c #CC1C25",
+"i= c #961423",
+"j= c #2C162C",
+"k= c #BD242E",
+"l= c #EF2C31",
+"m= c #F54C34",
+"n= c #F34037",
+"o= c #F5353A",
+"p= c #F7413D",
+"q= c #F8423D",
+"r= c #F93A39",
+"s= c #F95731",
+"t= c #341425",
+"u= c #110A1D",
+"v= c #140619",
+"w= c #18051B",
+"x= c #200F26",
+"y= c #864833",
+"z= c #F8773F",
+"A= c #FC7445",
+"B= c #FF7E48",
+"C= c #FF7E49",
+"D= c #FF7D49",
+"E= c #FF7D48",
+"F= c #FE8347",
+"G= c #FE8743",
+"H= c #FE893B",
+"I= c #6E452F",
+"J= c #100E23",
+"K= c #14041A",
+"L= c #55041D",
+"M= c #540921",
+"N= c #161124",
+"O= c #CE6A25",
+"P= c #3F1129",
+"Q= c #170A29",
+"R= c #0F0F29",
+"S= c #15132B",
+"T= c #1E182D",
+"U= c #A82B3D",
+"V= c #CB6633",
+"W= c #CC6932",
+"X= c #CC3D2D",
+"Y= c #331225",
+"Z= c #0F091C",
+"`= c #120417",
+" - c #160216",
+".- c #190419",
+"+- c #210F26",
+"@- c #8C4934",
+"#- c #F97A40",
+"$- c #FC7545",
+"%- c #FF7B49",
+"&- c #FE7D46",
+"*- c #FE7E43",
+"=- c #FD7B3E",
+"-- c #FA6934",
+";- c #532328",
+">- c #130B1D",
+",- c #150519",
+"'- c #14041C",
+")- c #120920",
+"!- c #C43624",
+"~- c #A21E23",
+"{- c #F87C30",
+"]- c #C9302D",
+"^- c #300F2A",
+"/- c #591129",
+"(- c #171328",
+"_- c #171628",
+":- c #141829",
+"<- c #101A2B",
+"[- c #0F172B",
+"}- c #0F1226",
+"|- c #0E0C20",
+"1- c #100619",
+"2- c #140316",
+"3- c #19051B",
+"4- c #3C1428",
+"5- c #E04B36",
+"6- c #FA7B41",
+"7- c #FD7346",
+"8- c #FE7548",
+"9- c #FF7849",
+"0- c #FF7749",
+"a- c #FE7B47",
+"b- c #FE7945",
+"c- c #FC7740",
+"d- c #FA7E39",
+"e- c #C1432F",
+"f- c #131523",
+"g- c #130A1C",
+"h- c #420621",
+"i- c #D08423",
+"j- c #F87739",
+"k- c #C03D37",
+"l- c #962B34",
+"m- c #A14332",
+"n- c #E54B30",
+"o- c #9E3E2F",
+"p- c #7F262E",
+"q- c #922D2E",
+"r- c #9C4B2E",
+"s- c #65212C",
+"t- c #101628",
+"u- c #101022",
+"v- c #11091C",
+"w- c #130619",
+"x- c #160A1E",
+"y- c #43252C",
+"z- c #F66439",
+"A- c #FA6942",
+"B- c #FD6C47",
+"C- c #FE6E48",
+"D- c #FE6F48",
+"E- c #FE7049",
+"F- c #FE714A",
+"G- c #FE744A",
+"H- c #FE7846",
+"I- c #FD7243",
+"J- c #FC703E",
+"K- c #FA6C37",
+"L- c #81312B",
+"M- c #121123",
+"N- c #15071D",
+"O- c #16031A",
+"P- c #17021B",
+"Q- c #8F3D22",
+"R- c #F8393E",
+"S- c #E42A3D",
+"T- c #E7473B",
+"U- c #FB503B",
+"V- c #FB4F3A",
+"W- c #F95439",
+"X- c #ED4C38",
+"Y- c #F45938",
+"Z- c #FB6537",
+"`- c #EA5236",
+" ; c #CE6232",
+".; c #CD392C",
+"+; c #181425",
+"@; c #120F21",
+"#; c #130D20",
+"$; c #151225",
+"%; c #903431",
+"&; c #F8703D",
+"*; c #FB6344",
+"=; c #FD6748",
+"-; c #FE6849",
+";; c #FE6949",
+">; c #FE6A49",
+",; c #FE6C4A",
+"'; c #FE704A",
+"); c #FE734A",
+"!; c #FE7449",
+"~; c #FE7347",
+"{; c #FE7145",
+"]; c #FD6C42",
+"^; c #FD753D",
+"/; c #F36E35",
+"(; c #CB452C",
+"_; c #600D24",
+":; c #1C061F",
+"<; c #1E031F",
+"[; c #5B3821",
+"}; c #CE9822",
+"|; c #FA4341",
+"1; c #FB4341",
+"2; c #FC4541",
+"3; c #FC4542",
+"4; c #FC4143",
+"5; c #FC4D42",
+"6; c #FB5042",
+"7; c #FB5342",
+"8; c #FC5242",
+"9; c #FD4F40",
+"0; c #FD503E",
+"a; c #FB6339",
+"b; c #F45E33",
+"c; c #A12A2E",
+"d; c #401E2C",
+"e; c #452D2F",
+"f; c #F74F38",
+"g; c #FA5940",
+"h; c #FC6245",
+"i; c #FE6447",
+"j; c #FE6449",
+"k; c #FE6549",
+"l; c #FE6749",
+"m; c #FE6B49",
+"n; c #FE6D49",
+"o; c #FE6D48",
+"p; c #FE6D47",
+"q; c #FE6D45",
+"r; c #FE6C44",
+"s; c #FE6A42",
+"t; c #FE663C",
+"u; c #FC6233",
+"v; c #752129",
+"w; c #1F0922",
+"x; c #750520",
+"y; c #81061F",
+"z; c #FA3D42",
+"A; c #FB4142",
+"B; c #FD4543",
+"C; c #FD4844",
+"D; c #FD4A45",
+"E; c #FD4D45",
+"F; c #FD5045",
+"G; c #FD5345",
+"H; c #FE5346",
+"I; c #FE5445",
+"J; c #FD5444",
+"K; c #FC4F41",
+"L; c #FA513D",
+"M; c #F95339",
+"N; c #F63736",
+"O; c #F75737",
+"P; c #F95F3B",
+"Q; c #FB5840",
+"R; c #FD5F43",
+"S; c #FE6345",
+"T; c #FE6547",
+"U; c #FE6548",
+"V; c #FE6448",
+"W; c #FE6248",
+"X; c #FE6348",
+"Y; c #FE6748",
+"Z; c #FE6848",
+"`; c #FE6846",
+" > c #FE6A45",
+".> c #FE6D43",
+"+> c #FE703F",
+"@> c #FC6F36",
+"#> c #6F302B",
+"$> c #140A22",
+"%> c #FA3B42",
+"&> c #FC4243",
+"*> c #FD4744",
+"=> c #FE4A45",
+"-> c #FE4C47",
+";> c #FE4D47",
+">> c #FE5047",
+",> c #FE5347",
+"'> c #FE5447",
+")> c #FD5246",
+"!> c #FB503F",
+"~> c #FA543D",
+"{> c #9B3D3B",
+"]> c #A3433B",
+"^> c #F9683D",
+"/> c #FC6940",
+"(> c #FE6342",
+"_> c #FE6645",
+":> c #FE6646",
+"<> c #FE6147",
+"[> c #FE6048",
+"}> c #FE6148",
+"|> c #FE6746",
+"1> c #FE6A46",
+"2> c #FE6F45",
+"3> c #FE7441",
+"4> c #FC7D39",
+"5> c #6C422E",
+"6> c #0F0F23",
+"7> c #FA4142",
+"8> c #FC4643",
+"9> c #FE4D46",
+"0> c #FE4E47",
+"a> c #FE4F48",
+"b> c #FE5148",
+"c> c #FE5348",
+"d> c #FE5548",
+"e> c #FE5247",
+"f> c #FD5445",
+"g> c #FC5544",
+"h> c #F96041",
+"i> c #D33F3D",
+"j> c #392D39",
+"k> c #973C38",
+"l> c #F94E3A",
+"m> c #FD693E",
+"n> c #FE6C43",
+"o> c #FE6047",
+"p> c #FE5D47",
+"q> c #FE5E48",
+"r> c #FE6948",
+"s> c #FE6947",
+"t> c #FE6B47",
+"u> c #FE6E46",
+"v> c #FD6D43",
+"w> c #FB723D",
+"x> c #D54A33",
+"y> c #301C29",
+"z> c #FB4A42",
+"A> c #FD4B44",
+"B> c #FE4F47",
+"C> c #FE5048",
+"D> c #FE5648",
+"E> c #FE5848",
+"F> c #FE5747",
+"G> c #FE5547",
+"H> c #FC5945",
+"I> c #F95742",
+"J> c #F3543D",
+"K> c #A33336",
+"L> c #302032",
+"M> c #152433",
+"N> c #CD3E38",
+"O> c #FD5A3F",
+"P> c #FE6343",
+"Q> c #FE6446",
+"R> c #FE6247",
+"S> c #FE6A47",
+"T> c #FC6542",
+"U> c #FB6A3B",
+"V> c #FA6D34",
+"W> c #D73C2D",
+"X> c #442428",
+"Y> c #281323",
+"Z> c #FD4E42",
+"`> c #FD4D43",
+" , c #FE4D45",
+"., c #FE5248",
+"+, c #FE5947",
+"@, c #FE5C47",
+"#, c #FE5B47",
+"$, c #FE5A47",
+"%, c #FE5847",
+"&, c #FC5C45",
+"*, c #F95B43",
+"=, c #F3613F",
+"-, c #E74F37",
+";, c #8C2431",
+">, c #161E2F",
+",, c #CD4E33",
+"', c #FD503A",
+"), c #FE5D40",
+"!, c #FE6445",
+"~, c #FE6946",
+"{, c #FE6847",
+"], c #FE6747",
+"^, c #FD6644",
+"/, c #FD6241",
+"(, c #FD5B3D",
+"_, c #FE6739",
+":, c #FE6135",
+"<, c #AB4830",
+"[, c #733E2A",
+"}, c #161224",
+"|, c #FC4E42",
+"1, c #FE4D44",
+"2, c #FE4E46",
+"3, c #FE5147",
+"4, c #FE5E47",
+"5, c #FD5C46",
+"6, c #FA5B44",
+"7, c #F45441",
+"8, c #EB393A",
+"9, c #CC3433",
+"0, c #47212F",
+"a, c #59242F",
+"b, c #FC6734",
+"c, c #FC6F3A",
+"d, c #FC723E",
+"e, c #FD6540",
+"f, c #FE6442",
+"g, c #FE6643",
+"h, c #FE6944",
+"i, c #FE6546",
+"j, c #FE6444",
+"k, c #FE6143",
+"l, c #FE5E41",
+"m, c #FE613F",
+"n, c #FE683C",
+"o, c #FE7937",
+"p, c #A25030",
+"q, c #692629",
+"r, c #151122",
+"s, c #FA573F",
+"t, c #FB4D40",
+"u, c #FC4F43",
+"v, c #FE5246",
+"w, c #FF6347",
+"x, c #FE5F48",
+"y, c #F65942",
+"z, c #F0493D",
+"A, c #ED3736",
+"B, c #73262F",
+"C, c #10152C",
+"D, c #3B292F",
+"E, c #363034",
+"F, c #AC3938",
+"G, c #FC6B3B",
+"H, c #FD763C",
+"I, c #FE6D3F",
+"J, c #FE6341",
+"K, c #FE6642",
+"L, c #FE6745",
+"M, c #FE6245",
+"N, c #FE6244",
+"O, c #FE6841",
+"P, c #FF683B",
+"Q, c #EC7035",
+"R, c #D0412D",
+"S, c #3A1627",
+"T, c #CF3938",
+"U, c #F6543C",
+"V, c #FB5040",
+"W, c #FD5544",
+"X, c #FE5A48",
+"Y, c #FE5D48",
+"Z, c #FE5F47",
+"`, c #FF6147",
+" ' c #FD5C45",
+".' c #FB5B43",
+"+' c #FA5A42",
+"@' c #F76040",
+"#' c #F4623D",
+"$' c #F26D38",
+"%' c #EC4130",
+"&' c #380E2B",
+"*' c #13122C",
+"=' c #362D31",
+"-' c #353435",
+";' c #352E37",
+">' c #2D3337",
+",' c #CC5838",
+"'' c #CD6F3A",
+")' c #CE6E3D",
+"!' c #FE793F",
+"~' c #FD7541",
+"{' c #FD6243",
+"]' c #FE6545",
+"^' c #FF6543",
+"/' c #FF6240",
+"(' c #FE723B",
+"_' c #FE8034",
+":' c #442D2C",
+"<' c #311725",
+"[' c #222830",
+"}' c #B73B36",
+"|' c #F94C3D",
+"1' c #FD5543",
+"2' c #FE5B48",
+"3' c #FF5E47",
+"4' c #FE5C48",
+"5' c #FC5B44",
+"6' c #F95640",
+"7' c #C34E3D",
+"8' c #A45A3A",
+"9' c #F37438",
+"0' c #F28935",
+"a' c #AF422F",
+"b' c #240D2B",
+"c' c #88292F",
+"d' c #FA8E34",
+"e' c #FC7E38",
+"f' c #FC5939",
+"g' c #694A37",
+"h' c #693437",
+"i' c #382638",
+"j' c #142439",
+"k' c #9F483A",
+"l' c #C45E3C",
+"m' c #FD7240",
+"n' c #FF6645",
+"o' c #FF6245",
+"p' c #FF6045",
+"q' c #FF6146",
+"r' c #FF6246",
+"s' c #FF6446",
+"t' c #FF6545",
+"u' c #FE763F",
+"v' c #FE7237",
+"w' c #C65331",
+"x' c #3D272A",
+"y' c #0D1E2B",
+"z' c #683032",
+"A' c #F9453A",
+"B' c #FD5341",
+"C' c #FE5A46",
+"D' c #FF5A48",
+"E' c #FE5948",
+"F' c #FD5A47",
+"G' c #FC5D43",
+"H' c #F95B3D",
+"I' c #713F37",
+"J' c #1E2D32",
+"K' c #C44531",
+"L' c #EF7A2F",
+"M' c #6B2E2C",
+"N' c #0F0E2C",
+"O' c #F56633",
+"P' c #FA803A",
+"Q' c #FC673E",
+"R' c #FD673E",
+"S' c #FC6F3C",
+"T' c #FA6E3B",
+"U' c #C6633A",
+"V' c #A06739",
+"W' c #835638",
+"X' c #381F38",
+"Y' c #713B38",
+"Z' c #7B503C",
+"`' c #FE7741",
+" ) c #FE7344",
+".) c #FE6D46",
+"+) c #FF6946",
+"@) c #FF5E46",
+"#) c #FF5D46",
+"$) c #FF5D47",
+"%) c #FF5F48",
+"&) c #FF6248",
+"*) c #FE6941",
+"=) c #FC783C",
+"-) c #C46B35",
+";) c #892730",
+">) c #111629",
+",) c #1F2630",
+"') c #AD3939",
+")) c #FC5D41",
+"!) c #FE5946",
+"~) c #FF5848",
+"{) c #FE5549",
+"]) c #FC5E42",
+"^) c #FA673B",
+"/) c #DB7033",
+"() c #392E2B",
+"_) c #311A28",
+":) c #3C2127",
+"<) c #1D1027",
+"[) c #92102C",
+"}) c #F58336",
+"|) c #FA673E",
+"1) c #FD6642",
+"2) c #FD5A41",
+"3) c #FC6D41",
+"4) c #FC6D3F",
+"5) c #FD683E",
+"6) c #F38C39",
+"7) c #CE6535",
+"8) c #612E34",
+"9) c #1D2637",
+"0) c #71513E",
+"a) c #FF6847",
+"b) c #FF5F47",
+"c) c #FF5A46",
+"d) c #FF5847",
+"e) c #FF5748",
+"f) c #FF594A",
+"g) c #FF5E4B",
+"h) c #FE654C",
+"i) c #FE694B",
+"j) c #FE6B48",
+"k) c #FC6A43",
+"l) c #F7683E",
+"m) c #EC6E39",
+" ",
+" ",
+" ",
+" ",
+" ",
+" ",
+" ",
+" ",
+" . + @ # $ % ",
+" & * = - ; > , ' ) ! ~ ",
+" { ] ^ / ( _ : < [ } | 1 2 ",
+" 3 4 5 6 7 8 9 0 a b c d e f g h i j ",
+" k l m n o p q r s t u v w x y z A B C D ",
+" E F G H I J K L M N O P Q R S T U V W X Y Z ` ",
+" ...+.@.#.$.%.&.*.=.-.;.>.,.S '.).!.~.{.].^./.(._. ",
+" :.<.[.}.|.1.2.3.4.5.6.7.8.9.0.a.b.c.d.e.!.S f.g.h.i.j.k. ",
+" l.m.n.o.p.q.r.s.t.u.J v.w.x.y.z.A.c.d.d.B.C.D.E.F.G.H.I. ",
+" J.K.L.M.N.O.P.Q.R.t S.T.U.V.W.X.Y.Z.`. +d.d..+B.'.++@+#+$+%+ ",
+" &+*+=+-+;+>+,+'+)+!+~+{+]+^+/+(+_+:+<+[+}+|+1+d.2+3+4+d.5+6+7+8+9+0+ ",
+" a+b+c+d+e+f+g+h+i+j+k+l+m+n+^+o+p+q+r+s+t+u+v+b.w+x+y+z+A+w+B+C+D+E+F+G+ ",
+" H+I+J+K+L+M+N+O+P+Q+R+S+T+U+V+W+Q ,.X+Y+Z+`+ @.@+@@@#@$@%@&@*@=@#@-@;@>@,@'@ ",
+" )@!@~@{@]@^@/@(@_@:@<@[@}@|@1@2@3@R ,.4@5@6@7@8@9@0@a@#@b@c@=@d@e@f@g@>@h@i@j@ ",
+" k@l@m@n@o@p@q@r@s@t@u@v@w@x@y@^+R S z@A@z.q+B@C@D@E@F@G@H@#@e@#@#@f@g@I@J@K@L@ ",
+" M@N@O@P@Q@R@S@T@U@V@W@X@Y@Z@`@ #.#+#+#S A@@###$#%#&#*#=#-#f@B+B+B+f@;#>#,#'#)# ",
+" !#~#{#]#^#/#(#(#_#:#<#[#}#|#1#^+.#S +#+#z@2#3#4#5#6#7#8#9#0#A.B+B+a#A.@@b#c#d# ",
+" e#f#g#h#i#j#k#l#m#n#o#p#q#r#s#t#u#v#.#w#S R ^+x#y#z#A#B#C#D#-#A.a#`.`.b.g@E#d#F# ",
+" G#0@H#I#J#K#L#M#N#O#P#Q#R#S#T#U#V#>.W#3@v#R R X+X#Y#s#Z#`# $.$+$@$g@f@5+5+#$6+$$%$&$ ",
+" *$=$-$;$>$,$'$)$!$~${$]$^$/$($_$*$u#:$Q 3@,.X+z.<$[$}$|$1$2$3$4$5$6$7$e@8$#$G@9$0$a$ ",
+" ,.4@E#b$c$d$e$f$g$h$i$j$k$l$m$n$`@>.:$o$3@,. #a.p$q$r$s$t$u$v$w$x$y$z$A$B$#@C$D$E$F$G$ ",
+" R S H$v+I$J$K$n+L$:$o$o$M$N$L$O$P$Q$R$N$o$3@S$T$U$V$W$X$Y$Z$`$ %.%+%@%#%$%%%&%*%=%-%;%>% ",
+" E.,%~.'%Z.4@v W#o$)%)%)%Q !%~%{%]%^%Q$u u#/%(%_%:%<%[%}%|%1%2%3%4%=%5%6%7%8%9%0%a%b%c%d% ",
+" e%f%g%a#,%,%z@R 3@3@3@)%Q h%i%j%k%l%m%{+n%o%p%q%r%s%t%u%v%w%x%y%z%A%*%B%C%D%E%F%G%H%I% ",
+" J%K%L%M%N%D.S v#)%)%O%P%Q%R%S%T%U%V%W%X%Y%Z%`% &.&+&@&#&$&%&&&*&f@a##@=&-&;&>&,&'&)& ",
+" !&~&{&]&^&.#w#^&/%/&(&_&:&<&[&}&|&1&2&3&4&5&6&7&8&9&0&a&b&c&d&e&e@1+5+e@f&g&h&i&j& ",
+" k&l&m&n&o&p&q&r&i%s&3.t&u&v&w&x&y&z&A&B&C&D&E&F&G&H&I&J&K&L&M&N&O&P&1+`.e@f&Q&R&S&T& ",
+" 0 U&V&W&X&<&Y&j%Z&`& *.*+*@*#*$*%*&***=*-*;*>*>*,*'*)*!*~*{*]*^*/*(*a#B+#@_*:*<*[*}* ",
+" |*1*2*3*4*5*6*7*8*9*0*a*b*c*d*e*f*g*h*i*j*k*l*m*n*o*p*q*r*s*t*u*v*E.w*d.e@x*y*z*A*B* ",
+" C*D*E*F*G*H*I*J*K*L*M*N*O*P*Q*R*S*T*U*V*W*l*X*o*o*Y*Z*`* =.=+=@=#='%$=%=e@&=*===-=;= ",
+" >=,='=)=!=~={=]=^=/=(=_=:=<=[=}=|=1=2=3=4=5=p*6=6=7=8=9=0=a=b=c=d=A@~.b.B+e=f=g=h=i= ",
+" j=k=l=m=n=o=p=q=r=s=t=u=v=w=x=y=z=A=5=Z*B=C=D=E=8=F=G=H=I=J=K=S$R z@'%L=M=N=O= ",
+" P=Q=R=S=T=U=V=W=X=Y=Z=`= -.-+-@-#-$-5=p*E=D=%-%-q*&-*-=---;->-,-/%3@^+'-)-!-~- ",
+" {-]-^-/-(-_-:-<-[-}-|-1-2- -3-4-5-6-7-8-n*m*9-0-9-o*a-b-c-d-e-f-g-(&h%w c h-i- ",
+" j-k-l-m-n-o-p-q-r-s-t-u-v-w-,-x-y-z-A-B-C-D-E-E-F-G-_@m*H-I-J-K-L-M-N-O-P-(+Q- ",
+" R-S-T-U-V-W-X-Y-Z-`- ;.;+;@;#;$;%;&;*;=;-;-;;;>;,;';);!;~;{;];^;/;(;_;:;<;[;}; ",
+" |;1;2;3;4;5;6;7;8;9;0;a;b;c;d;e;f;g;h;i;j;j;k;k;l;m;n;o;p;q;r;s;t;u;v;w;x;y; ",
+" z;A;B;C;D;E;F;G;H;I;J;K;L;M;N;O;P;Q;R;S;T;U;V;W;X;k;Y;Z;`; >r;.>+>@>#>$> ",
+" %>&>*>=>->;>>>,>'>,>)>F;8;!>~>{>]>^>/>(>_>:>i;<>[>X;}>i;|>1>q;2>3>4>5>6> ",
+" 7>8>=>9>0>a>b>c>d>,>e>e>f>g>h>i>j>k>l>m>n>:>i;o>p>q>W;r>s>t>p;u>v>w>x>y> ",
+" z>A>9>0>B>C>c>D>E>F>G>G>F>H>I>J>K>L>M>N>O>P>Q>R>o>R>T;s>S>S>S>t>1>T>U>V>W>X>Y> ",
+" Z>`> ,9>B>.,D>+,@,#,$,%,$,&,*,=,-,;,>,,,',),P>!,!,_>~,t>s>{,],{,],^,/,(,_,:,<,[,}, ",
+" |,`>1,2,3,G>+,4,o>o>4,@,@,5,6,7,8,9,0,a,b,c,d,e,f,g,h, >~,|>T;T;T;i,j,k,l,m,n,o,p,q,r, ",
+" s,t,u,v,G>%,@,o>w,R>x,p>@,5,6,y,z,A,B,C,D,E,F,G,H,I,J,K,L,L,i,i;i;i;Q>S;M,N,P>O,P,Q,R,S, ",
+" T,U,V,W,%,X,Y,Z,`,[>q>@, '.'+'@'#'$'%'&'*'='-';'>',''')'!'~'{'N,i,:>_>]'M,M,Q>_>^'/'('_':'<' ",
+" ['}'|'1'$,X,2'p>3'4'2'@,5'6'7'8'9'0'a'b'c'd'e'f'g'h'i'j'k'l'd,m'g, > >n'o'p'q'r's't'.>u'v'w'x' ",
+" y'z'A'B'C'X,X,2'D'E'E'F'G'H'I'J'K'L'M'N'O'P'Q'R'S'T'U'V'W'X'Y'Z'`' ).)+)r'@)#)$)%)&)l;1>*)=)-);) ",
+" >),)')))!)X,E'X,~){)d>!)])^)/)()_):)<)[)})|)1)f,2)3)4)5)6)7)8)9)0)*--*a)b)c)d)e)f)g)h)i)j)k)l)m) ",
+" ",
+" ",
+" ",
+" ",
+" ",
+" ",
+" ",
+" "};
--- /dev/null
+require 'json/common'
+require 'json/pure/parser'
+require 'json/pure/generator'
+
+module JSON
+ # This module holds all the modules/classes that implement JSON's
+ # functionality in pure ruby.
+ module Pure
+ $DEBUG and warn "Using Pure library for JSON."
+ JSON.parser = Parser
+ JSON.generator = Generator
+ end
+
+ JSON_LOADED = true unless defined?(::JSON::JSON_LOADED)
+end
--- /dev/null
+module JSON
+ MAP = {
+ "\x0" => '\u0000',
+ "\x1" => '\u0001',
+ "\x2" => '\u0002',
+ "\x3" => '\u0003',
+ "\x4" => '\u0004',
+ "\x5" => '\u0005',
+ "\x6" => '\u0006',
+ "\x7" => '\u0007',
+ "\b" => '\b',
+ "\t" => '\t',
+ "\n" => '\n',
+ "\xb" => '\u000b',
+ "\f" => '\f',
+ "\r" => '\r',
+ "\xe" => '\u000e',
+ "\xf" => '\u000f',
+ "\x10" => '\u0010',
+ "\x11" => '\u0011',
+ "\x12" => '\u0012',
+ "\x13" => '\u0013',
+ "\x14" => '\u0014',
+ "\x15" => '\u0015',
+ "\x16" => '\u0016',
+ "\x17" => '\u0017',
+ "\x18" => '\u0018',
+ "\x19" => '\u0019',
+ "\x1a" => '\u001a',
+ "\x1b" => '\u001b',
+ "\x1c" => '\u001c',
+ "\x1d" => '\u001d',
+ "\x1e" => '\u001e',
+ "\x1f" => '\u001f',
+ '"' => '\"',
+ '\\' => '\\\\',
+ } # :nodoc:
+
+ # Convert a UTF8 encoded Ruby string _string_ to a JSON string, encoded with
+ # UTF16 big endian characters as \u????, and return it.
+ if defined?(::Encoding)
+ def utf8_to_json(string) # :nodoc:
+ string = string.dup
+ string << '' # XXX workaround: avoid buffer sharing
+ string.force_encoding(::Encoding::ASCII_8BIT)
+ string.gsub!(/["\\\x0-\x1f]/) { MAP[$&] }
+ string.force_encoding(::Encoding::UTF_8)
+ string
+ end
+
+ def utf8_to_json_ascii(string) # :nodoc:
+ string = string.dup
+ string << '' # XXX workaround: avoid buffer sharing
+ string.force_encoding(::Encoding::ASCII_8BIT)
+ string.gsub!(/["\\\x0-\x1f]/) { MAP[$&] }
+ string.gsub!(/(
+ (?:
+ [\xc2-\xdf][\x80-\xbf] |
+ [\xe0-\xef][\x80-\xbf]{2} |
+ [\xf0-\xf4][\x80-\xbf]{3}
+ )+ |
+ [\x80-\xc1\xf5-\xff] # invalid
+ )/nx) { |c|
+ c.size == 1 and raise GeneratorError, "invalid utf8 byte: '#{c}'"
+ s = JSON.iconv('utf-16be', 'utf-8', c).unpack('H*')[0]
+ s.gsub!(/.{4}/n, '\\\\u\&')
+ }
+ string.force_encoding(::Encoding::UTF_8)
+ string
+ rescue => e
+ raise GeneratorError, "Caught #{e.class}: #{e}"
+ end
+ else
+ def utf8_to_json(string) # :nodoc:
+ string.gsub(/["\\\x0-\x1f]/) { MAP[$&] }
+ end
+
+ def utf8_to_json_ascii(string) # :nodoc:
+ string = string.gsub(/["\\\x0-\x1f]/) { MAP[$&] }
+ string.gsub!(/(
+ (?:
+ [\xc2-\xdf][\x80-\xbf] |
+ [\xe0-\xef][\x80-\xbf]{2} |
+ [\xf0-\xf4][\x80-\xbf]{3}
+ )+ |
+ [\x80-\xc1\xf5-\xff] # invalid
+ )/nx) { |c|
+ c.size == 1 and raise GeneratorError, "invalid utf8 byte: '#{c}'"
+ s = JSON.iconv('utf-16be', 'utf-8', c).unpack('H*')[0]
+ s.gsub!(/.{4}/n, '\\\\u\&')
+ }
+ string
+ rescue => e
+ raise GeneratorError, "Caught #{e.class}: #{e}"
+ end
+ end
+ module_function :utf8_to_json, :utf8_to_json_ascii
+
+ module Pure
+ module Generator
+ # This class is used to create State instances, that are use to hold data
+ # while generating a JSON text from a Ruby data structure.
+ class State
+ # Creates a State object from _opts_, which ought to be Hash to create
+ # a new State instance configured by _opts_, something else to create
+ # an unconfigured instance. If _opts_ is a State object, it is just
+ # returned.
+ def self.from_state(opts)
+ case
+ when self === opts
+ opts
+ when opts.respond_to?(:to_hash)
+ new(opts.to_hash)
+ when opts.respond_to?(:to_h)
+ new(opts.to_h)
+ else
+ SAFE_STATE_PROTOTYPE.dup
+ end
+ end
+
+ # Instantiates a new State object, configured by _opts_.
+ #
+ # _opts_ can have the following keys:
+ #
+ # * *indent*: a string used to indent levels (default: ''),
+ # * *space*: a string that is put after, a : or , delimiter (default: ''),
+ # * *space_before*: a string that is put before a : pair delimiter (default: ''),
+ # * *object_nl*: a string that is put at the end of a JSON object (default: ''),
+ # * *array_nl*: a string that is put at the end of a JSON array (default: ''),
+ # * *check_circular*: is deprecated now, use the :max_nesting option instead,
+ # * *max_nesting*: sets the maximum level of data structure nesting in
+ # the generated JSON, max_nesting = 0 if no maximum should be checked.
+ # * *allow_nan*: true if NaN, Infinity, and -Infinity should be
+ # generated, otherwise an exception is thrown, if these values are
+ # encountered. This options defaults to false.
+ # * *quirks_mode*: Enables quirks_mode for parser, that is for example
+ # generating single JSON values instead of documents is possible.
+ def initialize(opts = {})
+ @indent = ''
+ @space = ''
+ @space_before = ''
+ @object_nl = ''
+ @array_nl = ''
+ @allow_nan = false
+ @ascii_only = false
+ @quirks_mode = false
+ configure opts
+ end
+
+ # This string is used to indent levels in the JSON text.
+ attr_accessor :indent
+
+ # This string is used to insert a space between the tokens in a JSON
+ # string.
+ attr_accessor :space
+
+ # This string is used to insert a space before the ':' in JSON objects.
+ attr_accessor :space_before
+
+ # This string is put at the end of a line that holds a JSON object (or
+ # Hash).
+ attr_accessor :object_nl
+
+ # This string is put at the end of a line that holds a JSON array.
+ attr_accessor :array_nl
+
+ # This integer returns the maximum level of data structure nesting in
+ # the generated JSON, max_nesting = 0 if no maximum is checked.
+ attr_accessor :max_nesting
+
+ # If this attribute is set to true, quirks mode is enabled, otherwise
+ # it's disabled.
+ attr_accessor :quirks_mode
+
+ # This integer returns the current depth data structure nesting in the
+ # generated JSON.
+ attr_accessor :depth
+
+ def check_max_nesting # :nodoc:
+ return if @max_nesting.zero?
+ current_nesting = depth + 1
+ current_nesting > @max_nesting and
+ raise NestingError, "nesting of #{current_nesting} is too deep"
+ end
+
+ # Returns true, if circular data structures are checked,
+ # otherwise returns false.
+ def check_circular?
+ !@max_nesting.zero?
+ end
+
+ # Returns true if NaN, Infinity, and -Infinity should be considered as
+ # valid JSON and output.
+ def allow_nan?
+ @allow_nan
+ end
+
+ # Returns true, if only ASCII characters should be generated. Otherwise
+ # returns false.
+ def ascii_only?
+ @ascii_only
+ end
+
+ # Returns true, if quirks mode is enabled. Otherwise returns false.
+ def quirks_mode?
+ @quirks_mode
+ end
+
+ # Configure this State instance with the Hash _opts_, and return
+ # itself.
+ def configure(opts)
+ @indent = opts[:indent] if opts.key?(:indent)
+ @space = opts[:space] if opts.key?(:space)
+ @space_before = opts[:space_before] if opts.key?(:space_before)
+ @object_nl = opts[:object_nl] if opts.key?(:object_nl)
+ @array_nl = opts[:array_nl] if opts.key?(:array_nl)
+ @allow_nan = !!opts[:allow_nan] if opts.key?(:allow_nan)
+ @ascii_only = opts[:ascii_only] if opts.key?(:ascii_only)
+ @depth = opts[:depth] || 0
+ @quirks_mode = opts[:quirks_mode] if opts.key?(:quirks_mode)
+ if !opts.key?(:max_nesting) # defaults to 19
+ @max_nesting = 19
+ elsif opts[:max_nesting]
+ @max_nesting = opts[:max_nesting]
+ else
+ @max_nesting = 0
+ end
+ self
+ end
+ alias merge configure
+
+ # Returns the configuration instance variables as a hash, that can be
+ # passed to the configure method.
+ def to_h
+ result = {}
+ for iv in %w[indent space space_before object_nl array_nl allow_nan max_nesting ascii_only quirks_mode depth]
+ result[iv.intern] = instance_variable_get("@#{iv}")
+ end
+ result
+ end
+
+ # Generates a valid JSON document from object +obj+ and returns the
+ # result. If no valid JSON document can be created this method raises a
+ # GeneratorError exception.
+ def generate(obj)
+ result = obj.to_json(self)
+ if !@quirks_mode && result !~ /\A\s*(?:\[.*\]|\{.*\})\s*\Z/m
+ raise GeneratorError, "only generation of JSON objects or arrays allowed"
+ end
+ result
+ end
+
+ # Return the value returned by method +name+.
+ def [](name)
+ __send__ name
+ end
+ end
+
+ module GeneratorMethods
+ module Object
+ # Converts this object to a string (calling #to_s), converts
+ # it to a JSON string, and returns the result. This is a fallback, if no
+ # special method #to_json was defined for some object.
+ def to_json(*) to_s.to_json end
+ end
+
+ module Hash
+ # Returns a JSON string containing a JSON object, that is unparsed from
+ # this Hash instance.
+ # _state_ is a JSON::State object, that can also be used to configure the
+ # produced JSON string output further.
+ # _depth_ is used to find out nesting depth, to indent accordingly.
+ def to_json(state = nil, *)
+ state = State.from_state(state)
+ state.check_max_nesting
+ json_transform(state)
+ end
+
+ private
+
+ def json_shift(state)
+ state.object_nl.empty? or return ''
+ state.indent * state.depth
+ end
+
+ def json_transform(state)
+ delim = ','
+ delim << state.object_nl
+ result = '{'
+ result << state.object_nl
+ depth = state.depth += 1
+ first = true
+ indent = !state.object_nl.empty?
+ each { |key,value|
+ result << delim unless first
+ result << state.indent * depth if indent
+ result << key.to_s.to_json(state)
+ result << state.space_before
+ result << ':'
+ result << state.space
+ result << value.to_json(state)
+ first = false
+ }
+ depth = state.depth -= 1
+ result << state.object_nl
+ result << state.indent * depth if indent if indent
+ result << '}'
+ result
+ end
+ end
+
+ module Array
+ # Returns a JSON string containing a JSON array, that is unparsed from
+ # this Array instance.
+ # _state_ is a JSON::State object, that can also be used to configure the
+ # produced JSON string output further.
+ def to_json(state = nil, *)
+ state = State.from_state(state)
+ state.check_max_nesting
+ json_transform(state)
+ end
+
+ private
+
+ def json_transform(state)
+ delim = ','
+ delim << state.array_nl
+ result = '['
+ result << state.array_nl
+ depth = state.depth += 1
+ first = true
+ indent = !state.array_nl.empty?
+ each { |value|
+ result << delim unless first
+ result << state.indent * depth if indent
+ result << value.to_json(state)
+ first = false
+ }
+ depth = state.depth -= 1
+ result << state.array_nl
+ result << state.indent * depth if indent
+ result << ']'
+ end
+ end
+
+ module Integer
+ # Returns a JSON string representation for this Integer number.
+ def to_json(*) to_s end
+ end
+
+ module Float
+ # Returns a JSON string representation for this Float number.
+ def to_json(state = nil, *)
+ state = State.from_state(state)
+ case
+ when infinite?
+ if state.allow_nan?
+ to_s
+ else
+ raise GeneratorError, "#{self} not allowed in JSON"
+ end
+ when nan?
+ if state.allow_nan?
+ to_s
+ else
+ raise GeneratorError, "#{self} not allowed in JSON"
+ end
+ else
+ to_s
+ end
+ end
+ end
+
+ module String
+ if defined?(::Encoding)
+ # This string should be encoded with UTF-8 A call to this method
+ # returns a JSON string encoded with UTF16 big endian characters as
+ # \u????.
+ def to_json(state = nil, *args)
+ state = State.from_state(state)
+ if encoding == ::Encoding::UTF_8
+ string = self
+ else
+ string = encode(::Encoding::UTF_8)
+ end
+ if state.ascii_only?
+ '"' << JSON.utf8_to_json_ascii(string) << '"'
+ else
+ '"' << JSON.utf8_to_json(string) << '"'
+ end
+ end
+ else
+ # This string should be encoded with UTF-8 A call to this method
+ # returns a JSON string encoded with UTF16 big endian characters as
+ # \u????.
+ def to_json(state = nil, *args)
+ state = State.from_state(state)
+ if state.ascii_only?
+ '"' << JSON.utf8_to_json_ascii(self) << '"'
+ else
+ '"' << JSON.utf8_to_json(self) << '"'
+ end
+ end
+ end
+
+ # Module that holds the extinding methods if, the String module is
+ # included.
+ module Extend
+ # Raw Strings are JSON Objects (the raw bytes are stored in an
+ # array for the key "raw"). The Ruby String can be created by this
+ # module method.
+ def json_create(o)
+ o['raw'].pack('C*')
+ end
+ end
+
+ # Extends _modul_ with the String::Extend module.
+ def self.included(modul)
+ modul.extend Extend
+ end
+
+ # This method creates a raw object hash, that can be nested into
+ # other data structures and will be unparsed as a raw string. This
+ # method should be used, if you want to convert raw strings to JSON
+ # instead of UTF-8 strings, e. g. binary data.
+ def to_json_raw_object
+ {
+ JSON.create_id => self.class.name,
+ 'raw' => self.unpack('C*'),
+ }
+ end
+
+ # This method creates a JSON text from the result of
+ # a call to to_json_raw_object of this String.
+ def to_json_raw(*args)
+ to_json_raw_object.to_json(*args)
+ end
+ end
+
+ module TrueClass
+ # Returns a JSON string for true: 'true'.
+ def to_json(*) 'true' end
+ end
+
+ module FalseClass
+ # Returns a JSON string for false: 'false'.
+ def to_json(*) 'false' end
+ end
+
+ module NilClass
+ # Returns a JSON string for nil: 'null'.
+ def to_json(*) 'null' end
+ end
+ end
+ end
+ end
+end
--- /dev/null
+require 'strscan'
+
+module JSON
+ module Pure
+ # This class implements the JSON parser that is used to parse a JSON string
+ # into a Ruby data structure.
+ class Parser < StringScanner
+ STRING = /" ((?:[^\x0-\x1f"\\] |
+ # escaped special characters:
+ \\["\\\/bfnrt] |
+ \\u[0-9a-fA-F]{4} |
+ # match all but escaped special characters:
+ \\[\x20-\x21\x23-\x2e\x30-\x5b\x5d-\x61\x63-\x65\x67-\x6d\x6f-\x71\x73\x75-\xff])*)
+ "/nx
+ INTEGER = /(-?0|-?[1-9]\d*)/
+ FLOAT = /(-?
+ (?:0|[1-9]\d*)
+ (?:
+ \.\d+(?i:e[+-]?\d+) |
+ \.\d+ |
+ (?i:e[+-]?\d+)
+ )
+ )/x
+ NAN = /NaN/
+ INFINITY = /Infinity/
+ MINUS_INFINITY = /-Infinity/
+ OBJECT_OPEN = /\{/
+ OBJECT_CLOSE = /\}/
+ ARRAY_OPEN = /\[/
+ ARRAY_CLOSE = /\]/
+ PAIR_DELIMITER = /:/
+ COLLECTION_DELIMITER = /,/
+ TRUE = /true/
+ FALSE = /false/
+ NULL = /null/
+ IGNORE = %r(
+ (?:
+ //[^\n\r]*[\n\r]| # line comments
+ /\* # c-style comments
+ (?:
+ [^*/]| # normal chars
+ /[^*]| # slashes that do not start a nested comment
+ \*[^/]| # asterisks that do not end this comment
+ /(?=\*/) # single slash before this comment's end
+ )*
+ \*/ # the End of this comment
+ |[ \t\r\n]+ # whitespaces: space, horicontal tab, lf, cr
+ )+
+ )mx
+
+ UNPARSED = Object.new
+
+ # Creates a new JSON::Pure::Parser instance for the string _source_.
+ #
+ # It will be configured by the _opts_ hash. _opts_ can have the following
+ # keys:
+ # * *max_nesting*: The maximum depth of nesting allowed in the parsed data
+ # structures. Disable depth checking with :max_nesting => false|nil|0,
+ # it defaults to 19.
+ # * *allow_nan*: If set to true, allow NaN, Infinity and -Infinity in
+ # defiance of RFC 4627 to be parsed by the Parser. This option defaults
+ # to false.
+ # * *symbolize_names*: If set to true, returns symbols for the names
+ # (keys) in a JSON object. Otherwise strings are returned, which is also
+ # the default.
+ # * *create_additions*: If set to true, the Parser creates
+ # additions when if a matching class and create_id was found. This
+ # option defaults to false.
+ # * *object_class*: Defaults to Hash
+ # * *array_class*: Defaults to Array
+ # * *quirks_mode*: Enables quirks_mode for parser, that is for example
+ # parsing single JSON values instead of documents is possible.
+ def initialize(source, opts = {})
+ opts ||= {}
+ unless @quirks_mode = opts[:quirks_mode]
+ source = determine_encoding source
+ end
+ super source
+ if !opts.key?(:max_nesting) # defaults to 19
+ @max_nesting = 19
+ elsif opts[:max_nesting]
+ @max_nesting = opts[:max_nesting]
+ else
+ @max_nesting = 0
+ end
+ @allow_nan = !!opts[:allow_nan]
+ @symbolize_names = !!opts[:symbolize_names]
+ if opts.key?(:create_additions)
+ @create_additions = !!opts[:create_additions]
+ else
+ @create_additions = false
+ end
+ @create_id = @create_additions ? JSON.create_id : nil
+ @object_class = opts[:object_class] || Hash
+ @array_class = opts[:array_class] || Array
+ @match_string = opts[:match_string]
+ end
+
+ alias source string
+
+ def quirks_mode?
+ !!@quirks_mode
+ end
+
+ def reset
+ super
+ @current_nesting = 0
+ end
+
+ # Parses the current JSON string _source_ and returns the complete data
+ # structure as a result.
+ def parse
+ reset
+ obj = nil
+ if @quirks_mode
+ while !eos? && skip(IGNORE)
+ end
+ if eos?
+ raise ParserError, "source did not contain any JSON!"
+ else
+ obj = parse_value
+ obj == UNPARSED and raise ParserError, "source did not contain any JSON!"
+ end
+ else
+ until eos?
+ case
+ when scan(OBJECT_OPEN)
+ obj and raise ParserError, "source '#{peek(20)}' not in JSON!"
+ @current_nesting = 1
+ obj = parse_object
+ when scan(ARRAY_OPEN)
+ obj and raise ParserError, "source '#{peek(20)}' not in JSON!"
+ @current_nesting = 1
+ obj = parse_array
+ when skip(IGNORE)
+ ;
+ else
+ raise ParserError, "source '#{peek(20)}' not in JSON!"
+ end
+ end
+ obj or raise ParserError, "source did not contain any JSON!"
+ end
+ obj
+ end
+
+ private
+
+ def determine_encoding(source)
+ if defined?(::Encoding)
+ if source.encoding == ::Encoding::ASCII_8BIT
+ b = source[0, 4].bytes.to_a
+ source =
+ case
+ when b.size >= 4 && b[0] == 0 && b[1] == 0 && b[2] == 0
+ source.dup.force_encoding(::Encoding::UTF_32BE).encode!(::Encoding::UTF_8)
+ when b.size >= 4 && b[0] == 0 && b[2] == 0
+ source.dup.force_encoding(::Encoding::UTF_16BE).encode!(::Encoding::UTF_8)
+ when b.size >= 4 && b[1] == 0 && b[2] == 0 && b[3] == 0
+ source.dup.force_encoding(::Encoding::UTF_32LE).encode!(::Encoding::UTF_8)
+ when b.size >= 4 && b[1] == 0 && b[3] == 0
+ source.dup.force_encoding(::Encoding::UTF_16LE).encode!(::Encoding::UTF_8)
+ else
+ source.dup
+ end
+ else
+ source = source.encode(::Encoding::UTF_8)
+ end
+ source.force_encoding(::Encoding::ASCII_8BIT)
+ else
+ b = source
+ source =
+ case
+ when b.size >= 4 && b[0] == 0 && b[1] == 0 && b[2] == 0
+ JSON.iconv('utf-8', 'utf-32be', b)
+ when b.size >= 4 && b[0] == 0 && b[2] == 0
+ JSON.iconv('utf-8', 'utf-16be', b)
+ when b.size >= 4 && b[1] == 0 && b[2] == 0 && b[3] == 0
+ JSON.iconv('utf-8', 'utf-32le', b)
+ when b.size >= 4 && b[1] == 0 && b[3] == 0
+ JSON.iconv('utf-8', 'utf-16le', b)
+ else
+ b
+ end
+ end
+ source
+ end
+
+ # Unescape characters in strings.
+ UNESCAPE_MAP = Hash.new { |h, k| h[k] = k.chr }
+ UNESCAPE_MAP.update({
+ ?" => '"',
+ ?\\ => '\\',
+ ?/ => '/',
+ ?b => "\b",
+ ?f => "\f",
+ ?n => "\n",
+ ?r => "\r",
+ ?t => "\t",
+ ?u => nil,
+ })
+
+ EMPTY_8BIT_STRING = ''
+ if ::String.method_defined?(:encode)
+ EMPTY_8BIT_STRING.force_encoding Encoding::ASCII_8BIT
+ end
+
+ def parse_string
+ if scan(STRING)
+ return '' if self[1].empty?
+ string = self[1].gsub(%r((?:\\[\\bfnrt"/]|(?:\\u(?:[A-Fa-f\d]{4}))+|\\[\x20-\xff]))n) do |c|
+ if u = UNESCAPE_MAP[$&[1]]
+ u
+ else # \uXXXX
+ bytes = EMPTY_8BIT_STRING.dup
+ i = 0
+ while c[6 * i] == ?\\ && c[6 * i + 1] == ?u
+ bytes << c[6 * i + 2, 2].to_i(16) << c[6 * i + 4, 2].to_i(16)
+ i += 1
+ end
+ JSON.iconv('utf-8', 'utf-16be', bytes)
+ end
+ end
+ if string.respond_to?(:force_encoding)
+ string.force_encoding(::Encoding::UTF_8)
+ end
+ if @create_additions and @match_string
+ for (regexp, klass) in @match_string
+ klass.json_creatable? or next
+ string =~ regexp and return klass.json_create(string)
+ end
+ end
+ string
+ else
+ UNPARSED
+ end
+ rescue => e
+ raise ParserError, "Caught #{e.class} at '#{peek(20)}': #{e}"
+ end
+
+ def parse_value
+ case
+ when scan(FLOAT)
+ Float(self[1])
+ when scan(INTEGER)
+ Integer(self[1])
+ when scan(TRUE)
+ true
+ when scan(FALSE)
+ false
+ when scan(NULL)
+ nil
+ when (string = parse_string) != UNPARSED
+ string
+ when scan(ARRAY_OPEN)
+ @current_nesting += 1
+ ary = parse_array
+ @current_nesting -= 1
+ ary
+ when scan(OBJECT_OPEN)
+ @current_nesting += 1
+ obj = parse_object
+ @current_nesting -= 1
+ obj
+ when @allow_nan && scan(NAN)
+ NaN
+ when @allow_nan && scan(INFINITY)
+ Infinity
+ when @allow_nan && scan(MINUS_INFINITY)
+ MinusInfinity
+ else
+ UNPARSED
+ end
+ end
+
+ def parse_array
+ raise NestingError, "nesting of #@current_nesting is too deep" if
+ @max_nesting.nonzero? && @current_nesting > @max_nesting
+ result = @array_class.new
+ delim = false
+ until eos?
+ case
+ when (value = parse_value) != UNPARSED
+ delim = false
+ result << value
+ skip(IGNORE)
+ if scan(COLLECTION_DELIMITER)
+ delim = true
+ elsif match?(ARRAY_CLOSE)
+ ;
+ else
+ raise ParserError, "expected ',' or ']' in array at '#{peek(20)}'!"
+ end
+ when scan(ARRAY_CLOSE)
+ if delim
+ raise ParserError, "expected next element in array at '#{peek(20)}'!"
+ end
+ break
+ when skip(IGNORE)
+ ;
+ else
+ raise ParserError, "unexpected token in array at '#{peek(20)}'!"
+ end
+ end
+ result
+ end
+
+ def parse_object
+ raise NestingError, "nesting of #@current_nesting is too deep" if
+ @max_nesting.nonzero? && @current_nesting > @max_nesting
+ result = @object_class.new
+ delim = false
+ until eos?
+ case
+ when (string = parse_string) != UNPARSED
+ skip(IGNORE)
+ unless scan(PAIR_DELIMITER)
+ raise ParserError, "expected ':' in object at '#{peek(20)}'!"
+ end
+ skip(IGNORE)
+ unless (value = parse_value).equal? UNPARSED
+ result[@symbolize_names ? string.to_sym : string] = value
+ delim = false
+ skip(IGNORE)
+ if scan(COLLECTION_DELIMITER)
+ delim = true
+ elsif match?(OBJECT_CLOSE)
+ ;
+ else
+ raise ParserError, "expected ',' or '}' in object at '#{peek(20)}'!"
+ end
+ else
+ raise ParserError, "expected value in object at '#{peek(20)}'!"
+ end
+ when scan(OBJECT_CLOSE)
+ if delim
+ raise ParserError, "expected next name, value pair in object at '#{peek(20)}'!"
+ end
+ if @create_additions and klassname = result[@create_id]
+ klass = JSON.deep_const_get klassname
+ break unless klass and klass.json_creatable?
+ result = klass.json_create(result)
+ end
+ break
+ when skip(IGNORE)
+ ;
+ else
+ raise ParserError, "unexpected token in object at '#{peek(20)}'!"
+ end
+ end
+ result
+ end
+ end
+ end
+end
--- /dev/null
+module JSON
+ # JSON version
+ VERSION = '1.5.5'
+ VERSION_ARRAY = VERSION.split(/\./).map { |x| x.to_i } # :nodoc:
+ VERSION_MAJOR = VERSION_ARRAY[0] # :nodoc:
+ VERSION_MINOR = VERSION_ARRAY[1] # :nodoc:
+ VERSION_BUILD = VERSION_ARRAY[2] # :nodoc:
+end
--- /dev/null
+"A JSON payload should be an object or array, not a string."
\ No newline at end of file
--- /dev/null
+{"Extra value after close": true} "misplaced quoted value"
\ No newline at end of file
--- /dev/null
+{"Illegal expression": 1 + 2}
\ No newline at end of file
--- /dev/null
+{"Illegal invocation": alert()}
\ No newline at end of file
--- /dev/null
+{"Numbers cannot have leading zeroes": 013}
\ No newline at end of file
--- /dev/null
+{"Numbers cannot be hex": 0x14}
\ No newline at end of file
--- /dev/null
+[[[[[[[[[[[[[[[[[[[["Too deep"]]]]]]]]]]]]]]]]]]]]
--- /dev/null
+{"Missing colon" null}
\ No newline at end of file
--- /dev/null
+["Unclosed array"
\ No newline at end of file
--- /dev/null
+{"Double colon":: null}
\ No newline at end of file
--- /dev/null
+{"Comma instead of colon", null}
\ No newline at end of file
--- /dev/null
+["Colon instead of comma": false]
\ No newline at end of file
--- /dev/null
+["Bad value", truth]
\ No newline at end of file
--- /dev/null
+['single quote']
\ No newline at end of file
--- /dev/null
+["tab character in string "]
--- /dev/null
+["line
+break"]
\ No newline at end of file
--- /dev/null
+["line\
+break"]
\ No newline at end of file
--- /dev/null
+{unquoted_key: "keys must be quoted"}
\ No newline at end of file
--- /dev/null
+["extra comma",]
\ No newline at end of file
--- /dev/null
+["double extra comma",,]
\ No newline at end of file
--- /dev/null
+[ , "<-- missing value"]
\ No newline at end of file
--- /dev/null
+["Comma after the close"],
\ No newline at end of file
--- /dev/null
+["Extra close"]]
\ No newline at end of file
--- /dev/null
+{"Extra comma": true,}
\ No newline at end of file
--- /dev/null
+[
+ "JSON Test Pattern pass1",
+ {"object with 1 member":["array with 1 element"]},
+ {},
+ [],
+ -42,
+ true,
+ false,
+ null,
+ {
+ "integer": 1234567890,
+ "real": -9876.543210,
+ "e": 0.123456789e-12,
+ "E": 1.234567890E+34,
+ "": 23456789012E666,
+ "zero": 0,
+ "one": 1,
+ "space": " ",
+ "quote": "\"",
+ "backslash": "\\",
+ "controls": "\b\f\n\r\t",
+ "slash": "/ & \/",
+ "alpha": "abcdefghijklmnopqrstuvwyz",
+ "ALPHA": "ABCDEFGHIJKLMNOPQRSTUVWYZ",
+ "digit": "0123456789",
+ "special": "`1~!@#$%^&*()_+-={':[,]}|;.</>?",
+ "hex": "\u0123\u4567\u89AB\uCDEF\uabcd\uef4A",
+ "true": true,
+ "false": false,
+ "null": null,
+ "array":[ ],
+ "object":{ },
+ "address": "50 St. James Street",
+ "url": "http://www.JSON.org/",
+ "comment": "// /* <!-- --",
+ "# -- --> */": " ",
+ " s p a c e d " :[1,2 , 3
+
+,
+
+4 , 5 , 6 ,7 ],
+ "compact": [1,2,3,4,5,6,7],
+ "jsontext": "{\"object with 1 member\":[\"array with 1 element\"]}",
+ "quotes": "" \u0022 %22 0x22 034 "",
+ "\/\\\"\uCAFE\uBABE\uAB98\uFCDE\ubcda\uef4A\b\f\n\r\t`1~!@#$%^&*()_+-=[]{}|;:',./<>?"
+: "A key can be any string"
+ },
+ 0.5 ,98.6
+,
+99.44
+,
+
+1066
+
+
+,"rosebud"]
\ No newline at end of file
--- /dev/null
+["Illegal backslash escape: \x15"]
\ No newline at end of file
--- /dev/null
+["Illegal backslash escape: \'"]
\ No newline at end of file
--- /dev/null
+["Illegal backslash escape: \017"]
\ No newline at end of file
--- /dev/null
+[[[[[[[[[[[[[[[[[[["Not too deep"]]]]]]]]]]]]]]]]]]]
\ No newline at end of file
--- /dev/null
+["tab\ character\ in\ string\ "]
\ No newline at end of file
--- /dev/null
+{
+ "JSON Test Pattern pass3": {
+ "The outermost value": "must be an object or array.",
+ "In this test": "It is an object."
+ }
+}
--- /dev/null
+case ENV['JSON']
+when 'pure'
+ $:.unshift 'lib'
+ require 'json/pure'
+when 'ext'
+ $:.unshift 'ext', 'lib'
+ require 'json/ext'
+else
+ $:.unshift 'ext', 'lib'
+ require 'json'
+end
--- /dev/null
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+
+require 'test/unit'
+require File.join(File.dirname(__FILE__), 'setup_variant')
+require 'stringio'
+require 'tempfile'
+
+unless Array.method_defined?(:permutation)
+ begin
+ require 'enumerator'
+ require 'permutation'
+ class Array
+ def permutation
+ Permutation.for(self).to_enum.map { |x| x.project }
+ end
+ end
+ rescue LoadError
+ warn "Skipping permutation tests."
+ end
+end
+
+class TC_JSON < Test::Unit::TestCase
+ include JSON
+
+ def setup
+ @ary = [1, "foo", 3.14, 4711.0, 2.718, nil, [1,-2,3], false, true].map do
+ |x| [x]
+ end
+ @ary_to_parse = ["1", '"foo"', "3.14", "4711.0", "2.718", "null",
+ "[1,-2,3]", "false", "true"].map do
+ |x| "[#{x}]"
+ end
+ @hash = {
+ 'a' => 2,
+ 'b' => 3.141,
+ 'c' => 'c',
+ 'd' => [ 1, "b", 3.14 ],
+ 'e' => { 'foo' => 'bar' },
+ 'g' => "\"\0\037",
+ 'h' => 1000.0,
+ 'i' => 0.001
+ }
+ @json = '{"a":2,"b":3.141,"c":"c","d":[1,"b",3.14],"e":{"foo":"bar"},'\
+ '"g":"\\"\\u0000\\u001f","h":1.0E3,"i":1.0E-3}'
+ end
+
+ def test_construction
+ parser = JSON::Parser.new('test')
+ assert_equal 'test', parser.source
+ end
+
+ def assert_equal_float(expected, is)
+ assert_in_delta(expected.first, is.first, 1e-2)
+ end
+
+ def test_parse_simple_arrays
+ assert_equal([], parse('[]'))
+ assert_equal([], parse(' [ ] '))
+ assert_equal([nil], parse('[null]'))
+ assert_equal([false], parse('[false]'))
+ assert_equal([true], parse('[true]'))
+ assert_equal([-23], parse('[-23]'))
+ assert_equal([23], parse('[23]'))
+ assert_equal([0.23], parse('[0.23]'))
+ assert_equal([0.0], parse('[0e0]'))
+ assert_raises(JSON::ParserError) { parse('[+23.2]') }
+ assert_raises(JSON::ParserError) { parse('[+23]') }
+ assert_raises(JSON::ParserError) { parse('[.23]') }
+ assert_raises(JSON::ParserError) { parse('[023]') }
+ assert_equal_float [3.141], parse('[3.141]')
+ assert_equal_float [-3.141], parse('[-3.141]')
+ assert_equal_float [3.141], parse('[3141e-3]')
+ assert_equal_float [3.141], parse('[3141.1e-3]')
+ assert_equal_float [3.141], parse('[3141E-3]')
+ assert_equal_float [3.141], parse('[3141.0E-3]')
+ assert_equal_float [-3.141], parse('[-3141.0e-3]')
+ assert_equal_float [-3.141], parse('[-3141e-3]')
+ assert_raises(ParserError) { parse('[NaN]') }
+ assert parse('[NaN]', :allow_nan => true).first.nan?
+ assert_raises(ParserError) { parse('[Infinity]') }
+ assert_equal [1.0/0], parse('[Infinity]', :allow_nan => true)
+ assert_raises(ParserError) { parse('[-Infinity]') }
+ assert_equal [-1.0/0], parse('[-Infinity]', :allow_nan => true)
+ assert_equal([""], parse('[""]'))
+ assert_equal(["foobar"], parse('["foobar"]'))
+ assert_equal([{}], parse('[{}]'))
+ end
+
+ def test_parse_simple_objects
+ assert_equal({}, parse('{}'))
+ assert_equal({}, parse(' { } '))
+ assert_equal({ "a" => nil }, parse('{ "a" : null}'))
+ assert_equal({ "a" => nil }, parse('{"a":null}'))
+ assert_equal({ "a" => false }, parse('{ "a" : false } '))
+ assert_equal({ "a" => false }, parse('{"a":false}'))
+ assert_raises(JSON::ParserError) { parse('{false}') }
+ assert_equal({ "a" => true }, parse('{"a":true}'))
+ assert_equal({ "a" => true }, parse(' { "a" : true } '))
+ assert_equal({ "a" => -23 }, parse(' { "a" : -23 } '))
+ assert_equal({ "a" => -23 }, parse(' { "a" : -23 } '))
+ assert_equal({ "a" => 23 }, parse('{"a":23 } '))
+ assert_equal({ "a" => 23 }, parse(' { "a" : 23 } '))
+ assert_equal({ "a" => 0.23 }, parse(' { "a" : 0.23 } '))
+ assert_equal({ "a" => 0.23 }, parse(' { "a" : 0.23 } '))
+ end
+
+ def test_parse_json_primitive_values
+ assert_raise(JSON::ParserError) { JSON.parse('') }
+ assert_raise(JSON::ParserError) { JSON.parse('', :quirks_mode => true) }
+ assert_raise(JSON::ParserError) { JSON.parse(' /* foo */ ') }
+ assert_raise(JSON::ParserError) { JSON.parse(' /* foo */ ', :quirks_mode => true) }
+ parser = JSON::Parser.new('null')
+ assert_equal false, parser.quirks_mode?
+ assert_raise(JSON::ParserError) { parser.parse }
+ assert_raise(JSON::ParserError) { JSON.parse('null') }
+ assert_equal nil, JSON.parse('null', :quirks_mode => true)
+ parser = JSON::Parser.new('null', :quirks_mode => true)
+ assert_equal true, parser.quirks_mode?
+ assert_equal nil, parser.parse
+ assert_raise(JSON::ParserError) { JSON.parse('false') }
+ assert_equal false, JSON.parse('false', :quirks_mode => true)
+ assert_raise(JSON::ParserError) { JSON.parse('true') }
+ assert_equal true, JSON.parse('true', :quirks_mode => true)
+ assert_raise(JSON::ParserError) { JSON.parse('23') }
+ assert_equal 23, JSON.parse('23', :quirks_mode => true)
+ assert_raise(JSON::ParserError) { JSON.parse('1') }
+ assert_equal 1, JSON.parse('1', :quirks_mode => true)
+ assert_raise(JSON::ParserError) { JSON.parse('3.141') }
+ assert_in_delta 3.141, JSON.parse('3.141', :quirks_mode => true), 1E-3
+ assert_raise(JSON::ParserError) { JSON.parse('18446744073709551616') }
+ assert_equal 2 ** 64, JSON.parse('18446744073709551616', :quirks_mode => true)
+ assert_raise(JSON::ParserError) { JSON.parse('"foo"') }
+ assert_equal 'foo', JSON.parse('"foo"', :quirks_mode => true)
+ assert_raise(JSON::ParserError) { JSON.parse('NaN', :allow_nan => true) }
+ assert JSON.parse('NaN', :quirks_mode => true, :allow_nan => true).nan?
+ assert_raise(JSON::ParserError) { JSON.parse('Infinity', :allow_nan => true) }
+ assert JSON.parse('Infinity', :quirks_mode => true, :allow_nan => true).infinite?
+ assert_raise(JSON::ParserError) { JSON.parse('-Infinity', :allow_nan => true) }
+ assert JSON.parse('-Infinity', :quirks_mode => true, :allow_nan => true).infinite?
+ assert_raise(JSON::ParserError) { JSON.parse('[ 1, ]', :quirks_mode => true) }
+ end
+
+ if Array.method_defined?(:permutation)
+ def test_parse_more_complex_arrays
+ a = [ nil, false, true, "foßbar", [ "n€st€d", true ], { "nested" => true, "n€ßt€ð2" => {} }]
+ a.permutation.each do |perm|
+ json = pretty_generate(perm)
+ assert_equal perm, parse(json)
+ end
+ end
+
+ def test_parse_complex_objects
+ a = [ nil, false, true, "foßbar", [ "n€st€d", true ], { "nested" => true, "n€ßt€ð2" => {} }]
+ a.permutation.each do |perm|
+ s = "a"
+ orig_obj = perm.inject({}) { |h, x| h[s.dup] = x; s = s.succ; h }
+ json = pretty_generate(orig_obj)
+ assert_equal orig_obj, parse(json)
+ end
+ end
+ end
+
+ def test_parse_arrays
+ assert_equal([1,2,3], parse('[1,2,3]'))
+ assert_equal([1.2,2,3], parse('[1.2,2,3]'))
+ assert_equal([[],[[],[]]], parse('[[],[[],[]]]'))
+ end
+
+ def test_parse_values
+ assert_equal([""], parse('[""]'))
+ assert_equal(["\\"], parse('["\\\\"]'))
+ assert_equal(['"'], parse('["\""]'))
+ assert_equal(['\\"\\'], parse('["\\\\\\"\\\\"]'))
+ assert_equal(["\"\b\n\r\t\0\037"],
+ parse('["\"\b\n\r\t\u0000\u001f"]'))
+ for i in 0 ... @ary.size
+ assert_equal(@ary[i], parse(@ary_to_parse[i]))
+ end
+ end
+
+ def test_parse_array
+ assert_equal([], parse('[]'))
+ assert_equal([], parse(' [ ] '))
+ assert_equal([1], parse('[1]'))
+ assert_equal([1], parse(' [ 1 ] '))
+ assert_equal(@ary,
+ parse('[[1],["foo"],[3.14],[47.11e+2],[2718.0E-3],[null],[[1,-2,3]]'\
+ ',[false],[true]]'))
+ assert_equal(@ary, parse(%Q{ [ [1] , ["foo"] , [3.14] \t , [47.11e+2]\s
+ , [2718.0E-3 ],\r[ null] , [[1, -2, 3 ]], [false ],[ true]\n ] }))
+ end
+
+ class SubArray < Array
+ def <<(v)
+ @shifted = true
+ super
+ end
+
+ def shifted?
+ @shifted
+ end
+ end
+
+ class SubArray2 < Array
+ def to_json(*a)
+ {
+ JSON.create_id => self.class.name,
+ 'ary' => to_a,
+ }.to_json(*a)
+ end
+
+ def self.json_create(o)
+ o.delete JSON.create_id
+ o['ary']
+ end
+ end
+
+ def test_parse_array_custom_class
+ res = parse('[1,2]', :array_class => SubArray)
+ assert_equal([1,2], res)
+ assert_equal(SubArray, res.class)
+ assert res.shifted?
+ end
+
+ def test_parse_object
+ assert_equal({}, parse('{}'))
+ assert_equal({}, parse(' { } '))
+ assert_equal({'foo'=>'bar'}, parse('{"foo":"bar"}'))
+ assert_equal({'foo'=>'bar'}, parse(' { "foo" : "bar" } '))
+ end
+
+ class SubHash < Hash
+ def []=(k, v)
+ @item_set = true
+ super
+ end
+
+ def item_set?
+ @item_set
+ end
+ end
+
+ class SubHash2 < Hash
+ def to_json(*a)
+ {
+ JSON.create_id => self.class.name,
+ }.merge(self).to_json(*a)
+ end
+
+ def self.json_create(o)
+ o.delete JSON.create_id
+ self[o]
+ end
+ end
+
+ def test_parse_object_custom_class
+ res = parse('{"foo":"bar"}', :object_class => SubHash)
+ assert_equal({"foo" => "bar"}, res)
+ assert_equal(SubHash, res.class)
+ assert res.item_set?
+ end
+
+ def test_generation_of_core_subclasses_with_new_to_json
+ obj = SubHash2["foo" => SubHash2["bar" => true]]
+ obj_json = JSON(obj)
+ obj_again = JSON.parse(obj_json, :create_additions => true)
+ assert_kind_of SubHash2, obj_again
+ assert_kind_of SubHash2, obj_again['foo']
+ assert obj_again['foo']['bar']
+ assert_equal obj, obj_again
+ assert_equal ["foo"], JSON(JSON(SubArray2["foo"]), :create_additions => true)
+ end
+
+ def test_generation_of_core_subclasses_with_default_to_json
+ assert_equal '{"foo":"bar"}', JSON(SubHash["foo" => "bar"])
+ assert_equal '["foo"]', JSON(SubArray["foo"])
+ end
+
+ def test_generation_of_core_subclasses
+ obj = SubHash["foo" => SubHash["bar" => true]]
+ obj_json = JSON(obj)
+ obj_again = JSON(obj_json)
+ assert_kind_of Hash, obj_again
+ assert_kind_of Hash, obj_again['foo']
+ assert obj_again['foo']['bar']
+ assert_equal obj, obj_again
+ end
+
+ def test_parser_reset
+ parser = Parser.new(@json)
+ assert_equal(@hash, parser.parse)
+ assert_equal(@hash, parser.parse)
+ end
+
+ def test_comments
+ json = <<EOT
+{
+ "key1":"value1", // eol comment
+ "key2":"value2" /* multi line
+ * comment */,
+ "key3":"value3" /* multi line
+ // nested eol comment
+ * comment */
+}
+EOT
+ assert_equal(
+ { "key1" => "value1", "key2" => "value2", "key3" => "value3" },
+ parse(json))
+ json = <<EOT
+{
+ "key1":"value1" /* multi line
+ // nested eol comment
+ /* illegal nested multi line comment */
+ * comment */
+}
+EOT
+ assert_raises(ParserError) { parse(json) }
+ json = <<EOT
+{
+ "key1":"value1" /* multi line
+ // nested eol comment
+ closed multi comment */
+ and again, throw an Error */
+}
+EOT
+ assert_raises(ParserError) { parse(json) }
+ json = <<EOT
+{
+ "key1":"value1" /*/*/
+}
+EOT
+ assert_equal({ "key1" => "value1" }, parse(json))
+ end
+
+ def test_backslash
+ data = [ '\\.(?i:gif|jpe?g|png)$' ]
+ json = '["\\\\.(?i:gif|jpe?g|png)$"]'
+ assert_equal json, JSON.generate(data)
+ assert_equal data, JSON.parse(json)
+ #
+ data = [ '\\"' ]
+ json = '["\\\\\""]'
+ assert_equal json, JSON.generate(data)
+ assert_equal data, JSON.parse(json)
+ #
+ json = '["/"]'
+ data = JSON.parse(json)
+ assert_equal ['/'], data
+ assert_equal json, JSON.generate(data)
+ #
+ json = '["\""]'
+ data = JSON.parse(json)
+ assert_equal ['"'], data
+ assert_equal json, JSON.generate(data)
+ json = '["\\\'"]'
+ data = JSON.parse(json)
+ assert_equal ["'"], data
+ assert_equal '["\'"]', JSON.generate(data)
+ end
+
+ def test_wrong_inputs
+ assert_raises(ParserError) { JSON.parse('"foo"') }
+ assert_raises(ParserError) { JSON.parse('123') }
+ assert_raises(ParserError) { JSON.parse('[] bla') }
+ assert_raises(ParserError) { JSON.parse('[] 1') }
+ assert_raises(ParserError) { JSON.parse('[] []') }
+ assert_raises(ParserError) { JSON.parse('[] {}') }
+ assert_raises(ParserError) { JSON.parse('{} []') }
+ assert_raises(ParserError) { JSON.parse('{} {}') }
+ assert_raises(ParserError) { JSON.parse('[NULL]') }
+ assert_raises(ParserError) { JSON.parse('[FALSE]') }
+ assert_raises(ParserError) { JSON.parse('[TRUE]') }
+ assert_raises(ParserError) { JSON.parse('[07] ') }
+ assert_raises(ParserError) { JSON.parse('[0a]') }
+ assert_raises(ParserError) { JSON.parse('[1.]') }
+ assert_raises(ParserError) { JSON.parse(' ') }
+ end
+
+ def test_nesting
+ assert_raises(JSON::NestingError) { JSON.parse '[[]]', :max_nesting => 1 }
+ assert_raises(JSON::NestingError) { JSON.parser.new('[[]]', :max_nesting => 1).parse }
+ assert_equal [[]], JSON.parse('[[]]', :max_nesting => 2)
+ too_deep = '[[[[[[[[[[[[[[[[[[[["Too deep"]]]]]]]]]]]]]]]]]]]]'
+ too_deep_ary = eval too_deep
+ assert_raises(JSON::NestingError) { JSON.parse too_deep }
+ assert_raises(JSON::NestingError) { JSON.parser.new(too_deep).parse }
+ assert_raises(JSON::NestingError) { JSON.parse too_deep, :max_nesting => 19 }
+ ok = JSON.parse too_deep, :max_nesting => 20
+ assert_equal too_deep_ary, ok
+ ok = JSON.parse too_deep, :max_nesting => nil
+ assert_equal too_deep_ary, ok
+ ok = JSON.parse too_deep, :max_nesting => false
+ assert_equal too_deep_ary, ok
+ ok = JSON.parse too_deep, :max_nesting => 0
+ assert_equal too_deep_ary, ok
+ assert_raises(JSON::NestingError) { JSON.generate [[]], :max_nesting => 1 }
+ assert_equal '[[]]', JSON.generate([[]], :max_nesting => 2)
+ assert_raises(JSON::NestingError) { JSON.generate too_deep_ary }
+ assert_raises(JSON::NestingError) { JSON.generate too_deep_ary, :max_nesting => 19 }
+ ok = JSON.generate too_deep_ary, :max_nesting => 20
+ assert_equal too_deep, ok
+ ok = JSON.generate too_deep_ary, :max_nesting => nil
+ assert_equal too_deep, ok
+ ok = JSON.generate too_deep_ary, :max_nesting => false
+ assert_equal too_deep, ok
+ ok = JSON.generate too_deep_ary, :max_nesting => 0
+ assert_equal too_deep, ok
+ end
+
+ def test_symbolize_names
+ assert_equal({ "foo" => "bar", "baz" => "quux" },
+ JSON.parse('{"foo":"bar", "baz":"quux"}'))
+ assert_equal({ :foo => "bar", :baz => "quux" },
+ JSON.parse('{"foo":"bar", "baz":"quux"}', :symbolize_names => true))
+ end
+
+ def test_load
+ assert_equal @hash, JSON.load(@json)
+ tempfile = Tempfile.open('json')
+ tempfile.write @json
+ tempfile.rewind
+ assert_equal @hash, JSON.load(tempfile)
+ stringio = StringIO.new(@json)
+ stringio.rewind
+ assert_equal @hash, JSON.load(stringio)
+ assert_raise(NoMethodError) { JSON.load(nil) }
+ assert_raise(JSON::ParserError) {JSON.load('') }
+ end
+
+ def test_load_with_options
+ small_hash = JSON("foo" => 'bar')
+ symbol_hash = { :foo => 'bar' }
+ assert_equal symbol_hash, JSON.load(small_hash, nil, :symbolize_names => true)
+ end
+
+ def test_load_dump
+ too_deep = '[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]'
+ assert_equal too_deep, JSON.dump(eval(too_deep))
+ assert_kind_of String, Marshal.dump(eval(too_deep))
+ assert_raises(ArgumentError) { JSON.dump(eval(too_deep), 19) }
+ assert_raises(ArgumentError) { Marshal.dump(eval(too_deep), 19) }
+ assert_equal too_deep, JSON.dump(eval(too_deep), 20)
+ assert_kind_of String, Marshal.dump(eval(too_deep), 20)
+ output = StringIO.new
+ JSON.dump(eval(too_deep), output)
+ assert_equal too_deep, output.string
+ output = StringIO.new
+ JSON.dump(eval(too_deep), output, 20)
+ assert_equal too_deep, output.string
+ end
+
+ def test_big_integers
+ json1 = JSON([orig = (1 << 31) - 1])
+ assert_equal orig, JSON[json1][0]
+ json2 = JSON([orig = 1 << 31])
+ assert_equal orig, JSON[json2][0]
+ json3 = JSON([orig = (1 << 62) - 1])
+ assert_equal orig, JSON[json3][0]
+ json4 = JSON([orig = 1 << 62])
+ assert_equal orig, JSON[json4][0]
+ json5 = JSON([orig = 1 << 64])
+ assert_equal orig, JSON[json5][0]
+ end
+
+ if defined?(JSON::Ext::Parser)
+ def test_allocate
+ parser = JSON::Ext::Parser.new("{}")
+ assert_raise(TypeError, '[ruby-core:35079]') {parser.__send__(:initialize, "{}")}
+ parser = JSON::Ext::Parser.allocate
+ assert_raise(TypeError, '[ruby-core:35079]') {parser.source}
+ end
+ end
+
+ def test_argument_encoding
+ source = "{}".force_encoding("ascii-8bit")
+ JSON::Parser.new(source)
+ assert_equal Encoding::ASCII_8BIT, source.encoding
+ end if defined?(Encoding::ASCII_8BIT)
+end
--- /dev/null
+#!/usr/bin/env ruby
+# -*- coding:utf-8 -*-
+
+require 'test/unit'
+require File.join(File.dirname(__FILE__), 'setup_variant')
+require 'json/add/core'
+require 'json/add/complex'
+require 'json/add/rational'
+require 'date'
+
+class TC_JSONAddition < Test::Unit::TestCase
+ include JSON
+
+ class A
+ def initialize(a)
+ @a = a
+ end
+
+ attr_reader :a
+
+ def ==(other)
+ a == other.a
+ end
+
+ def self.json_create(object)
+ new(*object['args'])
+ end
+
+ def to_json(*args)
+ {
+ 'json_class' => self.class.name,
+ 'args' => [ @a ],
+ }.to_json(*args)
+ end
+ end
+
+ class A2 < A
+ def to_json(*args)
+ {
+ 'json_class' => self.class.name,
+ 'args' => [ @a ],
+ }.to_json(*args)
+ end
+ end
+
+ class B
+ def self.json_creatable?
+ false
+ end
+
+ def to_json(*args)
+ {
+ 'json_class' => self.class.name,
+ }.to_json(*args)
+ end
+ end
+
+ class C
+ def self.json_creatable?
+ false
+ end
+
+ def to_json(*args)
+ {
+ 'json_class' => 'TC_JSONAddition::Nix',
+ }.to_json(*args)
+ end
+ end
+
+ def test_extended_json
+ a = A.new(666)
+ assert A.json_creatable?
+ json = generate(a)
+ a_again = JSON.parse(json, :create_additions => true)
+ assert_kind_of a.class, a_again
+ assert_equal a, a_again
+ end
+
+ def test_extended_json_default
+ a = A.new(666)
+ assert A.json_creatable?
+ json = generate(a)
+ a_hash = JSON.parse(json)
+ assert_kind_of Hash, a_hash
+ end
+
+ def test_extended_json_disabled
+ a = A.new(666)
+ assert A.json_creatable?
+ json = generate(a)
+ a_again = JSON.parse(json, :create_additions => true)
+ assert_kind_of a.class, a_again
+ assert_equal a, a_again
+ a_hash = JSON.parse(json, :create_additions => false)
+ assert_kind_of Hash, a_hash
+ assert_equal(
+ {"args"=>[666], "json_class"=>"TC_JSONAddition::A"}.sort_by { |k,| k },
+ a_hash.sort_by { |k,| k }
+ )
+ end
+
+ def test_extended_json_fail1
+ b = B.new
+ assert !B.json_creatable?
+ json = generate(b)
+ assert_equal({ "json_class"=>"TC_JSONAddition::B" }, JSON.parse(json))
+ end
+
+ def test_extended_json_fail2
+ c = C.new
+ assert !C.json_creatable?
+ json = generate(c)
+ assert_raises(ArgumentError, NameError) { JSON.parse(json, :create_additions => true) }
+ end
+
+ def test_raw_strings
+ raw = ''
+ raw.respond_to?(:encode!) and raw.encode!(Encoding::ASCII_8BIT)
+ raw_array = []
+ for i in 0..255
+ raw << i
+ raw_array << i
+ end
+ json = raw.to_json_raw
+ json_raw_object = raw.to_json_raw_object
+ hash = { 'json_class' => 'String', 'raw'=> raw_array }
+ assert_equal hash, json_raw_object
+ assert_match(/\A\{.*\}\Z/, json)
+ assert_match(/"json_class":"String"/, json)
+ assert_match(/"raw":\[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255\]/, json)
+ raw_again = JSON.parse(json, :create_additions => true)
+ assert_equal raw, raw_again
+ end
+
+ MyJsonStruct = Struct.new 'MyJsonStruct', :foo, :bar
+
+ def test_core
+ t = Time.now
+ assert_equal t, JSON(JSON(t), :create_additions => true)
+ d = Date.today
+ assert_equal d, JSON(JSON(d), :create_additions => true)
+ d = DateTime.civil(2007, 6, 14, 14, 57, 10, Rational(1, 12), 2299161)
+ assert_equal d, JSON(JSON(d), :create_additions => true)
+ assert_equal 1..10, JSON(JSON(1..10), :create_additions => true)
+ assert_equal 1...10, JSON(JSON(1...10), :create_additions => true)
+ assert_equal "a".."c", JSON(JSON("a".."c"), :create_additions => true)
+ assert_equal "a"..."c", JSON(JSON("a"..."c"), :create_additions => true)
+ s = MyJsonStruct.new 4711, 'foot'
+ assert_equal s, JSON(JSON(s), :create_additions => true)
+ struct = Struct.new :foo, :bar
+ s = struct.new 4711, 'foot'
+ assert_raises(JSONError) { JSON(s) }
+ begin
+ raise TypeError, "test me"
+ rescue TypeError => e
+ e_json = JSON.generate e
+ e_again = JSON e_json, :create_additions => true
+ assert_kind_of TypeError, e_again
+ assert_equal e.message, e_again.message
+ assert_equal e.backtrace, e_again.backtrace
+ end
+ assert_equal(/foo/, JSON(JSON(/foo/), :create_additions => true))
+ assert_equal(/foo/i, JSON(JSON(/foo/i), :create_additions => true))
+ end
+
+ def test_utc_datetime
+ now = Time.now
+ d = DateTime.parse(now.to_s, :create_additions => true) # usual case
+ assert_equal d, JSON.parse(d.to_json, :create_additions => true)
+ d = DateTime.parse(now.utc.to_s) # of = 0
+ assert_equal d, JSON.parse(d.to_json, :create_additions => true)
+ d = DateTime.civil(2008, 6, 17, 11, 48, 32, Rational(1,24))
+ assert_equal d, JSON.parse(d.to_json, :create_additions => true)
+ d = DateTime.civil(2008, 6, 17, 11, 48, 32, Rational(12,24))
+ assert_equal d, JSON.parse(d.to_json, :create_additions => true)
+ end
+
+ def test_rational_complex
+ assert_equal Rational(2, 9), JSON.parse(JSON(Rational(2, 9)), :create_additions => true)
+ assert_equal Complex(2, 9), JSON.parse(JSON(Complex(2, 9)), :create_additions => true)
+ end
+end
--- /dev/null
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+
+require 'test/unit'
+require File.join(File.dirname(__FILE__), 'setup_variant')
+
+class TC_JSONEncoding < Test::Unit::TestCase
+ include JSON
+
+ def setup
+ @utf_8 = '["© ≠ €!"]'
+ @parsed = [ "© ≠ €!" ]
+ @generated = '["\u00a9 \u2260 \u20ac!"]'
+ if String.method_defined?(:encode)
+ @utf_16_data = [@parsed.first.encode('utf-16be', 'utf-8')]
+ @utf_8_ascii_8bit = @utf_8.dup.force_encoding(Encoding::ASCII_8BIT)
+ @utf_16be = @utf_8.encode('utf-16be', 'utf-8')
+ @utf_16be_ascii_8bit = @utf_16be.dup.force_encoding(Encoding::ASCII_8BIT)
+ @utf_16le = @utf_8.encode('utf-16le', 'utf-8')
+ @utf_16le_ascii_8bit = @utf_16le.dup.force_encoding(Encoding::ASCII_8BIT)
+ @utf_32be = @utf_8.encode('utf-32be', 'utf-8')
+ @utf_32be_ascii_8bit = @utf_32be.dup.force_encoding(Encoding::ASCII_8BIT)
+ @utf_32le = @utf_8.encode('utf-32le', 'utf-8')
+ @utf_32le_ascii_8bit = @utf_32le.dup.force_encoding(Encoding::ASCII_8BIT)
+ else
+ require 'iconv'
+ @utf_16_data = Iconv.iconv('utf-16be', 'utf-8', @parsed.first)
+ @utf_8_ascii_8bit = @utf_8.dup
+ @utf_16be, = Iconv.iconv('utf-16be', 'utf-8', @utf_8)
+ @utf_16be_ascii_8bit = @utf_16be.dup
+ @utf_16le, = Iconv.iconv('utf-16le', 'utf-8', @utf_8)
+ @utf_16le_ascii_8bit = @utf_16le.dup
+ @utf_32be, = Iconv.iconv('utf-32be', 'utf-8', @utf_8)
+ @utf_32be_ascii_8bit = @utf_32be.dup
+ @utf_32le, = Iconv.iconv('utf-32le', 'utf-8', @utf_8)
+ @utf_32le_ascii_8bit = @utf_32le.dup
+ end
+ end
+
+ def test_parse
+ assert_equal @parsed, JSON.parse(@utf_8)
+ assert_equal @parsed, JSON.parse(@utf_16be)
+ assert_equal @parsed, JSON.parse(@utf_16le)
+ assert_equal @parsed, JSON.parse(@utf_32be)
+ assert_equal @parsed, JSON.parse(@utf_32le)
+ end
+
+ def test_parse_ascii_8bit
+ assert_equal @parsed, JSON.parse(@utf_8_ascii_8bit)
+ assert_equal @parsed, JSON.parse(@utf_16be_ascii_8bit)
+ assert_equal @parsed, JSON.parse(@utf_16le_ascii_8bit)
+ assert_equal @parsed, JSON.parse(@utf_32be_ascii_8bit)
+ assert_equal @parsed, JSON.parse(@utf_32le_ascii_8bit)
+ end
+
+ def test_generate
+ assert_equal @generated, JSON.generate(@parsed, :ascii_only => true)
+ if defined?(::Encoding)
+ assert_equal @generated, JSON.generate(@utf_16_data, :ascii_only => true)
+ else
+ # XXX checking of correct utf8 data is not as strict (yet?) without :ascii_only
+ assert_raises(JSON::GeneratorError) { JSON.generate(@utf_16_data, :ascii_only => true) }
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+
+require 'test/unit'
+require File.join(File.dirname(__FILE__), 'setup_variant')
+
+class TC_JSONFixtures < Test::Unit::TestCase
+ def setup
+ fixtures = File.join(File.dirname(__FILE__), 'fixtures/*.json')
+ passed, failed = Dir[fixtures].partition { |f| f['pass'] }
+ @passed = passed.inject([]) { |a, f| a << [ f, File.read(f) ] }.sort
+ @failed = failed.inject([]) { |a, f| a << [ f, File.read(f) ] }.sort
+ end
+
+ def test_passing
+ for name, source in @passed
+ begin
+ assert JSON.parse(source),
+ "Did not pass for fixture '#{name}': #{source.inspect}"
+ rescue => e
+ warn "\nCaught #{e.class}(#{e}) for fixture '#{name}': #{source.inspect}\n#{e.backtrace * "\n"}"
+ raise e
+ end
+ end
+ end
+
+ def test_failing
+ for name, source in @failed
+ assert_raises(JSON::ParserError, JSON::NestingError,
+ "Did not fail for fixture '#{name}': #{source.inspect}") do
+ JSON.parse(source)
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+
+require 'test/unit'
+require File.join(File.dirname(__FILE__), 'setup_variant')
+
+class TC_JSONGenerate < Test::Unit::TestCase
+ include JSON
+
+ def setup
+ @hash = {
+ 'a' => 2,
+ 'b' => 3.141,
+ 'c' => 'c',
+ 'd' => [ 1, "b", 3.14 ],
+ 'e' => { 'foo' => 'bar' },
+ 'g' => "\"\0\037",
+ 'h' => 1000.0,
+ 'i' => 0.001
+ }
+ @json2 = '{"a":2,"b":3.141,"c":"c","d":[1,"b",3.14],"e":{"foo":"bar"},' +
+ '"g":"\\"\\u0000\\u001f","h":1000.0,"i":0.001}'
+ @json3 = <<'EOT'.chomp
+{
+ "a": 2,
+ "b": 3.141,
+ "c": "c",
+ "d": [
+ 1,
+ "b",
+ 3.14
+ ],
+ "e": {
+ "foo": "bar"
+ },
+ "g": "\"\u0000\u001f",
+ "h": 1000.0,
+ "i": 0.001
+}
+EOT
+ end
+
+ def test_generate
+ json = generate(@hash)
+ assert_equal(JSON.parse(@json2), JSON.parse(json))
+ parsed_json = parse(json)
+ assert_equal(@hash, parsed_json)
+ json = generate({1=>2})
+ assert_equal('{"1":2}', json)
+ parsed_json = parse(json)
+ assert_equal({"1"=>2}, parsed_json)
+ assert_raise(GeneratorError) { generate(666) }
+ assert_equal '666', generate(666, :quirks_mode => true)
+ end
+
+ def test_generate_pretty
+ json = pretty_generate(@hash)
+ # hashes aren't (insertion) ordered on every ruby implementation assert_equal(@json3, json)
+ assert_equal(JSON.parse(@json3), JSON.parse(json))
+ parsed_json = parse(json)
+ assert_equal(@hash, parsed_json)
+ json = pretty_generate({1=>2})
+ assert_equal(<<'EOT'.chomp, json)
+{
+ "1": 2
+}
+EOT
+ parsed_json = parse(json)
+ assert_equal({"1"=>2}, parsed_json)
+ assert_raise(GeneratorError) { pretty_generate(666) }
+ assert_equal '666', pretty_generate(666, :quirks_mode => true)
+ end
+
+ def test_fast_generate
+ json = fast_generate(@hash)
+ assert_equal(JSON.parse(@json2), JSON.parse(json))
+ parsed_json = parse(json)
+ assert_equal(@hash, parsed_json)
+ json = fast_generate({1=>2})
+ assert_equal('{"1":2}', json)
+ parsed_json = parse(json)
+ assert_equal({"1"=>2}, parsed_json)
+ assert_raise(GeneratorError) { fast_generate(666) }
+ assert_equal '666', fast_generate(666, :quirks_mode => true)
+ end
+
+ def test_own_state
+ state = State.new
+ json = generate(@hash, state)
+ assert_equal(JSON.parse(@json2), JSON.parse(json))
+ parsed_json = parse(json)
+ assert_equal(@hash, parsed_json)
+ json = generate({1=>2}, state)
+ assert_equal('{"1":2}', json)
+ parsed_json = parse(json)
+ assert_equal({"1"=>2}, parsed_json)
+ assert_raise(GeneratorError) { generate(666, state) }
+ state.quirks_mode = true
+ assert state.quirks_mode?
+ assert_equal '666', generate(666, state)
+ end
+
+ def test_states
+ json = generate({1=>2}, nil)
+ assert_equal('{"1":2}', json)
+ s = JSON.state.new
+ assert s.check_circular?
+ assert s[:check_circular?]
+ h = { 1=>2 }
+ h[3] = h
+ assert_raises(JSON::NestingError) { generate(h) }
+ assert_raises(JSON::NestingError) { generate(h, s) }
+ s = JSON.state.new
+ a = [ 1, 2 ]
+ a << a
+ assert_raises(JSON::NestingError) { generate(a, s) }
+ assert s.check_circular?
+ assert s[:check_circular?]
+ end
+
+ def test_pretty_state
+ state = PRETTY_STATE_PROTOTYPE.dup
+ assert_equal({
+ :allow_nan => false,
+ :array_nl => "\n",
+ :ascii_only => false,
+ :quirks_mode => false,
+ :depth => 0,
+ :indent => " ",
+ :max_nesting => 19,
+ :object_nl => "\n",
+ :space => " ",
+ :space_before => "",
+ }.sort_by { |n,| n.to_s }, state.to_h.sort_by { |n,| n.to_s })
+ end
+
+ def test_safe_state
+ state = SAFE_STATE_PROTOTYPE.dup
+ assert_equal({
+ :allow_nan => false,
+ :array_nl => "",
+ :ascii_only => false,
+ :quirks_mode => false,
+ :depth => 0,
+ :indent => "",
+ :max_nesting => 19,
+ :object_nl => "",
+ :space => "",
+ :space_before => "",
+ }.sort_by { |n,| n.to_s }, state.to_h.sort_by { |n,| n.to_s })
+ end
+
+ def test_fast_state
+ state = FAST_STATE_PROTOTYPE.dup
+ assert_equal({
+ :allow_nan => false,
+ :array_nl => "",
+ :ascii_only => false,
+ :quirks_mode => false,
+ :depth => 0,
+ :indent => "",
+ :max_nesting => 0,
+ :object_nl => "",
+ :space => "",
+ :space_before => "",
+ }.sort_by { |n,| n.to_s }, state.to_h.sort_by { |n,| n.to_s })
+ end
+
+ def test_allow_nan
+ assert_raises(GeneratorError) { generate([JSON::NaN]) }
+ assert_equal '[NaN]', generate([JSON::NaN], :allow_nan => true)
+ assert_raises(GeneratorError) { fast_generate([JSON::NaN]) }
+ assert_raises(GeneratorError) { pretty_generate([JSON::NaN]) }
+ assert_equal "[\n NaN\n]", pretty_generate([JSON::NaN], :allow_nan => true)
+ assert_raises(GeneratorError) { generate([JSON::Infinity]) }
+ assert_equal '[Infinity]', generate([JSON::Infinity], :allow_nan => true)
+ assert_raises(GeneratorError) { fast_generate([JSON::Infinity]) }
+ assert_raises(GeneratorError) { pretty_generate([JSON::Infinity]) }
+ assert_equal "[\n Infinity\n]", pretty_generate([JSON::Infinity], :allow_nan => true)
+ assert_raises(GeneratorError) { generate([JSON::MinusInfinity]) }
+ assert_equal '[-Infinity]', generate([JSON::MinusInfinity], :allow_nan => true)
+ assert_raises(GeneratorError) { fast_generate([JSON::MinusInfinity]) }
+ assert_raises(GeneratorError) { pretty_generate([JSON::MinusInfinity]) }
+ assert_equal "[\n -Infinity\n]", pretty_generate([JSON::MinusInfinity], :allow_nan => true)
+ end
+
+ def test_depth
+ ary = []; ary << ary
+ assert_equal 0, JSON::SAFE_STATE_PROTOTYPE.depth
+ assert_raises(JSON::NestingError) { JSON.generate(ary) }
+ assert_equal 0, JSON::SAFE_STATE_PROTOTYPE.depth
+ assert_equal 0, JSON::PRETTY_STATE_PROTOTYPE.depth
+ assert_raises(JSON::NestingError) { JSON.pretty_generate(ary) }
+ assert_equal 0, JSON::PRETTY_STATE_PROTOTYPE.depth
+ s = JSON.state.new
+ assert_equal 0, s.depth
+ assert_raises(JSON::NestingError) { ary.to_json(s) }
+ assert_equal 19, s.depth
+ end
+
+ def test_gc
+ bignum_too_long_to_embed_as_string = 1234567890123456789012345
+ expect = bignum_too_long_to_embed_as_string.to_s
+ stress, GC.stress = GC.stress, true
+
+ 10.times do |i|
+ tmp = bignum_too_long_to_embed_as_string.to_json
+ assert_equal expect, tmp
+ end
+ ensure
+ GC.stress = stress
+ end if GC.respond_to?(:stress=)
+end
--- /dev/null
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+
+require 'test/unit'
+require File.join(File.dirname(__FILE__), 'setup_variant')
+require 'stringio'
+require 'time'
+
+class TestJsonStringMatching < Test::Unit::TestCase
+ include JSON
+
+ class TestTime < ::Time
+ def self.json_create(string)
+ Time.parse(string)
+ end
+
+ def to_json(*)
+ %{"#{strftime('%FT%T%z')}"}
+ end
+
+ def ==(other)
+ to_i == other.to_i
+ end
+ end
+
+ def test_match_date
+ t = TestTime.new
+ t_json = [ t ].to_json
+ assert_equal [ t ],
+ JSON.parse(t_json, :create_additions => true,
+ :match_string => { /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{4}\z/ => TestTime })
+ assert_equal [ t.strftime('%FT%T%z') ],
+ JSON.parse(t_json, :create_additions => true,
+ :match_string => { /\A\d{3}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{4}\z/ => TestTime })
+ assert_equal [ t.strftime('%FT%T%z') ],
+ JSON.parse(t_json,
+ :match_string => { /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{4}\z/ => TestTime })
+ end
+end
--- /dev/null
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+
+require 'test/unit'
+require File.join(File.dirname(__FILE__), 'setup_variant')
+
+class TC_JSONUnicode < Test::Unit::TestCase
+ include JSON
+
+ def test_unicode
+ assert_equal '""', ''.to_json
+ assert_equal '"\\b"', "\b".to_json
+ assert_equal '"\u0001"', 0x1.chr.to_json
+ assert_equal '"\u001f"', 0x1f.chr.to_json
+ assert_equal '" "', ' '.to_json
+ assert_equal "\"#{0x7f.chr}\"", 0x7f.chr.to_json
+ utf8 = [ "© ≠ €! \01" ]
+ json = '["© ≠ €! \u0001"]'
+ assert_equal json, utf8.to_json(:ascii_only => false)
+ assert_equal utf8, parse(json)
+ json = '["\u00a9 \u2260 \u20ac! \u0001"]'
+ assert_equal json, utf8.to_json(:ascii_only => true)
+ assert_equal utf8, parse(json)
+ utf8 = ["\343\201\202\343\201\204\343\201\206\343\201\210\343\201\212"]
+ json = "[\"\343\201\202\343\201\204\343\201\206\343\201\210\343\201\212\"]"
+ assert_equal utf8, parse(json)
+ assert_equal json, utf8.to_json(:ascii_only => false)
+ utf8 = ["\343\201\202\343\201\204\343\201\206\343\201\210\343\201\212"]
+ assert_equal utf8, parse(json)
+ json = "[\"\\u3042\\u3044\\u3046\\u3048\\u304a\"]"
+ assert_equal json, utf8.to_json(:ascii_only => true)
+ assert_equal utf8, parse(json)
+ utf8 = ['საქართველო']
+ json = '["საქართველო"]'
+ assert_equal json, utf8.to_json(:ascii_only => false)
+ json = "[\"\\u10e1\\u10d0\\u10e5\\u10d0\\u10e0\\u10d7\\u10d5\\u10d4\\u10da\\u10dd\"]"
+ assert_equal json, utf8.to_json(:ascii_only => true)
+ assert_equal utf8, parse(json)
+ assert_equal '["Ã"]', JSON.generate(["Ã"], :ascii_only => false)
+ assert_equal '["\\u00c3"]', JSON.generate(["Ã"], :ascii_only => true)
+ assert_equal ["€"], JSON.parse('["\u20ac"]')
+ utf8 = ["\xf0\xa0\x80\x81"]
+ json = "[\"\xf0\xa0\x80\x81\"]"
+ assert_equal json, JSON.generate(utf8, :ascii_only => false)
+ assert_equal utf8, JSON.parse(json)
+ json = '["\ud840\udc01"]'
+ assert_equal json, JSON.generate(utf8, :ascii_only => true)
+ assert_equal utf8, JSON.parse(json)
+ end
+
+ def test_chars
+ (0..0x7f).each do |i|
+ json = '["\u%04x"]' % i
+ if RUBY_VERSION >= "1.9."
+ i = i.chr
+ end
+ assert_equal i, JSON.parse(json).first[0]
+ if i == ?\b
+ generated = JSON.generate(["" << i])
+ assert '["\b"]' == generated || '["\10"]' == generated
+ elsif [?\n, ?\r, ?\t, ?\f].include?(i)
+ assert_equal '[' << ('' << i).dump << ']', JSON.generate(["" << i])
+ elsif i.chr < 0x20.chr
+ assert_equal json, JSON.generate(["" << i])
+ end
+ end
+ assert_raise(JSON::GeneratorError) do
+ JSON.generate(["\x80"], :ascii_only => true)
+ end
+ assert_equal "\302\200", JSON.parse('["\u0080"]').first
+ end
+end
--- /dev/null
+require 'json'
+
+require 'iconv'
+ISO_8859_1_TO_UTF8 = Iconv.new('utf-8', 'iso-8859-15')
+class ::String
+ def to_utf8
+ ISO_8859_1_TO_UTF8.iconv self
+ end
+end
+
+class Fuzzer
+ def initialize(n, freqs = {})
+ sum = freqs.inject(0.0) { |s, x| s + x.last }
+ freqs.each_key { |x| freqs[x] /= sum }
+ s = 0.0
+ freqs.each_key do |x|
+ freqs[x] = s .. (s + t = freqs[x])
+ s += t
+ end
+ @freqs = freqs
+ @n = n
+ @alpha = (0..0xff).to_a
+ end
+
+ def random_string
+ s = ''
+ 30.times { s << @alpha[rand(@alpha.size)] }
+ s.to_utf8
+ end
+
+ def pick
+ r = rand
+ found = @freqs.find { |k, f| f.include? rand }
+ found && found.first
+ end
+
+ def make_pick
+ k = pick
+ case
+ when k == Hash, k == Array
+ k.new
+ when k == true, k == false, k == nil
+ k
+ when k == String
+ random_string
+ when k == Fixnum
+ rand(2 ** 30) - 2 ** 29
+ when k == Bignum
+ rand(2 ** 70) - 2 ** 69
+ end
+ end
+
+ def fuzz(current = nil)
+ if @n > 0
+ case current
+ when nil
+ @n -= 1
+ current = fuzz [ Hash, Array ][rand(2)].new
+ when Array
+ while @n > 0
+ @n -= 1
+ current << case p = make_pick
+ when Array, Hash
+ fuzz(p)
+ else
+ p
+ end
+ end
+ when Hash
+ while @n > 0
+ @n -= 1
+ current[random_string] = case p = make_pick
+ when Array, Hash
+ fuzz(p)
+ else
+ p
+ end
+ end
+ end
+ end
+ current
+ end
+end
+
+class MyState < JSON.state
+ WS = " \r\t\n"
+
+ def initialize
+ super(
+ :indent => make_spaces,
+ :space => make_spaces,
+ :space_before => make_spaces,
+ :object_nl => make_spaces,
+ :array_nl => make_spaces,
+ :max_nesting => false
+ )
+ end
+
+ def make_spaces
+ s = ''
+ rand(1).times { s << WS[rand(WS.size)] }
+ s
+ end
+end
+
+n = (ARGV.shift || 500).to_i
+loop do
+ fuzzer = Fuzzer.new(n,
+ Hash => 25,
+ Array => 25,
+ String => 10,
+ Fixnum => 10,
+ Bignum => 10,
+ nil => 5,
+ true => 5,
+ false => 5
+ )
+ o1 = fuzzer.fuzz
+ json = JSON.generate o1, MyState.new
+ if $DEBUG
+ puts "-" * 80
+ puts json, json.size
+ else
+ puts json.size
+ end
+ begin
+ o2 = JSON.parse(json, :max_nesting => false)
+ rescue JSON::ParserError => e
+ puts "Caught #{e.class}: #{e.message}\n#{e.backtrace * "\n"}"
+ puts "o1 = #{o1.inspect}", "json = #{json}", "json_str = #{json.inspect}"
+ puts "locals = #{local_variables.inspect}"
+ exit
+ end
+ if o1 != o2
+ puts "mismatch", "o1 = #{o1.inspect}", "o2 = #{o2.inspect}",
+ "json = #{json}", "json_str = #{json.inspect}"
+ puts "locals = #{local_variables.inspect}"
+ end
+end
--- /dev/null
+#!/usr/bin/env ruby
+
+require 'webrick'
+include WEBrick
+$:.unshift 'ext'
+$:.unshift 'lib'
+require 'json'
+
+class JSONServlet < HTTPServlet::AbstractServlet
+ @@count = 1
+
+ def do_GET(req, res)
+ obj = {
+ "TIME" => Time.now.strftime("%FT%T"),
+ "foo" => "Bär",
+ "bar" => "© ≠ €!",
+ 'a' => 2,
+ 'b' => 3.141,
+ 'COUNT' => @@count += 1,
+ 'c' => 'c',
+ 'd' => [ 1, "b", 3.14 ],
+ 'e' => { 'foo' => 'bar' },
+ 'g' => "松本行弘",
+ 'h' => 1000.0,
+ 'i' => 0.001,
+ 'j' => "\xf0\xa0\x80\x81",
+ }
+ res.body = JSON.generate obj
+ res['Content-Type'] = "application/json"
+ end
+end
+
+def create_server(err, dir, port)
+ dir = File.expand_path(dir)
+ err.puts "Surf to:", "http://#{Socket.gethostname}:#{port}"
+
+ s = HTTPServer.new(
+ :Port => port,
+ :DocumentRoot => dir,
+ :Logger => WEBrick::Log.new(err),
+ :AccessLog => [
+ [ err, WEBrick::AccessLog::COMMON_LOG_FORMAT ],
+ [ err, WEBrick::AccessLog::REFERER_LOG_FORMAT ],
+ [ err, WEBrick::AccessLog::AGENT_LOG_FORMAT ]
+ ]
+ )
+ s.mount("/json", JSONServlet)
+ s
+end
+
+default_dir = File.expand_path(File.join(File.dirname(__FILE__), '..', 'data'))
+dir = ARGV.shift || default_dir
+port = (ARGV.shift || 6666).to_i
+s = create_server(STDERR, dir, 6666)
+t = Thread.new { s.start }
+trap(:INT) do
+ s.shutdown
+ t.join
+ exit
+end
+sleep
--- /dev/null
+$: << File.join([File.dirname(__FILE__), "i18n/lib"])
--- /dev/null
+$: << File.join([File.dirname(__FILE__), "json/lib"])
--- /dev/null
+$: << File.join([File.dirname(__FILE__), "systemu/lib"])
--- /dev/null
+require 'systemu'
+require 'json'
+require 'i18n'
--- /dev/null
+same as Ruby's
+
+http://www.ruby-lang.org/en/LICENSE.txt
--- /dev/null
+NAME
+
+ systemu
+
+SYNOPSIS
+
+ universal capture of stdout and stderr and handling of child process pid for
+ windows, *nix, etc.
+
+URIS
+
+ http://github.com/ahoward/systemu
+ http://rubyforge.org/projects/codeforpeople/
+
+INSTALL
+
+ gem install systemu
+
+HISTORY
+ 2.0.0
+ - versioning issue. new gem release.
+
+ 1.3.1
+ - updates for ruby 1.9.1
+
+ 1.3.0
+ - move to github
+
+ 1.2.0
+
+ - fixed handling of background thread management - needed
+ Thread.current.abort_on_exception = true
+
+ - fixed reporting of child pid, it was reported as the parent's pid before
+
+SAMPLES
+
+
+ <========< samples/a.rb >========>
+
+ ~ > cat samples/a.rb
+
+ #
+ # systemu can be used on any platform to return status, stdout, and stderr of
+ # any command. unlike other methods like open3/popen4 there is zero danger of
+ # full pipes or threading issues hanging your process or subprocess.
+ #
+ require 'systemu'
+
+ date = %q( ruby -e" t = Time.now; STDOUT.puts t; STDERR.puts t " )
+
+ status, stdout, stderr = systemu date
+ p [ status, stdout, stderr ]
+
+ ~ > ruby samples/a.rb
+
+ [#<Process::Status: pid 50931 exit 0>, "2011-12-11 22:07:30 -0700\n", "2011-12-11 22:07:30 -0700\n"]
+
+
+ <========< samples/b.rb >========>
+
+ ~ > cat samples/b.rb
+
+ #
+ # quite a few keys can be passed to the command to alter it's behaviour. if
+ # either stdout or stderr is supplied those objects should respond_to? '<<'
+ # and only status will be returned
+ #
+ require 'systemu'
+
+ date = %q( ruby -e" t = Time.now; STDOUT.puts t; STDERR.puts t " )
+
+ stdout, stderr = '', ''
+ status = systemu date, 'stdout' => stdout, 'stderr' => stderr
+ p [ status, stdout, stderr ]
+
+ ~ > ruby samples/b.rb
+
+ [#<Process::Status: pid 50936 exit 0>, "2011-12-11 22:07:30 -0700\n", "2011-12-11 22:07:30 -0700\n"]
+
+
+ <========< samples/c.rb >========>
+
+ ~ > cat samples/c.rb
+
+ #
+ # of course stdin can be supplied too. synonyms for 'stdin' include '0' and
+ # 0. the other stdio streams have similar shortcuts
+ #
+ require 'systemu'
+
+ cat = %q( ruby -e" ARGF.each{|line| puts line} " )
+
+ status = systemu cat, 0=>'the stdin for cat', 1=>stdout=''
+ puts stdout
+
+ ~ > ruby samples/c.rb
+
+ the stdin for cat
+
+
+ <========< samples/d.rb >========>
+
+ ~ > cat samples/d.rb
+
+ #
+ # the cwd can be supplied
+ #
+ require 'systemu'
+ require 'tmpdir'
+
+ pwd = %q( ruby -e" STDERR.puts Dir.pwd " )
+
+ status = systemu pwd, 2=>(stderr=''), :cwd=>Dir.tmpdir
+ puts stderr
+
+
+ ~ > ruby samples/d.rb
+
+ /private/var/folders/sp/nwtflj890qnb6z4b53dqxvlw0000gp/T
+
+
+ <========< samples/e.rb >========>
+
+ ~ > cat samples/e.rb
+
+ #
+ # any environment vars specified are merged into the child's environment
+ #
+ require 'systemu'
+
+ env = %q( ruby -r yaml -e" puts ENV[ 'answer' ] " )
+
+ status = systemu env, 1=>stdout='', 'env'=>{ 'answer' => 0b101010 }
+ puts stdout
+
+ ~ > ruby samples/e.rb
+
+ 42
+
+
+ <========< samples/f.rb >========>
+
+ ~ > cat samples/f.rb
+
+ #
+ # if a block is specified then it is passed the child pid and run in a
+ # background thread. note that this thread will __not__ be blocked during the
+ # execution of the command so it may do useful work such as killing the child
+ # if execution time passes a certain threshold
+ #
+ require 'systemu'
+
+ looper = %q( ruby -e" loop{ STDERR.puts Time.now.to_i; sleep 1 } " )
+
+ status, stdout, stderr =
+ systemu looper do |cid|
+ sleep 3
+ Process.kill 9, cid
+ end
+
+ p status
+ p stderr
+
+ ~ > ruby samples/f.rb
+
+ #<Process::Status: pid 50956 SIGKILL (signal 9)>
+ "1323666451\n1323666452\n1323666453\n"
+
+
--- /dev/null
+NAME
+
+ systemu
+
+SYNOPSIS
+
+ univeral capture of stdout and stderr and handling of child process pid for windows, *nix, etc.
+
+URIS
+
+ http://github.com/ahoward/systemu
+ http://rubyforge.org/projects/codeforpeople/
+
+INSTALL
+
+ gem install systemu
+
+HISTORY
+ 2.0.0
+ - versioning issue. new gem release.
+
+ 1.3.1
+ - updates for ruby 1.9.1
+
+ 1.3.0
+ - move to github
+
+ 1.2.0
+
+ - fixed handling of background thread management - needed
+ Thread.current.abort_on_exception = true
+
+ - fixed reporting of child pid, it was reported as the parent's pid before
+
+SAMPLES
+
+<%= samples %>
--- /dev/null
+This.rubyforge_project = 'codeforpeople'
+This.author = "Ara T. Howard"
+This.email = "ara.t.howard@gmail.com"
+This.homepage = "https://github.com/ahoward/#{ This.lib }"
+
+task :license do
+ open('LICENSE', 'w'){|fd| fd.puts "same as ruby's"}
+end
+
+task :default do
+ puts((Rake::Task.tasks.map{|task| task.name.gsub(/::/,':')} - ['default']).sort)
+end
+
+task :test do
+ run_tests!
+end
+
+namespace :test do
+ task(:unit){ run_tests!(:unit) }
+ task(:functional){ run_tests!(:functional) }
+ task(:integration){ run_tests!(:integration) }
+end
+
+def run_tests!(which = nil)
+ which ||= '**'
+ test_dir = File.join(This.dir, "test")
+ test_glob ||= File.join(test_dir, "#{ which }/**_test.rb")
+ test_rbs = Dir.glob(test_glob).sort
+
+ div = ('=' * 119)
+ line = ('-' * 119)
+
+ test_rbs.each_with_index do |test_rb, index|
+ testno = index + 1
+ command = "#{ This.ruby } -I ./lib -I ./test/lib #{ test_rb }"
+
+ puts
+ say(div, :color => :cyan, :bold => true)
+ say("@#{ testno } => ", :bold => true, :method => :print)
+ say(command, :color => :cyan, :bold => true)
+ say(line, :color => :cyan, :bold => true)
+
+ system(command)
+
+ say(line, :color => :cyan, :bold => true)
+
+ status = $?.exitstatus
+
+ if status.zero?
+ say("@#{ testno } <= ", :bold => true, :color => :white, :method => :print)
+ say("SUCCESS", :color => :green, :bold => true)
+ else
+ say("@#{ testno } <= ", :bold => true, :color => :white, :method => :print)
+ say("FAILURE", :color => :red, :bold => true)
+ end
+ say(line, :color => :cyan, :bold => true)
+
+ exit(status) unless status.zero?
+ end
+end
+
+
+task :gemspec do
+ ignore_extensions = ['git', 'svn', 'tmp', /sw./, 'bak', 'gem']
+ ignore_directories = ['pkg']
+ ignore_files = ['test/log', 'a.rb']
+
+ shiteless =
+ lambda do |list|
+ list.delete_if do |entry|
+ next unless test(?e, entry)
+ extension = File.basename(entry).split(%r/[.]/).last
+ ignore_extensions.any?{|ext| ext === extension}
+ end
+ list.delete_if do |entry|
+ next unless test(?d, entry)
+ dirname = File.expand_path(entry)
+ ignore_directories.any?{|dir| File.expand_path(dir) == dirname}
+ end
+ list.delete_if do |entry|
+ next unless test(?f, entry)
+ filename = File.expand_path(entry)
+ ignore_files.any?{|file| File.expand_path(file) == filename}
+ end
+ end
+
+ lib = This.lib
+ object = This.object
+ version = This.version
+ files = shiteless[Dir::glob("**/**")]
+ executables = shiteless[Dir::glob("bin/*")].map{|exe| File.basename(exe)}
+ #has_rdoc = true #File.exist?('doc')
+ test_files = "test/#{ lib }.rb" if File.file?("test/#{ lib }.rb")
+ summary = object.respond_to?(:summary) ? object.summary : "summary: #{ lib } kicks the ass"
+ description = object.respond_to?(:description) ? object.description : "description: #{ lib } kicks the ass"
+
+ if This.extensions.nil?
+ This.extensions = []
+ extensions = This.extensions
+ %w( Makefile configure extconf.rb ).each do |ext|
+ extensions << ext if File.exists?(ext)
+ end
+ end
+ extensions = [extensions].flatten.compact
+
+ template =
+ if test(?e, 'gemspec.erb')
+ Template{ IO.read('gemspec.erb') }
+ else
+ Template {
+ <<-__
+ ## #{ lib }.gemspec
+ #
+
+ Gem::Specification::new do |spec|
+ spec.name = #{ lib.inspect }
+ spec.version = #{ version.inspect }
+ spec.platform = Gem::Platform::RUBY
+ spec.summary = #{ lib.inspect }
+ spec.description = #{ description.inspect }
+
+ spec.files =\n#{ files.sort.pretty_inspect }
+ spec.executables = #{ executables.inspect }
+
+ spec.require_path = "lib"
+
+ spec.test_files = #{ test_files.inspect }
+
+ ### spec.add_dependency 'lib', '>= version'
+ #### spec.add_dependency 'map'
+
+ spec.extensions.push(*#{ extensions.inspect })
+
+ spec.rubyforge_project = #{ This.rubyforge_project.inspect }
+ spec.author = #{ This.author.inspect }
+ spec.email = #{ This.email.inspect }
+ spec.homepage = #{ This.homepage.inspect }
+ end
+ __
+ }
+ end
+
+ Fu.mkdir_p(This.pkgdir)
+ gemspec = "#{ lib }.gemspec"
+ open(gemspec, "w"){|fd| fd.puts(template)}
+ This.gemspec = gemspec
+end
+
+task :gem => [:clean, :gemspec] do
+ Fu.mkdir_p(This.pkgdir)
+ before = Dir['*.gem']
+ cmd = "gem build #{ This.gemspec }"
+ `#{ cmd }`
+ after = Dir['*.gem']
+ gem = ((after - before).first || after.first) or abort('no gem!')
+ Fu.mv(gem, This.pkgdir)
+ This.gem = File.join(This.pkgdir, File.basename(gem))
+end
+
+task :readme do
+ samples = ''
+ prompt = '~ > '
+ lib = This.lib
+ version = This.version
+
+ Dir['sample*/*'].sort.each do |sample|
+ samples << "\n" << " <========< #{ sample } >========>" << "\n\n"
+
+ cmd = "cat #{ sample }"
+ samples << Util.indent(prompt + cmd, 2) << "\n\n"
+ samples << Util.indent(`#{ cmd }`, 4) << "\n"
+
+ cmd = "ruby #{ sample }"
+ samples << Util.indent(prompt + cmd, 2) << "\n\n"
+
+ cmd = "ruby -e'STDOUT.sync=true; exec %(ruby -I ./lib #{ sample })'"
+ samples << Util.indent(`#{ cmd } 2>&1`, 4) << "\n"
+ end
+
+ template =
+ if test(?e, 'readme.erb')
+ Template{ IO.read('readme.erb') }
+ else
+ Template {
+ <<-__
+ NAME
+ #{ lib }
+
+ DESCRIPTION
+
+ INSTALL
+ gem install #{ lib }
+
+ SAMPLES
+ #{ samples }
+ __
+ }
+ end
+
+ open("README", "w"){|fd| fd.puts template}
+end
+
+
+task :clean do
+ Dir[File.join(This.pkgdir, '**/**')].each{|entry| Fu.rm_rf(entry)}
+end
+
+
+task :release => [:clean, :gemspec, :gem] do
+ gems = Dir[File.join(This.pkgdir, '*.gem')].flatten
+ raise "which one? : #{ gems.inspect }" if gems.size > 1
+ raise "no gems?" if gems.size < 1
+
+ cmd = "gem push #{ This.gem }"
+ puts cmd
+ puts
+ system(cmd)
+ abort("cmd(#{ cmd }) failed with (#{ $?.inspect })") unless $?.exitstatus.zero?
+
+ cmd = "rubyforge login && rubyforge add_release #{ This.rubyforge_project } #{ This.lib } #{ This.version } #{ This.gem }"
+ puts cmd
+ puts
+ system(cmd)
+ abort("cmd(#{ cmd }) failed with (#{ $?.inspect })") unless $?.exitstatus.zero?
+end
+
+
+
+
+
+BEGIN {
+# support for this rakefile
+#
+ $VERBOSE = nil
+
+ require 'ostruct'
+ require 'erb'
+ require 'fileutils'
+ require 'rbconfig'
+ require 'pp'
+
+# fu shortcut
+#
+ Fu = FileUtils
+
+# cache a bunch of stuff about this rakefile/environment
+#
+ This = OpenStruct.new
+
+ This.file = File.expand_path(__FILE__)
+ This.dir = File.dirname(This.file)
+ This.pkgdir = File.join(This.dir, 'pkg')
+
+# grok lib
+#
+ lib = ENV['LIB']
+ unless lib
+ lib = File.basename(Dir.pwd).sub(/[-].*$/, '')
+ end
+ This.lib = lib
+
+# grok version
+#
+ version = ENV['VERSION']
+ unless version
+ require "./lib/#{ This.lib }"
+ This.name = lib.capitalize
+ This.object = eval(This.name)
+ version = This.object.send(:version)
+ end
+ This.version = version
+
+# we need to know the name of the lib an it's version
+#
+ abort('no lib') unless This.lib
+ abort('no version') unless This.version
+
+# discover full path to this ruby executable
+#
+ c = begin; ::RbConfig::CONFIG; rescue NameError; ::Config::CONFIG; end
+ bindir = c["bindir"] || c['BINDIR']
+ ruby_install_name = c['ruby_install_name'] || c['RUBY_INSTALL_NAME'] || 'ruby'
+ ruby_ext = c['EXEEXT'] || ''
+ ruby = File.join(bindir, (ruby_install_name + ruby_ext))
+ This.ruby = ruby
+
+# some utils
+#
+ module Util
+ def indent(s, n = 2)
+ s = unindent(s)
+ ws = ' ' * n
+ s.gsub(%r/^/, ws)
+ end
+
+ def unindent(s)
+ indent = nil
+ s.each_line do |line|
+ next if line =~ %r/^\s*$/
+ indent = line[%r/^\s*/] and break
+ end
+ indent ? s.gsub(%r/^#{ indent }/, "") : s
+ end
+ extend self
+ end
+
+# template support
+#
+ class Template
+ def initialize(&block)
+ @block = block
+ @template = block.call.to_s
+ end
+ def expand(b=nil)
+ ERB.new(Util.unindent(@template)).result((b||@block).binding)
+ end
+ alias_method 'to_s', 'expand'
+ end
+ def Template(*args, &block) Template.new(*args, &block) end
+
+# colored console output support
+#
+ This.ansi = {
+ :clear => "\e[0m",
+ :reset => "\e[0m",
+ :erase_line => "\e[K",
+ :erase_char => "\e[P",
+ :bold => "\e[1m",
+ :dark => "\e[2m",
+ :underline => "\e[4m",
+ :underscore => "\e[4m",
+ :blink => "\e[5m",
+ :reverse => "\e[7m",
+ :concealed => "\e[8m",
+ :black => "\e[30m",
+ :red => "\e[31m",
+ :green => "\e[32m",
+ :yellow => "\e[33m",
+ :blue => "\e[34m",
+ :magenta => "\e[35m",
+ :cyan => "\e[36m",
+ :white => "\e[37m",
+ :on_black => "\e[40m",
+ :on_red => "\e[41m",
+ :on_green => "\e[42m",
+ :on_yellow => "\e[43m",
+ :on_blue => "\e[44m",
+ :on_magenta => "\e[45m",
+ :on_cyan => "\e[46m",
+ :on_white => "\e[47m"
+ }
+ def say(phrase, *args)
+ options = args.last.is_a?(Hash) ? args.pop : {}
+ options[:color] = args.shift.to_s.to_sym unless args.empty?
+ keys = options.keys
+ keys.each{|key| options[key.to_s.to_sym] = options.delete(key)}
+
+ color = options[:color]
+ bold = options.has_key?(:bold)
+
+ parts = [phrase]
+ parts.unshift(This.ansi[color]) if color
+ parts.unshift(This.ansi[:bold]) if bold
+ parts.push(This.ansi[:clear]) if parts.size > 1
+
+ method = options[:method] || :puts
+
+ Kernel.send(method, parts.join)
+ end
+
+# always run out of the project dir
+#
+ Dir.chdir(This.dir)
+}
--- /dev/null
+# encoding: utf-8
+
+require 'tmpdir'
+require 'socket'
+require 'fileutils'
+require 'rbconfig'
+require 'thread'
+
+class Object
+ def systemu(*a, &b) SystemUniversal.new(*a, &b).systemu end
+end
+
+class SystemUniversal
+#
+# constants
+#
+ SystemUniversal::VERSION = '2.5.2' unless SystemUniversal.send(:const_defined?, :VERSION)
+ def SystemUniversal.version() SystemUniversal::VERSION end
+ def version() SystemUniversal::VERSION end
+#
+# class methods
+#
+
+ @host = Socket.gethostname
+ @ppid = Process.ppid
+ @pid = Process.pid
+ @turd = ENV['SYSTEMU_TURD']
+
+ c = begin; ::RbConfig::CONFIG; rescue NameError; ::Config::CONFIG; end
+ ruby = File.join(c['bindir'], c['ruby_install_name']) << c['EXEEXT']
+ @ruby = if system('%s -e 42' % ruby)
+ ruby
+ else
+ system('%s -e 42' % 'ruby') ? 'ruby' : warn('no ruby in PATH/CONFIG')
+ end
+
+ class << SystemUniversal
+ %w( host ppid pid ruby turd ).each{|a| attr_accessor a}
+
+ def quote(*words)
+ words.map{|word| word.inspect}.join(' ')
+ end
+ end
+
+#
+# instance methods
+#
+
+ def initialize argv, opts = {}, &block
+ getopt = getopts opts
+
+ @argv = argv
+ @block = block
+
+ @stdin = getopt[ ['stdin', 'in', '0', 0] ]
+ @stdout = getopt[ ['stdout', 'out', '1', 1] ]
+ @stderr = getopt[ ['stderr', 'err', '2', 2] ]
+ @env = getopt[ 'env' ]
+ @cwd = getopt[ 'cwd' ]
+
+ @host = getopt[ 'host', self.class.host ]
+ @ppid = getopt[ 'ppid', self.class.ppid ]
+ @pid = getopt[ 'pid', self.class.pid ]
+ @ruby = getopt[ 'ruby', self.class.ruby ]
+ end
+
+ def systemu
+ tmpdir do |tmp|
+ c = child_setup tmp
+ status = nil
+
+ begin
+ thread = nil
+
+ quietly{
+ IO.popen "#{ quote(@ruby) } #{ quote(c['program']) }", 'r+' do |pipe|
+ line = pipe.gets
+ case line
+ when %r/^pid: \d+$/
+ cid = Integer line[%r/\d+/]
+ else
+ begin
+ buf = pipe.read
+ buf = "#{ line }#{ buf }"
+ e = Marshal.load buf
+ raise unless Exception === e
+ raise e
+ rescue
+ raise "systemu: Error - process interrupted!\n#{ buf }\n"
+ end
+ end
+ thread = new_thread cid, @block if @block
+ pipe.read rescue nil
+ end
+ }
+ status = $?
+ ensure
+ if thread
+ begin
+ class << status
+ attr 'thread'
+ end
+ status.instance_eval{ @thread = thread }
+ rescue
+ 42
+ end
+ end
+ end
+
+ if @stdout or @stderr
+ open(c['stdout']){|f| relay f => @stdout} if @stdout
+ open(c['stderr']){|f| relay f => @stderr} if @stderr
+ status
+ else
+ [status, IO.read(c['stdout']), IO.read(c['stderr'])]
+ end
+ end
+ end
+
+ def quote *args, &block
+ SystemUniversal.quote(*args, &block)
+ end
+
+ def new_thread cid, block
+ q = Queue.new
+ Thread.new(cid) do |cid|
+ current = Thread.current
+ current.abort_on_exception = true
+ q.push current
+ block.call cid
+ end
+ q.pop
+ end
+
+ def child_setup tmp
+ stdin = File.expand_path(File.join(tmp, 'stdin'))
+ stdout = File.expand_path(File.join(tmp, 'stdout'))
+ stderr = File.expand_path(File.join(tmp, 'stderr'))
+ program = File.expand_path(File.join(tmp, 'program'))
+ config = File.expand_path(File.join(tmp, 'config'))
+
+ if @stdin
+ open(stdin, 'w'){|f| relay @stdin => f}
+ else
+ FileUtils.touch stdin
+ end
+ FileUtils.touch stdout
+ FileUtils.touch stderr
+
+ c = {}
+ c['argv'] = @argv
+ c['env'] = @env
+ c['cwd'] = @cwd
+ c['stdin'] = stdin
+ c['stdout'] = stdout
+ c['stderr'] = stderr
+ c['program'] = program
+ open(config, 'w'){|f| Marshal.dump(c, f)}
+
+ open(program, 'w'){|f| f.write child_program(config)}
+
+ c
+ end
+
+ def quietly
+ v = $VERBOSE
+ $VERBOSE = nil
+ yield
+ ensure
+ $VERBOSE = v
+ end
+
+ def child_program config
+ <<-program
+ # encoding: utf-8
+
+ PIPE = STDOUT.dup
+ begin
+ config = Marshal.load(IO.read('#{ config }'))
+
+ argv = config['argv']
+ env = config['env']
+ cwd = config['cwd']
+ stdin = config['stdin']
+ stdout = config['stdout']
+ stderr = config['stderr']
+
+ Dir.chdir cwd if cwd
+ env.each{|k,v| ENV[k.to_s] = v.to_s} if env
+
+ STDIN.reopen stdin
+ STDOUT.reopen stdout
+ STDERR.reopen stderr
+
+ PIPE.puts "pid: \#{ Process.pid }"
+ PIPE.flush ### the process is ready yo!
+ PIPE.close
+
+ exec *argv
+ rescue Exception => e
+ PIPE.write Marshal.dump(e) rescue nil
+ exit 42
+ end
+ program
+ end
+
+ def relay srcdst
+ src, dst, ignored = srcdst.to_a.first
+ if src.respond_to? 'read'
+ while((buf = src.read(8192))); dst << buf; end
+ else
+ if src.respond_to?(:each_line)
+ src.each_line{|buf| dst << buf}
+ else
+ src.each{|buf| dst << buf}
+ end
+ end
+ end
+
+ def tmpdir d = Dir.tmpdir, max = 42, &b
+ i = -1 and loop{
+ i += 1
+
+ tmp = File.join d, "systemu_#{ @host }_#{ @ppid }_#{ @pid }_#{ rand }_#{ i += 1 }"
+
+ begin
+ Dir.mkdir tmp
+ rescue Errno::EEXIST
+ raise if i >= max
+ next
+ end
+
+ break(
+ if b
+ begin
+ b.call tmp
+ ensure
+ FileUtils.rm_rf tmp unless SystemU.turd
+ end
+ else
+ tmp
+ end
+ )
+ }
+ end
+
+ def getopts opts = {}
+ lambda do |*args|
+ keys, default, ignored = args
+ catch(:opt) do
+ [keys].flatten.each do |key|
+ [key, key.to_s, key.to_s.intern].each do |key|
+ throw :opt, opts[key] if opts.has_key?(key)
+ end
+ end
+ default
+ end
+ end
+ end
+end
+
+# some monkeypatching for JRuby
+if defined? JRUBY_VERSION
+ require 'jruby'
+ java_import org.jruby.RubyProcess
+
+ class SystemUniversal
+ def systemu
+ split_argv = JRuby::PathHelper.smart_split_command @argv
+ process = java.lang.Runtime.runtime.exec split_argv.to_java(:string)
+
+ stdout, stderr = [process.input_stream, process.error_stream].map do |stream|
+ StreamReader.new(stream)
+ end
+
+ exit_code = process.wait_for
+ field = process.get_class.get_declared_field("pid")
+ field.set_accessible(true)
+ pid = field.get(process)
+ [
+ RubyProcess::RubyStatus.new_process_status(JRuby.runtime, exit_code, pid),
+ stdout.join,
+ stderr.join
+ ]
+ end
+
+ class StreamReader
+ def initialize(stream)
+ @data = ""
+ @thread = Thread.new do
+ reader = java.io.BufferedReader.new java.io.InputStreamReader.new(stream)
+
+ while line = reader.read_line
+ @data << line << "\n"
+ end
+ end
+ end
+
+ def join
+ @thread.join
+ @data
+ end
+ end
+ end
+end
+
+
+
+SystemU = SystemUniversal unless defined? SystemU
+Systemu = SystemUniversal unless defined? Systemu
+
+
+
+
+
+
+
+
+
+
+
+
+
+if $0 == __FILE__
+#
+# date
+#
+ date = %q( ruby -e" t = Time.now; STDOUT.puts t; STDERR.puts t " )
+
+ status, stdout, stderr = systemu date
+ p [status, stdout, stderr]
+
+ status = systemu date, 1=>(stdout = '')
+ p [status, stdout]
+
+ status = systemu date, 2=>(stderr = '')
+ p [status, stderr]
+#
+# sleep
+#
+ sleep = %q( ruby -e" p(sleep(1)) " )
+ status, stdout, stderr = systemu sleep
+ p [status, stdout, stderr]
+
+ sleep = %q( ruby -e" p(sleep(42)) " )
+ status, stdout, stderr = systemu(sleep){|cid| Process.kill 9, cid}
+ p [status, stdout, stderr]
+#
+# env
+#
+ env = %q( ruby -e" p ENV['A'] " )
+ status, stdout, stderr = systemu env, :env => {'A' => 42}
+ p [status, stdout, stderr]
+#
+# cwd
+#
+ env = %q( ruby -e" p Dir.pwd " )
+ status, stdout, stderr = systemu env, :cwd => Dir.tmpdir
+ p [status, stdout, stderr]
+end
--- /dev/null
+#
+# systemu can be used on any platform to return status, stdout, and stderr of
+# any command. unlike other methods like open3/popen4 there is zero danger of
+# full pipes or threading issues hanging your process or subprocess.
+#
+ require 'systemu'
+
+ date = %q( ruby -e" t = Time.now; STDOUT.puts t; STDERR.puts t " )
+
+ status, stdout, stderr = systemu date
+ p [ status, stdout, stderr ]
--- /dev/null
+#
+# quite a few keys can be passed to the command to alter it's behaviour. if
+# either stdout or stderr is supplied those objects should respond_to? '<<'
+# and only status will be returned
+#
+ require 'systemu'
+
+ date = %q( ruby -e" t = Time.now; STDOUT.puts t; STDERR.puts t " )
+
+ stdout, stderr = '', ''
+ status = systemu date, 'stdout' => stdout, 'stderr' => stderr
+ p [ status, stdout, stderr ]
--- /dev/null
+#
+# of course stdin can be supplied too. synonyms for 'stdin' include '0' and
+# 0. the other stdio streams have similar shortcuts
+#
+ require 'systemu'
+
+ cat = %q( ruby -e" ARGF.each{|line| puts line} " )
+
+ status = systemu cat, 0=>'the stdin for cat', 1=>stdout=''
+ puts stdout
--- /dev/null
+#
+# the cwd can be supplied
+#
+ require 'systemu'
+ require 'tmpdir'
+
+ pwd = %q( ruby -e" STDERR.puts Dir.pwd " )
+
+ status = systemu pwd, 2=>(stderr=''), :cwd=>Dir.tmpdir
+ puts stderr
+
--- /dev/null
+#
+# any environment vars specified are merged into the child's environment
+#
+ require 'systemu'
+
+ env = %q( ruby -r yaml -e" puts ENV[ 'answer' ] " )
+
+ status = systemu env, 1=>stdout='', 'env'=>{ 'answer' => 0b101010 }
+ puts stdout
--- /dev/null
+#
+# if a block is specified then it is passed the child pid and run in a
+# background thread. note that this thread will __not__ be blocked during the
+# execution of the command so it may do useful work such as killing the child
+# if execution time passes a certain threshold
+#
+ require 'systemu'
+
+ looper = %q( ruby -e" loop{ STDERR.puts Time.now.to_i; sleep 1 } " )
+
+ status, stdout, stderr =
+ systemu looper do |cid|
+ sleep 3
+ Process.kill 9, cid
+ end
+
+ p status
+ p stderr
--- /dev/null
+## systemu.gemspec
+#
+
+Gem::Specification::new do |spec|
+ spec.name = "systemu"
+ spec.version = "2.5.2"
+ spec.platform = Gem::Platform::RUBY
+ spec.summary = "systemu"
+ spec.description = "description: systemu kicks the ass"
+
+ spec.files =
+["LICENSE",
+ "README",
+ "README.erb",
+ "Rakefile",
+ "lib",
+ "lib/systemu.rb",
+ "samples",
+ "samples/a.rb",
+ "samples/b.rb",
+ "samples/c.rb",
+ "samples/d.rb",
+ "samples/e.rb",
+ "samples/f.rb",
+ "systemu.gemspec",
+ "test",
+ "test/systemu_test.rb",
+ "test/testing.rb"]
+
+ spec.executables = []
+
+ spec.require_path = "lib"
+
+ spec.test_files = nil
+
+### spec.add_dependency 'lib', '>= version'
+#### spec.add_dependency 'map'
+
+ spec.extensions.push(*[])
+
+ spec.rubyforge_project = "codeforpeople"
+ spec.author = "Ara T. Howard"
+ spec.email = "ara.t.howard@gmail.com"
+ spec.homepage = "https://github.com/ahoward/systemu"
+end
--- /dev/null
+require 'win32/daemon'
+
+module MCollective
+ class WindowsDaemon < Win32::Daemon
+ def self.daemonize_runner(pid=nil)
+ raise "Writing pid files are not supported on the Windows Platform" if pid
+ raise "The Windows Daemonizer should only be used on the Windows Platform" unless Util.windows?
+
+ WindowsDaemon.mainloop
+ end
+
+ def service_main
+ Log.debug("Starting Windows Service Daemon")
+
+ runner = Runner.new(nil)
+ runner.run
+ end
+
+ def service_stop
+ Log.info("Windows service stopping")
+ PluginManager["connector_plugin"].disconnect
+ exit! 0
+ end
+ end
+end
--- /dev/null
+#!/bin/sh
+#
+# mcollective Application Server for STOMP based agents
+#
+#
+# description: mcollective lets you build powerful Stomp compatible middleware clients in ruby without having to worry too
+# much about all the setup and management of a Stomp connection, it also provides stats, logging and so forth
+# as a bonus.
+#
+### BEGIN INIT INFO
+# Provides: mcollective
+# Required-Start: $remote_fs
+# Required-Stop: $remote_fs
+# Default-Start: 2 3 4 5
+# Default-Stop: 0 1 6
+# Short-Description: Start daemon at boot time
+# Description: Enable service provided by mcollective.
+### END INIT INFO
+
+# check permissions
+
+uid=`id -u`
+[ $uid -gt 0 ] && { echo "You need to be root to run file" ; exit 4 ; }
+
+
+# PID directory
+pidfile="/var/run/mcollective.pid"
+
+name="mcollective"
+mcollectived=/usr/sbin/mcollectived
+daemonopts="--config=/etc/mcollective/server.cfg"
+
+
+# Source function library.
+. /lib/lsb/init-functions
+
+# Check that binary exists
+if ! [ -f $mcollectived ]
+then
+ echo "mcollectived binary not found"
+ exit 5
+fi
+
+# See how we were called.
+case "$1" in
+ start)
+ echo "Starting daemon: " $name
+ # start the program
+ if [ -f $pidfile ]; then
+ if [ -f $(cat /proc/$(cat $pidfile)/exe > /dev/null) ] ; then
+ echo MCollective appears to be running
+ exit 1
+ else
+ /sbin/start-stop-daemon --start -b --quiet --oknodo -m --pidfile $pidfile --exec $mcollectived -- $daemonopts
+ [ $? = 0 ] && { exit 0 ; } || { exit 1 ; }
+ fi
+ else
+ /sbin/start-stop-daemon --start -b --quiet --oknodo -m --pidfile $pidfile --exec $mcollectived -- $daemonopts
+ fi
+ log_success_msg "mcollective started"
+ ;;
+ stop)
+ echo "Stopping daemon: " $name
+ /sbin/start-stop-daemon --stop -q --pidfile $pidfile
+ if [ -f $pidfile ]; then
+ rm -f $pidfile
+ fi
+ [ $? = 0 ] && { exit 0 ; } || { exit 1 ; }
+ log_success_msg "mcollective stopped"
+ ;;
+ restart)
+ echo "Restarting daemon: " $name
+ $0 stop
+ sleep 2
+ $0 start
+ [ $? = 0 ] && { echo "mcollective restarted" ; exit 0 ; }
+ ;;
+ status)
+ status_of_proc -p ${pidfile} ${mcollectived} ${name} && exit 0 || exit $?
+ ;;
+ *)
+ echo "Usage: mcollectived {start|stop|restart|condrestart|status}"
+ exit 2
+ ;;
+esac
--- /dev/null
+module MCollective
+ module Agent
+ # Discovery agent for The Marionette Collective
+ #
+ # Released under the Apache License, Version 2
+ class Discovery
+ attr_reader :timeout, :meta
+
+ def initialize
+ config = Config.instance.pluginconf
+
+ @timeout = 5
+ @meta = {:license => "Apache License, Version 2",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :timeout => @timeout,
+ :name => "Discovery Agent",
+ :version => MCollective.version,
+ :url => "http://www.marionette-collective.org",
+ :description => "MCollective Discovery Agent"}
+ end
+
+ def handlemsg(msg, stomp)
+ reply = "unknown request"
+
+ case msg[:body]
+ when "ping"
+ reply = "pong"
+
+ else
+ reply = "Unknown Request: #{msg[:body]}"
+ end
+
+ reply
+ end
+ end
+ end
+end
--- /dev/null
+metadata :name => "rpcutil",
+ :description => "General helpful actions that expose stats and internals to SimpleRPC clients",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :license => "Apache License, Version 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 10
+
+action "collective_info", :description => "Info about the main and sub collectives" do
+ display :always
+
+ output :main_collective,
+ :description => "The main Collective",
+ :display_as => "Main Collective"
+
+ output :collectives,
+ :description => "All Collectives",
+ :display_as => "All Collectives"
+
+ summarize do
+ aggregate summary(:collectives)
+ end
+end
+
+action "inventory", :description => "System Inventory" do
+ display :always
+
+ output :agents,
+ :description => "List of agent names",
+ :display_as => "Agents"
+
+ output :facts,
+ :description => "List of facts and values",
+ :display_as => "Facts"
+
+ output :classes,
+ :description => "List of classes on the system",
+ :display_as => "Classes"
+
+ output :version,
+ :description => "MCollective Version",
+ :display_as => "Version"
+
+ output :main_collective,
+ :description => "The main Collective",
+ :display_as => "Main Collective"
+
+ output :collectives,
+ :description => "All Collectives",
+ :display_as => "All Collectives"
+
+ output :data_plugins,
+ :description => "List of data plugin names",
+ :display_as => "Data Plugins"
+end
+
+action "get_fact", :description => "Retrieve a single fact from the fact store" do
+ display :always
+
+ input :fact,
+ :prompt => "The name of the fact",
+ :description => "The fact to retrieve",
+ :type => :string,
+ :validation => '^[\w\-\.]+$',
+ :optional => false,
+ :maxlength => 40
+
+ output :fact,
+ :description => "The name of the fact being returned",
+ :display_as => "Fact"
+
+ output :value,
+ :description => "The value of the fact",
+ :display_as => "Value"
+
+ summarize do
+ aggregate summary(:value)
+ end
+end
+
+action "daemon_stats", :description => "Get statistics from the running daemon" do
+ display :always
+
+ output :threads,
+ :description => "List of threads active in the daemon",
+ :display_as => "Threads"
+
+ output :agents,
+ :description => "List of agents loaded",
+ :display_as => "Agents"
+
+ output :pid,
+ :description => "Process ID of the daemon",
+ :display_as => "PID"
+
+ output :times,
+ :description => "Processor time consumed by the daemon",
+ :display_as => "Times"
+
+ output :validated,
+ :description => "Messages that passed security validation",
+ :display_as => "Security Validated"
+
+ output :unvalidated,
+ :description => "Messages that failed security validation",
+ :display_as => "Failed Security"
+
+ output :passed,
+ :description => "Passed filter checks",
+ :display_as => "Passed Filter"
+
+ output :filtered,
+ :description => "Didn't pass filter checks",
+ :display_as => "Failed Filter"
+
+ output :starttime,
+ :description => "Time the server started",
+ :display_as => "Start Time"
+
+ output :total,
+ :description => "Total messages received",
+ :display_as => "Total Messages"
+
+ output :replies,
+ :description => "Replies sent back to clients",
+ :display_as => "Replies"
+
+ output :configfile,
+ :description => "Config file used to start the daemon",
+ :display_as => "Config File"
+
+ output :version,
+ :description => "MCollective Version",
+ :display_as => "Version"
+
+ output :ttlexpired,
+ :description => "Messages that did pass TTL checks",
+ :display_as => "TTL Expired"
+
+ summarize do
+ aggregate summary(:version)
+ aggregate summary(:agents)
+ end
+end
+
+action "agent_inventory", :description => "Inventory of all agents on the server" do
+ display :always
+
+ output :agents,
+ :description => "List of agents on the server",
+ :display_as => "Agents"
+end
+
+action "get_config_item", :description => "Get the active value of a specific config property" do
+ display :always
+
+ input :item,
+ :prompt => "Configuration Item",
+ :description => "The item to retrieve from the server",
+ :type => :string,
+ :validation => '^.+$',
+ :optional => false,
+ :maxlength => 50
+
+ output :item,
+ :description => "The config property being retrieved",
+ :display_as => "Property"
+
+ output :value,
+ :description => "The value that is in use",
+ :display_as => "Value"
+
+ summarize do
+ aggregate summary(:value)
+ end
+end
+
+action "get_data", :description => "Get data from a data plugin" do
+ display :always
+
+ input :source,
+ :prompt => "Data Source",
+ :description => "The data plugin to retrieve information from",
+ :type => :string,
+ :validation => '^\w+$',
+ :optional => false,
+ :maxlength => 50
+
+ input :query,
+ :prompt => "Query",
+ :description => "The query argument to supply to the data plugin",
+ :type => :string,
+ :validation => '^.+$',
+ :optional => true,
+ :maxlength => 200
+end
+
+action "ping", :description => "Responds to requests for PING with PONG" do
+ display :always
+
+ output :pong,
+ :description => "The local timestamp",
+ :display_as => "Timestamp"
+end
--- /dev/null
+module MCollective
+ module Agent
+ class Rpcutil<RPC::Agent
+ # Basic system inventory, same as the basic discovery agent
+ action "inventory" do
+ reply[:agents] = Agents.agentlist
+ reply[:facts] = PluginManager["facts_plugin"].get_facts
+ reply[:version] = MCollective.version
+ reply[:classes] = []
+ reply[:main_collective] = config.main_collective
+ reply[:collectives] = config.collectives
+ reply[:data_plugins] = PluginManager.grep(/_data$/)
+
+ cfile = Config.instance.classesfile
+ if File.exist?(cfile)
+ reply[:classes] = File.readlines(cfile).map {|i| i.chomp}
+ end
+ end
+
+ # Retrieve a single fact from the node
+ action "get_fact" do
+ reply[:fact] = request[:fact]
+ reply[:value] = Facts[request[:fact]]
+ end
+
+ # Get the global stats for this mcollectied
+ action "daemon_stats" do
+ stats = PluginManager["global_stats"].to_hash
+
+ reply[:threads] = stats[:threads]
+ reply[:agents] = stats[:agents]
+ reply[:pid] = stats[:pid]
+ reply[:times] = stats[:times]
+ reply[:configfile] = Config.instance.configfile
+ reply[:version] = MCollective.version
+
+ reply.data.merge!(stats[:stats])
+ end
+
+ # Builds an inventory of all agents on teh machine
+ # including license, version and timeout information
+ action "agent_inventory" do
+ reply[:agents] = []
+
+ Agents.agentlist.sort.each do |target_agent|
+ agent = PluginManager["#{target_agent}_agent"]
+ actions = agent.methods.grep(/_agent/)
+
+ agent_data = {:agent => target_agent,
+ :license => "unknown",
+ :timeout => agent.timeout,
+ :description => "unknown",
+ :name => target_agent,
+ :url => "unknown",
+ :version => "unknown",
+ :author => "unknown"}
+
+ agent_data.merge!(agent.meta)
+
+ reply[:agents] << agent_data
+ end
+ end
+
+ # Retrieves a single config property that is in effect
+ action "get_config_item" do
+ reply.fail! "Unknown config property #{request[:item]}" unless config.respond_to?(request[:item])
+
+ reply[:item] = request[:item]
+ reply[:value] = config.send(request[:item])
+ end
+
+ # Responds to PING requests with the local timestamp
+ action "ping" do
+ reply[:pong] = Time.now.to_i
+ end
+
+ # Returns all configured collectives
+ action "collective_info" do
+ config = Config.instance
+ reply[:main_collective] = config.main_collective
+ reply[:collectives] = config.collectives
+ end
+
+ action "get_data" do
+ if request[:query]
+ query = Data.ddl_transform_input(Data.ddl(request[:source]), request[:query].to_s)
+ else
+ query = nil
+ end
+
+ data = Data[ request[:source] ].lookup(query)
+
+ data.keys.each do |key|
+ reply[key] = data[key]
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Aggregate
+ class Average<Base
+ # Before function is run processing
+ def startup_hook
+ @result[:value] = 0
+ @result[:type] = :numeric
+
+ @count = 0
+
+ # Set default aggregate_function if it is undefined
+ @aggregate_format = "Average of #{@result[:output]}: %f" unless @aggregate_format
+ end
+
+ # Determines the average of a set of numerical values
+ def process_result(value, reply)
+ @result[:value] += value
+ @count += 1
+ end
+
+ # Stops execution of the function and returns a ResultObject
+ def summarize
+ @result[:value] /= @count
+
+ result_class(:numeric).new(@result, @aggregate_format, @action)
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Aggregate
+ class Sum<Base
+ def startup_hook
+ @result[:value] = 0
+ @result[:type] = :numeric
+
+ # Set default aggregate_function if it is undefined
+ @aggregate_format = "Sum of #{@result[:output]}: %f" unless @aggregate_format
+ end
+
+ # Determines the average of a set of numerical values
+ def process_result(value, reply)
+ @result[:value] += value
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Aggregate
+ class Summary<Base
+ # Before function is run processing
+ def startup_hook
+ @result[:value] = {}
+ @result[:type] = :collection
+
+ # set default aggregate_format if it is undefined
+ @aggregate_format = :calculate unless @aggregate_format
+ end
+
+ # Increments the value field if value has been seen before
+ # Else create a new value field
+ def process_result(value, reply)
+ unless value.nil?
+ if value.is_a? Array
+ value.map{|val| add_value(val)}
+ else
+ add_value(value)
+ end
+ end
+ end
+
+ def add_value(value)
+ if @result[:value].keys.include?(value)
+ @result[:value][value] += 1
+ else
+ @result[:value][value] = 1
+ end
+ end
+
+ def summarize
+ if @aggregate_format == :calculate
+ max_key_length = @result[:value].keys.map do |k|
+
+ # Response values retain their types. Here we check
+ # if the response is a string and turn it into a string
+ # if it isn't one.
+ if k.respond_to?(:length)
+ k.length
+ elsif k.respond_to?(:to_s)
+ k.to_s.length
+ end
+ end.max
+ @aggregate_format = "%#{max_key_length}s = %s"
+ end
+
+ super
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Application::Completion<MCollective::Application
+ description "Helper for shell completion systems"
+
+ exclude_argument_sections "common", "filter", "rpc"
+
+ option :list_agents,
+ :description => "List all known agents",
+ :arguments => "--list-agents",
+ :required => false,
+ :type => :boolean
+
+ option :list_actions,
+ :description => "List all actions for an agent",
+ :arguments => "--list-actions",
+ :required => false,
+ :type => :boolean
+
+ option :list_inputs,
+ :description => "List all inputs for an action",
+ :arguments => "--list-inputs",
+ :required => false,
+ :type => :boolean
+
+ option :list_applications,
+ :description => "List all known applications",
+ :arguments => "--list-applications",
+ :required => false,
+ :type => :boolean
+
+ option :agent,
+ :description => "The agent to operate on",
+ :arguments => "--agent AGENT",
+ :required => false
+
+ option :action,
+ :description => "The action to operate on",
+ :arguments => "--action ACTION",
+ :required => false
+
+ def list_agents
+ if options[:verbose]
+ PluginManager.find(:agent, "ddl").each do |agent|
+ begin
+ ddl = DDL.new(agent)
+ puts "%s:%s" % [ agent, ddl.meta[:description] ]
+ rescue
+ end
+ end
+ else
+ PluginManager.find(:agent, "ddl").each {|p| puts p}
+ end
+ end
+
+ def list_actions
+ abort "Please specify an agent to list actions for" unless configuration[:agent]
+
+ if options[:verbose]
+ ddl = DDL.new(configuration[:agent], :agent)
+
+ ddl.actions.sort.each do |action|
+ puts "%s:%s" % [action, ddl.action_interface(action)[:description]]
+ end
+ else
+ DDL.new(configuration[:agent], :agent).actions.sort.each {|a| puts a}
+ end
+ rescue
+ end
+
+ def list_inputs
+ abort "Please specify an action and agent to list inputs for" unless configuration[:agent] && configuration[:action]
+
+ if options[:verbose]
+ ddl = DDL.new(configuration[:agent], :agent)
+ action = ddl.action_interface(configuration[:action])
+ action[:input].keys.sort.each do |input|
+ puts "%s:%s" % [input, action[:input][input][:description]]
+ end
+ else
+ DDL.new(configuration[:agent], :agent).action_interface(configuration[:action])[:input].keys.sort.each {|i| puts i}
+ end
+ rescue
+ end
+
+ def list_applications
+ if options[:verbose]
+ Applications.list.each do |app|
+ puts "%s:%s" % [app, Applications[app].application_description]
+ end
+ else
+ Applications.list.each {|a| puts a}
+ end
+ end
+
+ def main
+ actions = configuration.keys.map{|k| k.to_s}.grep(/^list_/)
+
+ abort "Please choose either --list-[agents|actions|inputs|applications]" if actions.empty?
+ abort "Please choose only one of --list-[agents|actions|inputs|applications]" if actions.size > 1
+
+ send actions.first
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Application::Doc<Application
+ description "Error message help"
+ usage "rpc help [ERROR]"
+
+ def post_option_parser(configuration)
+ configuration[:query] = ARGV.shift if ARGV.size > 0
+ end
+
+ def msg_template
+ File.read(File.join(Config.instance.helptemplatedir, "msg-help.erb"))
+ end
+
+ def main
+ if configuration[:query] =~ /^PLMC\d+$/i
+ msg_code = configuration[:query].upcase
+ msg_example = Util.t("%s.example" % msg_code, :raise => true) rescue Util.t("%s.pattern" % msg_code)
+ msg_detail = Util.t("%s.expanded" % msg_code)
+
+ helptext = ERB.new(msg_template, 0, '%')
+ puts helptext.result(binding)
+ end
+ end
+ end
+end
--- /dev/null
+class MCollective::Application::Facts<MCollective::Application
+ description "Reports on usage for a specific fact"
+
+ def post_option_parser(configuration)
+ configuration[:fact] = ARGV.shift if ARGV.size > 0
+ end
+
+ def validate_configuration(configuration)
+ raise "Please specify a fact to report for" unless configuration.include?(:fact)
+ end
+
+ def show_single_fact_report(fact, facts, verbose=false)
+ puts("Report for fact: #{fact}\n\n")
+
+ facts.keys.sort.each do |k|
+ printf(" %-40sfound %d times\n", k, facts[k].size)
+
+ if verbose
+ puts
+
+ facts[k].sort.each do |f|
+ puts(" #{f}")
+ end
+
+ puts
+ end
+ end
+ end
+
+ def main
+ rpcutil = rpcclient("rpcutil")
+ rpcutil.progress = false
+
+ facts = {}
+
+ rpcutil.get_fact(:fact => configuration[:fact]) do |resp|
+ begin
+ value = resp[:body][:data][:value]
+ if value
+ facts.include?(value) ? facts[value] << resp[:senderid] : facts[value] = [ resp[:senderid] ]
+ end
+ rescue Exception => e
+ STDERR.puts "Could not parse facts for #{resp[:senderid]}: #{e.class}: #{e}"
+ end
+ end
+
+ show_single_fact_report(configuration[:fact], facts, options[:verbose])
+
+ printrpcstats
+
+ halt rpcutil.stats
+ end
+end
--- /dev/null
+class MCollective::Application::Find<MCollective::Application
+ description "Find hosts using the discovery system matching filter criteria"
+
+ def main
+ mc = rpcclient("rpcutil")
+
+ starttime = Time.now
+
+ nodes = mc.discover
+
+ discoverytime = Time.now - starttime
+
+ STDERR.puts if options[:verbose]
+
+ nodes.each {|c| puts c}
+
+ STDERR.puts "\nDiscovered %s nodes in %.2f seconds using the %s discovery plugin" % [nodes.size, discoverytime, mc.client.discoverer.discovery_method] if options[:verbose]
+
+ nodes.size > 0 ? exit(0) : exit(1)
+ end
+end
--- /dev/null
+module MCollective
+ class Application::Help<Application
+ description "Application list and help"
+ usage "rpc help [application name]"
+
+ def post_option_parser(configuration)
+ configuration[:application] = ARGV.shift if ARGV.size > 0
+ end
+
+ def main
+ if configuration.include?(:application)
+ puts Applications[configuration[:application]].help
+ else
+ puts "The Marionette Collective version #{MCollective.version}"
+ puts
+
+ Applications.list.sort.each do |app|
+ begin
+ puts " %-15s %s" % [app, Applications[app].application_description]
+ rescue
+ end
+ end
+
+ puts
+ end
+ end
+ end
+end
--- /dev/null
+class MCollective::Application::Inventory<MCollective::Application
+ description "General reporting tool for nodes, collectives and subcollectives"
+
+ option :script,
+ :description => "Script to run",
+ :arguments => ["--script SCRIPT"]
+
+ option :collectives,
+ :description => "List all known collectives",
+ :arguments => ["--list-collectives", "--lc"],
+ :default => false,
+ :type => :bool
+
+ option :collectivemap,
+ :description => "Create a DOT graph of all collectives",
+ :arguments => ["--collective-graph MAP", "--cg MAP", "--map MAP"]
+
+ def post_option_parser(configuration)
+ configuration[:node] = ARGV.shift if ARGV.size > 0
+ end
+
+ def validate_configuration(configuration)
+ unless configuration[:node] || configuration[:script] || configuration[:collectives] || configuration[:collectivemap]
+ raise "Need to specify either a node name, script to run or other options"
+ end
+ end
+
+ # Get all the known collectives and nodes that belong to them
+ def get_collectives
+ util = rpcclient("rpcutil")
+ util.progress = false
+
+ collectives = {}
+ nodes = 0
+ total = 0
+
+ util.collective_info do |r, cinfo|
+ begin
+ if cinfo[:data] && cinfo[:data][:collectives]
+ cinfo[:data][:collectives].each do |collective|
+ collectives[collective] ||= []
+ collectives[collective] << cinfo[:sender]
+ end
+
+ nodes += 1
+ total += 1
+ end
+ end
+ end
+
+ {:collectives => collectives, :nodes => nodes, :total_nodes => total}
+ end
+
+ # Writes a crude DOT graph to a file
+ def collectives_map(file)
+ File.open(file, "w") do |graph|
+ puts "Retrieving collective info...."
+ collectives = get_collectives
+
+ graph.puts 'graph {'
+
+ collectives[:collectives].keys.sort.each do |collective|
+ graph.puts ' subgraph "%s" {' % [ collective ]
+
+ collectives[:collectives][collective].each do |member|
+ graph.puts ' "%s" -- "%s"' % [ member, collective ]
+ end
+
+ graph.puts ' }'
+ end
+
+ graph.puts '}'
+
+ puts "Graph of #{collectives[:total_nodes]} nodes has been written to #{file}"
+ end
+ end
+
+ # Prints a report of all known sub collectives
+ def collectives_report
+ collectives = get_collectives
+
+ puts " %-30s %s" % [ "Collective", "Nodes" ]
+ puts " %-30s %s" % [ "==========", "=====" ]
+
+ collectives[:collectives].sort_by {|key,count| count.size}.each do |collective|
+ puts " %-30s %d" % [ collective[0], collective[1].size ]
+ end
+
+ puts
+ puts " %30s %d" % [ "Total nodes:", collectives[:nodes] ]
+ puts
+ end
+
+ def node_inventory
+ node = configuration[:node]
+
+ util = rpcclient("rpcutil")
+ util.identity_filter node
+ util.progress = false
+
+ nodestats = util.custom_request("daemon_stats", {}, node, {"identity" => node}).first
+
+ unless nodestats
+ STDERR.puts "Did not receive any results from node #{node}"
+ exit 1
+ end
+
+ unless nodestats[:statuscode] == 0
+ STDERR.puts "Failed to retrieve daemon_stats from #{node}: #{nodestats[:statusmsg]}"
+ else
+ util.custom_request("inventory", {}, node, {"identity" => node}).each do |resp|
+ unless resp[:statuscode] == 0
+ STDERR.puts "Failed to retrieve inventory for #{node}: #{resp[:statusmsg]}"
+ next
+ end
+
+ data = resp[:data]
+
+ begin
+ puts "Inventory for #{resp[:sender]}:"
+ puts
+
+ nodestats = nodestats[:data]
+
+ puts " Server Statistics:"
+ puts " Version: #{nodestats[:version]}"
+ puts " Start Time: #{Time.at(nodestats[:starttime])}"
+ puts " Config File: #{nodestats[:configfile]}"
+ puts " Collectives: #{data[:collectives].join(', ')}" if data.include?(:collectives)
+ puts " Main Collective: #{data[:main_collective]}" if data.include?(:main_collective)
+ puts " Process ID: #{nodestats[:pid]}"
+ puts " Total Messages: #{nodestats[:total]}"
+ puts " Messages Passed Filters: #{nodestats[:passed]}"
+ puts " Messages Filtered: #{nodestats[:filtered]}"
+ puts " Expired Messages: #{nodestats[:ttlexpired]}"
+ puts " Replies Sent: #{nodestats[:replies]}"
+ puts " Total Processor Time: #{nodestats[:times][:utime]} seconds"
+ puts " System Time: #{nodestats[:times][:stime]} seconds"
+
+ puts
+
+ puts " Agents:"
+ if data[:agents].size > 0
+ data[:agents].sort.in_groups_of(3, "") do |agents|
+ puts " %-15s %-15s %-15s" % agents
+ end
+ else
+ puts " No agents installed"
+ end
+
+ puts
+
+ puts " Data Plugins:"
+ if data[:data_plugins].size > 0
+ data[:data_plugins].sort.in_groups_of(3, "") do |plugins|
+ puts " %-15s %-15s %-15s" % plugins.map{|p| p.gsub("_data", "")}
+ end
+ else
+ puts " No data plugins installed"
+ end
+
+ puts
+
+ puts " Configuration Management Classes:"
+ if data[:classes].size > 0
+ data[:classes].sort.in_groups_of(2, "") do |klasses|
+ puts " %-30s %-30s" % klasses
+ end
+ else
+ puts " No classes applied"
+ end
+
+ puts
+
+ puts " Facts:"
+ if data[:facts].size > 0
+ data[:facts].sort_by{|f| f[0]}.each do |f|
+ puts " #{f[0]} => #{f[1]}"
+ end
+ else
+ puts " No facts known"
+ end
+
+ break
+ rescue Exception => e
+ STDERR.puts "Failed to display node inventory: #{e.class}: #{e}"
+ end
+ end
+ end
+
+ halt util.stats
+ end
+
+ # Helpers to create a simple DSL for scriptlets
+ def format(fmt)
+ @fmt = fmt
+ end
+
+ def fields(&blk)
+ @flds = blk
+ end
+
+ def identity
+ @node[:identity]
+ end
+
+ def facts
+ @node[:facts]
+ end
+
+ def classes
+ @node[:classes]
+ end
+
+ def agents
+ @node[:agents]
+ end
+
+ def page_length(len)
+ @page_length = len
+ end
+
+ def page_heading(fmt)
+ @page_heading = fmt
+ end
+
+ def page_body(fmt)
+ @page_body = fmt
+ end
+
+ # Expects a simple printf style format and apply it to
+ # each node:
+ #
+ # inventory do
+ # format "%s:\t\t%s\t\t%s"
+ #
+ # fields { [ identity, facts["serialnumber"], facts["productname"] ] }
+ # end
+ def inventory(&blk)
+ raise "Need to give a block to inventory" unless block_given?
+
+ blk.call if block_given?
+
+ raise "Need to define a format" if @fmt.nil?
+ raise "Need to define inventory fields" if @flds.nil?
+
+ util = rpcclient("rpcutil")
+ util.progress = false
+
+ util.inventory do |t, resp|
+ @node = {:identity => resp[:sender],
+ :facts => resp[:data][:facts],
+ :classes => resp[:data][:classes],
+ :agents => resp[:data][:agents]}
+
+ puts @fmt % @flds.call
+ end
+ end
+
+ # Use the ruby formatr gem to build reports using Perls formats
+ #
+ # It is kind of ugly but brings a lot of flexibility in report
+ # writing without building an entire reporting language.
+ #
+ # You need to have formatr installed to enable reports like:
+ #
+ # formatted_inventory do
+ # page_length 20
+ #
+ # page_heading <<TOP
+ #
+ # Node Report @<<<<<<<<<<<<<<<<<<<<<<<<<
+ # time
+ #
+ # Hostname: Customer: Distribution:
+ # -------------------------------------------------------------------------
+ # TOP
+ #
+ # page_body <<BODY
+ #
+ # @<<<<<<<<<<<<<<<< @<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+ # identity, facts["customer"], facts["lsbdistdescription"]
+ # @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+ # facts["processor0"]
+ # BODY
+ # end
+ def formatted_inventory(&blk)
+ require 'formatr'
+
+ raise "Need to give a block to formatted_inventory" unless block_given?
+
+ blk.call if block_given?
+
+ raise "Need to define page body format" if @page_body.nil?
+
+ body_fmt = FormatR::Format.new(@page_heading, @page_body)
+ body_fmt.setPageLength(@page_length)
+ time = Time.now
+
+ util = rpcclient("rpcutil")
+ util.progress = false
+
+ util.inventory do |t, resp|
+ @node = {:identity => resp[:sender],
+ :facts => resp[:data][:facts],
+ :classes => resp[:data][:classes],
+ :agents => resp[:data][:agents]}
+
+ body_fmt.printFormat(binding)
+ end
+ rescue Exception => e
+ STDERR.puts "Could not create report: #{e.class}: #{e}"
+ exit 1
+ end
+
+ @fmt = nil
+ @flds = nil
+ @page_heading = nil
+ @page_body = nil
+ @page_length = 40
+
+ def main
+ if configuration[:script]
+ if File.exist?(configuration[:script])
+ eval(File.read(configuration[:script]))
+ else
+ raise "Could not find script to run: #{configuration[:script]}"
+ end
+
+ elsif configuration[:collectivemap]
+ collectives_map(configuration[:collectivemap])
+
+ elsif configuration[:collectives]
+ collectives_report
+
+ else
+ node_inventory
+ end
+ end
+end
--- /dev/null
+# encoding: utf-8
+module MCollective
+ class Application::Ping<Application
+ description "Ping all nodes"
+
+ option :graph,
+ :description => "Shows a graph of ping distribution",
+ :arguments => ["--graph", "-g"],
+ :default => false,
+ :type => :bool
+
+ # Convert the times structure into a array representing
+ # buckets of responses in 50 ms intervals. Return a small
+ # sparkline graph using UTF8 characters
+ def spark(resp_times)
+ return "" unless configuration[:graph] || Config.instance.pluginconf["rpc.graph"]
+
+ ticks=%w[▁ ▂ ▃ ▄ ▅ ▆ ▇]
+
+ histo = {}
+
+ # round each time to its nearest 50ms
+ # and keep a count for each 50ms
+ resp_times.each do |time|
+ time = Integer(time + 50 - (time % 50))
+ histo[time] ||= 0
+ histo[time] += 1
+ end
+
+ # set the 50ms intervals that saw no traffic to 0
+ ((histo.keys.max - histo.keys.min) / 50).times do |i|
+ time = (i * 50) + histo.keys.min
+ histo[time] = 0 unless histo[time]
+ end
+
+ # get a numerically sorted list of times
+ histo = histo.keys.sort.map{|k| histo[k]}
+
+ range = histo.max - histo.min
+ scale = ticks.size - 1
+ distance = histo.max.to_f / scale
+
+ histo.map do |val|
+ tick = (val / distance).round
+ tick = 0 if tick < 0
+
+ ticks[tick]
+ end.join
+ end
+
+ def main
+ client = MCollective::Client.new(options[:config])
+ client.options = options
+
+ start = Time.now.to_f
+ times = []
+
+ client.req("ping", "discovery") do |resp|
+ times << (Time.now.to_f - start) * 1000
+
+ puts "%-40s time=%.2f ms" % [resp[:senderid], times.last]
+ end
+
+ puts("\n\n---- ping statistics ----")
+
+ if times.size > 0
+ sum = times.inject(0){|acc,i|acc +i}
+ avg = sum / times.length.to_f
+
+ puts "%d replies max: %.2f min: %.2f avg: %.2f %s" % [times.size, times.max, times.min, avg, spark(times)]
+ else
+ puts("No responses received")
+ end
+
+ halt client.stats
+ end
+ end
+end
--- /dev/null
+module MCollective
+ class Application::Plugin<Application
+
+ exclude_argument_sections "common", "filter", "rpc"
+
+ description "MCollective Plugin Application"
+ usage <<-END_OF_USAGE
+mco plugin package [options] <directory>
+ mco plugin info <directory>
+ mco plugin doc <plugin>
+ mco plugin doc <type/plugin>
+ mco plugin generate agent <pluginname> [actions=val,val]
+ mco plugin generate data <pluginname> [outputs=val,val]
+
+ info : Display plugin information including package details.
+ package : Create all available plugin packages.
+ doc : Display documentation for a specific plugin.
+ END_OF_USAGE
+
+ option :pluginname,
+ :description => "Plugin name",
+ :arguments => ["-n", "--name NAME"],
+ :type => String
+
+ option :postinstall,
+ :description => "Post install script",
+ :arguments => ["--postinstall POSTINSTALL"],
+ :type => String
+
+ option :preinstall,
+ :description => "Pre install script",
+ :arguments => ["--preinstall PREINSTALL"],
+ :type => String
+
+ option :iteration,
+ :description => "Iteration number",
+ :arguments => ["--iteration ITERATION"],
+ :type => String
+
+ option :vendor,
+ :description => "Vendor name",
+ :arguments => ["--vendor VENDOR"],
+ :type => String
+
+ option :pluginpath,
+ :description => "MCollective plugin path",
+ :arguments => ["--pluginpath PATH"],
+ :type => String
+
+ option :mcname,
+ :description => "MCollective type (mcollective, pe-mcollective) that the packages depend on",
+ :arguments => ["--mcname NAME"],
+ :type => String
+
+ option :mcversion,
+ :description => "Version of MCollective that the packages depend on",
+ :arguments => "--mcversion MCVERSION",
+ :type => String
+
+ option :dependency,
+ :description => "Adds a dependency to the plugin",
+ :arguments => ["--dependency DEPENDENCIES"],
+ :type => :array
+
+ option :format,
+ :description => "Package output format. Defaults to rpmpackage or debpackage",
+ :arguments => ["--format OUTPUTFORMAT"],
+ :type => String
+
+ option :sign,
+ :description => "Embed a signature in the package",
+ :arguments => ["--sign"],
+ :type => :boolean
+
+ option :rpctemplate,
+ :description => "Template to use.",
+ :arguments => ["--template HELPTEMPLATE"],
+ :type => String
+
+ option :description,
+ :description => "Plugin description",
+ :arguments => ["--description DESCRIPTION"],
+ :type => String
+
+ option :author,
+ :description => "The author of the plugin",
+ :arguments => ["--author AUTHOR"],
+ :type => String
+
+ option :license,
+ :description => "The license under which the plugin is distributed",
+ :arguments => ["--license LICENSE"],
+ :type => String
+
+ option :version,
+ :description => "The version of the plugin",
+ :arguments => ["--pluginversion VERSION"],
+ :type => String
+
+ option :url,
+ :description => "Url at which information about the plugin can be found",
+ :arguments => ["--url URL"],
+ :type => String
+
+ option :timeout,
+ :description => "The plugin's timeout",
+ :arguments => ["--timeout TIMEOUT"],
+ :type => Integer
+
+ option :actions,
+ :description => "Actions to be generated for an Agent Plugin",
+ :arguments => ["--actions [ACTIONS]"],
+ :type => Array
+
+ option :outputs,
+ :description => "Outputs to be generated for an Data Plugin",
+ :arguments => ["--outputs [OUTPUTS]"],
+ :type => Array
+
+ # Handle alternative format that optparser can't parse.
+ def post_option_parser(configuration)
+ if ARGV.length >= 1
+ configuration[:action] = ARGV.delete_at(0)
+
+ configuration[:target] = ARGV.delete_at(0) || "."
+
+ if configuration[:action] == "generate"
+ unless ARGV[0] && ARGV[0].match(/(actions|outputs)=(.+)/i)
+ unless configuration[:pluginname]
+ configuration[:pluginname] = ARGV.delete_at(0)
+ else
+ ARGV.delete_at(0)
+ end
+ end
+
+ ARGV.each do |argument|
+ if argument.match(/(actions|outputs)=(.+)/i)
+ configuration[$1.downcase.to_sym]= $2.split(",")
+ else
+ raise "Could not parse --arg '#{argument}'"
+ end
+ end
+ end
+ end
+ end
+
+ # Display info about plugin
+ def info_command
+ plugin = prepare_plugin
+ packager = PluginPackager["#{configuration[:format].capitalize}Packager"]
+ packager.new(plugin).package_information
+ end
+
+ # Generate a plugin skeleton
+ def generate_command
+ raise "undefined plugin type. cannot generate plugin. valid types are 'agent' and 'data'" if configuration["target"] == '.'
+
+ unless configuration[:pluginname]
+ puts "No plugin name specified. Using 'new_plugin'"
+ configuration[:pluginname] = "new_plugin"
+ end
+
+ load_plugin_config_values
+
+ case configuration[:target].downcase
+ when 'agent'
+ Generators::AgentGenerator.new(configuration[:pluginname], configuration[:actions], configuration[:pluginname],
+ configuration[:description], configuration[:author], configuration[:license],
+ configuration[:version], configuration[:url], configuration[:timeout])
+ when 'data'
+ raise "data plugin must have at least one output" unless configuration[:outputs]
+ Generators::DataGenerator.new(configuration[:pluginname], configuration[:outputs], configuration[:pluginname],
+ configuration[:description], configuration[:author], configuration[:license],
+ configuration[:version], configuration[:url], configuration[:timeout])
+ else
+ raise "invalid plugin type. cannot generate plugin '#{configuration[:target]}'"
+ end
+ end
+
+ # Package plugin
+ def package_command
+ if configuration[:sign] && Config.instance.pluginconf.include?("debian_packager.keyname")
+ configuration[:sign] = Config.instance.pluginconf["debian_packager.keyname"]
+ configuration[:sign] = "\"#{configuration[:sign]}\"" unless configuration[:sign].match(/\".*\"/)
+ end
+
+ plugin = prepare_plugin
+ (configuration[:pluginpath] = configuration[:pluginpath] + "/") if (configuration[:pluginpath] && !configuration[:pluginpath].match(/^.*\/$/))
+ packager = PluginPackager["#{configuration[:format].capitalize}Packager"]
+ packager.new(plugin, configuration[:pluginpath], configuration[:sign], options[:verbose]).create_packages
+ end
+
+ # Agents are just called 'agent' but newer plugin types are
+ # called plugin_plugintype for example facter_facts etc so
+ # this will first try the old way then the new way.
+ def load_plugin_ddl(plugin, type)
+ [plugin, "#{plugin}_#{type}"].each do |p|
+ ddl = DDL.new(p, type, false)
+ if ddl.findddlfile(p, type)
+ ddl.loadddlfile
+ return ddl
+ end
+ end
+ end
+
+ # Show application list and plugin help
+ def doc_command
+ known_plugin_types = [["Agents", :agent], ["Data Queries", :data], ["Discovery Methods", :discovery], ["Validator Plugins", :validator]]
+
+ if configuration.include?(:target) && configuration[:target] != "."
+ if configuration[:target] =~ /^(.+?)\/(.+)$/
+ ddl = load_plugin_ddl($2.to_sym, $1)
+ else
+ found_plugin_type = nil
+
+ known_plugin_types.each do |plugin_type|
+ PluginManager.find(plugin_type[1], "ddl").each do |ddl|
+ pluginname = ddl.gsub(/_#{plugin_type[1]}$/, "")
+ if pluginname == configuration[:target]
+ abort "Duplicate plugin name found, please specify a full path like agent/rpcutil" if found_plugin_type
+ found_plugin_type = plugin_type[1]
+ end
+ end
+ end
+
+ abort "Could not find a plugin named %s in any supported plugin type" % configuration[:target] unless found_plugin_type
+
+ ddl = load_plugin_ddl(configuration[:target], found_plugin_type)
+ end
+
+ puts ddl.help(configuration[:rpctemplate])
+ else
+ puts "Please specify a plugin. Available plugins are:"
+ puts
+
+ load_errors = []
+
+ known_plugin_types.each do |plugin_type|
+ puts "%s:" % plugin_type[0]
+
+ PluginManager.find(plugin_type[1], "ddl").each do |ddl|
+ begin
+ help = DDL.new(ddl, plugin_type[1])
+ pluginname = ddl.gsub(/_#{plugin_type[1]}$/, "")
+ puts " %-25s %s" % [pluginname, help.meta[:description]]
+ rescue => e
+ load_errors << [plugin_type[1], ddl, e]
+ end
+ end
+
+ puts
+ end
+
+ unless load_errors.empty?
+ puts "Plugin Load Errors:"
+
+ load_errors.each do |e|
+ puts " %-25s %s" % ["#{e[0]}/#{e[1]}", Util.colorize(:yellow, e[2])]
+ end
+ end
+ end
+ end
+
+ # Creates the correct package plugin object.
+ def prepare_plugin
+ plugintype = set_plugin_type unless configuration[:plugintype]
+ configuration[:format] = "ospackage" unless configuration[:format]
+ PluginPackager.load_packagers
+ plugin_class = PluginPackager[configuration[:plugintype]]
+ configuration[:dependency] = configuration[:dependency][0].split(" ") if configuration[:dependency] && configuration[:dependency].size == 1
+ configuration[:dependency].map!{|dep| {:name => dep, :version => nil}} if configuration[:dependency]
+ mcdependency = {:mcname => configuration[:mcname], :mcversion => configuration[:mcversion]}
+
+ plugin_class.new(configuration[:target], configuration[:pluginname],
+ configuration[:vendor], configuration[:preinstall],
+ configuration[:postinstall], configuration[:iteration],
+ configuration[:dependency], mcdependency , plugintype)
+ end
+
+ def directory_for_type(type)
+ File.directory?(File.join(configuration[:target], type))
+ end
+
+ # Identify plugin type if not provided.
+ def set_plugin_type
+ if directory_for_type("agent") || directory_for_type("application")
+ configuration[:plugintype] = "AgentDefinition"
+ return "Agent"
+ elsif directory_for_type(plugintype = identify_plugin)
+ configuration[:plugintype] = "StandardDefinition"
+ return plugintype
+ else
+ raise RuntimeError, "target directory is not a valid mcollective plugin"
+ end
+ end
+
+ # If plugintype is StandardDefinition, identify which of the special
+ # plugin types we are dealing with based on directory structure.
+ # To keep it simple we limit it to one type per target directory.
+ def identify_plugin
+ plugintype = Dir.glob(File.join(configuration[:target], "*")).select do |file|
+ File.directory?(file) && file.match(/(connector|facts|registration|security|audit|pluginpackager|data|discovery|validator)/)
+ end
+
+ raise RuntimeError, "more than one plugin type detected in directory" if plugintype.size > 1
+ raise RuntimeError, "no plugins detected in directory" if plugintype.size < 1
+
+ File.basename(plugintype[0])
+ end
+
+ # Load preset metadata values from config if they are present
+ # This makes it possible to override metadata values in a local
+ # client config file.
+ #
+ # Example : plugin.metadata.license = Apache 2
+ def load_plugin_config_values
+ config = Config.instance
+ [:pluginname, :description, :author, :license, :version, :url, :timeout].each do |confoption|
+ configuration[confoption] = config.pluginconf["metadata.#{confoption}"] unless configuration[confoption]
+ end
+ end
+
+ def main
+ abort "No action specified, please run 'mco help plugin' for help" unless configuration.include?(:action)
+
+ cmd = "#{configuration[:action]}_command"
+
+ if respond_to? cmd
+ send cmd
+ else
+ abort "Invalid action #{configuration[:action]}, please run 'mco help plugin' for help."
+ end
+ end
+ end
+end
--- /dev/null
+class MCollective::Application::Rpc<MCollective::Application
+ description "Generic RPC agent client application"
+
+ usage "mco rpc [options] [filters] --agent <agent> --action <action> [--argument <key=val> --argument ...]"
+ usage "mco rpc [options] [filters] <agent> <action> [<key=val> <key=val> ...]"
+
+ option :show_results,
+ :description => "Do not process results, just send request",
+ :arguments => ["--no-results", "--nr"],
+ :default => true,
+ :type => :bool
+
+ option :agent,
+ :description => "Agent to call",
+ :arguments => ["-a", "--agent AGENT"]
+
+ option :action,
+ :description => "Action to call",
+ :arguments => ["--action ACTION"]
+
+ option :arguments,
+ :description => "Arguments to pass to agent",
+ :arguments => ["--arg", "--argument ARGUMENT"],
+ :type => :array,
+ :default => [],
+ :validate => Proc.new {|val| val.match(/^(.+?)=(.+)$/) ? true : "Could not parse --arg #{val} should be of the form key=val" }
+
+ def post_option_parser(configuration)
+ # handle the alternative format that optparse cant parse
+ unless (configuration.include?(:agent) && configuration.include?(:action))
+ if ARGV.length >= 2
+ configuration[:agent] = ARGV[0]
+ ARGV.delete_at(0)
+
+ configuration[:action] = ARGV[0]
+ ARGV.delete_at(0)
+
+ ARGV.each do |v|
+ if v =~ /^(.+?)=(.+)$/
+ configuration[:arguments] = [] unless configuration.include?(:arguments)
+ configuration[:arguments] << v
+ else
+ STDERR.puts("Could not parse --arg #{v}")
+ exit(1)
+ end
+ end
+ else
+ STDERR.puts("No agent, action and arguments specified")
+ exit(1)
+ end
+ end
+
+ # convert arguments to symbols for keys to comply with simplerpc conventions
+ args = configuration[:arguments].clone
+ configuration[:arguments] = {}
+
+ args.each do |v|
+ if v =~ /^(.+?)=(.+)$/
+ configuration[:arguments][$1.to_sym] = $2
+ end
+ end
+ end
+
+ def string_to_ddl_type(arguments, ddl)
+ return if ddl.empty?
+
+ arguments.keys.each do |key|
+ if ddl[:input].keys.include?(key)
+ case ddl[:input][key][:type]
+ when :boolean
+ arguments[key] = MCollective::DDL.string_to_boolean(arguments[key])
+
+ when :number, :integer, :float
+ arguments[key] = MCollective::DDL.string_to_number(arguments[key])
+ end
+ end
+ end
+ end
+
+ def main
+ mc = rpcclient(configuration[:agent])
+
+ mc.agent_filter(configuration[:agent])
+
+ string_to_ddl_type(configuration[:arguments], mc.ddl.action_interface(configuration[:action])) if mc.ddl
+
+ mc.validate_request(configuration[:action], configuration[:arguments])
+
+ if mc.reply_to
+ configuration[:arguments][:process_results] = true
+
+ puts "Request sent with id: " + mc.send(configuration[:action], configuration[:arguments]) + " replies to #{mc.reply_to}"
+ elsif !configuration[:show_results]
+ configuration[:arguments][:process_results] = false
+
+ puts "Request sent with id: " + mc.send(configuration[:action], configuration[:arguments])
+ else
+ # if there's stuff on STDIN assume its JSON that came from another
+ # rpc or printrpc, we feed that in as discovery data
+ discover_args = {:verbose => true}
+
+ unless STDIN.tty?
+ discovery_data = STDIN.read.chomp
+ discover_args = {:json => discovery_data} unless discovery_data == ""
+ end
+
+ mc.discover discover_args
+
+ printrpc mc.send(configuration[:action], configuration[:arguments])
+
+ printrpcstats :summarize => true, :caption => "#{configuration[:agent]}##{configuration[:action]} call stats" if mc.discover.size > 0
+
+ halt mc.stats
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module RPC
+ # An audit plugin that just logs to a file
+ #
+ # You can configure which file it logs to with the setting
+ #
+ # plugin.rpcaudit.logfile
+ #
+ class Logfile<Audit
+ require 'pp'
+
+ def audit_request(request, connection)
+ logfile = Config.instance.pluginconf["rpcaudit.logfile"] || "/var/log/mcollective-audit.log"
+
+ now = Time.now
+ now_tz = tz = now.utc? ? "Z" : now.strftime("%z")
+ now_iso8601 = "%s.%06d%s" % [now.strftime("%Y-%m-%dT%H:%M:%S"), now.tv_usec, now_tz]
+
+ File.open(logfile, "a") do |f|
+ f.puts("#{now_iso8601}: reqid=#{request.uniqid}: reqtime=#{request.time} caller=#{request.caller}@#{request.sender} agent=#{request.agent} action=#{request.action} data=#{request.data.pretty_print_inspect}")
+ end
+ end
+ end
+ end
+end
--- /dev/null
+require 'stomp'
+
+module MCollective
+ module Connector
+ # Handles sending and receiving messages over the Stomp protocol for ActiveMQ
+ # servers specifically, we take advantages of ActiveMQ specific features and
+ # enhancements to the Stomp protocol. For best results in a clustered environment
+ # use ActiveMQ 5.5.0 at least.
+ #
+ # This plugin takes an entirely different approach to dealing with ActiveMQ
+ # from the more generic stomp connector.
+ #
+ # - Agents use /topic/<collective>.<agent>.agent
+ # - Replies use temp-topics so they are private and transient.
+ # - Point to Point messages using topics are supported by subscribing to
+ # /queue/<collective>.nodes with a selector "mc_identity = 'identity'
+ #
+ # The use of temp-topics for the replies is a huge improvement over the old style.
+ # In the old way all clients got replies for all clients that were active at that
+ # time, this would mean that they would need to decrypt, validate etc in order to
+ # determine if they need to ignore the message, this was computationally expensive
+ # and on large busy networks the messages were being sent all over the show cross
+ # broker boundaries.
+ #
+ # The new way means the messages go point2point back to only whoever requested the
+ # message, they only get their own replies and this is ap private channel that
+ # casual observers cannot just snoop into.
+ #
+ # This plugin supports 1.1.6 and newer of the Stomp rubygem.
+ #
+ # connector = activemq
+ # plugin.activemq.pool.size = 2
+ #
+ # plugin.activemq.pool.1.host = stomp1.your.net
+ # plugin.activemq.pool.1.port = 61613
+ # plugin.activemq.pool.1.user = you
+ # plugin.activemq.pool.1.password = secret
+ # plugin.activemq.pool.1.ssl = true
+ # plugin.activemq.pool.1.ssl.cert = /path/to/your.cert
+ # plugin.activemq.pool.1.ssl.key = /path/to/your.key
+ # plugin.activemq.pool.1.ssl.ca = /path/to/your.ca
+ # plugin.activemq.pool.1.ssl.fallback = true
+ #
+ # plugin.activemq.pool.2.host = stomp2.your.net
+ # plugin.activemq.pool.2.port = 61613
+ # plugin.activemq.pool.2.user = you
+ # plugin.activemq.pool.2.password = secret
+ # plugin.activemq.pool.2.ssl = false
+ #
+ # Using this method you can supply just STOMP_USER and STOMP_PASSWORD. The port will
+ # default to 61613 if not specified.
+ #
+ # The ssl options are only usable in version of the Stomp gem newer than 1.2.2 where these
+ # will imply full SSL validation will be done and you'll only be able to connect to a
+ # ActiveMQ server that has a cert signed by the same CA. If you only set ssl = true
+ # and do not supply the cert, key and ca properties or if you have an older gem it
+ # will fall back to unverified mode only if ssl.fallback is true
+ #
+ # In addition you can set the following options for the rubygem:
+ #
+ # plugin.activemq.initial_reconnect_delay = 0.01
+ # plugin.activemq.max_reconnect_delay = 30.0
+ # plugin.activemq.use_exponential_back_off = true
+ # plugin.activemq.back_off_multiplier = 2
+ # plugin.activemq.max_reconnect_attempts = 0
+ # plugin.activemq.randomize = false
+ # plugin.activemq.timeout = -1
+ #
+ # You can set the initial connetion timeout - this is when your stomp server is simply
+ # unreachable - after which it would failover to the next in the pool:
+ #
+ # plugin.activemq.connect_timeout = 30
+ #
+ # ActiveMQ JMS message priorities can be set:
+ #
+ # plugin.activemq.priority = 4
+ #
+ class Activemq<Base
+ attr_reader :connection
+
+ # Older stomp gems do not have these error classes, in order to be able to
+ # handle these exceptions if they are present and still support older gems
+ # we're assigning the constants to a dummy exception that will never be thrown
+ # by us. End result is that the code catching these exceptions become noops on
+ # older gems but on newer ones they become usable and handle those new errors
+ # intelligently
+ class DummyError<RuntimeError; end
+
+ ::Stomp::Error = DummyError unless defined?(::Stomp::Error)
+ ::Stomp::Error::NoCurrentConnection = DummyError unless defined?(::Stomp::Error::NoCurrentConnection)
+ ::Stomp::Error::DuplicateSubscription = DummyError unless defined?(::Stomp::Error::DuplicateSubscription)
+
+ # Class for Stomp 1.1.9 callback based logging
+ class EventLogger
+ def on_connecting(params=nil)
+ Log.info("TCP Connection attempt %d to %s" % [params[:cur_conattempts], stomp_url(params)])
+ rescue
+ end
+
+ def on_connected(params=nil)
+ Log.info("Conncted to #{stomp_url(params)}")
+ rescue
+ end
+
+ def on_disconnect(params=nil)
+ Log.info("Disconnected from #{stomp_url(params)}")
+ rescue
+ end
+
+ def on_connectfail(params=nil)
+ Log.info("TCP Connection to #{stomp_url(params)} failed on attempt #{params[:cur_conattempts]}")
+ rescue
+ end
+
+ def on_miscerr(params, errstr)
+ Log.error("Unexpected error on connection #{stomp_url(params)}: #{errstr}")
+ rescue
+ end
+
+ def on_ssl_connecting(params)
+ Log.info("Estblishing SSL session with #{stomp_url(params)}")
+ rescue
+ end
+
+ def on_ssl_connected(params)
+ Log.info("SSL session established with #{stomp_url(params)}")
+ rescue
+ end
+
+ def on_ssl_connectfail(params)
+ Log.error("SSL session creation with #{stomp_url(params)} failed: #{params[:ssl_exception]}")
+ end
+
+ def stomp_url(params)
+ "%s://%s@%s:%d" % [ params[:cur_ssl] ? "stomp+ssl" : "stomp", params[:cur_login], params[:cur_host], params[:cur_port]]
+ end
+ end
+
+ def initialize
+ @config = Config.instance
+ @subscriptions = []
+ @msgpriority = 0
+ @base64 = false
+ end
+
+ # Connects to the ActiveMQ middleware
+ def connect(connector = ::Stomp::Connection)
+ if @connection
+ Log.debug("Already connection, not re-initializing connection")
+ return
+ end
+
+ begin
+ @base64 = get_bool_option("activemq.base64", false)
+ @msgpriority = get_option("activemq.priority", 0).to_i
+
+ pools = @config.pluginconf["activemq.pool.size"].to_i
+ hosts = []
+
+ 1.upto(pools) do |poolnum|
+ host = {}
+
+ host[:host] = get_option("activemq.pool.#{poolnum}.host")
+ host[:port] = get_option("activemq.pool.#{poolnum}.port", 61613).to_i
+ host[:login] = get_env_or_option("STOMP_USER", "activemq.pool.#{poolnum}.user")
+ host[:passcode] = get_env_or_option("STOMP_PASSWORD", "activemq.pool.#{poolnum}.password")
+ host[:ssl] = get_bool_option("activemq.pool.#{poolnum}.ssl", false)
+
+ host[:ssl] = ssl_parameters(poolnum, get_bool_option("activemq.pool.#{poolnum}.ssl.fallback", false)) if host[:ssl]
+
+ Log.debug("Adding #{host[:host]}:#{host[:port]} to the connection pool")
+ hosts << host
+ end
+
+ raise "No hosts found for the ActiveMQ connection pool" if hosts.size == 0
+
+ connection = {:hosts => hosts}
+
+ # Various STOMP gem options, defaults here matches defaults for 1.1.6 the meaning of
+ # these can be guessed, the documentation isn't clear
+ connection[:initial_reconnect_delay] = Float(get_option("activemq.initial_reconnect_delay", 0.01))
+ connection[:max_reconnect_delay] = Float(get_option("activemq.max_reconnect_delay", 30.0))
+ connection[:use_exponential_back_off] = get_bool_option("activemq.use_exponential_back_off", true)
+ connection[:back_off_multiplier] = Integer(get_option("activemq.back_off_multiplier", 2))
+ connection[:max_reconnect_attempts] = Integer(get_option("activemq.max_reconnect_attempts", 0))
+ connection[:randomize] = get_bool_option("activemq.randomize", false)
+ connection[:backup] = get_bool_option("activemq.backup", false)
+ connection[:timeout] = Integer(get_option("activemq.timeout", -1))
+ connection[:connect_timeout] = Integer(get_option("activemq.connect_timeout", 30))
+ connection[:reliable] = true
+
+ connection[:logger] = EventLogger.new
+
+ @connection = connector.new(connection)
+ rescue Exception => e
+ raise("Could not connect to ActiveMQ Server: #{e}")
+ end
+ end
+
+ # Sets the SSL paramaters for a specific connection
+ def ssl_parameters(poolnum, fallback)
+ params = {:cert_file => get_option("activemq.pool.#{poolnum}.ssl.cert", false),
+ :key_file => get_option("activemq.pool.#{poolnum}.ssl.key", false),
+ :ts_files => get_option("activemq.pool.#{poolnum}.ssl.ca", false)}
+
+ raise "cert, key and ca has to be supplied for verified SSL mode" unless params[:cert_file] && params[:key_file] && params[:ts_files]
+
+ raise "Cannot find certificate file #{params[:cert_file]}" unless File.exist?(params[:cert_file])
+ raise "Cannot find key file #{params[:key_file]}" unless File.exist?(params[:key_file])
+
+ params[:ts_files].split(",").each do |ca|
+ raise "Cannot find CA file #{ca}" unless File.exist?(ca)
+ end
+
+ begin
+ Stomp::SSLParams.new(params)
+ rescue NameError
+ raise "Stomp gem >= 1.2.2 is needed"
+ end
+
+ rescue Exception => e
+ if fallback
+ Log.warn("Failed to set full SSL verified mode, falling back to unverified: #{e.class}: #{e}")
+ return true
+ else
+ Log.error("Failed to set full SSL verified mode: #{e.class}: #{e}")
+ raise(e)
+ end
+ end
+
+ # Receives a message from the ActiveMQ connection
+ def receive
+ Log.debug("Waiting for a message from ActiveMQ")
+
+ # When the Stomp library > 1.2.0 is mid reconnecting due to its reliable connection
+ # handling it sets the connection to closed. If we happen to be receiving at just
+ # that time we will get an exception warning about the closed connection so handling
+ # that here with a sleep and a retry.
+ begin
+ msg = @connection.receive
+ rescue ::Stomp::Error::NoCurrentConnection
+ sleep 1
+ retry
+ end
+
+ Message.new(msg.body, msg, :base64 => @base64, :headers => msg.headers)
+ end
+
+ # Sends a message to the ActiveMQ connection
+ def publish(msg)
+ msg.base64_encode! if @base64
+
+ target = target_for(msg)
+
+ if msg.type == :direct_request
+ msg.discovered_hosts.each do |node|
+ target[:headers] = headers_for(msg, node)
+
+ Log.debug("Sending a direct message to ActiveMQ target '#{target[:name]}' with headers '#{target[:headers].inspect}'")
+
+ @connection.publish(target[:name], msg.payload, target[:headers])
+ end
+ else
+ target[:headers].merge!(headers_for(msg))
+
+ Log.debug("Sending a broadcast message to ActiveMQ target '#{target[:name]}' with headers '#{target[:headers].inspect}'")
+
+ @connection.publish(target[:name], msg.payload, target[:headers])
+ end
+ end
+
+ # Subscribe to a topic or queue
+ def subscribe(agent, type, collective)
+ source = make_target(agent, type, collective)
+
+ unless @subscriptions.include?(source[:id])
+ Log.debug("Subscribing to #{source[:name]} with headers #{source[:headers].inspect.chomp}")
+ @connection.subscribe(source[:name], source[:headers], source[:id])
+ @subscriptions << source[:id]
+ end
+ rescue ::Stomp::Error::DuplicateSubscription
+ Log.error("Received subscription request for #{source.inspect.chomp} but already had a matching subscription, ignoring")
+ end
+
+ # Subscribe to a topic or queue
+ def unsubscribe(agent, type, collective)
+ source = make_target(agent, type, collective)
+
+ Log.debug("Unsubscribing from #{source[:name]}")
+ @connection.unsubscribe(source[:name], source[:headers], source[:id])
+ @subscriptions.delete(source[:id])
+ end
+
+ def target_for(msg)
+ if msg.type == :reply
+ target = {:name => msg.request.headers["reply-to"], :headers => {}}
+ elsif [:request, :direct_request].include?(msg.type)
+ target = make_target(msg.agent, msg.type, msg.collective)
+ else
+ raise "Don't now how to create a target for message type #{msg.type}"
+ end
+
+ return target
+ end
+
+ # Disconnects from the ActiveMQ connection
+ def disconnect
+ Log.debug("Disconnecting from ActiveMQ")
+ @connection.disconnect
+ @connection = nil
+ end
+
+ def headers_for(msg, identity=nil)
+ headers = {}
+ headers = {"priority" => @msgpriority} if @msgpriority > 0
+
+ if [:request, :direct_request].include?(msg.type)
+ target = make_target(msg.agent, :reply, msg.collective)
+
+ if msg.reply_to
+ headers["reply-to"] = msg.reply_to
+ else
+ headers["reply-to"] = target[:name]
+ end
+
+ headers["mc_identity"] = identity if msg.type == :direct_request
+ end
+
+ return headers
+ end
+
+ def make_target(agent, type, collective)
+ raise("Unknown target type #{type}") unless [:directed, :broadcast, :reply, :request, :direct_request].include?(type)
+ raise("Unknown collective '#{collective}' known collectives are '#{@config.collectives.join ', '}'") unless @config.collectives.include?(collective)
+
+ target = {:name => nil, :headers => {}}
+
+ case type
+ when :reply
+ target[:name] = ["/queue/" + collective, :reply, "#{Config.instance.identity}_#{$$}"].join(".")
+
+ when :broadcast
+ target[:name] = ["/topic/" + collective, agent, :agent].join(".")
+
+ when :request
+ target[:name] = ["/topic/" + collective, agent, :agent].join(".")
+
+ when :direct_request
+ target[:name] = ["/queue/" + collective, :nodes].join(".")
+
+ when :directed
+ target[:name] = ["/queue/" + collective, :nodes].join(".")
+ target[:headers]["selector"] = "mc_identity = '#{@config.identity}'"
+ target[:id] = "%s_directed_to_identity" % collective
+ end
+
+ target[:id] = target[:name] unless target[:id]
+
+ target
+ end
+
+ # looks in the environment first then in the config file
+ # for a specific option, accepts an optional default.
+ #
+ # raises an exception when it cant find a value anywhere
+ def get_env_or_option(env, opt, default=nil)
+ return ENV[env] if ENV.include?(env)
+ return @config.pluginconf[opt] if @config.pluginconf.include?(opt)
+ return default if default
+
+ raise("No #{env} environment or plugin.#{opt} configuration option given")
+ end
+
+ # looks for a config option, accepts an optional default
+ #
+ # raises an exception when it cant find a value anywhere
+ def get_option(opt, default=nil)
+ return @config.pluginconf[opt] if @config.pluginconf.include?(opt)
+ return default unless default.nil?
+
+ raise("No plugin.#{opt} configuration option given")
+ end
+
+ # gets a boolean option from the config, supports y/n/true/false/1/0
+ def get_bool_option(opt, default)
+ return default unless @config.pluginconf.include?(opt)
+
+ val = @config.pluginconf[opt]
+
+ if val =~ /^1|yes|true/
+ return true
+ elsif val =~ /^0|no|false/
+ return false
+ else
+ return default
+ end
+ end
+ end
+ end
+end
+
+# vi:tabstop=4:expandtab:ai
--- /dev/null
+require 'stomp'
+
+module MCollective
+ module Connector
+ class Rabbitmq<Base
+ attr_reader :connection
+
+ class EventLogger
+ def on_connecting(params=nil)
+ Log.info("TCP Connection attempt %d to %s" % [params[:cur_conattempts], stomp_url(params)])
+ rescue
+ end
+
+ def on_connected(params=nil)
+ Log.info("Conncted to #{stomp_url(params)}")
+ rescue
+ end
+
+ def on_disconnect(params=nil)
+ Log.info("Disconnected from #{stomp_url(params)}")
+ rescue
+ end
+
+ def on_connectfail(params=nil)
+ Log.info("TCP Connection to #{stomp_url(params)} failed on attempt #{params[:cur_conattempts]}")
+ rescue
+ end
+
+ def on_miscerr(params, errstr)
+ Log.error("Unexpected error on connection #{stomp_url(params)}: #{errstr}")
+ rescue
+ end
+
+ def on_ssl_connecting(params)
+ Log.info("Estblishing SSL session with #{stomp_url(params)}")
+ rescue
+ end
+
+ def on_ssl_connected(params)
+ Log.info("SSL session established with #{stomp_url(params)}")
+ rescue
+ end
+
+ def on_ssl_connectfail(params)
+ Log.error("SSL session creation with #{stomp_url(params)} failed: #{params[:ssl_exception]}")
+ end
+
+ def stomp_url(params)
+ "%s://%s@%s:%d" % [ params[:cur_ssl] ? "stomp+ssl" : "stomp", params[:cur_login], params[:cur_host], params[:cur_port]]
+ end
+ end
+
+ def initialize
+ @config = Config.instance
+ @subscriptions = []
+ @base64 = false
+ end
+
+ # Connects to the RabbitMQ middleware
+ def connect(connector = ::Stomp::Connection)
+ if @connection
+ Log.debug("Already connection, not re-initializing connection")
+ return
+ end
+
+ begin
+ @base64 = get_bool_option("rabbitmq.base64", false)
+
+ pools = @config.pluginconf["rabbitmq.pool.size"].to_i
+ hosts = []
+
+ 1.upto(pools) do |poolnum|
+ host = {}
+
+ host[:host] = get_option("rabbitmq.pool.#{poolnum}.host")
+ host[:port] = get_option("rabbitmq.pool.#{poolnum}.port", 61613).to_i
+ host[:login] = get_env_or_option("STOMP_USER", "rabbitmq.pool.#{poolnum}.user")
+ host[:passcode] = get_env_or_option("STOMP_PASSWORD", "rabbitmq.pool.#{poolnum}.password")
+ host[:ssl] = get_bool_option("rabbitmq.pool.#{poolnum}.ssl", false)
+
+ host[:ssl] = ssl_parameters(poolnum, get_bool_option("rabbitmq.pool.#{poolnum}.ssl.fallback", false)) if host[:ssl]
+
+ Log.debug("Adding #{host[:host]}:#{host[:port]} to the connection pool")
+ hosts << host
+ end
+
+ raise "No hosts found for the RabbitMQ connection pool" if hosts.size == 0
+
+ connection = {:hosts => hosts}
+
+ # Various STOMP gem options, defaults here matches defaults for 1.1.6 the meaning of
+ # these can be guessed, the documentation isn't clear
+ connection[:initial_reconnect_delay] = Float(get_option("rabbitmq.initial_reconnect_delay", 0.01))
+ connection[:max_reconnect_delay] = Float(get_option("rabbitmq.max_reconnect_delay", 30.0))
+ connection[:use_exponential_back_off] = get_bool_option("rabbitmq.use_exponential_back_off", true)
+ connection[:back_off_multiplier] = Integer(get_option("rabbitmq.back_off_multiplier", 2))
+ connection[:max_reconnect_attempts] = Integer(get_option("rabbitmq.max_reconnect_attempts", 0))
+ connection[:randomize] = get_bool_option("rabbitmq.randomize", false)
+ connection[:backup] = get_bool_option("rabbitmq.backup", false)
+ connection[:timeout] = Integer(get_option("rabbitmq.timeout", -1))
+ connection[:connect_timeout] = Integer(get_option("rabbitmq.connect_timeout", 30))
+ connection[:reliable] = true
+
+ # RabbitMQ and Stomp supports vhosts, this sets it in a way compatible with RabbitMQ and
+ # force the version to 1.0, 1.1 support will be added in future
+ connection[:connect_headers] = {"accept-version" => '1.0', "host" => get_option("rabbitmq.vhost", "/")}
+
+ connection[:logger] = EventLogger.new
+
+ @connection = connector.new(connection)
+ rescue Exception => e
+ raise("Could not connect to RabbitMQ Server: #{e}")
+ end
+ end
+
+ # Sets the SSL paramaters for a specific connection
+ def ssl_parameters(poolnum, fallback)
+ params = {:cert_file => get_option("rabbitmq.pool.#{poolnum}.ssl.cert", false),
+ :key_file => get_option("rabbitmq.pool.#{poolnum}.ssl.key", false),
+ :ts_files => get_option("rabbitmq.pool.#{poolnum}.ssl.ca", false)}
+
+ raise "cert, key and ca has to be supplied for verified SSL mode" unless params[:cert_file] && params[:key_file] && params[:ts_files]
+
+ raise "Cannot find certificate file #{params[:cert_file]}" unless File.exist?(params[:cert_file])
+ raise "Cannot find key file #{params[:key_file]}" unless File.exist?(params[:key_file])
+
+ params[:ts_files].split(",").each do |ca|
+ raise "Cannot find CA file #{ca}" unless File.exist?(ca)
+ end
+
+ begin
+ Stomp::SSLParams.new(params)
+ rescue NameError
+ raise "Stomp gem >= 1.2.2 is needed"
+ end
+
+ rescue Exception => e
+ if fallback
+ Log.warn("Failed to set full SSL verified mode, falling back to unverified: #{e.class}: #{e}")
+ return true
+ else
+ Log.error("Failed to set full SSL verified mode: #{e.class}: #{e}")
+ raise(e)
+ end
+ end
+
+ # Receives a message from the RabbitMQ connection
+ def receive
+ Log.debug("Waiting for a message from RabbitMQ")
+
+ # When the Stomp library > 1.2.0 is mid reconnecting due to its reliable connection
+ # handling it sets the connection to closed. If we happen to be receiving at just
+ # that time we will get an exception warning about the closed connection so handling
+ # that here with a sleep and a retry.
+ begin
+ msg = @connection.receive
+ rescue ::Stomp::Error::NoCurrentConnection
+ sleep 1
+ retry
+ end
+
+ raise "Received a processing error from RabbitMQ: '%s'" % msg.body.chomp if msg.body =~ /Processing error/
+
+ Message.new(msg.body, msg, :base64 => @base64, :headers => msg.headers)
+ end
+
+ # Sends a message to the RabbitMQ connection
+ def publish(msg)
+ msg.base64_encode! if @base64
+
+ if msg.type == :direct_request
+ msg.discovered_hosts.each do |node|
+ target = target_for(msg, node)
+
+ Log.debug("Sending a direct message to RabbitMQ target '#{target[:name]}' with headers '#{target[:headers].inspect}'")
+
+ @connection.publish(target[:name], msg.payload, target[:headers])
+ end
+ else
+ target = target_for(msg)
+
+ Log.debug("Sending a broadcast message to RabbitMQ target '#{target[:name]}' with headers '#{target[:headers].inspect}'")
+
+ @connection.publish(target[:name], msg.payload, target[:headers])
+ end
+ end
+
+ def target_for(msg, node=nil)
+ if msg.type == :reply
+ target = {:name => msg.request.headers["reply-to"], :headers => {}, :id => ""}
+
+ elsif [:request, :direct_request].include?(msg.type)
+ target = make_target(msg.agent, msg.type, msg.collective, msg.reply_to, node)
+
+ else
+ raise "Don't now how to create a target for message type #{msg.type}"
+
+ end
+
+ return target
+ end
+
+ def make_target(agent, type, collective, reply_to=nil, node=nil)
+ raise("Unknown target type #{type}") unless [:directed, :broadcast, :reply, :request, :direct_request].include?(type)
+ raise("Unknown collective '#{collective}' known collectives are '#{@config.collectives.join ', '}'") unless @config.collectives.include?(collective)
+
+ target = {:name => "", :headers => {}, :id => nil}
+
+ case type
+ when :reply # receiving replies on a temp queue
+ target[:name] = "/temp-queue/mcollective_reply_%s" % agent
+ target[:id] = "mcollective_%s_replies" % agent
+
+ when :broadcast, :request # publishing a request to all nodes with an agent
+ target[:name] = "/exchange/%s_broadcast/%s" % [collective, agent]
+ if reply_to
+ target[:headers]["reply-to"] = reply_to
+ else
+ target[:headers]["reply-to"] = "/temp-queue/mcollective_reply_%s" % agent
+ end
+ target[:id] = "%s_broadcast_%s" % [collective, agent]
+
+ when :direct_request # a request to a specific node
+ raise "Directed requests need to have a node identity" unless node
+
+ target[:name] = "/exchange/%s_directed/%s" % [ collective, node]
+ target[:headers]["reply-to"] = "/temp-queue/mcollective_reply_%s" % agent
+
+ when :directed # subscribing to directed messages
+ target[:name] = "/exchange/%s_directed/%s" % [ collective, @config.identity ]
+ target[:id] = "%s_directed_to_identity" % @config.identity
+ end
+
+ target
+ end
+
+ # Subscribe to a topic or queue
+ def subscribe(agent, type, collective)
+ return if type == :reply
+
+ source = make_target(agent, type, collective)
+
+ unless @subscriptions.include?(source[:id])
+ Log.debug("Subscribing to #{source[:name]} with headers #{source[:headers].inspect.chomp}")
+ @connection.subscribe(source[:name], source[:headers], source[:id])
+ @subscriptions << source[:id]
+ end
+ rescue ::Stomp::Error::DuplicateSubscription
+ Log.error("Received subscription request for #{source.inspect.chomp} but already had a matching subscription, ignoring")
+ end
+
+ # Subscribe to a topic or queue
+ def unsubscribe(agent, type, collective)
+ return if type == :reply
+
+ source = make_target(agent, type, collective)
+
+ Log.debug("Unsubscribing from #{source[:name]}")
+ @connection.unsubscribe(source[:name], source[:headers], source[:id])
+ @subscriptions.delete(source[:id])
+ end
+
+ # Disconnects from the RabbitMQ connection
+ def disconnect
+ Log.debug("Disconnecting from RabbitMQ")
+ @connection.disconnect
+ @connection = nil
+ end
+
+ # looks in the environment first then in the config file
+ # for a specific option, accepts an optional default.
+ #
+ # raises an exception when it cant find a value anywhere
+ def get_env_or_option(env, opt, default=nil)
+ return ENV[env] if ENV.include?(env)
+ return @config.pluginconf[opt] if @config.pluginconf.include?(opt)
+ return default if default
+
+ raise("No #{env} environment or plugin.#{opt} configuration option given")
+ end
+
+ # looks for a config option, accepts an optional default
+ #
+ # raises an exception when it cant find a value anywhere
+ def get_option(opt, default=nil)
+ return @config.pluginconf[opt] if @config.pluginconf.include?(opt)
+ return default unless default.nil?
+
+ raise("No plugin.#{opt} configuration option given")
+ end
+
+ # gets a boolean option from the config, supports y/n/true/false/1/0
+ def get_bool_option(opt, default)
+ return default unless @config.pluginconf.include?(opt)
+
+ val = @config.pluginconf[opt]
+
+ if val =~ /^1|yes|true/
+ return true
+ elsif val =~ /^0|no|false/
+ return false
+ else
+ return default
+ end
+ end
+ end
+ end
+end
+
+# vi:tabstop=4:expandtab:ai
--- /dev/null
+metadata :name => "Agent",
+ :description => "Meta data about installed MColletive Agents",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 1
+
+dataquery :description => "Agent Meta Data" do
+ input :query,
+ :prompt => "Agent Name",
+ :description => "Valid agent name",
+ :type => :string,
+ :validation => /^[\w\_]+$/,
+ :maxlength => 20
+
+ [:license, :timeout, :description, :url, :version, :author].each do |item|
+ output item,
+ :description => "Agent #{item}",
+ :display_as => item.to_s.capitalize
+ end
+end
--- /dev/null
+module MCollective
+ module Data
+ class Agent_data<Base
+ query do |plugin|
+ raise "No agent called #{plugin} found" unless PluginManager.include?("#{plugin}_agent")
+
+ agent = PluginManager["#{plugin}_agent"]
+
+ result[:agent] = plugin
+
+ [:license, :timeout, :description, :url, :version, :author].each do |item|
+ result[item] = agent.meta[item]
+ end
+ end
+ end
+ end
+end
--- /dev/null
+metadata :name => "File Stat",
+ :description => "Retrieve file stat data for a given file",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 1
+
+dataquery :description => "File stat information" do
+ input :query,
+ :prompt => "File Name",
+ :description => "Valid File Name",
+ :type => :string,
+ :validation => /.+/,
+ :maxlength => 120
+
+ output :name,
+ :description => "File name",
+ :display_as => "Name"
+
+ output :output,
+ :description => "Human readable information about the file",
+ :display_as => "Status"
+
+ output :present,
+ :description => "Indicates if the file exist using 0 or 1",
+ :display_as => "Present"
+
+ output :size,
+ :description => "File size",
+ :display_as => "Size"
+
+ output :mode,
+ :description => "File mode",
+ :display_as => "Mode"
+
+ output :md5,
+ :description => "File MD5 digest",
+ :display_as => "MD5"
+
+ output :mtime,
+ :description => "File modification time",
+ :display_as => "Modification time"
+
+ output :ctime,
+ :description => "File change time",
+ :display_as => "Change time"
+
+ output :atime,
+ :description => "File access time",
+ :display_as => "Access time"
+
+ output :mtime_seconds,
+ :description => "File modification time in seconds",
+ :display_as => "Modification time"
+
+ output :ctime_seconds,
+ :description => "File change time in seconds",
+ :display_as => "Change time"
+
+ output :atime_seconds,
+ :description => "File access time in seconds",
+ :display_as => "Access time"
+
+ output :mtime_age,
+ :description => "File modification age in seconds",
+ :display_as => "Modification age"
+
+ output :ctime_age,
+ :description => "File change age in seconds",
+ :display_as => "Change age"
+
+ output :atime_age,
+ :description => "File access age in seconds",
+ :display_as => "Access age"
+
+ output :uid,
+ :description => "File owner",
+ :display_as => "Owner"
+
+ output :gid,
+ :description => "File group",
+ :display_as => "Group"
+
+ output :type,
+ :description => "File type",
+ :display_as => "Type"
+end
+
--- /dev/null
+module MCollective
+ module Data
+ class Fstat_data<Base
+ query do |file|
+ result[:name] = file
+ result[:output] = "not present"
+ result[:type] = "unknown"
+ result[:mode] = "0000"
+ result[:present] = 0
+ result[:size] = 0
+ result[:mtime] = 0
+ result[:ctime] = 0
+ result[:atime] = 0
+ result[:mtime_seconds] = 0
+ result[:ctime_seconds] = 0
+ result[:atime_seconds] = 0
+ result[:md5] = 0
+ result[:uid] = 0
+ result[:gid] = 0
+
+
+ if File.exists?(file)
+ result[:output] = "present"
+ result[:present] = 1
+
+ if File.symlink?(file)
+ stat = File.lstat(file)
+ else
+ stat = File.stat(file)
+ end
+
+ [:size, :uid, :gid].each do |item|
+ result[item] = stat.send(item)
+ end
+
+ [:mtime, :ctime, :atime].each do |item|
+ result[item] = stat.send(item).strftime("%F %T")
+ result["#{item}_seconds".to_sym] = stat.send(item).to_i
+ result["#{item}_age".to_sym] = Time.now.to_i - stat.send(item).to_i
+ end
+
+ result[:mode] = "%o" % [stat.mode]
+ result[:md5] = Digest::MD5.hexdigest(File.read(file)) if stat.file?
+
+ result[:type] = "directory" if stat.directory?
+ result[:type] = "file" if stat.file?
+ result[:type] = "symlink" if stat.symlink?
+ result[:type] = "socket" if stat.socket?
+ result[:type] = "chardev" if stat.chardev?
+ result[:type] = "blockdev" if stat.blockdev?
+ end
+ end
+ end
+ end
+end
+
--- /dev/null
+metadata :name => "flatfile",
+ :description => "Flatfile based discovery for node identities",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :license => "ASL 2.0",
+ :version => "0.1",
+ :url => "http://marionette-collective.org/",
+ :timeout => 0
+
+discovery do
+ capabilities :identity
+end
--- /dev/null
+# discovers against a flatfile instead of the traditional network discovery
+# the flat file must have a node name per line which should match identities
+# as configured
+module MCollective
+ class Discovery
+ class Flatfile
+ def self.discover(filter, timeout, limit=0, client=nil)
+ unless client.options[:discovery_options].empty?
+ file = client.options[:discovery_options].first
+ else
+ raise "The flatfile discovery method needs a path to a text file"
+ end
+
+ raise "Cannot read the file %s specified as discovery source" % file unless File.readable?(file)
+
+ discovered = []
+
+ hosts = File.readlines(file).map{|l| l.chomp}
+
+ # this plugin only supports identity filters, do regex matches etc against
+ # the list found in the flatfile
+ unless filter["identity"].empty?
+ filter["identity"].each do |identity|
+ identity = Regexp.new(identity.gsub("\/", "")) if identity.match("^/")
+
+ if identity.is_a?(Regexp)
+ discovered = hosts.grep(identity)
+ elsif hosts.include?(identity)
+ discovered << identity
+ end
+ end
+ else
+ discovered = hosts
+ end
+
+ discovered
+ end
+ end
+ end
+end
--- /dev/null
+metadata :name => "mc",
+ :description => "MCollective Broadcast based discovery",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :license => "ASL 2.0",
+ :version => "0.1",
+ :url => "http://marionette-collective.org/",
+ :timeout => 2
+
+discovery do
+ capabilities [:classes, :facts, :identity, :agents, :compound]
+end
--- /dev/null
+module MCollective
+ class Discovery
+ class Mc
+ def self.discover(filter, timeout, limit, client)
+ begin
+ hosts = []
+ Timeout.timeout(timeout) do
+ reqid = client.sendreq("ping", "discovery", filter)
+ Log.debug("Waiting #{timeout} seconds for discovery replies to request #{reqid}")
+
+ loop do
+ reply = client.receive(reqid)
+ Log.debug("Got discovery reply from #{reply.payload[:senderid]}")
+ hosts << reply.payload[:senderid]
+
+ return hosts if limit > 0 && hosts.size == limit
+ end
+ end
+ rescue Timeout::Error => e
+ rescue Exception => e
+ raise
+ ensure
+ client.unsubscribe("discovery", :reply)
+ end
+
+ hosts
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Facts
+ require 'yaml'
+
+ # A factsource that reads a hash of facts from a YAML file
+ #
+ # Multiple files can be specified seperated with a : in the
+ # config file, they will be merged with later files overriding
+ # earlier ones in the list.
+ class Yaml_facts<Base
+ def initialize
+ @yaml_file_mtimes = {}
+
+ super
+ end
+
+ def load_facts_from_source
+ config = Config.instance
+
+ fact_files = config.pluginconf["yaml"].split(File::PATH_SEPARATOR)
+ facts = {}
+
+ fact_files.each do |file|
+ begin
+ if File.exist?(file)
+ facts.merge!(YAML.load_file(file))
+ else
+ raise("Can't find YAML file to load: #{file}")
+ end
+ rescue Exception => e
+ Log.error("Failed to load yaml facts from #{file}: #{e.class}: #{e}")
+ end
+ end
+
+ facts
+ end
+
+ # force fact reloads when the mtime on the yaml file change
+ def force_reload?
+ config = Config.instance
+
+ fact_files = config.pluginconf["yaml"].split(File::PATH_SEPARATOR)
+
+ fact_files.each do |file|
+ @yaml_file_mtimes[file] ||= File.stat(file).mtime
+ mtime = File.stat(file).mtime
+
+ if mtime > @yaml_file_mtimes[file]
+ @yaml_file_mtimes[file] = mtime
+
+ Log.debug("Forcing fact reload due to age of #{file}")
+
+ return true
+ end
+ end
+
+ false
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module PluginPackager
+ class DebpackagePackager
+
+ require 'erb'
+ attr_accessor :plugin, :current_package, :tmpdir, :verbose, :libdir
+ attr_accessor :workingdir, :preinstall, :postinstall, :current_package_type
+ attr_accessor :current_package_data, :current_package_shortname
+ attr_accessor :current_package_fullname, :build_dir, :signature
+
+ def initialize(plugin, pluginpath = nil, signature = nil, verbose = false)
+ raise RuntimeError, "package 'debuild' is not installed" unless PluginPackager.build_tool?("debuild")
+ @plugin = plugin
+ @verbose = verbose
+ @libdir = pluginpath || "/usr/share/mcollective/plugins/mcollective/"
+ @signature = signature
+ @tmpdir = ""
+ @build_dir = ""
+ @targetdir = ""
+ end
+
+ def create_packages
+ @plugin.packagedata.each do |type, data|
+ begin
+ @tmpdir = Dir.mktmpdir("mcollective_packager")
+ @current_package_type = type
+ @current_package_data = data
+ @current_package_shortname = "#{@plugin.mcname}-#{@plugin.metadata[:name]}-#{@current_package_type}"
+ @current_package_fullname = "#{@plugin.mcname}-#{@plugin.metadata[:name]}-#{@current_package_type}" +
+ "_#{@plugin.metadata[:version]}-#{@plugin.iteration}"
+
+ @build_dir = File.join(@tmpdir, "#{@current_package_shortname}_#{@plugin.metadata[:version]}")
+ Dir.mkdir @build_dir
+
+ prepare_tmpdirs data
+ create_package
+ move_packages
+ rescue Exception => e
+ raise e
+ ensure
+ cleanup_tmpdirs
+ end
+ end
+ end
+
+ def create_package
+ begin
+ ["control", "Makefile", "compat", "rules", "copyright", "changelog"].each do |filename|
+ create_file(filename)
+ end
+ create_tar
+ create_install
+ create_preandpost_install
+
+ FileUtils.cd @build_dir do |f|
+ PluginPackager.do_quietly?(@verbose) do
+ if @signature
+ if @signature.is_a? String
+ PluginPackager.safe_system "debuild -i -k#{@signature}"
+ else
+ PluginPackager.safe_system "debuild -i"
+ end
+ else
+ PluginPackager.safe_system "debuild -i -us -uc"
+ end
+ end
+ end
+
+ puts "Created package #{@current_package_fullname}"
+ rescue Exception => e
+ raise RuntimeError, "Could not build package - #{e}"
+ end
+ end
+
+ def move_packages
+ begin
+ FileUtils.cp(Dir.glob(File.join(@tmpdir, "*.{deb,dsc,diff.gz,orig.tar.gz,changes}")), ".")
+ rescue Exception => e
+ raise RuntimeError, "Could not copy packages to working directory: '#{e}'"
+ end
+ end
+
+ def create_preandpost_install
+ if @plugin.preinstall
+ raise RuntimeError, "pre-install script '#{@plugin.preinstall}' not found" unless File.exists?(@plugin.preinstall)
+ FileUtils.cp(@plugin.preinstall, File.join(@build_dir, 'debian', "#{@current_package_shortname}.preinst"))
+ end
+
+ if @plugin.postinstall
+ raise RuntimeError, "post-install script '#{@plugin.postinstall}' not found" unless File.exists?(@plugin.postinstall)
+ FileUtils.cp(@plugin.postinstall, File.join(@build_dir, 'debian', "#{@current_package_shortname}.postinst"))
+ end
+
+ end
+
+ def create_install
+ begin
+ File.open(File.join(@build_dir, "debian", "#{@current_package_shortname}.install"), "w") do |f|
+ @current_package_data[:files].each do |filename|
+ extended_filename = File.join(@libdir, File.expand_path(filename).gsub(/#{File.expand_path(plugin.path)}|\.\//, ''))
+ f.puts "#{extended_filename} #{File.dirname(extended_filename)}"
+ end
+ end
+ rescue Exception => e
+ raise RuntimeError, "Could not create install file - #{e}"
+ end
+ end
+
+ def create_tar
+ begin
+ PluginPackager.do_quietly?(@verbose) do
+ Dir.chdir(@tmpdir) do
+ PluginPackager.safe_system "tar -Pcvzf #{File.join(@tmpdir,"#{@current_package_shortname}_#{@plugin.metadata[:version]}.orig.tar.gz")} #{@current_package_shortname}_#{@plugin.metadata[:version]}"
+ end
+ end
+ rescue Exception => e
+ raise "Could not create tarball - #{e}"
+ end
+ end
+
+ def create_file(filename)
+ begin
+ file = ERB.new(File.read(File.join(File.dirname(__FILE__), "templates", "debian", "#{filename}.erb")), nil, "-")
+ File.open(File.join(@build_dir, "debian", filename), "w") do |f|
+ f.puts file.result(binding)
+ end
+ rescue Exception => e
+ raise RuntimeError, "could not create #{filename} file - #{e}"
+ end
+ end
+
+ def prepare_tmpdirs(data)
+ data[:files].each do |file|
+ @targetdir = File.join(@build_dir, @libdir, File.dirname(File.expand_path(file)).gsub(@plugin.target_path, ""))
+ FileUtils.mkdir_p(@targetdir) unless File.directory? @targetdir
+ FileUtils.cp_r(file, @targetdir)
+ end
+
+ FileUtils.mkdir_p(File.join(@build_dir, "debian"))
+ end
+
+ def cleanup_tmpdirs
+ FileUtils.rm_r @tmpdir if File.directory? @tmpdir
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module PluginPackager
+ # MCollective plugin packager general OS implementation.
+ class OspackagePackager
+
+ attr_accessor :package, :verbose, :packager, :package_type
+
+ # Create packager object with package parameter containing list of files,
+ # dependencies and package metadata.
+ def initialize(package, pluginpath = nil, signature = nil, verbose = false)
+
+ if File.exists?("/etc/redhat-release")
+ @packager = PluginPackager["RpmpackagePackager"].new(package, pluginpath, signature, verbose)
+ @package_type = "RPM"
+ elsif File.exists?("/etc/debian_version")
+ @packager = PluginPackager["DebpackagePackager"].new(package, pluginpath, signature, verbose)
+ @package_type = "Deb"
+ else
+ raise "cannot identify operating system."
+ end
+
+ @package = package
+ @verbose = verbose
+ end
+
+ # Hands over package creation to the detected packager implementation
+ # based on operating system.
+ def create_packages
+ @packager.create_packages
+ end
+
+ # Displays the package metadata and detected files
+ def package_information
+ puts
+ puts "%30s%s" % ["Plugin information : ", @package.metadata[:name]]
+ puts "%30s%s" % ["-" * 22, "-" * 22]
+ puts "%30s%s" % ["Plugin Type : ", @package.plugintype.capitalize]
+ puts "%30s%s" % ["Package Output Format : ", @package_type]
+ puts "%30s%s" % ["Version : ", @package.metadata[:version]]
+ puts "%30s%s" % ["Iteration : ", @package.iteration]
+ puts "%30s%s" % ["Vendor : ", @package.vendor]
+ puts "%30s%s" % ["Post Install Script : ", @package.postinstall] if @package.postinstall
+ puts "%30s%s" % ["Author : ", @package.metadata[:author]]
+ puts "%30s%s" % ["License : ", @package.metadata[:license]]
+ puts "%30s%s" % ["URL : ", @package.metadata[:url]]
+
+ if @package.packagedata.size > 0
+ @package.packagedata.each_with_index do |values, i|
+ if i == 0
+ puts "%30s%s" % ["Identified Packages : ", values[0]]
+ else
+ puts "%30s%s" % [" ", values[0]]
+ end
+ end
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module PluginPackager
+ class RpmpackagePackager
+
+ require 'erb'
+ attr_accessor :plugin, :tmpdir, :verbose, :libdir, :workingdir
+ attr_accessor :current_package_type, :current_package_data
+ attr_accessor :current_package_name, :signature
+
+ def initialize(plugin, pluginpath = nil, signature = nil, verbose = false)
+ if(PluginPackager.build_tool?("rpmbuild-md5"))
+ @buildtool = "rpmbuild-md5"
+ elsif(PluginPackager.build_tool?("rpmbuild"))
+ @buildtool = "rpmbuild"
+ else
+ raise RuntimeError, "creating rpms require 'rpmbuild' or 'rpmbuild-md5' to be installed"
+ end
+
+ @plugin = plugin
+ @verbose = verbose
+ @libdir = pluginpath || "/usr/libexec/mcollective/mcollective/"
+ @signature = signature
+ @rpmdir = rpmdir
+ @srpmdir = srpmdir
+ end
+
+ def rpmdir
+ `rpm --eval '%_rpmdir'`.chomp
+ end
+
+ def srpmdir
+ `rpm --eval '%_srcrpmdir'`.chomp
+ end
+
+ def create_packages
+ @plugin.packagedata.each do |type, data|
+ begin
+ @current_package_type = type
+ @current_package_data = data
+ @current_package_name = "#{@plugin.mcname}-#{@plugin.metadata[:name]}-#{@current_package_type}"
+ @tmpdir = Dir.mktmpdir("mcollective_packager")
+ prepare_tmpdirs data
+ create_package type, data
+ rescue Exception => e
+ raise e
+ ensure
+ cleanup_tmpdirs
+ end
+ end
+ end
+
+ def create_package(type, data)
+ begin
+ tarfile = "#{@current_package_name}-#{@plugin.metadata[:version]}.tgz"
+ make_spec_file
+ PluginPackager.do_quietly?(verbose) do
+ Dir.chdir(@tmpdir) do
+ PluginPackager.safe_system("tar -cvzf #{File.join(@tmpdir, tarfile)} #{@current_package_name}-#{@plugin.metadata[:version]}")
+ end
+
+ PluginPackager.safe_system("#{@buildtool} -ta #{"--quiet" unless verbose} #{"--sign" if @signature} #{File.join(@tmpdir, tarfile)}")
+ end
+
+ FileUtils.cp(File.join(@rpmdir, "noarch", "#{@current_package_name}-#{@plugin.metadata[:version]}-#{@plugin.iteration}.noarch.rpm"), ".")
+ FileUtils.cp(File.join(@srpmdir, "#{@current_package_name}-#{@plugin.metadata[:version]}-#{@plugin.iteration}.src.rpm"), ".")
+
+ puts "Created RPM and SRPM packages for #{@current_package_name}"
+ rescue Exception => e
+ raise RuntimeError, "Could not build package. Reason - #{e}"
+ end
+ end
+
+ def make_spec_file
+ begin
+ spec_template = ERB.new(File.read(File.join(File.dirname(__FILE__), "templates", "redhat", "rpm_spec.erb")), nil, "-")
+ File.open(File.join(@tmpdir, "#{@current_package_name}-#{@plugin.metadata[:version]}" ,"#{@current_package_name}-#{@plugin.metadata[:version]}.spec"), "w") do |f|
+ f.puts spec_template.result(binding)
+ end
+ rescue Exception => e
+ raise RuntimeError, "Could not create specfile - #{e}"
+ end
+ end
+
+ def prepare_tmpdirs(data)
+ data[:files].each do |file|
+ targetdir = File.join(@tmpdir, "#{@current_package_name}-#{@plugin.metadata[:version]}", @libdir, File.dirname(File.expand_path(file)).gsub(@plugin.target_path, ""))
+ FileUtils.mkdir_p(targetdir) unless File.directory? targetdir
+ FileUtils.cp_r(file, targetdir)
+ end
+ end
+
+ def cleanup_tmpdirs
+ FileUtils.rm_r @tmpdir if File.directory? @tmpdir
+ end
+ end
+ end
+end
--- /dev/null
+DESTDIR=
+
+build:
+
+clean:
+
+install:
--- /dev/null
+<%= @current_package_shortname -%> (<%= @plugin.metadata[:version]-%>-<%= @plugin.iteration-%>) unstable; urgency=low
+
+ * Automated release for <%= @current_package_shortname-%> by mco plugin packager.
+
+ -- The Marionette Collective <mcollective-dev@googlegroups.com> <%= Time.new.strftime('%a, %d %b %Y %H:%M:%S %z') %>
--- /dev/null
+<%
+ @current_package_data[:dependencies].map! do |dep|
+ if dep[:version] && dep[:iteration]
+ dep[:name] + '(>=' + dep[:version] + '-' + dep[:iteration].to_s + ')'
+ elsif dep[:version]
+ dep[:name] + '(>=' + dep[:version] + ')'
+ else
+ dep[:name]
+ end
+ end
+
+ if @current_package_data[:plugindependency]
+ @current_package_data[:plugindependency] = "#{@current_package_data[:plugindependency][:name]} (= #{current_package_data[:plugindependency][:version]}-#{@current_package_data[:plugindependency][:iteration]})"
+ @current_package_data[:dependencies].push(@current_package_data[:plugindependency])
+ end
+-%>
+Source: <%= @current_package_shortname %>
+Homepage: <%= @plugin.metadata[:url]%>
+Section: utils
+Priority: extra
+Maintainer: <%= @plugin.metadata[:author] %>
+Standards-Version: 3.8.0
+
+Package: <%= @current_package_shortname %>
+Architecture: all
+Depends: <%= @current_package_data[:dependencies].join(", ") %>
+Description: <%= @plugin.metadata[:description] %>
--- /dev/null
+This package was generated by the MCollective plugin packager on
+<%= Time.now%>
+
+Upstream Author:
+ <%= @plugin.metadata[:author] %> <%= @plugin.metadata[:url] %>
+
+License:
+ <%= @plugin.metadata[:license] -%>
--- /dev/null
+#!/usr/bin/make -f
+
+include /usr/share/cdbs/1/rules/debhelper.mk
+include /usr/share/cdbs/1/rules/dpatch.mk
+include /usr/share/cdbs/1/class/makefile.mk
+DEB_MAKE_INVOKE = $(DEB_MAKE_ENVVARS) make -f debian/Makefile -C $(DEB_BUILDDIR)
--- /dev/null
+Summary: <%= @plugin.metadata[:description] %>
+Name: <%= @current_package_name%>
+Version: <%= @plugin.metadata[:version] %>
+Release: <%= @plugin.iteration %>
+License: <%= @plugin.metadata[:license] %>
+URL: <%= @plugin.metadata[:url] %>
+Vendor: <%= @plugin.vendor %>
+Packager: <%= @plugin.metadata[:author] %>
+BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
+BuildArch: noarch
+Group: System Tools
+Source0: <%= "#{@current_package_name}-#{@plugin.metadata[:version]}.tgz" %>
+<% @current_package_data[:dependencies].each do |dep|-%>
+Requires: <%= dep[:name] -%> <%= ">= #{dep[:version]}" if dep[:version]%><%="-#{dep[:iteration]}" if dep[:iteration]%>
+<% end -%>
+<% if @current_package_data[:plugindependency] %>
+Requires: <%= @current_package_data[:plugindependency][:name] -%> = <%= @current_package_data[:plugindependency][:version]%>-<%= @current_package_data[:plugindependency][:iteration]%>
+<% end %>
+
+%description
+<%= @plugin.metadata[:description] %>
+
+%prep
+%setup
+
+%build
+
+%install
+<% @dirs = [] -%>
+<% @package_files = [] -%>
+<% @current_package_data[:files].each do |file| -%>
+<% @dirs << File.dirname(File.join(@libdir, File.expand_path(file).gsub(/#{File.expand_path(plugin.path)}|\.\//, ''))) -%>
+<% @package_files << File.join(@libdir, File.expand_path(file).gsub(/#{File.expand_path(plugin.path)}|\.\//, '')) -%>
+<% end -%>
+rm -rf %{buildroot}
+<% @dirs.uniq.each do |dir| -%>
+%{__install} -d -m0755 %{buildroot}<%= dir%>
+<% end -%>
+<% @package_files -= @dirs %>
+<% @package_files.each do |package_file| -%>
+%{__install} -m0644 <%= (package_file[0].chr == '/') ? package_file[1..package_file.size-1]: package_file%> %{buildroot}<%=package_file %>
+<% end -%>
+
+%files
+%defattr(-,root,root,-)
+<% @package_files.each do |file| -%>
+<%= file %>
+<% end -%>
+
+<% if @plugin.preinstall -%>
+%pre
+<%= @plugin.preinstall %>
+<% end -%>
+<% if @plugin.postinstall -%>
+%post
+<%= @plugin.postinstall%>
+<% end -%>
+
+%changelog
+* <%= Time.now.strftime("%a %b %d %Y") -%> <%= @plugin.metadata[:author]%> - <%= @plugin.metadata[:version]%>-<%= @plugin.iteration%>
+- Built Package <%= @current_package_name-%>
--- /dev/null
+module MCollective
+ module Registration
+ # A registration plugin that simply sends in the list of agents we have
+ class Agentlist<Base
+ def body
+ Agents.agentlist
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Security
+ # Impliments a security system that encrypts payloads using AES and secures
+ # the AES encrypted data using RSA public/private key encryption.
+ #
+ # The design goals of this plugin are:
+ #
+ # - Each actor - clients and servers - can have their own set of public and
+ # private keys
+ # - All actors are uniquely and cryptographically identified
+ # - Requests are encrypted using the clients private key and anyone that has
+ # the public key can see the request. Thus an atacker may see the requests
+ # given access to network or machine due to the broadcast nature of mcollective
+ # - The message time and TTL of messages are cryptographically secured making the
+ # ensuring messages can not be replayed with fake TTLs or times
+ # - Replies are encrypted using the calling clients public key. Thus no-one but
+ # the caller can view the contents of replies.
+ # - Servers can all have their own RSA keys, or share one, or reuse keys created
+ # by other PKI using software like Puppet
+ # - Requests from servers - like registration data - can be secured even to external
+ # eaves droppers depending on the level of configuration you are prepared to do
+ # - Given a network where you can ensure third parties are not able to access the
+ # middleware public key distribution can happen automatically
+ #
+ # Configuration Options:
+ # ======================
+ #
+ # Common Options:
+ #
+ # # Enable this plugin
+ # securityprovider = aes_security
+ #
+ # # Use YAML as serializer
+ # plugin.aes.serializer = yaml
+ #
+ # # Send our public key with every request so servers can learn it
+ # plugin.aes.send_pubkey = 1
+ #
+ # Clients:
+ #
+ # # The clients public and private keys
+ # plugin.aes.client_private = /home/user/.mcollective.d/user-private.pem
+ # plugin.aes.client_public = /home/user/.mcollective.d/user.pem
+ #
+ # Servers:
+ #
+ # # Where to cache client keys or find manually distributed ones
+ # plugin.aes.client_cert_dir = /etc/mcollective/ssl/clients
+ #
+ # # Cache public keys promiscuously from the network
+ # plugin.aes.learn_pubkeys = 1
+ #
+ # # Log but accept messages that may have been tampered with
+ # plugin.aes.enforce_ttl = 0
+ #
+ # # The servers public and private keys
+ # plugin.aes.server_private = /etc/mcollective/ssl/server-private.pem
+ # plugin.aes.server_public = /etc/mcollective/ssl/server-public.pem
+ #
+ class Aes_security<Base
+ def decodemsg(msg)
+ body = deserialize(msg.payload)
+
+ should_process_msg?(msg, body[:requestid])
+
+ # if we get a message that has a pubkey attached and we're set to learn
+ # then add it to the client_cert_dir this should only happen on servers
+ # since clients will get replies using their own pubkeys
+ if @config.pluginconf.include?("aes.learn_pubkeys") && @config.pluginconf["aes.learn_pubkeys"] == "1"
+ if body.include?(:sslpubkey)
+ if client_cert_dir
+ certname = certname_from_callerid(body[:callerid])
+ if certname
+ certfile = "#{client_cert_dir}/#{certname}.pem"
+ unless File.exist?(certfile)
+ Log.debug("Caching client cert in #{certfile}")
+ File.open(certfile, "w") {|f| f.print body[:sslpubkey]}
+ end
+ end
+ end
+ end
+ end
+
+ cryptdata = {:key => body[:sslkey], :data => body[:body]}
+
+ if @initiated_by == :client
+ body[:body] = deserialize(decrypt(cryptdata, nil))
+ else
+ body[:body] = deserialize(decrypt(cryptdata, body[:callerid]))
+
+ # If we got a hash it's possible that this is a message with secure
+ # TTL and message time, attempt to decode that and transform into a
+ # traditional message.
+ #
+ # If it's not a hash it might be a old style message like old discovery
+ # ones that would just be a string so we allow that unaudited but only
+ # if enforce_ttl is disabled. This is primarly to allow a mixed old and
+ # new plugin infrastructure to work
+ if body[:body].is_a?(Hash)
+ update_secure_property(body, :aes_ttl, :ttl, "TTL")
+ update_secure_property(body, :aes_msgtime, :msgtime, "Message Time")
+
+ body[:body] = body[:body][:aes_msg] if body[:body].include?(:aes_msg)
+ else
+ unless @config.pluginconf["aes.enforce_ttl"] == "0"
+ raise "Message %s is in an unknown or older security protocol, ignoring" % [request_description(body)]
+ end
+ end
+ end
+
+ return body
+ rescue MsgDoesNotMatchRequestID
+ raise
+
+ rescue OpenSSL::PKey::RSAError
+ raise MsgDoesNotMatchRequestID, "Could not decrypt message using our key, possibly directed at another client"
+
+ rescue Exception => e
+ Log.warn("Could not decrypt message from client: #{e.class}: #{e}")
+ raise SecurityValidationFailed, "Could not decrypt message"
+ end
+
+ # To avoid tampering we turn the origin body into a hash and copy some of the protocol keys
+ # like :ttl and :msg_time into the hash before encrypting it.
+ #
+ # This function compares and updates the unencrypted ones based on the encrypted ones. By
+ # default it enforces matching and presense by raising exceptions, if aes.enforce_ttl is set
+ # to 0 it will only log warnings about violations
+ def update_secure_property(msg, secure_property, property, description)
+ req = request_description(msg)
+
+ unless @config.pluginconf["aes.enforce_ttl"] == "0"
+ raise "Request #{req} does not have a secure #{description}" unless msg[:body].include?(secure_property)
+ raise "Request #{req} #{description} does not match encrypted #{description} - possible tampering" unless msg[:body][secure_property] == msg[property]
+ else
+ if msg[:body].include?(secure_property)
+ Log.warn("Request #{req} #{description} does not match encrypted #{description} - possible tampering") unless msg[:body][secure_property] == msg[property]
+ else
+ Log.warn("Request #{req} does not have a secure #{description}") unless msg[:body].include?(secure_property)
+ end
+ end
+
+ msg[property] = msg[:body][secure_property] if msg[:body].include?(secure_property)
+ msg[:body].delete(secure_property)
+ end
+
+ # Encodes a reply
+ def encodereply(sender, msg, requestid, requestcallerid)
+ crypted = encrypt(serialize(msg), requestcallerid)
+
+ req = create_reply(requestid, sender, crypted[:data])
+ req[:sslkey] = crypted[:key]
+
+ serialize(req)
+ end
+
+ # Encodes a request msg
+ def encoderequest(sender, msg, requestid, filter, target_agent, target_collective, ttl=60)
+ req = create_request(requestid, filter, nil, @initiated_by, target_agent, target_collective, ttl)
+
+ # embed the ttl and msgtime in the crypted data later we will use these in
+ # the decoding of a message to set the message ones from secure sources. this
+ # is to ensure messages are not tampered with to facility replay attacks etc
+ aes_msg = {:aes_msg => msg,
+ :aes_ttl => ttl,
+ :aes_msgtime => req[:msgtime]}
+
+ crypted = encrypt(serialize(aes_msg), callerid)
+
+ req[:body] = crypted[:data]
+ req[:sslkey] = crypted[:key]
+
+ if @config.pluginconf.include?("aes.send_pubkey") && @config.pluginconf["aes.send_pubkey"] == "1"
+ if @initiated_by == :client
+ req[:sslpubkey] = File.read(client_public_key)
+ else
+ req[:sslpubkey] = File.read(server_public_key)
+ end
+ end
+
+ serialize(req)
+ end
+
+ # Serializes a message using the configured encoder
+ def serialize(msg)
+ serializer = @config.pluginconf["aes.serializer"] || "marshal"
+
+ Log.debug("Serializing using #{serializer}")
+
+ case serializer
+ when "yaml"
+ return YAML.dump(msg)
+ else
+ return Marshal.dump(msg)
+ end
+ end
+
+ # De-Serializes a message using the configured encoder
+ def deserialize(msg)
+ serializer = @config.pluginconf["aes.serializer"] || "marshal"
+
+ Log.debug("De-Serializing using #{serializer}")
+
+ case serializer
+ when "yaml"
+ return YAML.load(msg)
+ else
+ return Marshal.load(msg)
+ end
+ end
+
+ # sets the caller id to the md5 of the public key
+ def callerid
+ if @initiated_by == :client
+ id = "cert=#{File.basename(client_public_key).gsub(/\.pem$/, '')}"
+ raise "Invalid callerid generated from client public key" unless valid_callerid?(id)
+ else
+ # servers need to set callerid as well, not usually needed but
+ # would be if you're doing registration or auditing or generating
+ # requests for some or other reason
+ id = "cert=#{File.basename(server_public_key).gsub(/\.pem$/, '')}"
+ raise "Invalid callerid generated from server public key" unless valid_callerid?(id)
+ end
+
+ return id
+ end
+
+ def encrypt(string, certid)
+ if @initiated_by == :client
+ @ssl ||= SSL.new(client_public_key, client_private_key)
+
+ Log.debug("Encrypting message using private key")
+ return @ssl.encrypt_with_private(string)
+ else
+ # when the server is initating requests like for registration
+ # then the certid will be our callerid
+ if certid == callerid
+ Log.debug("Encrypting message using private key #{server_private_key}")
+
+ ssl = SSL.new(server_public_key, server_private_key)
+ return ssl.encrypt_with_private(string)
+ else
+ Log.debug("Encrypting message using public key for #{certid}")
+
+ ssl = SSL.new(public_key_path_for_client(certid))
+ return ssl.encrypt_with_public(string)
+ end
+ end
+ end
+
+ def decrypt(string, certid)
+ if @initiated_by == :client
+ @ssl ||= SSL.new(client_public_key, client_private_key)
+
+ Log.debug("Decrypting message using private key")
+ return @ssl.decrypt_with_private(string)
+ else
+ Log.debug("Decrypting message using public key for #{certid}")
+
+ ssl = SSL.new(public_key_path_for_client(certid))
+ return ssl.decrypt_with_public(string)
+ end
+ end
+
+ # On servers this will look in the aes.client_cert_dir for public
+ # keys matching the clientid, clientid is expected to be in the format
+ # set by callerid
+ def public_key_path_for_client(clientid)
+ raise "Unknown callerid format in '#{clientid}'" unless clientid.match(/^cert=(.+)$/)
+
+ clientid = $1
+
+ client_cert_dir + "/#{clientid}.pem"
+ end
+
+ # Figures out the client private key either from MCOLLECTIVE_AES_PRIVATE or the
+ # plugin.aes.client_private config option
+ def client_private_key
+ return ENV["MCOLLECTIVE_AES_PRIVATE"] if ENV.include?("MCOLLECTIVE_AES_PRIVATE")
+
+ raise("No plugin.aes.client_private configuration option specified") unless @config.pluginconf.include?("aes.client_private")
+
+ return @config.pluginconf["aes.client_private"]
+ end
+
+ # Figures out the client public key either from MCOLLECTIVE_AES_PUBLIC or the
+ # plugin.aes.client_public config option
+ def client_public_key
+ return ENV["MCOLLECTIVE_AES_PUBLIC"] if ENV.include?("MCOLLECTIVE_AES_PUBLIC")
+
+ raise("No plugin.aes.client_public configuration option specified") unless @config.pluginconf.include?("aes.client_public")
+
+ return @config.pluginconf["aes.client_public"]
+ end
+
+ # Figures out the server public key from the plugin.aes.server_public config option
+ def server_public_key
+ raise("No aes.server_public configuration option specified") unless @config.pluginconf.include?("aes.server_public")
+ return @config.pluginconf["aes.server_public"]
+ end
+
+ # Figures out the server private key from the plugin.aes.server_private config option
+ def server_private_key
+ raise("No plugin.aes.server_private configuration option specified") unless @config.pluginconf.include?("aes.server_private")
+ @config.pluginconf["aes.server_private"]
+ end
+
+ # Figures out where to get client public certs from the plugin.aes.client_cert_dir config option
+ def client_cert_dir
+ raise("No plugin.aes.client_cert_dir configuration option specified") unless @config.pluginconf.include?("aes.client_cert_dir")
+ @config.pluginconf["aes.client_cert_dir"]
+ end
+
+ def request_description(msg)
+ "%s from %s@%s" % [msg[:requestid], msg[:callerid], msg[:senderid]]
+ end
+
+ # Takes our cert=foo callerids and return the foo bit else nil
+ def certname_from_callerid(id)
+ if id =~ /^cert=([\w\.\-]+)/
+ return $1
+ else
+ Log.warn("Received a callerid in an unexpected format: '#{id}', ignoring")
+ return nil
+ end
+ end
+ end
+ end
+end
--- /dev/null
+module MCollective
+ module Security
+ # Impliments message authentication using digests and shared keys
+ #
+ # You should configure a psk in the configuration file and all requests
+ # will be validated for authenticity with this.
+ #
+ # Serialization uses Marshal, this is the default security module that is
+ # supported out of the box.
+ #
+ # Validation is as default and is provided by MCollective::Security::Base
+ #
+ # You can configure the caller id being created, this can adjust how you
+ # create authorization plugins. For example you can use a unix group instead
+ # of uid to do authorization.
+ class Psk < Base
+ require 'etc'
+
+ # Decodes a message by unserializing all the bits etc, it also validates
+ # it as valid using the psk etc
+ def decodemsg(msg)
+ body = Marshal.load(msg.payload)
+
+ should_process_msg?(msg, body[:requestid])
+
+ if validrequest?(body)
+ body[:body] = Marshal.load(body[:body])
+ return body
+ else
+ nil
+ end
+ end
+
+ # Encodes a reply
+ def encodereply(sender, msg, requestid, requestcallerid=nil)
+ serialized = Marshal.dump(msg)
+ digest = makehash(serialized)
+
+ req = create_reply(requestid, sender, serialized)
+ req[:hash] = digest
+
+ Marshal.dump(req)
+ end
+
+ # Encodes a request msg
+ def encoderequest(sender, msg, requestid, filter, target_agent, target_collective, ttl=60)
+ serialized = Marshal.dump(msg)
+ digest = makehash(serialized)
+
+ req = create_request(requestid, filter, serialized, @initiated_by, target_agent, target_collective, ttl)
+ req[:hash] = digest
+
+ Marshal.dump(req)
+ end
+
+ # Checks the md5 hash in the request body against our psk, the request sent for validation
+ # should not have been deserialized already
+ def validrequest?(req)
+ digest = makehash(req[:body])
+
+ if digest == req[:hash]
+ @stats.validated
+
+ return true
+ else
+ @stats.unvalidated
+
+ raise(SecurityValidationFailed, "Received an invalid signature in message")
+ end
+ end
+
+ def callerid
+ if @config.pluginconf.include?("psk.callertype")
+ callertype = @config.pluginconf["psk.callertype"].to_sym if @config.pluginconf.include?("psk.callertype")
+ else
+ callertype = :uid
+ end
+
+ case callertype
+ when :gid
+ id = "gid=#{Process.gid}"
+
+ when :group
+ raise "Cannot use the 'group' callertype for the PSK security plugin on the Windows platform" if Util.windows?
+
+ id = "group=#{Etc.getgrgid(Process.gid).name}"
+
+ when :user
+ id = "user=#{Etc.getlogin}"
+
+ when :identity
+ id = "identity=#{@config.identity}"
+
+ else
+ id ="uid=#{Process.uid}"
+ end
+
+ Log.debug("Setting callerid to #{id} based on callertype=#{callertype}")
+
+ id
+ end
+
+ private
+ # Retrieves the value of plugin.psk and builds a hash with it and the passed body
+ def makehash(body)
+ if ENV.include?("MCOLLECTIVE_PSK")
+ psk = ENV["MCOLLECTIVE_PSK"]
+ else
+ raise("No plugin.psk configuration option specified") unless @config.pluginconf.include?("psk")
+ psk = @config.pluginconf["psk"]
+ end
+
+ Digest::MD5.hexdigest(body.to_s + psk)
+ end
+ end
+ end
+end
--- /dev/null
+require 'base64'
+require 'openssl'
+
+module MCollective
+ module Security
+ # Impliments a public/private key based message validation system using SSL
+ # public and private keys.
+ #
+ # The design goal of the plugin is two fold:
+ #
+ # - give different security credentials to clients and servers to avoid
+ # a compromised server from sending new client requests.
+ # - create a token that uniquely identify the client - based on the filename
+ # of the public key
+ #
+ # To setup you need to create a SSL key pair that is shared by all nodes.
+ #
+ # openssl genrsa -out mcserver-private.pem 1024
+ # openssl rsa -in mcserver-private.pem -out mcserver-public.pem -outform PEM -pubout
+ #
+ # Distribute the private and public file to /etc/mcollective/ssl on all the nodes.
+ # Distribute the public file to /etc/mcollective/ssl everywhere the client code runs.
+ #
+ # Now you should create a key pair for every one of your clients, here we create one
+ # for user john - you could also if you are less concerned with client id create one
+ # pair and share it with all clients:
+ #
+ # openssl genrsa -out john-private.pem 1024
+ # openssl rsa -in john-private.pem -out john-public.pem -outform PEM -pubout
+ #
+ # Each user has a unique userid, this is based on the name of the public key.
+ # In this example case the userid would be 'john-public'.
+ #
+ # Store these somewhere like:
+ #
+ # /home/john/.mc/john-private.pem
+ # /home/john/.mc/john-public.pem
+ #
+ # Every users public key needs to be distributed to all the nodes, save the john one
+ # in a file called:
+ #
+ # /etc/mcollective/ssl/clients/john-public.pem
+ #
+ # If you wish to use registration or auditing that sends connections over MC to a
+ # central host you will need also put the server-public.pem in the clients directory.
+ #
+ # You should be aware if you do add the node public key to the clients dir you will in
+ # effect be weakening your overall security. You should consider doing this only if
+ # you also set up an Authorization method that limits the requests the nodes can make.
+ #
+ # client.cfg:
+ #
+ # securityprovider = ssl
+ # plugin.ssl_server_public = /etc/mcollective/ssl/server-public.pem
+ # plugin.ssl_client_private = /home/john/.mc/john-private.pem
+ # plugin.ssl_client_public = /home/john/.mc/john-public.pem
+ #
+ # If you have many clients per machine and dont want to configure the main config file
+ # with the public/private keys you can set the following environment variables:
+ #
+ # export MCOLLECTIVE_SSL_PRIVATE=/home/john/.mc/john-private.pem
+ # export MCOLLECTIVE_SSL_PUBLIC=/home/john/.mc/john-public.pem
+ #
+ # server.cfg:
+ #
+ # securityprovider = ssl
+ # plugin.ssl_server_private = /etc/mcollective/ssl/server-private.pem
+ # plugin.ssl_server_public = /etc/mcollective/ssl/server-public.pem
+ # plugin.ssl_client_cert_dir = /etc/mcollective/etc/ssl/clients/
+ #
+ # # Log but accept messages that may have been tampered with
+ # plugin.ssl.enforce_ttl = 0
+ #
+ # Serialization can be configured to use either Marshal or YAML, data types
+ # in and out of mcollective will be preserved from client to server and reverse
+ #
+ # You can configure YAML serialization:
+ #
+ # plugins.ssl_serializer = yaml
+ #
+ # else the default is Marshal. Use YAML if you wish to write a client using
+ # a language other than Ruby that doesn't support Marshal.
+ #
+ # Validation is as default and is provided by MCollective::Security::Base
+ #
+ # Initial code was contributed by Vladimir Vuksan and modified by R.I.Pienaar
+ class Ssl < Base
+ # Decodes a message by unserializing all the bits etc, it also validates
+ # it as valid using the psk etc
+ def decodemsg(msg)
+ body = deserialize(msg.payload)
+
+ should_process_msg?(msg, body[:requestid])
+
+ if validrequest?(body)
+ body[:body] = deserialize(body[:body])
+
+ unless @initiated_by == :client
+ if body[:body].is_a?(Hash)
+ update_secure_property(body, :ssl_ttl, :ttl, "TTL")
+ update_secure_property(body, :ssl_msgtime, :msgtime, "Message Time")
+
+ body[:body] = body[:body][:ssl_msg] if body[:body].include?(:ssl_msg)
+ else
+ unless @config.pluginconf["ssl.enforce_ttl"] == nil
+ raise "Message %s is in an unknown or older security protocol, ignoring" % [request_description(body)]
+ end
+ end
+ end
+
+ return body
+ else
+ nil
+ end
+ end
+
+ # To avoid tampering we turn the origin body into a hash and copy some of the protocol keys
+ # like :ttl and :msg_time into the hash before hashing it.
+ #
+ # This function compares and updates the unhashed ones based on the hashed ones. By
+ # default it enforces matching and presense by raising exceptions, if ssl.enforce_ttl is set
+ # to 0 it will only log warnings about violations
+ def update_secure_property(msg, secure_property, property, description)
+ req = request_description(msg)
+
+ unless @config.pluginconf["ssl.enforce_ttl"] == "0"
+ raise "Request #{req} does not have a secure #{description}" unless msg[:body].include?(secure_property)
+ raise "Request #{req} #{description} does not match encrypted #{description} - possible tampering" unless msg[:body][secure_property] == msg[property]
+ else
+ if msg[:body].include?(secure_property)
+ Log.warn("Request #{req} #{description} does not match encrypted #{description} - possible tampering") unless msg[:body][secure_property] == msg[property]
+ else
+ Log.warn("Request #{req} does not have a secure #{description}") unless msg[:body].include?(secure_property)
+ end
+ end
+
+ msg[property] = msg[:body][secure_property] if msg[:body].include?(secure_property)
+ msg[:body].delete(secure_property)
+ end
+
+ # Encodes a reply
+ def encodereply(sender, msg, requestid, requestcallerid=nil)
+ serialized = serialize(msg)
+ digest = makehash(serialized)
+
+
+ req = create_reply(requestid, sender, serialized)
+ req[:hash] = digest
+
+ serialize(req)
+ end
+
+ # Encodes a request msg
+ def encoderequest(sender, msg, requestid, filter, target_agent, target_collective, ttl=60)
+ req = create_request(requestid, filter, "", @initiated_by, target_agent, target_collective, ttl)
+
+ ssl_msg = {:ssl_msg => msg,
+ :ssl_ttl => ttl,
+ :ssl_msgtime => req[:msgtime]}
+
+ serialized = serialize(ssl_msg)
+ digest = makehash(serialized)
+
+ req[:hash] = digest
+ req[:body] = serialized
+
+ serialize(req)
+ end
+
+ # Checks the SSL signature in the request body
+ def validrequest?(req)
+ message = req[:body]
+ signature = req[:hash]
+
+ Log.debug("Validating request from #{req[:callerid]}")
+
+ if verify(public_key_file(req[:callerid]), signature, message.to_s)
+ @stats.validated
+ return true
+ else
+ @stats.unvalidated
+ raise(SecurityValidationFailed, "Received an invalid signature in message")
+ end
+ end
+
+
+ # sets the caller id to the md5 of the public key
+ def callerid
+ if @initiated_by == :client
+ id = "cert=#{File.basename(client_public_key).gsub(/\.pem$/, '')}"
+ raise "Invalid callerid generated from client public key" unless valid_callerid?(id)
+ else
+ # servers need to set callerid as well, not usually needed but
+ # would be if you're doing registration or auditing or generating
+ # requests for some or other reason
+ id = "cert=#{File.basename(server_public_key).gsub(/\.pem$/, '')}"
+ raise "Invalid callerid generated from server public key" unless valid_callerid?(id)
+ end
+
+ return id
+ end
+
+ private
+ # Serializes a message using the configured encoder
+ def serialize(msg)
+ serializer = @config.pluginconf["ssl_serializer"] || "marshal"
+
+ Log.debug("Serializing using #{serializer}")
+
+ case serializer
+ when "yaml"
+ return YAML.dump(msg)
+ else
+ return Marshal.dump(msg)
+ end
+ end
+
+ # De-Serializes a message using the configured encoder
+ def deserialize(msg)
+ serializer = @config.pluginconf["ssl_serializer"] || "marshal"
+
+ Log.debug("De-Serializing using #{serializer}")
+
+ case serializer
+ when "yaml"
+ return YAML.load(msg)
+ else
+ return Marshal.load(msg)
+ end
+ end
+
+ # Figures out where to get our private key
+ def private_key_file
+ if ENV.include?("MCOLLECTIVE_SSL_PRIVATE")
+ return ENV["MCOLLECTIVE_SSL_PRIVATE"]
+ else
+ if @initiated_by == :node
+ return server_private_key
+ else
+ return client_private_key
+ end
+ end
+ end
+
+ # Figures out the public key to use
+ #
+ # If the node is asking do it based on caller id
+ # If the client is asking just get the node public key
+ def public_key_file(callerid = nil)
+ if @initiated_by == :client
+ return server_public_key
+ else
+ if callerid =~ /cert=([\w\.\-]+)/
+ cid = $1
+
+ if File.exist?("#{client_cert_dir}/#{cid}.pem")
+ return "#{client_cert_dir}/#{cid}.pem"
+ else
+ raise("Could not find a public key for #{cid} in #{client_cert_dir}/#{cid}.pem")
+ end
+ else
+ raise("Caller id is not in the expected format")
+ end
+ end
+ end
+
+ # Figures out the client private key either from MCOLLECTIVE_SSL_PRIVATE or the
+ # plugin.ssl_client_private config option
+ def client_private_key
+ return ENV["MCOLLECTIVE_SSL_PRIVATE"] if ENV.include?("MCOLLECTIVE_SSL_PRIVATE")
+
+ raise("No plugin.ssl_client_private configuration option specified") unless @config.pluginconf.include?("ssl_client_private")
+
+ return @config.pluginconf["ssl_client_private"]
+ end
+
+ # Figures out the client public key either from MCOLLECTIVE_SSL_PUBLIC or the
+ # plugin.ssl_client_public config option
+ def client_public_key
+ return ENV["MCOLLECTIVE_SSL_PUBLIC"] if ENV.include?("MCOLLECTIVE_SSL_PUBLIC")
+
+ raise("No plugin.ssl_client_public configuration option specified") unless @config.pluginconf.include?("ssl_client_public")
+
+ return @config.pluginconf["ssl_client_public"]
+ end
+
+ # Figures out the server private key from the plugin.ssl_server_private config option
+ def server_private_key
+ raise("No plugin.ssl_server_private configuration option specified") unless @config.pluginconf.include?("ssl_server_private")
+ @config.pluginconf["ssl_server_private"]
+ end
+
+ # Figures out the server public key from the plugin.ssl_server_public config option
+ def server_public_key
+ raise("No ssl_server_public configuration option specified") unless @config.pluginconf.include?("ssl_server_public")
+ return @config.pluginconf["ssl_server_public"]
+ end
+
+ # Figures out where to get client public certs from the plugin.ssl_client_cert_dir config option
+ def client_cert_dir
+ raise("No plugin.ssl_client_cert_dir configuration option specified") unless @config.pluginconf.include?("ssl_client_cert_dir")
+ @config.pluginconf["ssl_client_cert_dir"]
+ end
+
+ # Retrieves the value of plugin.psk and builds a hash with it and the passed body
+ def makehash(body)
+ Log.debug("Creating message hash using #{private_key_file}")
+
+ sign(private_key_file, body.to_s)
+ end
+
+ # Code adapted from http://github.com/adamcooke/basicssl
+ # signs a message
+ def sign(key, string)
+ SSL.new(nil, key).sign(string, true)
+ end
+
+ # verifies a signature
+ def verify(key, signature, string)
+ SSL.new(key).verify_signature(signature, string, true)
+ end
+
+ def request_description(msg)
+ "%s from %s@%s" % [msg[:requestid], msg[:callerid], msg[:senderid]]
+ end
+ end
+ end
+end
--- /dev/null
+metadata :name => "Array",
+ :description => "Validates that a value is included in a list",
+ :author => "P. Loubser <pieter.loubser@puppetlabs.com>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 1
--- /dev/null
+module MCollective
+ module Validator
+ class ArrayValidator
+ def self.validate(validator, array)
+ raise ValidatorError, "value should be one of %s" % [ array.join(", ") ] unless array.include?(validator)
+ end
+ end
+ end
+end
--- /dev/null
+metadata :name => "IPv4 Address",
+ :description => "Validates that a value is an ipv4 address",
+ :author => "P. Loubser <pieter.loubser@puppetlabs.com>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 1
--- /dev/null
+module MCollective
+ module Validator
+ class Ipv4addressValidator
+ require 'ipaddr'
+
+ def self.validate(validator)
+ begin
+ ip = IPAddr.new(validator)
+ raise ValidatorError, "value should be an ipv4 adddress" unless ip.ipv4?
+ rescue
+ raise ValidatorError, "value should be an ipv4 address"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+metadata :name => "IPv6 Address",
+ :description => "Validates that a value is an ipv6 address",
+ :author => "P. Loubser <pieter.loubser@puppetlabs.com>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 1
--- /dev/null
+module MCollective
+ module Validator
+ class Ipv6addressValidator
+ require 'ipaddr'
+
+ def self.validate(validator)
+ begin
+ ip = IPAddr.new(validator)
+ raise ValidatorError, "value should be an ipv6 adddress" unless ip.ipv6?
+ rescue
+ raise ValidatorError, "value should be an ipv6 address"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+metadata :name => "Length",
+ :description => "Validates that the length of a string is less or equal to a specified value",
+ :author => "P. Loubser <pieter.loubser@puppetlabs.com>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 1
--- /dev/null
+module MCollective
+ module Validator
+ class LengthValidator
+ def self.validate(validator, length)
+ if (validator.size > length) && (length > 0)
+ raise ValidatorError, "Input string is longer than #{length} character(s)"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+metadata :name => "Regex",
+ :description => "Validates that a string matches a supplied regular expression",
+ :author => "P. Loubser <pieter.loubser@puppetlabs.com>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 1
--- /dev/null
+module MCollective
+ module Validator
+ class RegexValidator
+ def self.validate(validator, regex)
+ raise ValidatorError, "value should match #{regex}" unless validator.match(regex)
+ end
+ end
+ end
+end
--- /dev/null
+metadata :name => "Shellsafe",
+ :description => "Validates that a string is shellsafe",
+ :author => "P. Loubser <pieter.loubser@puppetlabs.com>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 1
--- /dev/null
+module MCollective
+ module Validator
+ class ShellsafeValidator
+ def self.validate(validator)
+ raise ValidatorError, "value should be a String" unless validator.is_a?(String)
+
+ ['`', '$', ';', '|', '&&', '>', '<'].each do |chr|
+ raise ValidatorError, "value should not have #{chr} in it" if validator.match(Regexp.escape(chr))
+ end
+ end
+ end
+ end
+end
--- /dev/null
+metadata :name => "Typecheck",
+ :description => "Validates that a value is of a certain type",
+ :author => "P. Loubser <pieter.loubser@puppetlabs.com>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 1
--- /dev/null
+module MCollective
+ module Validator
+ class TypecheckValidator
+ def self.validate(validator, validation_type)
+ raise ValidatorError, "value should be a #{validation_type.to_s}" unless check_type(validator, validation_type)
+ end
+
+ def self.check_type(validator, validation_type)
+ case validation_type
+ when Class
+ validator.is_a?(validation_type)
+ when :integer
+ validator.is_a?(Fixnum)
+ when :float
+ validator.is_a?(Float)
+ when :number
+ validator.is_a?(Numeric)
+ when :string
+ validator.is_a?(String)
+ when :boolean
+ [TrueClass, FalseClass].include?(validator.class)
+ else
+ false
+ end
+ end
+ end
+ end
+end
--- /dev/null
+require File.join(File.dirname(__FILE__), "spec_helper.rb")
+require 'rake'
+require 'rspec/core/rake_task'
+
+desc "Run all specs"
+RSpec::Core::RakeTask.new(:all) do |t|
+ t.pattern = 'unit/**/*_spec.rb'
+
+ if MCollective::Util.windows?
+ t.rspec_opts = File.read("windows_spec.opts").chomp
+ else
+ t.rspec_opts = File.read("spec.opts").chomp
+ end
+end
+
+task :default => [:all]
--- /dev/null
+module MCollective
+ class Application::Test < Application
+ def run
+ true
+ end
+ end
+end
--- /dev/null
+-----BEGIN CERTIFICATE-----
+MIICRDCCAa2gAwIBAgIBAjANBgkqhkiG9w0BAQUFADAfMR0wGwYDVQQDDBRQdXBw
+ZXQgQ0E6IGRldmNvLm5ldDAeFw0xMjAzMzAyMTQ2NDRaFw0xNzAzMjkyMTQ2NDRa
+MA4xDDAKBgNVBAMMA2ZvbzCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA0X8t
+vvXk9Tmfhw5T87svZ1Oc2xORR1YbPTBCpNR3dvg2Pdurt16dLMXIGT5EeXkZy6cF
+MHG9pg/0ubCA7L/EdPI/Xq6n+O2kUUX8+ca0qwmj9/TwqxMSxVlKGr6SSqcCcHfi
+0hsDsHQaKyzAPbr69gbHhBzskd9bkh51Tyg96JECAwEAAaOBoDCBnTA4BglghkgB
+hvhCAQ0EKxYpUHVwcGV0IFJ1YnkvT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNh
+dGUwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUjZoFfKlgsGXuldlQQPOnBNrX9oMw
+CwYDVR0PBAQDAgWgMCcGA1UdJQQgMB4GCCsGAQUFBwMBBggrBgEFBQcDAgYIKwYB
+BQUHAwQwDQYJKoZIhvcNAQEFBQADgYEApfZYKcYrMUtStJ0jjoZryn4ssghs8mkl
+C+njqTt/X8/7drqiSHB/0WXN0366QfVEgotxh0akhi2GJxs+eD/iCIRSL29hwDjL
+LpcsM/Rk0s8jG/XJDmS+BdY+wGOezasaCaqQlsU/Sjv8TQ+8D89OBfh+ZYW3gL9E
+JWp47gFO/RA=
+-----END CERTIFICATE-----
--- /dev/null
+-----BEGIN RSA PRIVATE KEY-----
+MIICXQIBAAKBgQDMMGfwPZO8Xwn8XvrOy3Nc4nTkY/8lJrF3wvoGWdrcuQsNjYvZ
+O59PqXGxu/No5Lt5ymMd8orxR3bPRy2uobmZ2yXPP5mh48lZkMazGHPUeZkWOWbR
+7hgAvOVpNGp6NyR7D3PxZ96jzAbH8ttJVZAt/T2ImMJrzUG1Oj0/XO9fNQIDAQAB
+AoGALnA/41KN3ASdZ8lOL2P0C8bxINRhPdjL+dndNT6QWSy4h8+OY8x8kgiOdAaz
++EI1JSDTZAc6dF91dPTSPepIJH5n+ueBDHFTGUnbxCw25+6X9CaLig52fNh7vs53
+sjtEvpju4Qe49FjnGe+UZ6jhd5b10pFtiuTVhMUTE77HqmECQQD4iSmj5/OVz3qk
+zrJ9Z8z6NjQFjklA1reHWhYal9k+475xOyLQz4yXk+pzq+9uYMo+LBIvDaY+mzDb
+I3Sp8/vJAkEA0lJKANLqkRsGqp2BITzFS3vpoYQxL13pghFfvQPkpD+JbHB0wI4a
+E1xv1kr/4bQ3aMEohJnaLbAf/O20nS3mDQJBAOwZ0Tbl+J7OlSHPQfykCTOBHnZQ
+rwIrd/na+LiWnEiELbx/gxl+sX6lg8oTAceHp1jcoQGWI+HBp+3lhsSVBRECQHz2
+jae9qcc7kpNu79lqvSLjZeYkoACvwN0aK5MnrAL3CVTX4FbEV7PnOT/O4ggdxspD
+8ioPK7X3rpneNnEpY0UCQQCM38gg75soJTS9W3BY04gLdjLd1h5rXe28bTgnwRM2
+Y2HMzPDtCWdjWvmpZMfAg+U4Rkr24Gbd1V3bq6OlJIeB
+-----END RSA PRIVATE KEY-----
--- /dev/null
+-----BEGIN PUBLIC KEY-----
+MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDMMGfwPZO8Xwn8XvrOy3Nc4nTk
+Y/8lJrF3wvoGWdrcuQsNjYvZO59PqXGxu/No5Lt5ymMd8orxR3bPRy2uobmZ2yXP
+P5mh48lZkMazGHPUeZkWOWbR7hgAvOVpNGp6NyR7D3PxZ96jzAbH8ttJVZAt/T2I
+mMJrzUG1Oj0/XO9fNQIDAQAB
+-----END PUBLIC KEY-----
--- /dev/null
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
+
+ foo
+ bar
+
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
--- /dev/null
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
+
+ foo
+ bar
+
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
+ This is a string This is a string This is a string This is a string This is a string This is a string This is a string
--- /dev/null
+foooooooooooooooooooooooooooooooooooooooooooooooooooo
--- /dev/null
+ foooooooooooooooooooooooooooooooooooooooooooooooooooo
--- /dev/null
+ Strings Strings Strings Strings Strings Strings
--- /dev/null
+ Strings Strings Strings
+ Strings Strings Strings
--- /dev/null
+this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text
+
+ foo(bar)
+
+this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text this is usage text
--- /dev/null
+ this is usage text this is usage text this is usage text this is usage text
+ this is usage text this is usage text this is usage text this is usage text
+ this is usage text this is usage text this is usage text this is usage text
+
+ foo(bar)
+
+ this is usage text this is usage text this is usage text this is usage text
+ this is usage text this is usage text this is usage text this is usage text
+ this is usage text this is usage text this is usage text this is usage text
--- /dev/null
+module MCollective
+ module Matchers
+ def raise_code(*args)
+ CodedExceptionMatcher.new(args)
+ end
+
+ class CodedExceptionMatcher
+ def initialize(args)
+ @args = args
+
+ raise "Need at least an exception to match" if args.size == 0
+
+ @expected_code = @args.shift
+ @expected_data = @args.shift
+
+ @failure_type = nil
+ @failure_expected = nil
+ @failure_got = nil
+ end
+
+ def matches?(actual)
+ begin
+ actual.call
+ rescue => e
+ unless e.is_a?(MCollective::CodedError)
+ @failure_type = :type
+ @failure_expected = "MCollective::CodedError"
+ @failure_got = e.class
+ return false
+ end
+
+ unless [e.code, e.default].include?(@expected_code)
+ @failure_type = :code
+ @failure_expected = @expected_code
+ @failure_got = e.code
+ return false
+ end
+
+ if @expected_data
+ unless e.args == @expected_data
+ @failure_type = :arguments
+ @failure_expected = @expected_data.inspect
+ @failure_got = e.args.inspect
+ return false
+ end
+ end
+ end
+
+ true
+ end
+
+ def failure_message
+ case @failure_type
+ when :type
+ "Expected an exception of type %s but got %s" % [@failure_expected, @failure_got]
+ when :code
+ "Expected a message code %s but got %s" % [@failure_expected, @failure_got]
+ when :arguments
+ "Expected arguments %s but got %s" % [@failure_expected, @failure_got]
+ end
+ end
+
+ def negative_failure_message
+ case @failure_type
+ when :type
+ "Expected an exception of type %s but got %s" % [@failure_got, @failure_expected]
+ when :code
+ "Expected a message code %s but got %s" % [@failure_got, @failure_expected]
+ when :arguments
+ "Expected arguments %s but got %s" % [@failure_got, @failure_expected]
+ end
+ end
+ end
+ end
+end
--- /dev/null
+unless Object.respond_to?("instance_variable_defined?")
+ class Object
+ def instance_variable_defined?(meth)
+ instance_variables.include?(meth)
+ end
+ end
+end
--- /dev/null
+--colour --backtrace
--- /dev/null
+dir = File.expand_path(File.dirname(__FILE__))
+$LOAD_PATH.unshift("#{dir}/")
+$LOAD_PATH.unshift("#{dir}/../lib")
+$LOAD_PATH.unshift("#{dir}/../plugins")
+
+require 'rubygems'
+
+gem 'mocha'
+
+require 'rspec'
+require 'mcollective'
+require 'rspec/mocks'
+require 'mocha'
+require 'ostruct'
+require 'tmpdir'
+require 'tempfile'
+require 'fileutils'
+
+require 'monkey_patches/instance_variable_defined'
+require 'matchers/exception_matchers'
+
+RSpec.configure do |config|
+ config.mock_with :mocha
+ config.include(MCollective::Matchers)
+
+ config.before :each do
+ MCollective::Config.instance.set_config_defaults("")
+ MCollective::PluginManager.clear
+ MCollective::Log.stubs(:log)
+ MCollective::Log.stubs(:logmsg)
+ MCollective::Log.stubs(:logexception)
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Agents do
+ before do
+ tmpfile = Tempfile.new("mc_agents_spec")
+ path = tmpfile.path
+ tmpfile.close!
+
+ @tmpdir = FileUtils.mkdir_p(path)
+ @tmpdir = @tmpdir[0] if @tmpdir.is_a?(Array) # ruby 1.9.2
+
+ @agentsdir = File.join([@tmpdir, "mcollective", "agent"])
+ FileUtils.mkdir_p(@agentsdir)
+
+ logger = mock
+ logger.stubs(:log)
+ logger.stubs(:start)
+ Log.configure(logger)
+ end
+
+ after do
+ FileUtils.rm_r(@tmpdir)
+ end
+
+ describe "#initialize" do
+ it "should fail if configuration has not been loaded" do
+ Config.instance.expects(:configured).returns(false)
+
+ expect {
+ Agents.new
+ }.to raise_error("Configuration has not been loaded, can't load agents")
+ end
+
+ it "should load agents" do
+ Config.instance.expects(:configured).returns(true)
+ Agents.any_instance.expects(:loadagents).once
+
+ Agents.new
+ end
+ end
+
+ describe "#clear!" do
+ it "should delete and unsubscribe all loaded agents" do
+ Config.instance.expects(:configured).returns(true).at_least_once
+ Config.instance.expects(:libdir).returns([@tmpdir])
+ PluginManager.expects(:delete).with("foo_agent").once
+ Util.expects(:make_subscriptions).with("foo", :broadcast).returns("foo_target")
+ Util.expects(:unsubscribe).with("foo_target")
+
+ a = Agents.new({"foo" => 1})
+ end
+ end
+
+ describe "#loadagents" do
+ before do
+ Config.instance.stubs(:configured).returns(true)
+ Config.instance.stubs(:libdir).returns([@tmpdir])
+ Agents.any_instance.stubs("clear!").returns(true)
+ end
+
+ it "should delete all existing agents" do
+ Agents.any_instance.expects("clear!").once
+ a = Agents.new
+ end
+
+ it "should attempt to load agents from all libdirs" do
+ Config.instance.expects(:libdir).returns(["/nonexisting", "/nonexisting"])
+ File.expects("directory?").with("/nonexisting/mcollective/agent").twice
+
+ a = Agents.new
+ end
+
+ it "should load found agents" do
+ Agents.any_instance.expects("loadagent").with("test").once
+
+ FileUtils.touch(File.join([@agentsdir, "test.rb"]))
+
+ a = Agents.new
+ end
+
+ it "should load each agent unless already loaded" do
+ Agents.any_instance.expects("loadagent").with("test").never
+
+ FileUtils.touch(File.join([@agentsdir, "test.rb"]))
+
+ PluginManager << {:type => "test_agent", :class => String.new}
+ a = Agents.new
+ end
+ end
+
+ describe "#loadagent" do
+ before do
+ FileUtils.touch(File.join([@agentsdir, "test.rb"]))
+ Config.instance.stubs(:configured).returns(true)
+ Config.instance.stubs(:libdir).returns([@tmpdir])
+ Agents.any_instance.stubs("clear!").returns(true)
+ PluginManager.stubs(:loadclass).returns(true)
+ PluginManager.stubs("[]").with("test_agent").returns(true)
+ Util.stubs(:make_subscriptions).with("test", :broadcast).returns([{:agent => "test", :type => :broadcast, :collective => "test"}])
+ Util.stubs(:subscribe).with([{:agent => "test", :type => :broadcast, :collective => "test"}]).returns(true)
+ Agents.stubs(:findagentfile).returns(File.join([@agentsdir, "test.rb"]))
+ Agents.any_instance.stubs("activate_agent?").returns(true)
+
+ @a = Agents.new
+ end
+
+ it "should return false if the agent file is missing" do
+ Agents.any_instance.expects(:findagentfile).returns(false).once
+ @a.loadagent("test").should == false
+ end
+
+ it "should delete the agent before loading again" do
+ PluginManager.expects(:delete).with("test_agent").twice
+ @a.loadagent("test")
+ end
+
+ it "should load the agent class from disk" do
+ PluginManager.expects(:loadclass).with("MCollective::Agent::Test")
+ @a.loadagent("test")
+ end
+
+ it "should check if the agent should be activated" do
+ Agents.any_instance.expects(:findagentfile).with("test").returns(File.join([@agentsdir, "test.rb"]))
+ Agents.any_instance.expects("activate_agent?").with("test").returns(true)
+ @a.loadagent("test").should == true
+ end
+
+ it "should set discovery and registration to be single instance plugins" do
+ PluginManager.expects("<<").with({:type => "registration_agent", :class => "MCollective::Agent::Registration", :single_instance => true}).once
+ PluginManager.expects("<<").with({:type => "discovery_agent", :class => "MCollective::Agent::Discovery", :single_instance => true}).once
+ Agents.any_instance.expects("activate_agent?").with("registration").returns(true)
+ Agents.any_instance.expects("activate_agent?").with("discovery").returns(true)
+
+ PluginManager.expects(:loadclass).with("MCollective::Agent::Registration").returns(true).once
+ PluginManager.expects(:loadclass).with("MCollective::Agent::Discovery").returns(true).once
+
+ FileUtils.touch(File.join([@agentsdir, "registration.rb"]))
+ FileUtils.touch(File.join([@agentsdir, "discovery.rb"]))
+
+ @a.loadagent("registration")
+ @a.loadagent("discovery")
+ end
+
+ it "should add general plugins as multiple instance plugins" do
+ PluginManager.expects("<<").with({:type => "test_agent", :class => "MCollective::Agent::Test", :single_instance => false}).once
+ @a.loadagent("test")
+ end
+
+ it "should add the agent to the plugin manager and subscribe" do
+ PluginManager.expects("<<").with({:type => "foo_agent", :class => "MCollective::Agent::Foo", :single_instance => false})
+ Util.stubs(:make_subscriptions).with("foo", :broadcast).returns([{:agent => "foo", :type => :broadcast, :collective => "test"}])
+ Util.expects("subscribe").with([{:type => :broadcast, :agent => 'foo', :collective => 'test'}]).returns(true)
+ Agents.any_instance.expects(:findagentfile).with("foo").returns(File.join([@agentsdir, "foo.rb"]))
+ FileUtils.touch(File.join([@agentsdir, "foo.rb"]))
+ Agents.any_instance.expects("activate_agent?").with("foo").returns(true)
+ PluginManager.stubs("[]").with("foo_agent").returns(true)
+
+ @a.loadagent("foo")
+ end
+
+ it "should check if an agent is loadable and remove it from the list if not" do
+ PluginManager.expects("<<").with({:type => "foo_agent", :class => "MCollective::Agent::Foo", :single_instance => false})
+ Agents.any_instance.expects(:findagentfile).with("foo").returns(File.join([@agentsdir, "foo.rb"]))
+ FileUtils.touch(File.join([@agentsdir, "foo.rb"]))
+ Agents.any_instance.expects("activate_agent?").with("foo").returns(true)
+ PluginManager.stubs("[]").with("foo_agent").raises("rspec")
+
+ Log.expects(:error).once.with("Loading agent foo failed: rspec")
+
+ @a.loadagent("foo").should == false
+ Agents.agentlist.include?("foo").should == false
+ end
+
+ it "should add the agent to the agent list" do
+ Agents.agentlist.should == ["test"]
+ end
+
+ it "should return true on success" do
+ @a.loadagent("test").should == true
+ end
+
+ it "should handle load exceptions" do
+ Agents.any_instance.expects(:findagentfile).with("foo").returns(File.join([@agentsdir, "foo.rb"]))
+ Log.expects(:error).with(regexp_matches(/Loading agent foo failed/))
+ @a.loadagent("foo").should == false
+ end
+
+ it "should delete plugins that failed to load" do
+ Agents.any_instance.expects(:findagentfile).with("foo").returns(File.join([@agentsdir, "foo.rb"]))
+ PluginManager.expects(:delete).with("foo_agent").twice
+
+ @a.loadagent("foo").should == false
+ end
+ end
+
+ describe "#class_for_agent" do
+ it "should return the correct class" do
+ Config.instance.stubs(:configured).returns(true)
+ Agents.any_instance.stubs(:loadagents).returns(true)
+ Agents.new.class_for_agent("foo").should == "MCollective::Agent::Foo"
+ end
+ end
+
+ describe "#activate_agent?" do
+ before do
+ Config.instance.stubs(:configured).returns(true)
+ Agents.any_instance.stubs(:loadagents).returns(true)
+ @a = Agents.new
+
+ module MCollective::Agent;end
+ class MCollective::Agent::Test; end
+ end
+
+ it "should check if the correct class has an activation method" do
+ Agent::Test.expects("respond_to?").with("activate?").once
+
+ @a.activate_agent?("test")
+ end
+
+ it "should call the activation method" do
+ Agent::Test.expects("activate?").returns(true).once
+ @a.activate_agent?("test")
+ end
+
+ it "should log a debug message and return true if the class has no activation method" do
+ Agent::Test.expects("respond_to?").with("activate?").returns(false).once
+ Log.expects(:debug).with("MCollective::Agent::Test does not have an activate? method, activating as default")
+
+ @a.activate_agent?("test").should == true
+ end
+
+ it "should handle exceptions in the activation as false" do
+ Agent::Test.expects("activate?").raises(RuntimeError)
+ @a.activate_agent?("test").should == false
+ end
+ end
+
+ describe "#findagentfile" do
+ before do
+ Config.instance.stubs(:configured).returns(true)
+ Config.instance.stubs(:libdir).returns([@tmpdir])
+ Agents.any_instance.stubs(:loadagents).returns(true)
+ @a = Agents.new
+ end
+
+ it "should support multiple libdirs" do
+ Config.instance.expects(:libdir).returns([@tmpdir, @tmpdir]).once
+ File.expects("exist?").returns(false).twice
+ @a.findagentfile("test")
+ end
+
+ it "should look for the correct filename in the libdir" do
+ File.expects("exist?").with(File.join([@tmpdir, "mcollective", "agent", "test.rb"])).returns(false).once
+ @a.findagentfile("test")
+ end
+
+ it "should return the full path if the agent is found" do
+ agentfile = File.join([@tmpdir, "mcollective", "agent", "test.rb"])
+ File.expects("exist?").with(agentfile).returns(true).once
+ @a.findagentfile("test").should == agentfile
+ end
+
+ it "should return false if no agent is found" do
+ @a.findagentfile("foo").should == false
+ end
+ end
+
+ describe "#include?" do
+ it "should correctly report the plugin state" do
+ Config.instance.stubs(:configured).returns(true)
+ Config.instance.stubs(:libdir).returns([@tmpdir])
+ Agents.any_instance.stubs(:loadagents).returns(true)
+ PluginManager.expects("include?").with("test_agent").returns(true)
+
+ @a = Agents.new
+
+ @a.include?("test").should == true
+ end
+ end
+
+ describe "#agentlist" do
+ it "should return the correct agent list" do
+ Config.instance.stubs(:configured).returns(true)
+ Config.instance.stubs(:libdir).returns([@tmpdir])
+ Agents.any_instance.stubs(:loadagents).returns(true)
+
+ @a = Agents.new("test" => true)
+ Agents.agentlist.should == ["test"]
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ class Aggregate
+ describe Base do
+ describe "#initialize" do
+ it "should set the correct instance variables and call the startup hook" do
+ Base.any_instance.expects(:startup_hook).once
+ base = Base.new("value", [], "%s%s", "rspec")
+ base.name.should == "MCollective::Aggregate::Base"
+ base.output_name.should == "value"
+ base.aggregate_format.should == "%s%s"
+ base.action.should == "rspec"
+ end
+ end
+
+ describe "#startup_hook and #process_result" do
+ it "should raise an exception for an unimplemented startup_hook method " do
+ expect{
+ base = Base.new("value", [], "", "rspec")
+ }.to raise_error(RuntimeError, "'startup_hook' method of function class MCollective::Aggregate::Base has not yet been implemented")
+ end
+
+ it "should raise an exception for an unimplemented process_result method" do
+ Base.any_instance.stubs(:startup_hook)
+ base = Base.new("value", [], "", "rspec")
+ expect{
+ base.process_result
+ }.to raise_error(RuntimeError,"'process_result' method of function class MCollective::Aggregate::Base has not yet been implemented")
+ end
+ end
+
+ describe "summarize" do
+ it "should raise and exception if the result type has not been set" do
+ Base.any_instance.stubs(:startup_hook)
+ base = Base.new("value", [], "", "rspec")
+ expect{
+ base.summarize
+ }.to raise_error(RuntimeError, "Result type is not set while trying to summarize aggregate function results")
+ end
+
+ it "should return the correct result class if result type has been set" do
+ result_object = mock
+ result_object.stubs(:new)
+
+ Base.any_instance.stubs(:startup_hook)
+ base = Base.new("value", [], "", "rspec")
+ base.result[:type] = :result_type
+ base.expects(:result_class).with(:result_type).returns(result_object)
+ base.summarize
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ class Aggregate
+ module Result
+ describe Base do
+ describe "#initialize" do
+ it "should raise an exception if neither the ddl or the aggregate function defines a format" do
+ expect{
+ base = Base.new(:structure, nil, :action)
+ }.to raise_error(RuntimeError, "No aggregate_format defined in ddl or aggregate function")
+ end
+ end
+
+ describe "#to_s" do
+ it "should raise an exception if the to_s method isn't implemented" do
+ base = Base.new(:structure, :aggregate_format, :action)
+ expect{
+ base.to_s
+ }.to raise_error(RuntimeError, "'to_s' method not implemented for result class 'MCollective::Aggregate::Result::Base'")
+ end
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ class Aggregate
+ module Result
+ describe CollectionResult do
+ describe "#to_s" do
+ it "should return the correctly formatted string" do
+ result = CollectionResult.new({:output => [:test], :value => {"foo" => 3, "bar" => 2}}, "%s:%s", :action).to_s
+ result.should == "foo:3\nbar:2"
+ end
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ class Aggregate
+ module Result
+ describe NumericResult do
+ describe "#to_s" do
+ it "should return empty string when no results were computed" do
+ NumericResult.new({}, "test %d", :action).to_s.should == ""
+ end
+
+ it "should return the correctly formatted string" do
+ num = NumericResult.new({:value => 1}, "test %d", :action).to_s
+ num.should == "test 1"
+ end
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Aggregate do
+ let(:ddl) do
+ {
+ :aggregate => [{:function => :func, :args=>[:foo, {:format => "%s"}]}],
+ :action => "test_action",
+ :output => {:foo => nil, :bar => nil}
+ }
+ end
+
+ describe '#create_functions' do
+ let(:function){mock}
+
+ it "should load all the functions with a format if defined" do
+ function.expects(:new).with(:foo, {}, "%s", 'test_action')
+ Aggregate.any_instance.expects(:contains_output?).returns(true)
+ Aggregate.any_instance.expects(:load_function).once.returns(function)
+ Aggregate.new(ddl)
+ end
+
+ it "should load all the functions without a format if it isn't defined" do
+ function.expects(:new).with(:foo, {}, nil, 'test_action')
+ Aggregate.any_instance.expects(:load_function).once.returns(function)
+ ddl[:aggregate].first[:args][1][:format] = nil
+ Aggregate.new(ddl)
+ end
+
+ it "should not summarize functions where the output is not specified in the ddl" do
+ invalid_ddl = { :aggregate => [{:function => :func, :args=>[:foo], :format => "%s"}, {:function => :func, :args=>[:fail], :format => "%s"}],
+ :action => "test_action",
+ :output => {:foo => nil, :bar => nil}}
+
+ function.stubs(:new).returns("function")
+ Aggregate.any_instance.stubs(:load_function).returns(function)
+
+ Log.expects(:error)
+ aggregate = Aggregate.new(invalid_ddl)
+ aggregate.functions.should == ["function"]
+ aggregate.failed.should == [{:type=>:create, :name=>:fail}]
+ end
+
+ it "should pass additional arguments if specified in the ddl" do
+ function.expects(:new).with(:foo, {:extra => "extra"}, "%s", 'test_action')
+ Aggregate.any_instance.expects(:load_function).once.returns(function)
+ ddl[:aggregate].first[:args][1][:extra] = "extra"
+ Aggregate.new(ddl)
+ end
+
+ it "should not summarize functions if the startup hook raises an exception" do
+ function.stubs(:new).raises("rspec")
+ Aggregate.any_instance.expects(:load_function).returns(function)
+
+ Log.expects(:error)
+ aggregate = Aggregate.new(ddl)
+ aggregate.failed.should == [{:type=>:startup, :name =>:foo }]
+ end
+ end
+
+ describe '#contains_ouput?' do
+ before :all do
+ Aggregate.any_instance.stubs(:create_functions)
+ @aggregate = Aggregate.new(ddl)
+ end
+
+ it "should return false if the ddl output does not include the function's input" do
+ result = @aggregate.contains_output?(:baz)
+ result.should == false
+ end
+
+ it "should return true if the ddl output includes the function's input" do
+ result = @aggregate.contains_output?(:foo)
+ result.should == true
+ end
+ end
+
+ describe '#call_functions' do
+ let(:aggregate){ Aggregate.new(ddl)}
+ let(:result){ RPC::Result.new("rspec", "rspec", :sender => "rspec", :statuscode => 0, :statusmsg => "rspec", :data => {:test => :result})}
+ let(:function) {mock}
+
+ before :each do
+ Aggregate.any_instance.stubs(:create_functions)
+ end
+
+ it "should call all of the functions" do
+ function.expects(:process_result).with(:result, result).once
+ function.expects(:output_name).returns(:test)
+ aggregate.functions = [function]
+
+ aggregate.call_functions(result)
+ end
+
+ it "should not fail if 'process_result' method raises an exception" do
+ aggregate.functions = [function]
+ function.stubs(:output_name).returns(:test)
+ function.stubs(:process_result).raises("Failed")
+
+ Log.expects(:error)
+ aggregate.call_functions(result)
+ aggregate.failed.should == [:name => :test, :type => :process_result]
+ end
+
+ it "should not fail if 'summarize' method raises en exception" do
+ function.stubs(:summarize).raises("Failed")
+ function.stubs(:output_name).returns("rspec")
+ aggregate.functions = [function]
+
+ Log.expects(:error)
+ result = aggregate.summarize
+ end
+ end
+
+ describe '#summarize' do
+ it "should return the ordered function results" do
+ Aggregate.any_instance.stubs(:create_functions)
+ aggregate = Aggregate.new(ddl)
+
+ func1 = mock
+ func1.expects(:summarize).returns(func1)
+ func1.stubs(:result).returns(:output => 5)
+
+ func2 = mock
+ func2.expects(:summarize).returns(func2)
+ func2.stubs(:result).returns(:output => 2)
+
+ aggregate.functions = [func1, func2]
+
+ result = aggregate.summarize
+ result.should == [func2, func1]
+ end
+
+ it "should not summarise data that raises an exception" do
+ Aggregate.any_instance.stubs(:create_functions)
+ aggregate = Aggregate.new(ddl)
+ func = mock
+ func.stubs(:summarize).raises("rspec")
+ func.stubs(:output_name).returns("rspec")
+ aggregate.functions = [func]
+ Log.expects(:error)
+
+ aggregate.summarize
+ aggregate.failed.should == [{:name => "rspec", :type => :summarize}]
+ end
+ end
+
+ describe '#load_function' do
+ before :all do
+ Aggregate.any_instance.stubs(:create_functions)
+ @aggregate = Aggregate.new(ddl)
+ end
+
+ it "should return a class object if it can be loaded" do
+ PluginManager.expects(:loadclass).with("MCollective::Aggregate::Test")
+ Aggregate.expects(:const_get).with("Test")
+ function = @aggregate.load_function("test")
+ end
+
+ it "should raise an exception if the class object cannot be loaded" do
+ PluginManager.expects(:loadclass).with("MCollective::Aggregate::Test")
+ expect {
+ function = @aggregate.load_function("test")
+ }.to raise_error("Aggregate function file 'test.rb' cannot be loaded")
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Application do
+ before do
+ Application.intialize_application_options
+ @argv_backup = ARGV.clone
+ end
+
+ describe "#application_options" do
+ it "should return the application options" do
+ Application.application_options.should == {:description => nil,
+ :usage => [],
+ :cli_arguments => [],
+ :exclude_arg_sections => []}
+ end
+ end
+
+ describe "#[]=" do
+ it "should set the application option" do
+ Application["foo"] = "bar"
+ Application.application_options["foo"].should == "bar"
+ end
+ end
+
+ describe "#[]" do
+ it "should set the application option" do
+ Application[:cli_arguments].should == []
+ end
+ end
+
+ describe "#intialize_application_options" do
+ it "should initialize application options correctly" do
+ Application.intialize_application_options.should == {:description => nil,
+ :usage => [],
+ :cli_arguments => [],
+ :exclude_arg_sections => []}
+ end
+ end
+
+ describe "#description" do
+ it "should set the description correctly" do
+ Application.description "meh"
+ Application[:description].should == "meh"
+ end
+ end
+
+ describe "#usage" do
+ it "should set the usage correctly" do
+ Application.usage "meh"
+ Application.usage "foo"
+
+ Application[:usage].should == ["meh", "foo"]
+ end
+ end
+
+ describe "#exclude_argument_sections" do
+ it "should set the excluded sections correctly" do
+ Application.exclude_argument_sections "common", "rpc", "filter"
+ Application[:exclude_arg_sections].should == ["common", "rpc", "filter"]
+ Application.exclude_argument_sections ["common", "rpc", "filter"]
+ Application[:exclude_arg_sections].should == ["common", "rpc", "filter"]
+ end
+
+ it "should detect unknown sections" do
+ expect { Application.exclude_argument_sections "rspec" }.to raise_error("Unknown CLI argument section rspec")
+ end
+ end
+
+ describe "#option" do
+ it "should add an option correctly" do
+ Application.option :test,
+ :description => "description",
+ :arguments => "--config CONFIG",
+ :type => Integer,
+ :required => true
+
+ args = Application[:cli_arguments].first
+ args.delete(:validate)
+
+ args.should == {:name=>:test,
+ :arguments=>"--config CONFIG",
+ :required=>true,
+ :type=>Integer,
+ :description=>"description"}
+ end
+
+ it "should set correct defaults" do
+ Application.option :test, {}
+
+ args = Application[:cli_arguments].first
+ args.delete(:validate)
+
+ args.should == {:name=>:test,
+ :arguments=>[],
+ :required=>false,
+ :type=>String,
+ :description=>nil}
+ end
+ end
+
+ describe "#validate_option" do
+ it "should pass validations" do
+ a = Application.new
+ a.validate_option(Proc.new {|v| v == 1}, "key", 1)
+ end
+
+ it "should print an error to STDERR on error" do
+ IO.any_instance.expects(:puts).with("Validation of key failed: failed").at_least_once
+ Application.any_instance.stubs("exit").returns(true)
+
+ a = Application.new
+ a.validate_option(Proc.new {|v| "failed"}, "key", 1)
+ end
+
+ it "should exit on valdation error" do
+ IO.any_instance.expects(:puts).at_least_once
+ Application.any_instance.stubs("exit").returns(true)
+
+ a = Application.new
+ a.validate_option(Proc.new {|v| "failed"}, "key", 1)
+ end
+ end
+
+ describe "#application_parse_options" do
+ it "should pass the requested help value to the clioptions method" do
+ ARGV.clear
+
+ app = Application.new
+ app.expects(:clioptions).with(true)
+ app.application_parse_options(true)
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+
+ it "should support creating arrays of values" do
+ Application.any_instance.stubs("main").returns(true)
+
+ Application.option :foo,
+ :description => "meh",
+ :arguments => "--foo [FOO]",
+ :type => :array
+
+ ARGV.clear
+ ARGV << "--foo=bar" << "--foo=baz"
+
+ a = Application.new
+ a.run
+ a.configuration.should == {:foo=>["bar", "baz"]}
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+
+ it "should support boolean options" do
+ Application.any_instance.stubs("main").returns(true)
+
+ Application.option :foo,
+ :description => "meh",
+ :arguments => "--foo",
+ :type => :boolean
+
+ ARGV.clear
+ ARGV << "--foo"
+
+ a = Application.new
+ a.run
+ a.configuration.should == {:foo=>true}
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+
+ it "should support unsetting boolean options" do
+ Application.any_instance.stubs("main").returns(true)
+
+ Application.option :foo,
+ :description => "meh",
+ :arguments => "--[no-]foo",
+ :type => :boolean
+
+ ARGV.clear
+ ARGV << "--no-foo"
+
+ a = Application.new
+ a.run
+ a.configuration.should == {:foo=>false}
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+
+ it "should set the application description as head" do
+ OptionParser.any_instance.stubs(:define_head).with("meh")
+
+ ARGV.clear
+
+ Application.description "meh"
+ Application.new.application_parse_options
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+
+ it "should set the application usage as a banner" do
+ OptionParser.any_instance.stubs(:banner).with("meh")
+
+ ARGV.clear
+
+ Application.usage "meh"
+ Application.new.application_parse_options
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+
+ it "should support validation" do
+ IO.any_instance.expects(:puts).with("Validation of foo failed: failed").at_least_once
+ Application.any_instance.stubs("exit").returns(true)
+ Application.any_instance.stubs("main").returns(true)
+
+ Application.option :foo,
+ :description => "meh",
+ :required => true,
+ :default => "meh",
+ :arguments => "--foo [FOO]",
+ :validate => Proc.new {|v| "failed"}
+
+ ARGV.clear
+ ARGV << "--foo=bar"
+
+ a = Application.new
+ a.run
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+
+ it "should support default values" do
+ Application.any_instance.stubs("main").returns(true)
+
+ Application.option :foo,
+ :description => "meh",
+ :required => true,
+ :default => "meh",
+ :arguments => "--foo [FOO]"
+
+ a = Application.new
+ a.run
+ a.configuration.should == {:foo => "meh"}
+ end
+
+ it "should enforce required options" do
+ Application.any_instance.stubs("exit").returns(true)
+ Application.any_instance.stubs("main").returns(true)
+ OptionParser.any_instance.stubs("parse!").returns(true)
+ IO.any_instance.expects(:puts).with(anything).at_least_once
+ IO.any_instance.expects(:puts).with("The foo option is mandatory").at_least_once
+
+ ARGV.clear
+ ARGV << "--foo=bar"
+
+ Application.option :foo,
+ :description => "meh",
+ :required => true,
+ :arguments => "--foo [FOO]"
+
+ Application.new.run
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+
+ it "should call post_option_parser" do
+ OptionParser.any_instance.stubs("parse!").returns(true)
+ Application.any_instance.stubs("post_option_parser").returns(true).at_least_once
+ Application.any_instance.stubs("main").returns(true)
+
+ ARGV.clear
+ ARGV << "--foo=bar"
+
+ Application.option :foo,
+ :description => "meh",
+ :arguments => "--foo [FOO]"
+
+ Application.new.run
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+
+ it "should create an application option" do
+ OptionParser.any_instance.stubs("parse!").returns(true)
+ OptionParser.any_instance.expects(:on).with(anything, anything, anything, anything).at_least_once
+ OptionParser.any_instance.expects(:on).with('--foo [FOO]', String, 'meh').at_least_once
+ Application.any_instance.stubs("main").returns(true)
+
+ ARGV.clear
+ ARGV << "--foo=bar"
+
+ Application.option :foo,
+ :description => "meh",
+ :arguments => "--foo [FOO]"
+
+ Application.new.run
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+ end
+
+ describe "#initialize" do
+ it "should parse the command line options at application run" do
+ Application.any_instance.expects("application_parse_options").once
+ Application.any_instance.stubs("main").returns(true)
+
+ Application.new.run
+ end
+ end
+
+ describe "#application_options" do
+ it "sshould return the application options" do
+ Application.new.application_options.should == Application.application_options
+ end
+ end
+
+ describe "#application_description" do
+ it "should provide the right description" do
+ Application.description "Foo"
+ Application.new.application_description.should == "Foo"
+ end
+ end
+
+ describe "#application_usage" do
+ it "should provide the right usage" do
+ Application.usage "Foo"
+ Application.new.application_usage.should == ["Foo"]
+ end
+ end
+
+ describe "#application_cli_arguments" do
+ it "should provide the right usage" do
+ Application.option :foo,
+ :description => "meh",
+ :arguments => "--foo [FOO]"
+
+ args = Application.new.application_cli_arguments.first
+
+ # need to remove this cos we cant validate procs for equality afaik
+ args.delete(:validate)
+
+ args.should == {:description=>"meh",
+ :name=>:foo,
+ :arguments=>"--foo [FOO]",
+ :type=>String,
+ :required=>false}
+ end
+ end
+
+ describe "#help" do
+ it "should generate help using the full user supplied options" do
+ app = Application.new
+ app.expects(:clioptions).with(true).once
+ app.help
+ end
+ end
+
+ describe "#main" do
+ it "should detect applications without a #main" do
+ IO.any_instance.expects(:puts).with("Applications need to supply a 'main' method")
+
+ expect {
+ Application.new.run
+ }.to raise_error(SystemExit)
+ end
+
+ it "should raise SystemExit exceptions for exit events" do
+ connector = mock
+ connector.expects(:disconnect)
+ PluginManager.expects("[]").with("connector_plugin").returns(connector)
+
+ a = Application.new
+ a.expects(:main).raises(SystemExit)
+
+ expect {
+ a.run
+ }.to raise_error(SystemExit)
+ end
+ end
+
+ describe "#configuration" do
+ it "should return the correct configuration" do
+ Application.any_instance.stubs("main").returns(true)
+
+ ARGV.clear
+ ARGV << "--foo=bar"
+
+ Application.option :foo,
+ :description => "meh",
+ :arguments => "--foo [FOO]"
+
+ a = Application.new
+ a.run
+
+ a.configuration.should == {:foo => "bar"}
+
+ ARGV.clear
+ @argv_backup.each{|a| ARGV << a}
+ end
+ end
+
+ describe "#halt" do
+ before do
+ @stats = {:discoverytime => 0, :discovered => 0, :failcount => 0, :responses => 0, :okcount => 0}
+ end
+
+ it "should exit with code 0 if discovery was done and all responses passed" do
+ app = Application.new
+
+ @stats[:discoverytime] = 2
+ @stats[:discovered] = 2
+ @stats[:responses] = 2
+ @stats[:okcount] = 2
+
+ app.halt_code(@stats).should == 0
+ end
+
+ it "should exit with code 0 if no discovery were done but responses were received" do
+ app = Application.new
+
+ @stats[:responses] = 1
+ @stats[:okcount] = 1
+ @stats[:discovered] = 1
+
+ app.halt_code(@stats).should == 0
+ end
+
+ it "should exit with code 1 if discovery info is missing" do
+ app = Application.new
+
+ app.halt_code({}).should == 1
+ end
+
+ it "should exit with code 1 if no nodes were discovered and discovery was done" do
+ app = Application.new
+
+ @stats[:discoverytime] = 2
+
+ app.halt_code(@stats).should == 1
+ end
+
+ it "should exit with code 2 if a request failed for some nodes" do
+ app = Application.new
+
+ @stats[:discovered] = 2
+ @stats[:failcount] = 1
+ @stats[:discoverytime] = 2
+ @stats[:responses] = 2
+
+ app.halt_code(@stats).should == 2
+ end
+
+ it "should exit with code 2 when no discovery were done and there were failure results" do
+ app = Application.new
+
+ @stats[:discovered] = 1
+ @stats[:failcount] = 1
+ @stats[:discoverytime] = 0
+ @stats[:responses] = 1
+
+ app.halt_code(@stats).should == 2
+ end
+
+ it "should exit with code 3 if no responses were received after discovery" do
+ app = Application.new
+
+ @stats[:discovered] = 1
+ @stats[:discoverytime] = 2
+
+ app.halt_code(@stats).should == 3
+ end
+
+ it "should exit with code 4 if no discovery was done and no responses were received" do
+ app = Application.new
+
+ app.halt_code(@stats).should == 4
+ end
+ end
+
+ describe "#disconnect" do
+ it "should disconnect from the connector plugin" do
+ connector = mock
+ connector.expects(:disconnect)
+ PluginManager.expects("[]").with("connector_plugin").returns(connector)
+
+ Application.new.disconnect
+ end
+ end
+
+ describe "#clioptions" do
+ it "should pass the excluded argument section" do
+ oparser = mock
+ oparser.stubs(:parse)
+
+ Application.exclude_argument_sections "rpc"
+
+ Optionparser.expects(:new).with({:verbose => false, :progress_bar => true}, "filter", ["rpc"]).returns(oparser)
+
+ Application.new.clioptions(false)
+ end
+
+ it "should add the RPC options" do
+ oparser = mock
+ oparser.stubs(:parse).yields(oparser, {})
+ oparser.stubs(:banner=)
+ oparser.stubs(:define_tail)
+
+ Optionparser.stubs(:new).with({:verbose => false, :progress_bar => true}, "filter", []).returns(oparser)
+ RPC::Helpers.expects(:add_simplerpc_options).with(oparser, {})
+
+ Application.new.clioptions(false)
+ end
+
+ it "should support bypassing the RPC options" do
+ oparser = mock
+ oparser.stubs(:parse).yields(oparser, {})
+ oparser.stubs(:banner=)
+ oparser.stubs(:define_tail)
+
+ Application.exclude_argument_sections "rpc"
+
+ Optionparser.stubs(:new).with({:verbose => false, :progress_bar => true}, "filter", ["rpc"]).returns(oparser)
+ RPC::Helpers.expects(:add_simplerpc_options).never
+
+ Application.new.clioptions(false)
+ end
+
+ it "should return the help text if requested" do
+ parser = mock
+ parser.expects(:help)
+
+ oparser = mock
+ oparser.stubs(:parse).yields(oparser, {})
+ oparser.stubs(:banner=)
+ oparser.stubs(:define_tail)
+ oparser.expects(:parser).returns(parser)
+
+ Optionparser.stubs(:new).with({:verbose => false, :progress_bar => true}, "filter", []).returns(oparser)
+ RPC::Helpers.expects(:add_simplerpc_options).with(oparser, {})
+
+ Application.new.clioptions(true)
+ end
+ end
+
+ describe "#application_failure" do
+ before do
+ @app = Application.new
+ end
+
+ it "on SystemExit it should disconnect and exit without backtraces or error messages" do
+ @app.expects(:disconnect)
+ expect { @app.application_failure(SystemExit.new) }.to raise_error(SystemExit)
+ end
+
+ it "should print a single line error message" do
+ out = StringIO.new
+ @app.stubs(:disconnect)
+ @app.stubs(:exit).with(1)
+ @app.stubs(:options).returns({})
+
+ Config.instance.stubs(:color).returns(false)
+ e = mock
+ e.stubs(:backtrace).returns([])
+ e.stubs(:to_s).returns("rspec")
+
+ out.expects(:puts).with(regexp_matches(/rspec application failed to run/))
+
+ @app.application_failure(e, out)
+ end
+
+ it "should print a backtrace if options are unset or verbose is enabled" do
+ out = StringIO.new
+ @app.stubs(:disconnect)
+ @app.stubs(:exit).with(1)
+ @app.stubs(:options).returns(nil)
+
+ Config.instance.stubs(:color).returns(false)
+ e = mock
+ e.stubs(:backtrace).returns(["rspec"])
+ e.stubs(:to_s).returns("rspec")
+
+ @app.expects(:options).returns({:verbose => true}).times(3)
+ out.expects(:puts).with(regexp_matches(/ application failed to run/))
+ out.expects(:puts).with(regexp_matches(/from rspec <---/))
+ out.expects(:puts).with(regexp_matches(/rspec.+Mocha::Mock/))
+
+ @app.application_failure(e, out)
+ end
+ end
+
+ describe "#run" do
+ before do
+ @app = Application.new
+ end
+
+ it "should parse the application options, run main and disconnect" do
+ @app.expects(:application_parse_options)
+ @app.expects(:main)
+ @app.expects(:disconnect)
+
+ @app.run
+ end
+
+ it "should allow the application plugin to validate configuration variables" do
+ @app.expects("respond_to?").with(:validate_configuration).returns(true)
+ @app.expects(:validate_configuration).once
+
+ @app.stubs(:application_parse_options)
+ @app.stubs(:main)
+ @app.stubs(:disconnect)
+
+ @app.run
+ end
+
+ it "should start the sleeper thread on windows" do
+ Util.expects("windows?").returns(true)
+ Util.expects(:setup_windows_sleeper).once
+
+ @app.stubs(:application_parse_options)
+ @app.stubs(:main)
+ @app.stubs(:disconnect)
+
+ @app.run
+ end
+
+ it "should catch handle exit() correctly" do
+ @app.expects(:main).raises(SystemExit)
+ @app.expects(:disconnect).once
+
+ expect { @app.run }.to raise_error(SystemExit)
+ end
+
+ it "should catch all exceptions and process them correctly" do
+ @app.expects(:main).raises("rspec")
+ @app.expects(:application_failure).once
+ @app.run
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Applications do
+ before do
+ tmpfile = Tempfile.new("mc_applications_spec")
+ path = tmpfile.path
+ tmpfile.close!
+
+ @tmpdir = FileUtils.mkdir_p(path)
+ @tmpdir = @tmpdir[0] if @tmpdir.is_a?(Array) # ruby 1.9.2
+ $LOAD_PATH << @tmpdir
+
+ @appsdir = File.join([@tmpdir, "mcollective", "application"])
+ FileUtils.mkdir_p(@appsdir)
+
+ FileUtils.cp(File.join([File.dirname(__FILE__), "..", "fixtures", "application", "test.rb"]), @appsdir)
+ end
+
+ after do
+ FileUtils.rm_r(@tmpdir)
+ end
+
+ describe "[]" do
+ it "should load the config" do
+ Applications.expects(:load_config).once
+ PluginManager.expects("[]").once
+ Applications["test"]
+ end
+
+ it "should return the correct stored application" do
+ app = mock("app")
+ app.stubs(:run)
+
+ Applications.expects(:load_config).once
+ PluginManager.expects("[]").with("test_application").once.returns(app)
+ Applications["test"].should == app
+ end
+ end
+
+ describe "#run" do
+ it "should load the configuration" do
+ app = mock("app")
+ app.stubs(:run)
+
+ Applications.expects(:load_config).once
+ Applications.expects(:load_application).once
+ PluginManager.expects("[]").once.returns(app)
+
+ Applications.run("test")
+ end
+
+ it "should load the application" do
+ app = mock("app")
+ app.stubs(:run)
+
+ Applications.expects(:load_config).once
+ Applications.expects(:load_application).with("test").once
+ PluginManager.expects("[]").once.returns(app)
+
+ Applications.run("test")
+ end
+
+ it "should invoke the application run method" do
+ app = mock("app")
+ app.stubs(:run).returns("hello world")
+
+ Applications.expects(:load_config).once
+ Applications.expects(:load_application)
+ PluginManager.expects("[]").once.returns(app)
+
+ Applications.run("test").should == "hello world"
+ end
+ end
+
+ describe "#load_application" do
+ it "should return the existing application if already loaded" do
+ app = mock("app")
+ app.stubs(:run)
+
+ PluginManager << {:type => "test_application", :class => app}
+
+ Applications.expects("load_config").never
+
+ Applications.load_application("test")
+ end
+
+ it "should load the config" do
+ Applications.expects("load_config").returns(true).once
+ Applications.load_application("test")
+ end
+
+ it "should load the correct class from disk" do
+ PluginManager.expects("loadclass").with("MCollective::Application::Test")
+ Applications.expects("load_config").returns(true).once
+
+ Applications.load_application("test")
+ end
+
+ it "should add the class to the plugin manager" do
+ Applications.expects("load_config").returns(true).once
+
+ PluginManager.expects("<<").with({:type => "test_application", :class => "MCollective::Application::Test"})
+
+ Applications.load_application("test")
+ end
+ end
+
+ describe "#list" do
+ it "should load the configuration" do
+ Applications.expects("load_config").returns(true).once
+ Config.instance.expects("libdir").returns([@tmpdir])
+ Applications.list
+ end
+
+ it "should add found applications to the list" do
+ Applications.expects("load_config").returns(true).once
+ Config.instance.expects("libdir").returns([@tmpdir])
+
+ Applications.list.should == ["test"]
+ end
+
+ it "should print a friendly error and exit on failure" do
+ Applications.expects("load_config").raises(Exception)
+ IO.any_instance.expects(:puts).with(regexp_matches(/Failed to generate application list/)).once
+
+ expect {
+ Applications.list.should
+ }.to raise_error(SystemExit)
+ end
+ end
+
+ describe "#filter_extra_options" do
+ it "should parse --config=x" do
+ ["--config=x --foo=bar -f -f bar", "--foo=bar --config=x -f -f bar"].each do |t|
+ Applications.filter_extra_options(t).should == "--config=x"
+ end
+ end
+
+ it "should parse --config x" do
+ ["--config x --foo=bar -f -f bar", "--foo=bar --config x -f -f bar"].each do |t|
+ Applications.filter_extra_options(t).should == "--config=x"
+ end
+ end
+
+ it "should parse -c x" do
+ ["-c x --foo=bar -f -f bar", "--foo=bar -c x -f -f bar"].each do |t|
+ Applications.filter_extra_options(t).should == "--config=x"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+class Array
+ describe "#in_groups_of" do
+ it "should correctly group array members" do
+ [1,2,3,4,5,6,7,8,9,10].in_groups_of(5).should == [[1,2,3,4,5], [6,7,8,9,10]]
+ end
+
+ it "should padd missing data with correctly" do
+ arr = [1,2,3,4,5,6,7,8,9,10]
+
+ arr.in_groups_of(3).should == [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, nil, nil]]
+ arr.in_groups_of(3, 0).should == [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 0, 0]]
+ arr.in_groups_of(11).should == [[1,2,3,4,5, 6,7,8,9,10, nil]]
+ arr.in_groups_of(11, 0).should == [[1,2,3,4,5, 6,7,8,9,10, 0]]
+ end
+
+ it "should indicate when the last abtched was reached" do
+ arr = [1,2,3,4,5,6,7,8,9,10]
+
+ ctr = 0
+
+ [1,2,3,4,5,6,7,8,9,10].in_groups_of(3) {|a, last_batch| ctr += 1 unless last_batch}
+
+ ctr.should == 3
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Cache do
+ before do
+ @locks_mutex = Cache.instance_variable_set("@locks_mutex", Mutex.new)
+ @cache_locks = Cache.instance_variable_set("@cache_locks", {})
+ @cache = Cache.instance_variable_set("@cache", {})
+ end
+
+ describe "#check_cache!" do
+ it "should correctly check for valid caches" do
+ Cache.expects(:has_cache?).with("rspec").returns(true)
+ Cache.expects(:has_cache?).with("fail").returns(false)
+
+ Cache.check_cache!("rspec")
+ expect { Cache.check_cache!("fail") }.to raise_code("Could not find a cache called '%{cache_name}'", :cache_name => "fail")
+ end
+ end
+
+ describe "#setup" do
+ it "should use a mutex to manage access to the cache" do
+ @locks_mutex.expects(:synchronize).yields
+ Cache.setup("x").should == true
+ @cache.should == {"x" => {:max_age => 300.0}}
+ end
+
+ it "should correctly setup a new cache" do
+ @locks_mutex.expects(:synchronize).twice.yields
+ Cache.setup("rspec1", 300)
+ @cache["rspec1"].should == {:max_age => 300.0}
+
+ Cache.setup("rspec2")
+ @cache["rspec2"].should == {:max_age => 300.0}
+ end
+ end
+
+ describe "#has_cache?" do
+ it "should correctly report presense of a cache" do
+ Cache.setup("rspec")
+ Cache.has_cache?("rspec").should == true
+ Cache.has_cache?("fail").should == false
+ end
+ end
+
+ describe "#delete!" do
+ it "should delete the cache and return true" do
+ Cache.expects(:check_cache!).with("rspec")
+
+ Cache.setup("rspec")
+ Cache.delete!("rspec").should == true
+ end
+ end
+
+ describe "#write" do
+ it "should write to the cache" do
+ time = Time.now
+ Time.expects(:now).returns(time)
+ Cache.expects(:check_cache!).with("rspec")
+
+ Cache.setup("rspec")
+ Cache.write("rspec", :key, :val).should == :val
+
+ @cache["rspec"][:key][:value].should == :val
+ @cache["rspec"][:key][:cache_create_time].should == time
+ end
+ end
+
+ describe "#read" do
+ it "should read a written entry correctly" do
+ Cache.setup("rspec")
+ Cache.write("rspec", :key, :val)
+
+ Cache.expects(:check_cache!).with("rspec")
+ Cache.expects(:ttl).with("rspec", :key).returns(1)
+
+ Cache.read("rspec", :key).should == :val
+ end
+
+ it "should raise on expired reads" do
+ Cache.setup("rspec")
+ Cache.write("rspec", :key, :val)
+
+ Cache.expects(:ttl).returns(0)
+
+ Cache.expects(:check_cache!).with("rspec")
+
+ expect { Cache.read("rspec", :key) }.to raise_code("Cache expired on '%{cache_name}' key '%{item}'", :cache_name => "rspec", :item => :key)
+ end
+ end
+
+ describe "#invalidate!" do
+ it "should return false for unknown keys" do
+ Cache.expects(:check_cache!).with("rspec")
+
+ @locks_mutex.expects(:synchronize).yields
+
+ Cache.setup("rspec")
+ Cache.invalidate!("rspec", "no_such_key").should == false
+ end
+
+ it "should delete the key" do
+ Cache.setup("rspec")
+ Cache.write("rspec", "valid_key", "rspec")
+
+ Cache.expects(:check_cache!).with("rspec")
+ @cache["rspec"].expects(:delete).with("valid_key")
+
+ Cache.invalidate!("rspec", "valid_key")
+ end
+ end
+
+ describe "#ttl" do
+ it "should detect invalid key names" do
+ Cache.expects(:check_cache!).with("rspec")
+ Cache.setup("rspec", 300)
+ expect { Cache.ttl("rspec", :key) }.to raise_code("No item called '%{item}' for cache '%{cache_name}'", :cache_name => "rspec", :item => :key)
+ end
+
+ it "should return >0 for valid" do
+ Cache.setup("rspec", 300)
+ Cache.write("rspec", :key, :val)
+
+ Cache.expects(:check_cache!).with("rspec")
+ Cache.ttl("rspec", :key).should >= 0
+ end
+
+ it "should return <0 for expired messages" do
+ Cache.setup("rspec", 300)
+ Cache.write("rspec", :key, :val)
+
+ time = Time.now + 600
+ Time.expects(:now).returns(time)
+
+ Cache.expects(:check_cache!).with("rspec")
+ Cache.ttl("rspec", :key).should <= 0
+ end
+ end
+
+ describe "#synchronize" do
+ it "should use the correct mutex" do
+ Cache.expects(:check_cache!).with("rspec")
+ Cache.setup("rspec")
+
+ rspec_lock = @cache_locks["rspec"]
+ rspec_lock.expects(:synchronize).yields
+
+ @cache_locks.expects("[]").with("rspec").returns(rspec_lock)
+
+ ran = 0
+ Cache.synchronize("rspec") do
+ ran = 1
+ end
+
+ ran.should == 1
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Client do
+ before do
+ @security = mock
+ @security.stubs(:initiated_by=)
+ @connector = mock
+ @connector.stubs(:connect)
+ @connector.stubs(:subscribe)
+ @connector.stubs(:unsubscribe)
+ @ddl = mock
+ @ddl.stubs(:meta).returns({:timeout => 1})
+ @discoverer = mock
+
+ Discovery.expects(:new).returns(@discoverer)
+
+ Config.instance.instance_variable_set("@configured", true)
+ PluginManager.expects("[]").with("connector_plugin").returns(@connector)
+ PluginManager.expects("[]").with("security_plugin").returns(@security)
+
+ @client = Client.new("/nonexisting")
+ @client.options = Util.default_options
+ end
+
+ describe "#sendreq" do
+ it "should send the supplied message" do
+ message = Message.new("rspec", nil, {:agent => "rspec", :type => :request, :collective => "mcollective", :filter => Util.empty_filter, :options => Util.default_options})
+
+ message.expects(:encode!)
+ @client.expects(:subscribe).with("rspec", :reply)
+ message.expects(:publish)
+ message.expects(:requestid).twice
+ @client.sendreq(message, "rspec")
+ end
+
+ it "should not subscribe to a reply queue for a message with a reply-to" do
+ message = Message.new("rspec", nil, {:agent => "rspec", :type => :request, :collective => "mcollective", :filter => Util.empty_filter, :options => Util.default_options})
+ message.reply_to = "rspec"
+
+ message.expects(:encode!)
+ @client.expects(:subscribe).never
+ message.expects(:publish)
+ message.expects(:requestid).twice
+ @client.sendreq(message, "rspec")
+ end
+ end
+ describe "#req" do
+ it "should record the requestid" do
+ message = Message.new("rspec", nil, {:agent => "rspec", :type => :request, :collective => "mcollective", :filter => Util.empty_filter, :options => Util.default_options})
+ message.discovered_hosts = ["rspec"]
+
+ reply = mock
+ reply.stubs("payload").returns("rspec payload")
+
+ @client.expects(:sendreq).returns("823a3419a0975c3facbde121f72ab61f")
+ @client.expects(:receive).returns(reply)
+
+ @discoverer.expects(:discovery_timeout).with(message.options[:timeout], message.options[:filter]).returns(0)
+
+ Time.stubs(:now).returns(Time.at(1340621250), Time.at(1340621251))
+
+ @client.req(message){}.should == {:blocktime => 1.0, :discoverytime => 0, :noresponsefrom => [],
+ :requestid => "823a3419a0975c3facbde121f72ab61f", :responses => 1,
+ :starttime => 1340621250.0, :totaltime => 1.0}
+ end
+ end
+
+ describe "#discover" do
+ it "should delegate to the discovery plugins" do
+ @discoverer.expects(:discover).with({}, 1, 0).returns([])
+ @client.discover({}, 1).should == []
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Config do
+ describe "#loadconfig" do
+ it "should fail when no libdir is set" do
+ File.expects(:exists?).with("/nonexisting").returns(true)
+ File.expects(:readlines).with("/nonexisting").returns([])
+ Config.instance.stubs(:set_config_defaults)
+ expect { Config.instance.loadconfig("/nonexisting") }.to raise_error("The /nonexisting config file does not specify a libdir setting, cannot continue")
+ end
+
+ it "should only test that libdirs are absolute paths" do
+ Util.expects(:absolute_path?).with("/one").returns(true)
+ Util.expects(:absolute_path?).with("/two").returns(true)
+ Util.expects(:absolute_path?).with("/three").returns(true)
+ Util.expects(:absolute_path?).with("four").returns(false)
+
+ File.stubs(:exists?).with("/nonexisting").returns(true)
+ File.stubs(:exists?).with(File.join(File.dirname("/nonexisting"), "rpc-help.erb")).returns(true)
+
+ ["/one:/two", "/three"].each do |path|
+ File.expects(:readlines).with("/nonexisting").returns(["libdir = #{path}"])
+
+ Config.instance.loadconfig("/nonexisting")
+
+ PluginManager.clear
+ end
+
+ File.expects(:readlines).with("/nonexisting").returns(["libdir = four"])
+
+ expect { Config.instance.loadconfig("/nonexisting") }.to raise_error(/should be absolute paths/)
+ end
+
+ it "should not allow any path like construct for identities" do
+ # Taken from puppet test cases
+ ['../foo', '..\\foo', './../foo', '.\\..\\foo',
+ '/foo', '//foo', '\\foo', '\\\\goo',
+ "test\0/../bar", "test\0\\..\\bar",
+ "..\\/bar", "/tmp/bar", "/tmp\\bar", "tmp\\bar",
+ " / bar", " /../ bar", " \\..\\ bar",
+ "c:\\foo", "c:/foo", "\\\\?\\UNC\\bar", "\\\\foo\\bar",
+ "\\\\?\\c:\\foo", "//?/UNC/bar", "//foo/bar",
+ "//?/c:/foo"
+ ].each do |input|
+ File.expects(:readlines).with("/nonexisting").returns(["identity = #{input}", "libdir=/nonexistinglib"])
+ File.expects(:exists?).with("/nonexisting").returns(true)
+ File.expects(:exists?).with(File.join(File.dirname("/nonexisting"), "rpc-help.erb")).returns(true)
+
+ expect {
+ Config.instance.loadconfig("/nonexisting")
+ }.to raise_error('Identities can only match /\w\.\-/')
+ end
+ end
+
+ it "should allow valid identities" do
+ ["foo", "foo_bar", "foo-bar", "foo-bar-123", "foo.bar", "foo_bar_123"].each do |input|
+ File.expects(:readlines).with("/nonexisting").returns(["identity = #{input}", "libdir=/nonexistinglib"])
+ File.expects(:exists?).with("/nonexisting").returns(true)
+ File.expects(:exists?).with(File.join(File.dirname("/nonexisting"), "rpc-help.erb")).returns(true)
+ PluginManager.stubs(:loadclass)
+ PluginManager.stubs("<<")
+
+ Config.instance.loadconfig("/nonexisting")
+ end
+ end
+
+ it "should set direct_addressing to true by default" do
+ File.expects(:readlines).with("/nonexisting").returns(["libdir=/nonexistinglib"])
+ File.expects(:exists?).with("/nonexisting").returns(true)
+ File.expects(:exists?).with(File.join(File.dirname("/nonexisting"), "rpc-help.erb")).returns(true)
+ PluginManager.stubs(:loadclass)
+ PluginManager.stubs("<<")
+
+ Config.instance.loadconfig("/nonexisting")
+ Config.instance.direct_addressing.should == true
+ end
+
+ it "should allow direct_addressing to be disabled in the config file" do
+ File.expects(:readlines).with("/nonexisting").returns(["libdir=/nonexistinglib", "direct_addressing=n"])
+ File.expects(:exists?).with("/nonexisting").returns(true)
+ File.expects(:exists?).with(File.join(File.dirname("/nonexisting"), "rpc-help.erb")).returns(true)
+ PluginManager.stubs(:loadclass)
+ PluginManager.stubs("<<")
+
+ Config.instance.loadconfig("/nonexisting")
+ Config.instance.direct_addressing.should == false
+ end
+
+ it "should not allow the syslog logger type on windows" do
+ Util.expects("windows?").returns(true).twice
+ File.expects(:readlines).with("/nonexisting").returns(["libdir=/nonexistinglib", "logger_type=syslog"])
+ File.expects(:exists?).with("/nonexisting").returns(true)
+ File.expects(:exists?).with(File.join(File.dirname("/nonexisting"), "rpc-help.erb")).returns(true)
+ PluginManager.stubs(:loadclass)
+ PluginManager.stubs("<<")
+
+ expect { Config.instance.loadconfig("/nonexisting") }.to raise_error("The sylog logger is not usable on the Windows platform")
+ end
+
+ it "should default to finding the help template in the same dir as the config file" do
+ path = File.join(File.dirname("/nonexisting"), "rpc-help.erb")
+
+ File.expects(:readlines).with("/nonexisting").returns(["libdir=/nonexistinglib"])
+ File.expects(:exists?).with("/nonexisting").returns(true)
+ PluginManager.stubs(:loadclass)
+ PluginManager.stubs("<<")
+
+ File.expects(:exists?).with(path).returns(true)
+
+ Config.instance.loadconfig("/nonexisting")
+ Config.instance.rpchelptemplate.should == path
+ end
+
+ it "should fall back to old behavior if the help template file does not exist in the config dir" do
+ path = File.join(File.dirname("/nonexisting"), "rpc-help.erb")
+
+ File.expects(:readlines).with("/nonexisting").returns(["libdir=/nonexistinglib"])
+ File.expects(:exists?).with("/nonexisting").returns(true)
+ File.expects(:exists?).with(path).returns(false)
+ PluginManager.stubs(:loadclass)
+ PluginManager.stubs("<<")
+
+ Config.instance.loadconfig("/nonexisting")
+ Config.instance.rpchelptemplate.should == "/etc/mcollective/rpc-help.erb"
+ end
+
+ it "should support multiple default_discovery_options" do
+ File.expects(:readlines).with("/nonexisting").returns(["default_discovery_options = 1", "default_discovery_options = 2", "libdir=/nonexistinglib"])
+ File.expects(:exists?).with("/nonexisting").returns(true)
+ File.expects(:exists?).with(File.join(File.dirname("/nonexisting"), "rpc-help.erb")).returns(true)
+ PluginManager.stubs(:loadclass)
+ PluginManager.stubs("<<")
+
+ Config.instance.loadconfig("/nonexisting")
+ Config.instance.default_discovery_options.should == ["1", "2"]
+ end
+ end
+
+ describe "#read_plugin_config_dir" do
+ before do
+ @plugindir = File.join("/", "nonexisting", "plugin.d")
+
+ File.stubs(:directory?).with(@plugindir).returns(true)
+
+ Config.instance.set_config_defaults("")
+ end
+
+ it "should not fail if the supplied directory is missing" do
+ File.expects(:directory?).with(@plugindir).returns(false)
+ Config.instance.read_plugin_config_dir(@plugindir)
+ Config.instance.pluginconf.should == {}
+ end
+
+ it "should skip files that do not match the expected filename pattern" do
+ Dir.expects(:new).with(@plugindir).returns(["foo.txt"])
+
+ IO.expects(:open).with(File.join(@plugindir, "foo.txt")).never
+
+ Config.instance.read_plugin_config_dir(@plugindir)
+ end
+
+ it "should load the config files" do
+ Dir.expects(:new).with(@plugindir).returns(["foo.cfg"])
+ File.expects(:open).with(File.join(@plugindir, "foo.cfg"), "r").returns([]).once
+ Config.instance.read_plugin_config_dir(@plugindir)
+ end
+
+ it "should set config parameters correctly" do
+ Dir.expects(:new).with(@plugindir).returns(["foo.cfg"])
+ File.expects(:open).with(File.join(@plugindir, "foo.cfg"), "r").returns(["rspec = test"])
+ Config.instance.read_plugin_config_dir(@plugindir)
+ Config.instance.pluginconf.should == {"foo.rspec" => "test"}
+ end
+
+ it "should override main config file" do
+ configfile = File.join(@plugindir, "foo.cfg")
+ servercfg = File.join(File.dirname(@plugindir), "server.cfg")
+
+ PluginManager.stubs(:loadclass)
+
+ File.stubs(:exists?).returns(true)
+ File.stubs(:directory?).with(@plugindir).returns(true)
+ File.stubs(:exists?).with(servercfg).returns(true)
+ File.expects(:readlines).with(servercfg).returns(["plugin.rspec.key = default", "libdir=/nonexisting"])
+ File.stubs(:directory?).with("/nonexisting").returns(true)
+
+ Dir.expects(:new).with(@plugindir).returns(["rspec.cfg"])
+ File.expects(:open).with(File.join(@plugindir, "rspec.cfg"), "r").returns(["key = overridden"])
+
+ Config.instance.loadconfig(servercfg)
+ Config.instance.pluginconf.should == {"rspec.key" => "overridden"}
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module Data
+ describe Base do
+ before do
+ @ddl = mock
+ @ddl.stubs(:dataquery_interface).returns({:output => {'rspec' => {}}})
+ @ddl.stubs(:meta).returns({:timeout => 1})
+ end
+
+ describe "#initialize" do
+ it "should set the plugin name, ddl and timeout and call the startup hook" do
+ DDL.stubs(:new).returns(@ddl)
+ Base.any_instance.expects(:startup_hook).once
+ plugin = Base.new
+ plugin.name.should == "base"
+ plugin.timeout.should == 1
+ plugin.result.class.should == Result
+ end
+ end
+
+ describe "#lookup" do
+ before do
+ DDL.stubs(:new).returns(@ddl)
+ @plugin = Base.new
+ end
+
+ it "should validate the request" do
+ @plugin.expects(:ddl_validate).with("hello world").returns(true)
+ @plugin.stubs(:query_data)
+ @plugin.lookup("hello world")
+ end
+
+ it "should query the plugin" do
+ @plugin.stubs(:ddl_validate)
+ @plugin.expects(:query_data).with("hello world")
+ @plugin.lookup("hello world").class.should == Result
+ end
+
+ it "should raise MsgTTLExpired errors for Timeout errors" do
+ @plugin.stubs(:ddl_validate)
+ @plugin.expects(:query_data).raises(Timeout::Error)
+
+ msg = "Data plugin base timed out on query 'hello world'"
+ Log.expects(:error).with(msg)
+ expect { @plugin.lookup("hello world") }.to raise_error(msg)
+ end
+ end
+
+ describe "#query" do
+ it "should create a new method" do
+ class Rspec_data<Base; end
+ Rspec_data.query { "rspec test" }
+
+ DDL.stubs(:new).returns(@ddl)
+
+ data = Rspec_data.new
+ data.query_data.should == "rspec test"
+ end
+ end
+
+ describe "#ddl_validate" do
+ it "should validate the request using the Data class" do
+ DDL.stubs(:new).returns(@ddl)
+ plugin = Base.new
+ Data.expects(:ddl_validate).with(@ddl, "rspec")
+ plugin.ddl_validate("rspec")
+ end
+ end
+
+ describe "#activate_when" do
+ it "should create a new activate? method" do
+ class Rspec_data<Base;end
+
+ Rspec_data.activate_when { raise "rspec" }
+ DDL.stubs(:new).returns(@ddl)
+ expect { Rspec_data.activate? }.to raise_error("rspec")
+ end
+ end
+
+ describe "#activate?" do
+ it "should default to true" do
+ Base.activate?.should == true
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module Data
+ describe Result do
+ before(:each) do
+ @result = Result.new({})
+ end
+
+ describe "#initialize" do
+ it "should initialize empty values for all output fields" do
+ result = Result.new({:rspec1 => {}, :rspec2 => {}})
+ result[:rspec1].should == nil
+ result[:rspec2].should == nil
+ end
+
+ it "should set default values for all output fields" do
+ result = Result.new({:rspec1 => {:default => 1}, :rspec2 => {}})
+ result[:rspec1].should == 1
+ result[:rspec2].should == nil
+ end
+ end
+
+ describe "#[]=" do
+ it "should only allow trusted types of data to be saved" do
+ expect { @result["rspec"] = Time.now }.to raise_error
+ @result["rspec"] = 1
+ @result["rspec"] = 1.1
+ @result["rspec"] = "rspec"
+ @result["rspec"] = true
+ @result["rspec"] = false
+ end
+
+ it "should set the correct value" do
+ @result["rspec"] = "rspec value"
+ @result.instance_variable_get("@data").should == {:rspec => "rspec value"}
+ end
+
+ it "should only allow valid data types" do
+ expect { @result["rspec"] = Time.now }.to raise_error(/Can only store .+ data but got Time for key rspec/)
+ end
+ end
+
+ describe "#include" do
+ it "should return the correct list of keys" do
+ @result["x"] = "1"
+ @result[:y] = "2"
+ @result.keys.sort.should == [:x, :y]
+ end
+ end
+
+ describe "#include?" do
+ it "should correctly report that a key is present or absent" do
+ @result.include?("rspec").should == false
+ @result.include?(:rspec).should == false
+ @result["rspec"] = "rspec"
+ @result.include?("rspec").should == true
+ @result.include?(:rspec).should == true
+ end
+ end
+
+ describe "#[]" do
+ it "should retrieve the correct information" do
+ @result["rspec"].should == nil
+ @result[:rspec].should == nil
+ @result["rspec"] = "rspec value"
+ @result["rspec"].should == "rspec value"
+ @result[:rspec].should == "rspec value"
+ end
+ end
+
+ describe "#method_missing" do
+ it "should raise the correct exception for unknown keys" do
+ expect { @result.nosuchdata }.to raise_error(NoMethodError)
+ end
+
+ it "should retrieve the correct data" do
+ @result["rspec"] = "rspec value"
+ @result.rspec.should == "rspec value"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Data do
+ describe "#load_data_sources" do
+ it "should use the pluginmanager to load data sources" do
+ PluginManager.expects(:find_and_load).with("data").returns([])
+ Data.load_data_sources
+ end
+
+ it "should remove plugins that should not be active on this node" do
+ PluginManager.expects(:find_and_load).with("data").returns(["rspec_data"])
+ PluginManager.expects(:grep).returns(["rspec_data"])
+ PluginManager.expects(:delete).with("rspec_data")
+
+ ddl = mock
+ ddl.stubs(:meta).returns({:timeout => 1})
+ ddl.stubs(:dataquery_interface).returns({:rspec => nil})
+ ddl.stubs(:dataquery_interface).returns({:output => {}})
+ DDL.stubs(:new).returns(ddl)
+ Data::Base.expects(:activate?).returns(false)
+ PluginManager.expects("[]").with("rspec_data").returns(Data::Base.new)
+ Data.load_data_sources
+ end
+
+ it "should handle exceptions and delete broken plugins" do
+ PluginManager.expects(:find_and_load).with("data").returns(["rspec_data"])
+ PluginManager.expects(:grep).returns(["rspec_data"])
+ PluginManager.expects(:delete).with("rspec_data")
+
+ ddl = mock
+ ddl.stubs(:meta).returns({:timeout => 1})
+ ddl.stubs(:dataquery_interface).returns({:output => {}})
+ DDL.stubs(:new).returns(ddl)
+ Data::Base.expects(:activate?).raises("rspec failure")
+ Log.expects(:debug).once.with("Disabling data plugin rspec_data due to exception RuntimeError: rspec failure")
+ PluginManager.expects("[]").with("rspec_data").returns(Data::Base.new)
+ Data.load_data_sources
+ end
+ end
+
+ describe "#ddl" do
+ it "should load the right data ddl" do
+ DDL.expects(:new).with("fstat_data", :data).times(3)
+ Data.ddl("fstat")
+ Data.ddl("fstat_data")
+ Data.ddl(:fstat)
+ end
+ end
+
+ describe "#pluginname" do
+ it "should return the correct plugin name" do
+ Data.pluginname("Rspec").should == "rspec_data"
+ Data.pluginname("Rspec_data").should == "rspec_data"
+ end
+ end
+
+ describe "#[]" do
+ it "should return the correct plugin" do
+ PluginManager.expects("[]").with("rspec_data").times(4)
+ Data["Rspec"]
+ Data["rspec"]
+ Data["rspec_data"]
+ Data["rspec_Data"]
+ end
+ end
+
+ describe "#method_missing" do
+ it "should raise errors for unknown plugins" do
+ PluginManager.expects("include?").with("rspec_data").returns(false)
+ expect { Data.rspec_data }.to raise_error(NoMethodError)
+ end
+
+ it "should do a lookup on the right plugin" do
+ rspec_data = mock
+ rspec_data.expects(:lookup).returns("rspec")
+
+ PluginManager.expects("include?").with("rspec_data").returns(true)
+ PluginManager.expects("[]").with("rspec_data").returns(rspec_data)
+
+ Data.rspec_data("rspec").should == "rspec"
+ end
+ end
+
+ describe "#ddl_transform_intput" do
+ it "should convert boolean data" do
+ ddl = mock
+ ddl.stubs(:entities).returns({:data => {:input => {:query => {:type => :boolean}}}})
+ Data.ddl_transform_input(ddl, "1").should == true
+ Data.ddl_transform_input(ddl, "0").should == false
+ end
+
+ it "should conver numeric data" do
+ ddl = mock
+ ddl.stubs(:entities).returns({:data => {:input => {:query => {:type => :number}}}})
+ Data.ddl_transform_input(ddl, "1").should == 1
+ Data.ddl_transform_input(ddl, "0").should == 0
+ Data.ddl_transform_input(ddl, "1.1").should == 1.1
+ end
+
+ it "should return the original input on any failure" do
+ ddl = mock
+ ddl.expects(:entities).raises("rspec failure")
+ Data.ddl_transform_input(ddl, 1).should == 1
+ end
+ end
+
+ describe "#ddl_has_output?" do
+ it "should correctly verify output keys" do
+ ddl = mock
+ ddl.stubs(:entities).returns({:data => {:output => {:rspec => {}}}})
+ Data.ddl_has_output?(ddl, "rspec").should == true
+ Data.ddl_has_output?(ddl, :rspec).should == true
+ Data.ddl_has_output?(ddl, :foo).should == false
+ Data.ddl_has_output?(ddl, "foo").should == false
+ end
+
+ it "should return false for any exception" do
+ ddl = mock
+ ddl.stubs(:entities).returns(nil)
+ Data.ddl_has_output?(ddl, "rspec").should == false
+ end
+ end
+
+ describe "#ddl_validate" do
+ before do
+ @ddl = mock
+ @ddl.expects(:meta).returns({:name => "rspec test"})
+ end
+
+ it "should ensure the ddl has a dataquery" do
+ @ddl.expects(:entities).returns({})
+ expect { Data.ddl_validate(@ddl, "rspec") }.to raise_code(:PLMC31, :plugin => "rspec test")
+ end
+
+ it "should allow ddls without any input defined" do
+ @ddl.expects(:entities).returns({:data => {:input => {}, :output => {:x => {}}}})
+ Data.ddl_validate(@ddl, nil)
+ end
+
+ it "should not allow input arguments when no query were defined" do
+ @ddl.expects(:entities).returns({:data => {:input => {}, :output => {:x => {}}}})
+ expect { Data.ddl_validate(@ddl, "rspec") }.to raise_code(:PLMC33, :plugin => "rspec test")
+ end
+
+ it "should ensure the ddl has output" do
+ @ddl.expects(:entities).returns({:data => {:input => {:query => {}}, :output => {}}})
+ expect { Data.ddl_validate(@ddl, "rspec") }.to raise_code(:PLMC32, :plugin => "rspec test")
+ end
+
+ it "should skip optional arguments that were not supplied" do
+ @ddl.expects(:entities).returns({:data => {:input => {:query => {:optional => true}}, :output => {:test => {}}}})
+ @ddl.expects(:validate_input_argument).never
+ Data.ddl_validate(@ddl, nil).should == true
+ end
+
+ it "should validate the argument" do
+ @ddl.expects(:entities).returns({:data => {:input => {:query => {}}, :output => {:test => {}}}})
+ @ddl.expects(:validate_input_argument).returns("rspec validated")
+ Data.ddl_validate(@ddl, "rspec").should == "rspec validated"
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module DDL
+ describe AgentDDL do
+ before :each do
+ Cache.delete!(:ddl) rescue nil
+ @ddl = DDL.new("rspec", :agent, false)
+ @ddl.metadata(:name => "name", :description => "description", :author => "author", :license => "license", :version => "version", :url => "url", :timeout => "timeout")
+ end
+
+ describe "#input" do
+ it "should validate that an :optional property is set" do
+ expect { @ddl.input(:x, {:y => 1}) }.to raise_error("Input needs a :optional property")
+ end
+ end
+
+ describe "#set_default_input_arguments" do
+ before do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+
+ @ddl.input(:optional, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => true, :validation => "",
+ :maxlength => 1, :default => "default")
+ @ddl.input(:required, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => false, :validation => "",
+ :maxlength => 1, :default => "default")
+ end
+
+ it "should correctly add default arguments to required inputs" do
+ args = {}
+
+ @ddl.set_default_input_arguments(:test, args)
+
+ args.should == {:required => "default"}
+ end
+
+ it "should not override any existing arguments" do
+ args = {:required => "specified"}
+
+ @ddl.set_default_input_arguments(:test, args)
+
+ args.should == {:required => "specified"}
+ end
+ end
+
+ describe "#validate_rpc_request" do
+ it "should ensure the action is known" do
+ @ddl.action(:test, :description => "rspec")
+
+ expect {
+ @ddl.validate_rpc_request(:fail, {})
+ }.to raise_code(:PLMC29, :action => :fail, :plugin => "rspec")
+
+ @ddl.validate_rpc_request(:test, {})
+ end
+
+ it "should check all required arguments are present" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+ @ddl.input(:optional, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => true, :validation => "",
+ :maxlength => 1)
+ @ddl.input(:required, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => false, :validation => "",
+ :maxlength => 1)
+
+ @ddl.stubs(:validate_input_argument).returns(true)
+
+ expect {
+ @ddl.validate_rpc_request(:test, {})
+ }.to raise_code(:PLMC30, :action => :test, :key => :required)
+
+ @ddl.validate_rpc_request(:test, {:required => "f"}).should == true
+ end
+
+ it "should input validate every supplied key" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+ @ddl.input(:optional, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => true, :validation => "",
+ :maxlength => 1)
+ @ddl.input(:required, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => false, :validation => "",
+ :maxlength => 1)
+
+
+ @ddl.expects(:validate_input_argument).with(@ddl.entities[:test][:input], :required, "f")
+ @ddl.expects(:validate_input_argument).with(@ddl.entities[:test][:input], :optional, "f")
+
+ @ddl.validate_rpc_request(:test, {:required => "f", :optional => "f"}).should == true
+ end
+ end
+
+ describe "#is_function" do
+ before :each do
+ PluginManager.expects(:find).with("aggregate").returns(["plugin"])
+ end
+
+ it "should return true if the aggregate function is present" do
+ @ddl.is_function?("plugin").should == true
+ end
+
+ it "should return false if the aggregate function is not present" do
+ @ddl.is_function?("no_plugin").should == false
+ end
+ end
+
+ describe "#method_missing" do
+ it "should call super if the aggregate plugin isn't present" do
+ expect{
+ @ddl.test
+ }.to raise_error(NoMethodError)
+ end
+
+ it "should call super if @process_aggregate_function is false" do
+ expect{
+ result = @ddl.test(:value)
+ }.to raise_error(NoMethodError)
+ end
+
+ it "should return the function hash" do
+ Config.instance.mode = :client
+ @ddl.instance_variable_set(:@process_aggregate_functions, true)
+ result = @ddl.method_missing(:test_function, :rspec)
+ result.should == {:args => [:rspec], :function => :test_function }
+ end
+ end
+
+ describe "#actions" do
+ it "should return the correct list of actions" do
+ @ddl.action(:test1, :description => "rspec")
+ @ddl.action(:test2, :description => "rspec")
+
+ @ddl.actions.sort.should == [:test1, :test2]
+ end
+ end
+
+ describe "#action_interface" do
+ it "should return the correct interface" do
+ @ddl.action(:test1, :description => "rspec")
+ @ddl.action_interface(:test1).should == {:description=>"rspec", :output=>{}, :input=>{}, :action=>:test1, :display=>:failed}
+ end
+ end
+
+ describe "#display" do
+ it "should ensure a valid display property is set" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+
+ [:ok, :failed, :flatten, :always].each do |display|
+ @ddl.display(display)
+
+ action = @ddl.action_interface(:test)
+ action[:display].should == display
+ end
+
+ expect {
+ @ddl.display(:foo)
+ }.to raise_error(/Display preference foo is not valid/)
+ end
+ end
+
+ describe "#summarize" do
+ before :each do
+ @block_result = nil
+ @block = Proc.new{@block_result = :success}
+ end
+
+ after :each do
+ @block_result = nil
+ end
+
+ it "should call the block parameter if config mode is not server" do
+ Config.instance.mode = :client
+ result = @ddl.summarize(&@block)
+ @block_result.should == :success
+ end
+
+ it "should not call the block parameter if config mode is server" do
+ Config.instance.mode = :server
+ result = @ddl.summarize(&@block)
+ @block_result.should == nil
+ end
+ end
+
+ describe "#aggregate" do
+ it "should raise an exception if aggregate format isn't a hash" do
+ expect{
+ @ddl.aggregate(:foo, :format)
+ }.to raise_code(:PLMC28)
+ end
+
+ it "should raise an exception if format hash does not include a :format key" do
+ expect{
+ @ddl.aggregate(:foo, {})
+ }.to raise_code(:PLMC27)
+ end
+
+ it "should raise an exception if aggregate function is not a hash" do
+ expect{
+ @ddl.aggregate(:foo)
+ }.to raise_code(:PLMC26)
+ end
+
+ it "should raise an exception if function hash does not include a :args key" do
+ expect{
+ @ddl.stubs(:entities).returns({nil => {:action => :foo}})
+ @ddl.aggregate({})
+ }.to raise_code(:PLMC25, :action => :foo)
+ end
+
+ it "should correctly add an aggregate function to the function array" do
+ @ddl.stubs(:entities).returns({nil => {:aggregate => nil}})
+ @ddl.aggregate({:function => :foo, :args => [:bar]})
+ @ddl.entities.should == {nil => {:aggregate => [{:function => :foo, :args => [:bar]}]}}
+ end
+ end
+
+ describe "#action" do
+ it "should ensure a description is set" do
+ expect {
+ @ddl.action("act", {})
+ }.to raise_error("Action needs a :description property")
+ end
+
+ it "should create a default action structure" do
+ @ddl.action("act", :description => "rspec")
+
+ action = @ddl.action_interface("act")
+
+ action.class.should == Hash
+ action[:action].should == "act"
+ action[:input].should == {}
+ action[:output].should == {}
+ action[:display].should == :failed
+ action[:description].should == "rspec"
+ end
+
+ it "should call a block if one is given and set the correct action name" do
+ @ddl.action("act", :description => "rspec") { @ddl.instance_variable_get("@current_entity").should == "act" }
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module DDL
+ describe Base do
+ before :each do
+ Cache.delete!(:ddl) rescue nil
+ @ddl = DDL.new("rspec", :agent, false)
+ @ddl.metadata(:name => "name", :description => "description", :author => "author", :license => "license", :version => "version", :url => "url", :timeout => "timeout")
+ end
+
+ describe "#template_for_plugintype" do
+ it "should return the backward compat path for agent ddls" do
+ @ddl.template_for_plugintype.should == "rpc-help.erb"
+ end
+
+ it "should return correct new path for other ddls" do
+ @ddl.instance_variable_set("@plugintype", :data)
+ File.expects(:exists?).with("/etc/mcollective/data-help.erb").returns(true)
+ @ddl.template_for_plugintype.should == "data-help.erb"
+ end
+ end
+
+ describe "#help" do
+ it "should use conventional template paths when none is provided" do
+ File.expects(:read).with("/etc/mcollective/rpc-help.erb").returns("rspec")
+ File.expects(:read).with("/etc/mcollective/metadata-help.erb").returns("rspec")
+ @ddl.help.should == "rspec"
+ end
+
+ it "should use template from help template path when provided template name is not an absolute file path" do
+ File.expects(:read).with("/etc/mcollective/foo").returns("rspec")
+ File.expects(:read).with("/etc/mcollective/metadata-help.erb").returns("rspec")
+ @ddl.help("foo").should == "rspec"
+ end
+
+ it "should use supplied template path when one is provided" do
+ File.expects(:read).with("/foo").returns("rspec")
+ File.expects(:read).with("/etc/mcollective/metadata-help.erb").returns("rspec")
+ @ddl.help("/foo").should == "rspec"
+ end
+
+ it "should correctly execute the template with a valid binding" do
+ @ddl.instance_variable_set("@meta", "meta")
+ @ddl.instance_variable_set("@entities", "actions")
+ File.expects(:read).with("/template").returns("<%= meta %>:<%= entities %>")
+ File.expects(:read).with("/etc/mcollective/metadata-help.erb").returns("rspec")
+ @ddl.help("/template").should == "meta:actions"
+ end
+ end
+
+ describe "#validate_input_arguments" do
+ before :all do
+ Config.instance.stubs(:configured).returns(true)
+ Config.instance.stubs(:libdir).returns([File.join(File.dirname(__FILE__), "../../../plugins")])
+ end
+
+ it "should ensure strings are String" do
+ @ddl.action(:string, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :string)
+ @ddl.input(:string, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => true, :validation => "",
+ :maxlength => 1)
+
+ expect {
+ @ddl.validate_input_argument(@ddl.entities[:string][:input], :string, 1)
+ }.to raise_code(:PLMC21, :input => :string, :error => "value should be a string")
+
+ @ddl.validate_input_argument(@ddl.entities[:string][:input], :string, "1")
+ end
+
+ it "should ensure strings are not longer than maxlength" do
+ @ddl.action(:string, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :string)
+ @ddl.input(:string, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => true, :validation => "",
+ :maxlength => 1)
+
+ expect {
+ @ddl.validate_input_argument(@ddl.entities[:string][:input], :string, "too long")
+ }.to raise_code(:PLMC21, :input => :string, :error => "Input string is longer than 1 character(s)")
+
+ @ddl.validate_input_argument(@ddl.entities[:string][:input], :string, "1")
+ end
+
+ it "should validate strings using regular expressions" do
+ @ddl.action(:string, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :string)
+ @ddl.input(:string, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => true, :validation => "^regex$",
+ :maxlength => 100)
+
+ expect {
+ @ddl.validate_input_argument(@ddl.entities[:string][:input], :string, "doesnt validate")
+ }.to raise_code(:PLMC21, :input => :string, :error => "value should match ^regex$")
+
+ @ddl.validate_input_argument(@ddl.entities[:string][:input], :string, "regex")
+ end
+
+ it "should validate list arguments correctly" do
+ @ddl.action(:list, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :list)
+ @ddl.input(:list, :prompt => "prompt", :description => "descr",
+ :type => :list, :optional => true, :list => [1,2])
+
+ expect {
+ @ddl.validate_input_argument(@ddl.entities[:list][:input], :list, 3)
+ }.to raise_code(:PLMC21, :input => :list, :error => "value should be one of 1, 2")
+
+ @ddl.validate_input_argument(@ddl.entities[:list][:input], :list, 1)
+ end
+
+ it "should validate boolean arguments correctly" do
+ @ddl.action(:bool, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :bool)
+ @ddl.input(:bool, :prompt => "prompt", :description => "descr",
+ :type => :boolean, :optional => true)
+
+ expect {
+ @ddl.validate_input_argument(@ddl.entities[:bool][:input], :bool, 3)
+ }.to raise_code(:PLMC21, :input => :bool, :error => "value should be a boolean")
+
+ @ddl.validate_input_argument(@ddl.entities[:bool][:input], :bool, true)
+ @ddl.validate_input_argument(@ddl.entities[:bool][:input], :bool, false)
+ end
+
+ it "should validate integer arguments correctly" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+ @ddl.input(:int, :prompt => "prompt", :description => "descr",
+ :type => :integer, :optional => true)
+
+ expect {
+ @ddl.validate_input_argument(@ddl.entities[:test][:input], :int, "1")
+ }.to raise_code(:PLMC21, :input => :int, :error => "value should be a integer")
+
+ expect {
+ @ddl.validate_input_argument(@ddl.entities[:test][:input], :int, 1.1)
+ }.to raise_code(:PLMC21, :input => :int, :error => "value should be a integer")
+
+ @ddl.validate_input_argument(@ddl.entities[:test][:input], :int, 1)
+ end
+
+ it "should validate float arguments correctly" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+ @ddl.input(:float, :prompt => "prompt", :description => "descr",
+ :type => :float, :optional => true)
+
+ expect {
+ @ddl.validate_input_argument(@ddl.entities[:test][:input], :float, "1")
+ }.to raise_code(:PLMC21, :input => :float, :error => "value should be a float")
+
+ expect {
+ @ddl.validate_input_argument(@ddl.entities[:test][:input], :float, 1)
+ }.to raise_code(:PLMC21, :input => :float, :error => "value should be a float")
+
+ @ddl.validate_input_argument(@ddl.entities[:test][:input], :float, 1.1)
+ end
+
+ it "should validate number arguments correctly" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+ @ddl.input(:number, :prompt => "prompt", :description => "descr",
+ :type => :number, :optional => true)
+
+ expect {
+ @ddl.validate_input_argument(@ddl.entities[:test][:input], :number, "1")
+ }.to raise_code(:PLMC21, :input => :number, :error => "value should be a number")
+
+ @ddl.validate_input_argument(@ddl.entities[:test][:input], :number, 1)
+ @ddl.validate_input_argument(@ddl.entities[:test][:input], :number, 1.1)
+ end
+ end
+
+ describe "#requires" do
+ it "should only accept hashes as arguments" do
+ expect { @ddl.requires(1) }.to raise_error(/should be a hash/)
+ end
+
+ it "should only accept valid requirement types" do
+ expect { @ddl.requires(:rspec => "1") }.to raise_error(/is not a valid requirement/)
+ @ddl.requires(:mcollective => "1.0.0")
+ end
+
+ it "should save the requirement" do
+ @ddl.requires(:mcollective => "1.0.0")
+
+ @ddl.requirements.should == {:mcollective => "1.0.0"}
+ end
+ end
+
+ describe "#validate_requirements" do
+ it "should fail for older versions of mcollective" do
+ Util.stubs(:mcollective_version).returns("0.1")
+ expect { @ddl.requires(:mcollective => "2.0") }.to raise_error(/requires.+version 2.0/)
+ end
+
+ it "should pass for newer versions of mcollective" do
+ Util.stubs(:mcollective_version).returns("2.0")
+ @ddl.requires(:mcollective => "0.1")
+ @ddl.validate_requirements.should == true
+ end
+
+ it "should bypass checks in development" do
+ Util.stubs(:mcollective_version).returns("@DEVELOPMENT_VERSION@")
+ @ddl.expects(:log_code).with(:PLMC19, anything, :warn)
+ @ddl.requires(:mcollective => "0.1")
+ end
+ end
+
+ describe "#loaddlfile" do
+ it "should raise the correct error when a ddl isnt present" do
+ @ddl.expects(:findddlfile).returns(false)
+ expect { @ddl.loadddlfile }.to raise_error("Can't find DDL for agent plugin 'rspec'")
+ end
+ end
+
+ describe "#findddlfile" do
+ it "should construct the correct ddl file name" do
+ Config.instance.expects(:libdir).returns(["/nonexisting"])
+ File.expects("exist?").with("/nonexisting/mcollective/agent/foo.ddl").returns(false)
+
+ @ddl.findddlfile("foo").should == false
+ end
+
+ it "should check each libdir for a ddl file" do
+ Config.instance.expects(:libdir).returns(["/nonexisting1", "/nonexisting2"])
+ File.expects("exist?").with("/nonexisting1/mcollective/agent/foo.ddl").returns(false)
+ File.expects("exist?").with("/nonexisting2/mcollective/agent/foo.ddl").returns(false)
+
+ @ddl.findddlfile("foo").should == false
+ end
+
+ it "should return the ddl file path if found" do
+ Config.instance.expects(:libdir).returns(["/nonexisting"])
+ File.expects("exist?").with("/nonexisting/mcollective/agent/foo.ddl").returns(true)
+ @ddl.expects(:log_code).with(:PLMC18, anything, :debug, :ddlname => "foo", :ddlfile => "/nonexisting/mcollective/agent/foo.ddl")
+
+ @ddl.findddlfile("foo").should == "/nonexisting/mcollective/agent/foo.ddl"
+ end
+
+ it "should default to the current plugin and type" do
+ Config.instance.expects(:libdir).returns(["/nonexisting"])
+ File.expects("exist?").with("/nonexisting/mcollective/agent/rspec.ddl").returns(true)
+
+ @ddl.findddlfile.should == "/nonexisting/mcollective/agent/rspec.ddl"
+ end
+ end
+
+ describe "#metadata" do
+ it "should ensure minimum parameters are given" do
+ [:name, :description, :author, :license, :version, :url, :timeout].each do |tst|
+ metadata = {:name => "name", :description => "description", :author => "author",
+ :license => "license", :version => "version", :url => "url", :timeout => "timeout"}
+
+ metadata.delete(tst)
+
+ expect {
+ @ddl.metadata(metadata)
+ }.to raise_error("Metadata needs a :#{tst} property")
+ end
+ end
+
+ it "should should allow arbitrary metadata" do
+ metadata = {:name => "name", :description => "description", :author => "author", :license => "license",
+ :version => "version", :url => "url", :timeout => "timeout", :foo => "bar"}
+
+ @ddl.metadata(metadata)
+ @ddl.meta.should == metadata
+ end
+ end
+
+ describe "#input" do
+ it "should ensure required properties are set" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+
+ [:prompt, :description, :type, :optional].each do |arg|
+ args = {:prompt => "prompt", :description => "descr", :type => "type", :optional => true}
+ args.delete(arg)
+
+ expect {
+ @ddl.input(:test, args)
+ }.to raise_error("Input needs a :#{arg} property")
+ end
+
+ @ddl.input(:test, {:prompt => "prompt", :description => "descr", :type => "type", :optional => true})
+ end
+
+ it "should ensure strings have a validation and maxlength" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+
+ expect {
+ @ddl.input(:test, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => true)
+ }.to raise_error("Input type :string needs a :validation argument")
+
+ expect {
+ @ddl.input(:test, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => true, :validation => 1)
+ }.to raise_error("Input type :string needs a :maxlength argument")
+
+ @ddl.input(:test, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => true, :validation => 1, :maxlength => 1)
+ end
+
+ it "should ensure lists have a list argument" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+
+ expect {
+ @ddl.input(:test, :prompt => "prompt", :description => "descr",
+ :type => :list, :optional => true)
+ }.to raise_error("Input type :list needs a :list argument")
+
+ @ddl.input(:test, :prompt => "prompt", :description => "descr",
+ :type => :list, :optional => true, :list => [])
+ end
+
+ it "should save correct data for a list input" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+ @ddl.input(:test, :prompt => "prompt", :description => "descr",
+ :type => :list, :optional => true, :list => [])
+
+ action = @ddl.action_interface(:test)
+
+ action[:input][:test][:prompt].should == "prompt"
+ action[:input][:test][:description].should == "descr"
+ action[:input][:test][:type].should == :list
+ action[:input][:test][:optional].should == true
+ action[:input][:test][:list].should == []
+ end
+
+ it "should save correct data for a string input" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+ @ddl.input(:test, :prompt => "prompt", :description => "descr",
+ :type => :string, :optional => true, :validation => "",
+ :maxlength => 1)
+
+ action = @ddl.action_interface(:test)
+
+ action[:input][:test][:prompt].should == "prompt"
+ action[:input][:test][:description].should == "descr"
+ action[:input][:test][:type].should == :string
+ action[:input][:test][:optional].should == true
+ action[:input][:test][:validation].should == ""
+ action[:input][:test][:maxlength].should == 1
+ end
+ end
+
+ describe "#output" do
+ it "should ensure a :description is set" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+
+ expect {
+ @ddl.output(:test, {})
+ }.to raise_error("Output test needs a description argument")
+ end
+
+ it "should ensure a :display_as is set" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+
+ expect {
+ @ddl.output(:test, {:description => "rspec"})
+ }.to raise_error("Output test needs a display_as argument")
+ end
+
+ it "should save correct data for an output" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+
+ @ddl.output(:test, {:description => "rspec", :display_as => "RSpec", :default => "default"})
+
+ action = @ddl.action_interface(:test)
+
+ action[:output][:test][:description].should == "rspec"
+ action[:output][:test][:display_as].should == "RSpec"
+ action[:output][:test][:default].should == "default"
+ end
+
+ it "should set unsupplied defaults to our internal unset representation" do
+ @ddl.action(:test, :description => "rspec")
+ @ddl.instance_variable_set("@current_entity", :test)
+
+ @ddl.output(:test, {:description => "rspec", :display_as => "RSpec"})
+
+ action = @ddl.action_interface(:test)
+
+ action[:output][:test][:default].should == nil
+ end
+ end
+
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module DDL
+ describe DataDDL do
+ before :each do
+ Cache.delete!(:ddl) rescue nil
+ @ddl = DDL.new("rspec", :data, false)
+ @ddl.metadata(:name => "name", :description => "description", :author => "author", :license => "license", :version => "version", :url => "url", :timeout => "timeout")
+ end
+
+ describe "#input" do
+ it "should only allow 'query' as input for data plugins" do
+ ddl = DDL.new("rspec", :data, false)
+ ddl.dataquery(:description => "rspec")
+ ddl.instance_variable_set("@current_entity", :query)
+ ddl.instance_variable_set("@plugintype", :data)
+
+ expect { ddl.input(:rspec, {}) }.to raise_error("The only valid input name for a data query is 'query'")
+ end
+ end
+
+ describe "#dataquery_interface" do
+ it "should return the correct data" do
+ input = {:prompt => "Matcher", :description => "Augeas Matcher", :type => :string, :validation => /.+/, :maxlength => 0}
+ output = {:description=>"rspec", :display_as=>"rspec", :default => nil}
+
+ @ddl.instance_variable_set("@plugintype", :data)
+ @ddl.dataquery(:description => "rspec") do
+ @ddl.output :rspec, output
+ @ddl.input :query, input
+ end
+
+ @ddl.dataquery_interface.should == {:description => "rspec",
+ :input => {:query => input.merge(:optional => nil, :default => nil)},
+ :output => {:rspec => output}}
+ end
+ end
+
+ describe "#dataquery" do
+ it "should ensure a description is set" do
+ expect { @ddl.dataquery({}) }.to raise_error("Data queries need a :description")
+ end
+
+ it "should ensure only one definition" do
+ @ddl.dataquery(:description => "rspec")
+
+ expect { @ddl.dataquery(:description => "rspec") }.to raise_error("Data queries can only have one definition")
+ end
+
+ it "should create the default structure" do
+ @ddl.dataquery(:description => "rspec")
+ @ddl.instance_variable_set("@plugintype", :data)
+ @ddl.dataquery_interface.should == {:description => "rspec", :input => {}, :output => {}}
+ end
+
+ it "should call the block if given" do
+ @ddl.dataquery(:description => "rspec") { @ddl.instance_variable_get("@current_entity").should == :data }
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module DDL
+ describe DiscoveryDDL do
+ before :each do
+ Cache.delete!(:ddl) rescue nil
+ @ddl = DDL.new("rspec", :discovery, false)
+ @ddl.metadata(:name => "name", :description => "description", :author => "author", :license => "license", :version => "version", :url => "url", :timeout => "timeout")
+ end
+
+ describe "#discovery_interface" do
+ it "should return correct data" do
+ @ddl.instance_variable_set("@plugintype", :discovery)
+ @ddl.discovery do
+ @ddl.capabilities :identity
+ end
+
+ @ddl.discovery_interface.should == {:capabilities => [:identity]}
+ end
+ end
+
+ describe "#capabilities" do
+ it "should support non arrays" do
+ @ddl.instance_variable_set("@plugintype", :discovery)
+ @ddl.discovery do
+ @ddl.capabilities :identity
+ end
+ @ddl.discovery_interface.should == {:capabilities => [:identity]}
+ end
+
+ it "should not accept empty capability lists" do
+ @ddl.instance_variable_set("@plugintype", :discovery)
+ @ddl.discovery do
+ expect { @ddl.capabilities [] }.to raise_error("Discovery plugin capabilities can't be empty")
+ end
+ end
+
+ it "should only accept known capabilities" do
+ @ddl.instance_variable_set("@plugintype", :discovery)
+ @ddl.discovery do
+ expect { @ddl.capabilities :rspec }.to raise_error(/rspec is not a valid capability/)
+ end
+ end
+
+ it "should correctly store the capabilities" do
+ @ddl.instance_variable_set("@plugintype", :discovery)
+ @ddl.discovery do
+ @ddl.capabilities [:identity, :classes]
+ end
+ @ddl.discovery_interface.should == {:capabilities => [:identity, :classes]}
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe DDL do
+ before do
+ Cache.delete!(:ddl) rescue nil
+ end
+
+ describe "#new" do
+ it "should default to agent ddls" do
+ DDL::AgentDDL.expects(:new).once
+ DDL.new("rspec")
+ end
+
+ it "should return the correct plugin ddl class" do
+ DDL.new("rspec", :agent, false).class.should == DDL::AgentDDL
+ end
+
+ it "should default to base when no specific class exist" do
+ DDL.new("rspec", :rspec, false).class.should == DDL::Base
+ end
+ end
+
+ describe "#load_and_cache" do
+ it "should setup the cache" do
+ Cache.setup(:ddl)
+
+ Cache.expects(:setup).once.returns(true)
+ DDL.load_and_cache("rspec", :agent, false)
+ end
+
+ it "should attempt to read from the cache and return found ddl" do
+ Cache.expects(:setup)
+ Cache.expects(:read).with(:ddl, "agent/rspec").returns("rspec")
+ DDL.load_and_cache("rspec", :agent, false).should == "rspec"
+ end
+
+ it "should handle cache misses then create and save a new ddl object" do
+ Cache.expects(:setup)
+ Cache.expects(:read).with(:ddl, "agent/rspec").raises("failed")
+ Cache.expects(:write).with(:ddl, "agent/rspec", kind_of(DDL::AgentDDL)).returns("rspec")
+
+ DDL.load_and_cache("rspec", :agent, false).should == "rspec"
+ end
+ end
+
+ describe "#string_to_number" do
+ it "should turn valid strings into numbers" do
+ ["1", "0", "9999"].each do |i|
+ DDL.string_to_number(i).class.should == Fixnum
+ end
+
+ ["1.1", "0.0", "9999.99"].each do |i|
+ DDL.string_to_number(i).class.should == Float
+ end
+ end
+
+ it "should raise errors for invalid values" do
+ expect { DDL.string_to_number("rspec") }.to raise_error
+ end
+ end
+
+ describe "#string_to_boolean" do
+ it "should turn valid strings into boolean" do
+ ["true", "yes", "1"].each do |t|
+ DDL.string_to_boolean(t).should == true
+ DDL.string_to_boolean(t.upcase).should == true
+ end
+
+ ["false", "no", "0"].each do |f|
+ DDL.string_to_boolean(f).should == false
+ DDL.string_to_boolean(f.upcase).should == false
+ end
+ end
+
+ it "should raise errors for invalid values" do
+ expect { DDL.string_to_boolean("rspec") }.to raise_error
+ end
+ end
+
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Discovery do
+ before do
+ Config.instance.stubs(:default_discovery_method).returns("mc")
+ @client = mock
+
+ Discovery.any_instance.stubs(:find_known_methods).returns(["mc"])
+ @discovery = Discovery.new(@client)
+ end
+
+ describe "#timeout_for_compound_filter" do
+ it "should return the correct time" do
+ ddl = mock
+ ddl.stubs(:meta).returns({:timeout => 1})
+
+ filter = [Matcher.create_compound_callstack("test().size=1 and rspec().size=1")]
+
+ DDL.expects(:new).with("test_data", :data).returns(ddl)
+ DDL.expects(:new).with("rspec_data", :data).returns(ddl)
+
+ @discovery.timeout_for_compound_filter(filter).should == 2
+ end
+ end
+
+ describe "#discover" do
+ before do
+ ddl = mock
+ ddl.stubs(:meta).returns({:timeout => 2})
+
+ discoverer = mock
+
+ @discovery.stubs(:force_discovery_method_by_filter).returns(false)
+ @discovery.stubs(:ddl).returns(ddl)
+ @discovery.stubs(:check_capabilities)
+ @discovery.stubs(:discovery_class).returns(discoverer)
+ end
+
+ it "should error for non fixnum limits" do
+ expect { @discovery.discover(nil, 0, 1.1) }.to raise_error("Limit has to be an integer")
+ end
+
+ it "should use the DDL timeout if none is specified" do
+ filter = Util.empty_filter
+ @discovery.discovery_class.expects(:discover).with(filter, 2, 0, @client)
+ @discovery.discover(filter, nil, 0)
+ end
+
+ it "should check the discovery method is capable of serving the filter" do
+ @discovery.expects(:check_capabilities).with("filter").raises("capabilities check failed")
+ expect { @discovery.discover("filter", nil, 0) }.to raise_error("capabilities check failed")
+ end
+
+ it "should call the correct discovery plugin" do
+ @discovery.discovery_class.expects(:discover).with("filter", 2, 0, @client)
+ @discovery.discover("filter", nil, 0)
+ end
+
+ it "should handle limits correctly" do
+ @discovery.discovery_class.stubs(:discover).returns([1,2,3,4,5])
+ @discovery.discover(Util.empty_filter, 1, 1).should == [1]
+ @discovery.discover(Util.empty_filter, 1, 0).should == [1,2,3,4,5]
+ end
+ end
+
+ describe "#force_discovery_method_by_filter" do
+ it "should force mc plugin when needed" do
+ options = {:discovery_method => "rspec"}
+
+ Log.expects(:info).with("Switching to mc discovery method because compound filters are used")
+
+ @discovery.expects(:discovery_method).returns("rspec")
+ @client.expects(:options).returns(options)
+ @discovery.force_discovery_method_by_filter({"compound" => ["rspec"]}).should == true
+
+ options[:discovery_method].should == "mc"
+ end
+
+ it "should not force mc plugin when no compound filter is used" do
+ options = {:discovery_method => "rspec"}
+
+ @discovery.expects(:discovery_method).returns("rspec")
+ @discovery.force_discovery_method_by_filter({"compound" => []}).should == false
+
+ options[:discovery_method].should == "rspec"
+ end
+ end
+
+ describe "#check_capabilities" do
+ before do
+ @ddl = mock
+ @discovery.stubs(:ddl).returns(@ddl)
+ @discovery.stubs(:discovery_method).returns("rspec")
+ end
+
+ it "should fail for unsupported capabilities" do
+ @ddl.stubs(:discovery_interface).returns({:capabilities => []})
+
+ filter = Util.empty_filter
+
+ expect { @discovery.check_capabilities(filter.merge({"cf_class" => ["filter"]})) }.to raise_error(/Cannot use class filters/)
+
+ ["fact", "identity", "compound"].each do |type|
+ expect { @discovery.check_capabilities(filter.merge({type => ["filter"]})) }.to raise_error(/Cannot use #{type} filters/)
+ end
+ end
+ end
+
+ describe "#ddl" do
+ before do
+ @ddl = mock
+ @ddl.stubs(:meta).returns({:name => "mc"})
+ end
+
+ it "should create an instance of the right ddl" do
+ @discovery.instance_variable_set("@ddl", nil)
+ @client.stubs(:options).returns({})
+ DDL.expects(:new).with("mc", :discovery).returns(@ddl)
+ @discovery.ddl
+ end
+
+ it "should reload the ddl if the method has changed" do
+ @discovery.instance_variable_set("@ddl", @ddl)
+ @discovery.stubs(:discovery_method).returns("rspec")
+ DDL.expects(:new).with("rspec", :discovery).returns(@ddl)
+ @discovery.ddl
+ end
+ end
+
+ describe "#discovery_class" do
+ it "should try to load the class if not already loaded" do
+ @discovery.expects(:discovery_method).returns("mc")
+ PluginManager.expects(:loadclass).with("MCollective::Discovery::Mc")
+ Discovery.expects(:const_defined?).with("Mc").returns(false)
+ Discovery.expects(:const_get).with("Mc").returns("rspec")
+ @discovery.discovery_class.should == "rspec"
+ end
+
+ it "should not load the class again if its already loaded" do
+ @discovery.expects(:discovery_method).returns("mc")
+ PluginManager.expects(:loadclass).never
+ Discovery.expects(:const_defined?).with("Mc").returns(true)
+ Discovery.expects(:const_get).with("Mc").returns("rspec")
+ @discovery.discovery_class.should == "rspec"
+ end
+ end
+
+ describe "#initialize" do
+ it "should load all the known methods" do
+ @discovery.instance_variable_get("@known_methods").should == ["mc"]
+ end
+ end
+
+ describe "#find_known_methods" do
+ it "should use the PluginManager to find plugins of type 'discovery'" do
+ @discovery.find_known_methods.should == ["mc"]
+ end
+ end
+
+ describe "#has_method?" do
+ it "should correctly report the availability of a discovery method" do
+ @discovery.has_method?("mc").should == true
+ @discovery.has_method?("rspec").should == false
+ end
+ end
+
+ describe "#descovery_method" do
+ it "should default to 'mc'" do
+ @client.expects(:options).returns({})
+ @discovery.discovery_method.should == "mc"
+ end
+
+ it "should give preference to the client options" do
+ @client.expects(:options).returns({:discovery_method => "rspec"}).twice
+ Config.instance.expects(:direct_addressing).returns(true)
+ @discovery.expects(:has_method?).with("rspec").returns(true)
+ @discovery.discovery_method.should == "rspec"
+ end
+
+ it "should validate the discovery method exists" do
+ @client.expects(:options).returns({:discovery_method => "rspec"}).twice
+ expect { @discovery.discovery_method.should == "rspec" }.to raise_error("Unknown discovery method rspec")
+ end
+
+ it "should only allow custom discovery methods if direct_addressing is enabled" do
+ @client.expects(:options).returns({:discovery_method => "rspec"}).twice
+ Config.instance.expects(:direct_addressing).returns(false)
+ @discovery.expects(:has_method?).with("rspec").returns(true)
+ expect { @discovery.discovery_method.should == "rspec" }.to raise_error("Custom discovery methods require direct addressing mode")
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective::Facts
+ describe Base do
+ before do
+ class Testfacts<Base; end
+
+ MCollective::PluginManager.delete("facts_plugin")
+ MCollective::PluginManager << {:type => "facts_plugin", :class => "MCollective::Facts::Testfacts"}
+ end
+
+ describe "#inherited" do
+ it "should add classes to the plugin manager" do
+ MCollective::PluginManager.expects("<<").with({:type => "facts_plugin", :class => "MCollective::Facts::Bar"})
+
+ class Bar<Base; end
+ end
+
+ it "should be available in the PluginManager" do
+ MCollective::PluginManager["facts_plugin"].class.should == MCollective::Facts::Testfacts
+ end
+ end
+
+ describe "#get_fact" do
+ it "should call the fact provider #load_facts_from_source" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({"foo" => "bar"}).once
+
+ f = Testfacts.new
+ f.get_fact("foo")
+ end
+
+ it "should honor the cache timeout" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({"foo" => "bar"}).once
+
+ f = Testfacts.new
+ f.get_fact("foo")
+ f.get_fact("foo")
+ end
+
+ it "should detect empty facts" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({})
+ MCollective::Log.expects("error").with("Failed to load facts: RuntimeError: Got empty facts").once
+
+ f = Testfacts.new
+ f.get_fact("foo")
+ end
+
+ it "should convert non string facts to strings" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({:foo => "bar"})
+
+ f = Testfacts.new
+ f.get_fact("foo").should == "bar"
+ end
+
+ it "should not create duplicate facts while converting to strings" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({:foo => "bar"})
+
+ f = Testfacts.new
+ f.get_fact(nil).include?(:foo).should == false
+ end
+
+ it "should update last_facts_load on success" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({"foo" => "bar"}).once
+
+ f = Testfacts.new
+ f.get_fact("foo")
+
+ f.instance_variable_get("@last_facts_load").should_not == 0
+ end
+
+ it "should restore last known good facts on failure" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({}).once
+ MCollective::Log.expects("error").with("Failed to load facts: RuntimeError: Got empty facts").once
+
+ f = Testfacts.new
+ f.instance_variable_set("@last_good_facts", {"foo" => "bar"})
+
+ f.get_fact("foo").should == "bar"
+ end
+
+ it "should return all facts for nil parameter" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({"foo" => "bar", "bar" => "baz"})
+
+ f = Testfacts.new
+ f.get_fact(nil).keys.size.should == 2
+ end
+
+ it "should return a specific fact when specified" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({"foo" => "bar", "bar" => "baz"})
+
+ f = Testfacts.new
+ f.get_fact("bar").should == "baz"
+ end
+ end
+
+ describe "#get_facts" do
+ it "should load and return all facts" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({"foo" => "bar", "bar" => "baz"})
+
+ f = Testfacts.new
+ f.get_facts.should == {"foo" => "bar", "bar" => "baz"}
+ end
+ end
+
+ describe "#has_fact?" do
+ it "should correctly report fact presense" do
+ Testfacts.any_instance.stubs("load_facts_from_source").returns({"foo" => "bar"})
+
+ f = Testfacts.new
+ f.has_fact?("foo").should == true
+ f.has_fact?("bar").should == false
+ end
+ end
+
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Facts do
+ before do
+ class Facts::Testfacts<Facts::Base; end
+
+ PluginManager.delete("facts_plugin")
+ PluginManager << {:type => "facts_plugin", :class => "MCollective::Facts::Testfacts"}
+ end
+
+ describe "#has_fact?" do
+ it "should correctly report fact presense" do
+ Facts::Testfacts.any_instance.stubs("load_facts_from_source").returns({"foo" => "bar"})
+
+ Facts.has_fact?("foo", "foo").should == false
+ Facts.has_fact?("foo", "bar").should == true
+ end
+ end
+
+ describe "#get_fact" do
+ it "should return the correct fact" do
+ Facts::Testfacts.any_instance.stubs("load_facts_from_source").returns({"foo" => "bar"})
+
+ Facts.get_fact("foo").should == "bar"
+ end
+ end
+
+ describe "#[]" do
+ it "should return the correct fact" do
+ Facts::Testfacts.any_instance.stubs("load_facts_from_source").returns({"foo" => "bar"})
+
+ Facts["foo"].should == "bar"
+ end
+ end
+ end
+end
--- /dev/null
+#! /usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module Generators
+ describe AgentGenerator do
+
+ describe "#create_ddl" do
+
+ before :each do
+ AgentGenerator.any_instance.stubs(:create_plugin_content)
+ AgentGenerator.any_instance.stubs(:create_plugin_string)
+ AgentGenerator.any_instance.stubs(:write_plugins)
+ AgentGenerator.any_instance.expects(:create_metadata_string).returns("metadata\n")
+ end
+
+ it "should create a ddl with nothing but metadata if no actions are specified" do
+ result = AgentGenerator.new("foo").ddl
+ result.should == "metadata\n"
+ end
+
+ it "should add action strings to metadata if there are actions specfied" do
+ result = AgentGenerator.new("foo", ["action1", "action2"]).ddl
+ expected = File.read(File.join(File.dirname(__FILE__), "snippets", "agent_ddl"))
+ result.should == expected
+ end
+ end
+
+ describe "#create_plugin_content" do
+ before :each do
+ AgentGenerator.any_instance.stubs(:create_plugin_string)
+ AgentGenerator.any_instance.stubs(:write_plugins)
+ AgentGenerator.any_instance.stubs(:create_metadata_string).returns("metadata\n")
+ AgentGenerator.any_instance.stubs(:create_ddl)
+ end
+
+ it "should create the correct pluginf ile content with actions if they are specified" do
+ AgentGenerator.any_instance.stubs(:create_metadata_string).returns("meta\n")
+ result = AgentGenerator.new("foo", ["action1", "action2"]).content
+ result.should == " action \"action1\" do\n end\n\n action \"action2\" do\n end\n"
+ end
+ end
+
+ describe "#action_help" do
+ before :each do
+ AgentGenerator.any_instance.stubs(:create_plugin_content)
+ AgentGenerator.any_instance.stubs(:create_plugin_string)
+ AgentGenerator.any_instance.stubs(:write_plugins)
+ AgentGenerator.any_instance.stubs(:create_metadata_string).returns("metadata\n")
+ end
+
+ it "should load and return the action_help snippet" do
+ erb = mock
+ erb.stubs(:result).returns("result")
+ File.stubs(:dirname).returns("/tmp")
+ File.expects(:read).with("/tmp/templates/action_snippet.erb").returns("result")
+ ERB.expects(:new).with("result").returns(erb)
+ AgentGenerator.new("foo", ["action"])
+ end
+
+ it "should raise an error if the action_help snippet does not exist" do
+ File.stubs(:dirname).returns("/tmp")
+ File.stubs(:read).raises(Errno::ENOENT, "No such file or directory")
+ expect{
+ AgentGenerator.new("foo", ["action"])
+ }.to raise_error(Errno::ENOENT)
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require "spec_helper"
+
+module MCollective
+ module Generators
+ describe Base do
+ before :each do
+ @erb = mock
+ @erb.stubs(:result)
+ File.stubs(:dirname).returns("/tmp")
+ @base = Base.new(nil, nil, nil, nil, nil, nil, nil)
+ end
+
+ describe "#initialize" do
+ it "should set the correct metaparameters" do
+ res = Base.new("name", "description", "author", "license", "version", "url", "timeout")
+ res.meta.should == {:name => "name",
+ :description => "description",
+ :author => "author",
+ :license => "license",
+ :version => "version",
+ :url => "url",
+ :timeout => "timeout"}
+ end
+ end
+
+ describe "#create_metadata_string" do
+ it "should load the ddl template if it is present" do
+ File.expects(:read).returns("ddl")
+ ERB.expects(:new).with("ddl", nil, "-").returns(@erb)
+ @base.create_metadata_string
+ end
+
+ it "should raise an error if the template is not present" do
+ File.expects(:read).raises(Errno::ENOENT)
+ expect{
+ @base.create_metadata_string
+ }.to raise_error(Errno::ENOENT)
+ end
+ end
+
+ describe "#create_plugin_string " do
+ it "should load the plugin template if it is present" do
+ File.expects(:read).returns("plugin")
+ ERB.expects(:new).with("plugin", nil, "-").returns(@erb)
+ @base.create_plugin_string
+ end
+
+ it "should raise an error if the template is not present" do
+ File.expects(:read).raises(Errno::ENOENT)
+ expect{
+ @base.create_plugin_string
+ }.to raise_error(Errno::ENOENT)
+ end
+ end
+
+ describe "#write_plugins" do
+ it "should fail if the directory already exists" do
+ Dir.expects(:mkdir).raises(Errno::EEXIST)
+ @base.plugin_name = "foo"
+ expect{
+ @base.write_plugins
+ }.to raise_error(RuntimeError)
+ end
+
+ it "should create the directory and the plugin files if it doesn't exist" do
+ Dir.stubs(:pwd).returns("/tmp")
+ @base.stubs(:puts)
+
+ Dir.expects(:mkdir).with("foo")
+ Dir.expects(:mkdir).with("foo/agent")
+ File.expects(:open).with("foo/agent/foo.ddl", "w")
+ File.expects(:open).with("foo/agent/foo.rb", "w")
+
+ @base.plugin_name = "foo"
+ @base.mod_name = "agent"
+ @base.write_plugins
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#! /usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module Generators
+ describe DataGenerator do
+
+ before :each do
+ DataGenerator.stubs(:create_metadata_string).returns("meta\n")
+ end
+
+ describe "#create_ddl" do
+ it "create the correct ddl string" do
+ DataGenerator.any_instance.stubs(:create_plugin_content)
+ DataGenerator.any_instance.stubs(:create_plugin_string)
+ DataGenerator.any_instance.stubs(:write_plugins)
+
+ ddl = DataGenerator.new("foo", ["output"]).ddl
+ expected = File.read(File.join(File.dirname(__FILE__), "snippets", "data_ddl")).chop
+ ddl.should == expected
+ end
+ end
+
+ describe "#create_plugin_content" do
+ it "should create the correct plugin content" do
+ DataGenerator.any_instance.stubs(:create_ddl)
+ DataGenerator.any_instance.stubs(:create_plugin_string)
+ DataGenerator.any_instance.stubs(:write_plugins)
+
+ ddl = DataGenerator.new("foo", ["output"]).content
+ ddl.should == " query do |what|\n result[:output] = nil\n end\n"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+metadata
+action "action1", :description => "%ACTIONDESCRIPTION%" do
+ # Example Input
+ input :name,
+ :prompt => "%PROMPT%",
+ :description => "%DESCRIPTION%",
+ :type => %TYPE%,
+ :validation => '%VALIDATION%',
+ :optional => %OPTIONAL%,
+ :maxlength => %MAXLENGTH%
+
+ # Example output
+ output :name,
+ :description => "%DESCRIPTION%",
+ :display_as => "%DISPLAYAS%"
+end
+
+action "action2", :description => "%ACTIONDESCRIPTION%" do
+end
--- /dev/null
+metadata :name => "%FULLNANE%",
+ :description => "%DESCRIPTION%",
+ :author => "%AUTHOR%",
+ :license => "%LICENSE%",
+ :version => "%VERSION%",
+ :url => "%URL%",
+ :timeout => %TIMEOUT%
+
+dataquery :description => "Query information" do
+ input :query,
+ :prompt => "%PROMP%",
+ :description => "%DESCRIPTION%",
+ :type => %TYPE%,
+ :validation => %VALIDATION%,
+ :maxlength => %MAXLENGTH%
+
+ output :output,
+ :description => "%DESCRIPTION%",
+ :display_as => "%DESCRIPTION%"
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Log do
+ before do
+ @logger = mock("logger provider")
+ @logger.stubs(:start)
+ @logger.stubs(:set_logging_level)
+ @logger.stubs(:valid_levels)
+ @logger.stubs(:should_log?).returns(true)
+ @logger.stubs(:new).returns(@logger)
+
+ # we stub it out at the top of the test suite
+ Log.unstub(:log)
+ Log.unstub(:logexception)
+ Log.unstub(:logmsg)
+
+ Log.unconfigure
+ Log.set_logger(@logger)
+ end
+
+ describe "#config_and_check_level" do
+ it "should configure then not already configured" do
+ Log.expects(:configure)
+ Log.config_and_check_level(:debug)
+ end
+
+ it "should not reconfigure the logger" do
+ Log.configure(@logger)
+ Log.expects(:configure).never
+ Log.config_and_check_level(:debug)
+ end
+
+ it "should check the level is valid" do
+ Log.configure(@logger)
+ Log.expects(:check_level).with(:debug)
+ Log.config_and_check_level(:debug)
+ end
+
+ it "should respect the loggers decision about levels" do
+ Log.configure(@logger)
+ @logger.expects(:should_log?).returns(false)
+ Log.config_and_check_level(:debug).should == false
+ end
+ end
+
+ describe "#valid_level?" do
+ it "should correctly report for valid levels" do
+ [:error, :fatal, :debug, :warn, :info].each {|level| Log.valid_level?(level).should == true }
+ Log.valid_level?(:rspec).should == false
+ end
+ end
+
+ describe "#message_for" do
+ it "should return the code and retrieved message" do
+ Util.expects(:t).with(:PLMC1, {:rspec => true}).returns("this is PLMC1")
+ Log.message_for(:PLMC1, {:rspec => true}).should == "PLMC1: this is PLMC1"
+ end
+ end
+
+ describe "#logexception" do
+ it "should short circuit messages below current level" do
+ Log.expects(:config_and_check_level).with(:debug).returns(false)
+ Log.expects(:log).never
+ Log.logexception(:PLMC1, :debug, Exception.new, {})
+ end
+
+ it "should request the message including the exception string and log it" do
+ Log.stubs(:config_and_check_level).returns(true)
+ Log.expects(:message_for).with(:PLMC1, {:rspec => "test", :error => "Exception: this is a test"}).returns("This is a test")
+ Log.expects(:log).with(:debug, "This is a test", "test:2")
+
+ e = Exception.new("this is a test")
+ e.set_backtrace ["/some/dir/test:1", "/some/dir/test:2"]
+
+ Log.logexception(:PLMC1, :debug, e, false, {:rspec => "test"})
+ end
+ end
+
+ describe "#logmsg" do
+ it "should short circuit messages below current level" do
+ Log.expects(:config_and_check_level).with(:debug).returns(false)
+ Log.expects(:log).never
+ Log.logmsg(:PLMC1, "", :debug, {})
+ end
+
+ it "should request the message and log it" do
+ Log.stubs(:config_and_check_level).returns(true)
+ Log.expects(:message_for).with(:PLMC1, {:rspec => "test", :default => "default"}).returns("This is a test")
+ Log.expects(:log).with(:debug, "This is a test")
+ Log.logmsg(:PLMC1, "default", :debug, :rspec => "test")
+ end
+ end
+
+ describe "#check_level" do
+ it "should check for valid levels" do
+ Log.expects(:valid_level?).with(:debug).returns(true)
+ Log.check_level(:debug)
+ end
+
+ it "should raise for invalid levels" do
+ expect { Log.check_level(:rspec) }.to raise_error("Unknown log level")
+ end
+ end
+
+ describe "#configure" do
+ it "should default to console logging if called prior to configuration" do
+ Config.instance.instance_variable_set("@configured", false)
+ Log.configure
+ Log.logger.should == MCollective::Logger::Console_logger
+ end
+ end
+
+ describe "#instance" do
+ it "should return the correct reference" do
+ Log.configure(@logger)
+ Log.instance.should == MCollective::Log
+ end
+ end
+
+ describe "#log" do
+ it "should log at the right levels" do
+ Log.configure(@logger)
+
+ [:debug, :info, :fatal, :error, :warn].each do |level|
+ @logger.expects(:log).with(level, anything, regexp_matches(/#{level} test/))
+ @logger.expects(:should_log?).with(level).returns(true)
+ Log.send(level, "#{level} test")
+ end
+ end
+ end
+
+ describe "#cycle_level" do
+ it "should cycle logger class levels" do
+ @logger.expects(:cycle_level)
+
+ Log.configure(@logger)
+ Log.cycle_level
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective::Logger
+ describe Base do
+ before do
+ Base.any_instance.stubs(:set_logging_level).returns(true)
+ Base.any_instance.stubs(:valid_levels).returns({:info => :info_test,
+ :warn => :warning_test,
+ :debug => :debug_test,
+ :fatal => :crit_test,
+ :error => :err_test})
+ end
+
+ describe "#initialize" do
+ it "should check for valid levels" do
+ Base.any_instance.stubs(:valid_levels).returns({})
+
+ expect {
+ Base.new
+ }.to raise_error(/Logger class did not specify a map for/)
+ end
+
+ it "should accept correct levels" do
+ logger = Base.new
+ logger.set_level :warn
+
+ logger.should_log?(:debug).should == false
+ logger.should_log?(:error).should == true
+ end
+ end
+
+ describe "#should_log?" do
+ it "should correctly determine if a line should be logged" do
+ end
+ end
+ describe "#valid_levels" do
+ it "should report if valid_levels was not implimented" do
+ Base.any_instance.unstub(:valid_levels)
+
+ expect {
+ logger = Base.new
+ }.to raise_error("The logging class did not supply a valid_levels method")
+ end
+ end
+
+ describe "#log" do
+ it "should report if log was not implimented" do
+ logger = Base.new
+
+ expect {
+ logger.send(:log, nil, nil, nil)
+ }.to raise_error("The logging class did not supply a log method")
+ end
+ end
+
+ describe "#start" do
+ it "should report if log was not implimented" do
+ logger = Base.new
+
+ expect {
+ logger.send(:start)
+ }.to raise_error("The logging class did not supply a start method")
+ end
+ end
+
+ describe "#map_level" do
+ it "should map levels correctly" do
+ logger = Base.new
+
+ logger.send(:map_level, :info).should == :info_test
+ logger.send(:map_level, :warn).should == :warning_test
+ logger.send(:map_level, :debug).should == :debug_test
+ logger.send(:map_level, :fatal).should == :crit_test
+ logger.send(:map_level, :error).should == :err_test
+ end
+ end
+
+ describe "#get_next_level" do
+ it "should supply the correct next level" do
+ logger = Base.new
+ logger.set_level(:fatal)
+
+ logger.send(:get_next_level).should == :debug
+ end
+ end
+
+ describe "#cycle_level" do
+ it "should set the level to the next one and log the event" do
+ logger = Base.new
+
+ logger.stubs(:get_next_level).returns(:error)
+
+ logger.expects(:set_level).with(:error)
+ logger.expects(:log).with(:error, "", "Logging level is now ERROR")
+
+ logger.cycle_level
+ end
+ end
+
+ describe "#set_level" do
+ it "should set the active level" do
+ logger = Base.new
+
+ logger.set_level(:error)
+
+ logger.active_level.should == :error
+ end
+
+ it "should set the level on the logger" do
+ logger = Base.new
+
+ logger.set_level(:error)
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ require 'mcollective/logger/console_logger'
+
+ module Logger
+ describe Console_logger do
+ describe "#start" do
+ it "should default to :info and allow the config to override" do
+ logger = Console_logger.new
+ logger.expects(:set_level).with(:info)
+ Config.instance.expects(:configured).returns(true)
+ Config.instance.expects(:loglevel).returns("error")
+ logger.expects(:set_level).with(:error)
+ logger.start
+ end
+ end
+
+ describe "#color" do
+ it "should not colorize if color was disabled" do
+ logger = Console_logger.new
+ Config.instance.stubs(:color).returns(false)
+ logger.color(:error).should == ""
+ logger.color(:reset).should == ""
+ end
+
+ it "should correctly colorize by level" do
+ logger = Console_logger.new
+ Config.instance.stubs(:color).returns(true)
+ logger.color(:error).should == Util.color(:red)
+ logger.color(:reset).should == Util.color(:reset)
+ end
+ end
+
+ describe "#log" do
+ it "should log higher than configured levels" do
+ io = StringIO.new
+ io.expects(:puts).with("error 2012/07/03 15:11:35: rspec message")
+
+ time = stub
+ time.expects(:strftime).returns("2012/07/03 15:11:35")
+
+ Time.expects(:new).returns(time)
+
+ Config.instance.stubs(:color).returns(false)
+ logger = Console_logger.new
+ logger.set_level(:warn)
+ logger.log(:error, "rspec", "message", io)
+ end
+
+ it "should resort to STDERR output if all else fails" do
+ io = StringIO.new
+ io.expects(:puts).raises
+
+ last_resort_io = StringIO.new
+ last_resort_io.expects(:puts).with("warn: message")
+
+ logger = Console_logger.new
+ logger.set_level(:debug)
+ logger.log(:warn, "rspec", "message", io, last_resort_io)
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ unless Util.windows?
+ require 'mcollective/logger/syslog_logger'
+
+ module Logger
+ describe Syslog_logger do
+ describe "#start" do
+ before do
+ Config.instance.stubs(:logfacility).returns("user")
+ Config.instance.stubs(:loglevel).returns("error")
+ end
+
+ it "should close the syslog if already opened" do
+ Syslog.expects("opened?").returns(true)
+ Syslog.expects(:close).once
+ Syslog.expects(:open).once
+
+ logger = Syslog_logger.new
+ logger.start
+ end
+
+ it "should open syslog with the correct facility" do
+ logger = Syslog_logger.new
+ Syslog.expects(:open).with("rspec", 3, Syslog::LOG_USER).once
+ logger.start
+ end
+
+ it "should set the logger level correctly" do
+ logger = Syslog_logger.new
+ Syslog.expects(:open).with("rspec", 3, Syslog::LOG_USER).once
+ logger.expects(:set_level).with(:error).once
+ logger.start
+ end
+ end
+
+ describe "#syslog_facility" do
+ it "should support valid facilities" do
+ logger = Syslog_logger.new
+ logger.syslog_facility("LOCAL1").should == Syslog::LOG_LOCAL1
+ logger.syslog_facility("local1").should == Syslog::LOG_LOCAL1
+ end
+
+ it "should set LOG_USER for unknown facilities" do
+ logger = Syslog_logger.new
+ IO.any_instance.expects(:puts).with("Invalid syslog facility rspec supplied, reverting to USER")
+ logger.syslog_facility("rspec").should == Syslog::LOG_USER
+ end
+ end
+
+ describe "#log" do
+ it "should log higher than configured levels" do
+ logger = Syslog_logger.new
+ logger.set_level(:debug)
+ Syslog.expects(:err).once
+ logger.log(:error, "rspec", "rspec")
+ end
+
+ it "should log using the correctly mapped level" do
+ logger = Syslog_logger.new
+ Syslog.expects(:err).with("rspec rspec").once
+ logger.set_level(:debug)
+ logger.log(:error, "rspec", "rspec")
+ end
+
+ it "should resort to STDERR output if all else fails" do
+ logger = Syslog_logger.new
+ IO.any_instance.expects(:puts).with("error: rspec").once
+
+ logger.log(:error, "rspec", "rspec")
+ end
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#! /usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module Matcher
+ describe Parser do
+ before :each do
+ Config.instance.stubs(:color).returns(false)
+ end
+
+ describe '#parse' do
+ it "should parse statements seperated by '='" do
+ parser = Parser.new("foo=bar")
+ parser.execution_stack.should == [{"statement" => "foo=bar"}]
+ end
+
+ it "should parse statements seperated by '<'" do
+ parser = Parser.new("foo<bar")
+ parser.execution_stack.should == [{"statement" => "foo<bar"}]
+ end
+
+ it "should parse statements seperated by '>'" do
+ parser = Parser.new("foo>bar")
+ parser.execution_stack.should == [{"statement" => "foo>bar"}]
+ end
+
+ it "should parse statements seperated by '<='" do
+ parser = Parser.new("foo<=bar")
+ parser.execution_stack.should == [{"statement" => "foo<=bar"}]
+ end
+
+ it "should parse statements seperated by '>='" do
+ parser = Parser.new("foo>=bar")
+ parser.execution_stack.should == [{"statement" => "foo>=bar"}]
+ end
+
+ it "should parse class regex statements" do
+ parser = Parser.new("/foo/")
+ parser.execution_stack.should == [{"statement" => "/foo/"}]
+ end
+
+ it "should parse fact regex statements" do
+ parser = Parser.new("foo=/bar/")
+ parser.execution_stack.should == [{"statement" => "foo=/bar/"}]
+ end
+
+ it "should parse a correct 'and' token" do
+ parser = Parser.new("foo=bar and bar=foo")
+ parser.execution_stack.should == [{"statement" => "foo=bar"}, {"and" => "and"}, {"statement" => "bar=foo"}]
+ end
+
+ it "should not parse an incorrect and token" do
+ expect {
+ parser = Parser.new("and foo=bar")
+ }.to raise_error(RuntimeError, "Parse errors found while parsing -S input and foo=bar")
+ end
+
+ it "should parse a correct 'or' token" do
+ parser = Parser.new("foo=bar or bar=foo")
+ parser.execution_stack.should == [{"statement" => "foo=bar"}, {"or" => "or"}, {"statement" => "bar=foo"}]
+ end
+
+ it "should not parse an incorrect or token" do
+ expect{
+ parser = Parser.new("or foo=bar")
+ }.to raise_error(RuntimeError, "Parse errors found while parsing -S input or foo=bar")
+ end
+
+ it "should parse a correct 'not' token" do
+ parser = Parser.new("! bar=foo")
+ parser.execution_stack.should == [{"not" => "not"}, {"statement" => "bar=foo"}]
+ parser = Parser.new("not bar=foo")
+ parser.execution_stack.should == [{"not" => "not"}, {"statement" => "bar=foo"}]
+ end
+
+ it "should not parse an incorrect 'not' token" do
+ expect{
+ parser = Parser.new("foo=bar !")
+ }.to raise_error(RuntimeError, "Parse errors found while parsing -S input foo=bar !")
+ end
+
+ it "should parse correct parentheses" do
+ parser = Parser.new("(foo=bar)")
+ parser.execution_stack.should == [{"(" => "("}, {"statement" => "foo=bar"}, {")" => ")"}]
+ end
+
+ it "should fail on incorrect parentheses" do
+ expect{
+ parser = Parser.new(")foo=bar(")
+ }.to raise_error(RuntimeError, "Malformed token(s) found while parsing -S input )foo=bar(")
+ end
+
+ it "should fail on missing parentheses" do
+ expect{
+ parser = Parser.new("(foo=bar")
+ }.to raise_error(RuntimeError, "Missing parenthesis found while parsing -S input (foo=bar")
+ end
+
+ it "should parse correctly formatted compound statements" do
+ parser = Parser.new("(foo=bar or foo=rab) and (bar=foo)")
+ parser.execution_stack.should == [{"(" => "("}, {"statement"=>"foo=bar"}, {"or"=>"or"}, {"statement"=>"foo=rab"},
+ {")"=>")"}, {"and"=>"and"}, {"("=>"("}, {"statement"=>"bar=foo"},
+ {")"=>")"}]
+ end
+
+ it "should parse complex fstatements and statements with operators seperated by whitespaces" do
+ parser = Parser.new("foo('bar').value = 1 and foo=bar or foo = bar")
+ parser.execution_stack.should == [{"fstatement"=>"foo('bar').value=1"}, {"and"=>"and"}, {"statement"=>"foo=bar"}, {"or"=>"or"}, {"statement"=>"foo=bar"}]
+ end
+
+ it "should parse statements where classes are mixed with fact comparisons and fstatements" do
+ parser = Parser.new("klass and function('param').value = 1 and fact=value")
+ parser.execution_stack.should == [{"statement" => "klass"},
+ {"and" => "and"},
+ {"fstatement" => "function('param').value=1"},
+ {"and" => "and"},
+ {"statement" => "fact=value"}]
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#! /usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module Matcher
+ describe 'scanner' do
+ it "should identify a '(' token" do
+ scanner = Scanner.new("(")
+ token = scanner.get_token
+ token.should == ["(", "("]
+ end
+
+ it "should identify a ')' token" do
+ scanner = Scanner.new(")")
+ token = scanner.get_token
+ token.should == [")", ")"]
+ end
+
+ it "should identify a 'and' token" do
+ scanner = Scanner.new("and ")
+ token = scanner.get_token
+ token.should == ["and", "and"]
+ end
+
+ it "should identify a 'or' token" do
+ scanner = Scanner.new("or ")
+ token = scanner.get_token
+ token.should == ["or", "or"]
+ end
+
+ it "should identify a 'not' token" do
+ scanner = Scanner.new("not ")
+ token = scanner.get_token
+ token.should == ["not", "not"]
+ end
+
+ it "should identify a '!' token" do
+ scanner = Scanner.new("!")
+ token = scanner.get_token
+ token.should == ["not", "not"]
+ end
+
+ it "should identify a fact statement token" do
+ scanner = Scanner.new("foo=bar")
+ token = scanner.get_token
+ token.should == ["statement", "foo=bar"]
+ end
+
+ it "should identify a fact statement token" do
+ scanner = Scanner.new("foo=bar")
+ token = scanner.get_token
+ token.should == ["statement", "foo=bar"]
+ end
+
+ it "should identify a class statement token" do
+ scanner = Scanner.new("/class/")
+ token = scanner.get_token
+ token.should == ["statement", "/class/"]
+ end
+
+ it "should identify a function statement token with a dot value" do
+ scanner = Scanner.new("foo('bar').baz")
+ token = scanner.get_token
+ token.should == ["fstatement", "foo('bar').baz"]
+ end
+
+ it "should identify a function statement token without a dot value" do
+ scanner = Scanner.new("foo('bar')")
+ token = scanner.get_token
+ token.should == ["fstatement", "foo('bar')"]
+ end
+
+ it "should identify a function statement with multiple parameters" do
+ scanner = Scanner.new("foo('bar','baz')")
+ token = scanner.get_token
+ token.should == ["fstatement", "foo('bar','baz')"]
+ end
+
+ it "should identify a bad token when a function is missing its end bracket" do
+ scanner = Scanner.new("foo(")
+ token = scanner.get_token
+ token.should == ["bad_token", [0,3]]
+ end
+
+ it "should identify a bad token when there is a regex before a comparison operator" do
+ scanner = Scanner.new("/foo/=bar")
+ token = scanner.get_token
+ token.should == ["bad_token", [0,8]]
+ end
+
+ it "should identify a bad token where there is a forward slash before a comparison operator" do
+ scanner = Scanner.new("/foo=bar")
+ token = scanner.get_token
+ token.should == ["bad_token", [0,7]]
+ end
+
+ it "should identify a bad token where there is only one forward slash after a comparison operator" do
+ scanner = Scanner.new("foo=/bar")
+ token = scanner.get_token
+ token.should == ["bad_token", [0,7]]
+ end
+
+ it "should identify a bad token where function parameters are not in single quotes" do
+ scanner = Scanner.new("foo(bar)")
+ token = scanner.get_token
+ token.should == ["bad_token", [0,7]]
+ end
+
+ it "should identify a bad token where there are non alphanumerical or underscore chars in the dot value" do
+ scanner = Scanner.new("foo('bar').val-ue")
+ token = scanner.get_token
+ token.should == ["bad_token", [0,16]]
+ end
+
+ it "should identify a bad token where there are chained dot values" do
+ scanner = Scanner.new("foo('bar').a.b")
+ token = scanner.get_token
+ token.should == ["bad_token", [0,13]]
+ end
+
+ it "should identify bad tokens where function parameters are not comma seperated" do
+ scanner = Scanner.new("foo('a' 'b')")
+ token = scanner.get_token
+ token.should == ["bad_token", [0,11]]
+
+ scanner = Scanner.new("foo(\"a\" \"b\")")
+ token = scanner.get_token
+ token.should == ["bad_token", [0,11]]
+
+ end
+
+ it "should identify fstatement tokens where the values and the comparison operator are seperated by whitespaces" do
+ scanner = Scanner.new("foo('a').bar = 1")
+ token = scanner.get_token
+ token.should == ["fstatement", "foo('a').bar=1"]
+ end
+
+ it "should identify statement tokens where the values and the comparison operator are seperated by whitespaces" do
+ scanner = Scanner.new("a = c")
+ token = scanner.get_token
+ token.should == ["statement", "a=c"]
+ end
+
+ it "should idenify a function statement where a parameter is an empty string" do
+ scanner = Scanner.new("foo('')")
+ token = scanner.get_token
+ token.should == ["fstatement", "foo('')"]
+ end
+
+ it "should correctly tokenize a statement with escaped qoutes in parameters" do
+ scanner = Scanner.new("foo('\"bar\"')")
+ token = scanner.get_token
+ token.should == ["fstatement", "foo('\"bar\"')"]
+
+ scanner = Scanner.new('foo("\'bar\'")')
+ token =scanner.get_token
+ token.should == ["fstatement", "foo(\"'bar'\")"]
+ end
+
+ it "should correctly tokenize a statement where a regular expression contains parentheses" do
+ scanner = Scanner.new("foo=/bar(1|2)/")
+ token = scanner.get_token
+ token.should == ["statement", "foo=/bar(1|2)/"]
+ end
+
+ it "should correctly tokenize a statement with a comparison operator in a parameter" do
+ scanner = Scanner.new("foo('bar=baz')")
+ token = scanner.get_token
+ token.should == ["fstatement", "foo('bar=baz')"]
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Matcher do
+ describe "#create_function_hash" do
+ it "should create a correct hash for a 'normal' function call using single quotes" do
+ result = Matcher.create_function_hash("foo('bar').res=1")
+ result["value"].should == "res"
+ result["params"].should == "bar"
+ result["r_compare"].should == "1"
+ result["operator"].should == "=="
+ result["name"].should == "foo"
+ end
+
+ it "should create a correct hash for a 'normal' function call using double quotes" do
+ result = Matcher.create_function_hash('foo("bar").res=1')
+ result["value"].should == "res"
+ result["params"].should == "bar"
+ result["r_compare"].should == "1"
+ result["operator"].should == "=="
+ result["name"].should == "foo"
+ end
+
+ it "should create a correct hash when a paramater contains a dot value" do
+ result = Matcher.create_function_hash("foo('bar.1').res=1")
+ result["value"].should == "res"
+ result["params"].should == "bar.1"
+ result["r_compare"].should == "1"
+ result["operator"].should == "=="
+ result["name"].should == "foo"
+ end
+
+ it "should create a correct hash when right compare value is a regex" do
+ result = Matcher.create_function_hash("foo('bar').res=/reg/")
+ result["value"].should == "res"
+ result["params"].should == "bar"
+ result["r_compare"].should == /reg/
+ result["operator"].should == "=~"
+ result["name"].should == "foo"
+ end
+
+ it "should create a correct hash when the operator is >= or <=" do
+ result = Matcher.create_function_hash("foo('bar').res<=1")
+ result["value"].should == "res"
+ result["params"].should == "bar"
+ result["r_compare"].should == "1"
+ result["operator"].should == "<="
+ result["name"].should == "foo"
+
+ result = Matcher.create_function_hash("foo('bar').res>=1")
+ result["value"].should == "res"
+ result["params"].should == "bar"
+ result["r_compare"].should == "1"
+ result["operator"].should == ">="
+ result["name"].should == "foo"
+ end
+
+ it "should create a correct hash when no dot value is present" do
+ result = Matcher.create_function_hash("foo('bar')<=1")
+ result["value"].should == nil
+ result["params"].should == "bar"
+ result["r_compare"].should == "1"
+ result["operator"].should == "<="
+ result["name"].should == "foo"
+ end
+
+ it "should create a correct hash when a dot is present in a parameter but no dot value is present" do
+ result = Matcher.create_function_hash("foo('bar.one')<=1")
+ result["value"].should == nil
+ result["params"].should == "bar.one"
+ result["r_compare"].should == "1"
+ result["operator"].should == "<="
+ result["name"].should == "foo"
+ end
+
+ it "should create a correct hash when multiple dots are present in parameters but no dot value is present" do
+ result = Matcher.create_function_hash("foo('bar.one.two, bar.three.four')<=1")
+ result["value"].should == nil
+ result["params"].should == "bar.one.two, bar.three.four"
+ result["r_compare"].should == "1"
+ result["operator"].should == "<="
+ result["name"].should == "foo"
+ end
+
+ it "should create a correct hash when no parameters are given" do
+ result = Matcher.create_function_hash("foo()<=1")
+ result["value"].should == nil
+ result["params"].should == nil
+ result["r_compare"].should == "1"
+ result["operator"].should == "<="
+ result["name"].should == "foo"
+ end
+
+ it "should create a correct hash parameters are empty strings" do
+ result = Matcher.create_function_hash("foo('')=1")
+ result["value"].should == nil
+ result["params"].should == ""
+ result["r_compare"].should == "1"
+ result["operator"].should == "=="
+ result["name"].should == "foo"
+ end
+ end
+
+ describe "#execute_function" do
+ it "should return the result of an evaluated function with a dot value" do
+ data = mock
+ data.expects(:send).with("value").returns("success")
+ MCollective::Data.expects(:send).with("foo", "bar").returns(data)
+ result = Matcher.execute_function({"name" => "foo", "params" => "bar", "value" => "value"})
+ result.should == "success"
+ end
+
+ it "should return the result of an evaluated function without a dot value" do
+ MCollective::Data.expects(:send).with("foo", "bar").returns("success")
+ result = Matcher.execute_function({"name" => "foo", "params" => "bar"})
+ result.should == "success"
+ end
+ end
+
+ describe "#eval_compound_statement" do
+ it "should return correctly on a regex class statement" do
+ Util.expects(:has_cf_class?).with("/foo/").returns(true)
+ Matcher.eval_compound_statement({"statement" => "/foo/"}).should == true
+ Util.expects(:has_cf_class?).with("/foo/").returns(false)
+ Matcher.eval_compound_statement({"statement" => "/foo/"}).should == false
+ end
+
+ it "should return correcly for string and regex facts" do
+ Util.expects(:has_fact?).with("foo", "bar", "==").returns(true)
+ Matcher.eval_compound_statement({"statement" => "foo=bar"}).should == "true"
+ Util.expects(:has_fact?).with("foo", "/bar/", "=~").returns(false)
+ Matcher.eval_compound_statement({"statement" => "foo=/bar/"}).should == "false"
+ end
+
+ it "should return correctly on a string class statement" do
+ Util.expects(:has_cf_class?).with("foo").returns(true)
+ Matcher.eval_compound_statement({"statement" => "foo"}).should == true
+ Util.expects(:has_cf_class?).with("foo").returns(false)
+ Matcher.eval_compound_statement({"statement" => "foo"}).should == false
+ end
+ end
+
+ describe "#eval_compound_fstatement" do
+ describe "it should return false if a string, true or false are compared with > or <" do
+ let(:function_hash) do
+ {"name" => "foo",
+ "params" => "bar",
+ "value" => "value",
+ "operator" => "<",
+ "r_compare" => "teststring"}
+ end
+
+
+ it "should return false if a string is compare with a < or >" do
+ Matcher.expects(:execute_function).returns("teststring")
+ result = Matcher.eval_compound_fstatement(function_hash)
+ result.should == false
+ end
+
+ it "should return false if a TrueClass is compared with a < or > " do
+ Matcher.expects(:execute_function).returns(true)
+ result = Matcher.eval_compound_fstatement(function_hash)
+ result.should == false
+ end
+
+ it "should return false if a FalseClass is compared with a < or >" do
+ Matcher.expects(:execute_function).returns(false)
+ result = Matcher.eval_compound_fstatement(function_hash)
+ result.should == false
+ end
+ end
+
+ describe "it should return false if backticks are present in parameters and if non strings are compared with regex's" do
+ before :each do
+ @function_hash = {"name" => "foo",
+ "params" => "bar",
+ "value" => "value",
+ "operator" => "=",
+ "r_compare" => "1"}
+ end
+
+ it "should return false if a backtick is present in a parameter" do
+ @function_hash["params"] = "`bar`"
+ Matcher.expects(:execute_function).returns("teststring")
+ MCollective::Log.expects(:debug).with("Cannot use backticks in function parameters")
+ result = Matcher.eval_compound_fstatement(@function_hash)
+ result.should == false
+ end
+
+ it "should return false if left compare object isn't a string and right compare is a regex" do
+ Matcher.expects(:execute_function).returns(1)
+ @function_hash["r_compare"] = "/foo/"
+ @function_hash["operator"] = "=~"
+ MCollective::Log.expects(:debug).with("Cannot do a regex check on a non string value.")
+ result = Matcher.eval_compound_fstatement(@function_hash)
+ result.should == false
+ end
+ end
+
+ describe "it should return the expected result for valid function hashes" do
+ before :each do
+ @function_hash = {"name" => "foo",
+ "params" => "bar",
+ "value" => "value",
+ "operator" => "=",
+ "r_compare" => ""}
+ end
+
+ it "should return true if right value is a regex and matches the left value" do
+ Matcher.expects(:execute_function).returns("teststring")
+ @function_hash["r_compare"] = /teststring/
+ @function_hash["operator"] = "=~"
+ result = Matcher.eval_compound_fstatement(@function_hash)
+ result.should == true
+ end
+
+ it "should return false if right value is a regex, operator is != and regex equals left value" do
+ Matcher.expects(:execute_function).returns("teststring")
+ @function_hash["r_compare"] = /teststring/
+ @function_hash["operator"] = "!=~"
+ result = Matcher.eval_compound_fstatement(@function_hash)
+ result.should == false
+ end
+
+ it "should return false if right value is a regex and does not match left value" do
+ Matcher.expects(:execute_function).returns("testsnotstring")
+ @function_hash["r_compare"] = /teststring/
+ @function_hash["operator"] = "=~"
+ result = Matcher.eval_compound_fstatement(@function_hash)
+ result.should == false
+ end
+
+ it "should return true if left value logically compares to the right value" do
+ Matcher.expects(:execute_function).returns(1)
+ @function_hash["r_compare"] = 1
+ @function_hash["operator"] = ">="
+ result = Matcher.eval_compound_fstatement(@function_hash)
+ result.should == true
+ end
+
+ it "should return false if left value does not logically compare to right value" do
+ Matcher.expects(:execute_function).returns("1")
+ @function_hash["r_compare"] = "1"
+ @function_hash["operator"] = ">"
+ result = Matcher.eval_compound_fstatement(@function_hash)
+ result.should == false
+ end
+ end
+ end
+
+ describe "#create_compound_callstack" do
+ it "should create a callstack from a valid call_string" do
+ result = Matcher.create_compound_callstack("foo('bar')=1 and bar=/bar/")
+ result.should == [{"fstatement" => {"params"=>"bar", "name"=>"foo", "operator"=>"==", "r_compare"=>"1"}}, {"and" => "and"}, {"statement" => "bar=/bar/"}]
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Message do
+ before do
+ Config.instance.set_config_defaults("")
+ end
+
+ describe "#initialize" do
+ it "should set defaults" do
+ m = Message.new("payload", "message")
+ m.payload.should == "payload"
+ m.message.should == "message"
+ m.request.should == nil
+ m.headers.should == {}
+ m.agent.should == nil
+ m.collective.should == nil
+ m.type.should == :message
+ m.filter.should == Util.empty_filter
+ m.requestid.should == nil
+ m.base64?.should == false
+ m.options.should == {}
+ m.discovered_hosts.should == nil
+ m.ttl.should == 60
+ m.validated.should == false
+ m.msgtime.should == 0
+ m.expected_msgid == nil
+ end
+
+ it "should set all supplied options" do
+ Message.any_instance.expects(:base64_decode!)
+
+ m = Message.new("payload", "message", :base64 => true,
+ :agent => "rspecagent",
+ :headers => {:rspec => "test"},
+ :type => :rspec,
+ :filter => "filter",
+ :options => {:ttl => 30},
+ :collective => "collective")
+ m.payload.should == "payload"
+ m.message.should == "message"
+ m.request.should == nil
+ m.headers.should == {:rspec => "test"}
+ m.agent.should == "rspecagent"
+ m.collective.should == "collective"
+ m.type.should == :rspec
+ m.filter.should == "filter"
+ m.base64?.should == true
+ m.options.should == {:ttl => 30}
+ m.ttl.should == 30
+ end
+
+ it "if given a request it should set options based on the request" do
+ request = mock
+ request.expects(:agent).returns("request")
+ request.expects(:collective).returns("collective")
+
+ m = Message.new("payload", "message", :request => request)
+ m.agent.should == "request"
+ m.collective.should == "collective"
+ m.type.should == :reply
+ m.request.should == request
+ end
+ end
+
+ describe "#reply_to=" do
+ it "should only set the reply-to header for requests" do
+ Config.instance.expects(:direct_addressing).returns(true)
+ m = Message.new("payload", "message", :type => :reply)
+ m.discovered_hosts = ["foo"]
+ expect { m.reply_to = "foo" }.to raise_error(/reply targets/)
+
+ [:request, :direct_request].each do |t|
+ m.type = t
+ m.reply_to = "foo"
+ m.reply_to.should == "foo"
+ end
+ end
+ end
+
+ describe "#expected_msgid=" do
+ it "should correctly set the property" do
+ m = Message.new("payload", "message", :type => :reply)
+ m.expected_msgid = "rspec test"
+ m.expected_msgid.should == "rspec test"
+ end
+
+ it "should only be set for reply messages" do
+ m = Message.new("payload", "message", :type => :request)
+
+ expect {
+ m.expected_msgid = "rspec test"
+ }.to raise_error("Can only store the expected msgid for reply messages")
+ end
+ end
+
+ describe "#base64_decode!" do
+ it "should not decode if not encoded" do
+ SSL.expects(:base64_decode).never
+ m = Message.new("payload", "message")
+ end
+
+ it "should decode encoded messages" do
+ SSL.expects(:base64_decode)
+ m = Message.new("payload", "message", :base64 => true)
+ end
+
+ it "should set base64 to false after decoding" do
+ SSL.expects(:base64_decode).with("payload")
+ m = Message.new("payload", "message", :base64 => true)
+ m.base64?.should == false
+ end
+ end
+
+ describe "#base64_encode" do
+ it "should not encode already encoded messages" do
+ SSL.expects(:base64_encode).never
+ Message.any_instance.stubs(:base64_decode!)
+ m = Message.new("payload", "message", :base64 => true)
+ m.base64_encode!
+ end
+
+ it "should encode plain messages" do
+ SSL.expects(:base64_encode).with("payload")
+ m = Message.new("payload", "message")
+ m.base64_encode!
+ end
+
+ it "should set base64 to false after encoding" do
+ SSL.expects(:base64_encode)
+ m = Message.new("payload", "message")
+ m.base64_encode!
+ m.base64?.should == true
+ end
+ end
+
+ describe "#base64?" do
+ it "should correctly report base64 state" do
+ m = Message.new("payload", "message")
+ m.base64?.should == m.instance_variable_get("@base64")
+ end
+ end
+
+ describe "#type=" do
+ it "should only allow types to be set when discovered hosts were given" do
+ m = Message.new("payload", "message")
+ Config.instance.stubs(:direct_addressing).returns(true)
+
+ expect {
+ m.type = :direct_request
+ }.to raise_error("Can only set type to :direct_request if discovered_hosts have been set")
+ end
+
+ it "should not allow direct_request to be set if direct addressing isnt enabled" do
+ m = Message.new("payload", "message")
+ Config.instance.stubs(:direct_addressing).returns(false)
+
+ expect {
+ m.type = :direct_request
+ }.to raise_error("Direct requests is not enabled using the direct_addressing config option")
+ end
+
+ it "should only accept valid types" do
+ m = Message.new("payload", "message")
+ Config.instance.stubs(:direct_addressing).returns(true)
+
+ expect {
+ m.type = :foo
+ }.to raise_error("Unknown message type foo")
+ end
+
+ it "should clear the filter in direct_request mode and add just an agent filter" do
+ m = Message.new("payload", "message")
+ m.discovered_hosts = ["rspec"]
+ Config.instance.stubs(:direct_addressing).returns(true)
+
+ m.filter = Util.empty_filter.merge({"cf_class" => ["test"]})
+ m.agent = "rspec"
+ m.type = :direct_request
+ m.filter.should == Util.empty_filter.merge({"agent" => ["rspec"]})
+ end
+
+ it "should set the type" do
+ m = Message.new("payload", "message")
+ m.type = :request
+ m.type.should == :request
+ end
+ end
+
+ describe "#encode!" do
+ it "should encode replies using the security plugin #encodereply" do
+ request = mock
+ request.stubs(:agent).returns("rspec_agent")
+ request.stubs(:collective).returns("collective")
+ request.stubs(:payload).returns({:requestid => "123", :callerid => "id=callerid"})
+
+ security = mock
+ security.expects(:encodereply).with('rspec_agent', 'payload', '123', 'id=callerid')
+ security.expects(:valid_callerid?).with("id=callerid").returns(true)
+
+ PluginManager.expects("[]").with("security_plugin").returns(security).twice
+
+ m = Message.new("payload", "message", :request => request, :type => :reply)
+
+ m.encode!
+ end
+
+ it "should encode requests using the security plugin #encoderequest" do
+ security = mock
+ security.expects(:encoderequest).with("identity", 'payload', '123', Util.empty_filter, 'rspec_agent', 'mcollective', 60).twice
+ PluginManager.expects("[]").with("security_plugin").returns(security).twice
+
+ Config.instance.expects(:identity).returns("identity").twice
+
+ Message.any_instance.expects(:requestid).returns("123").twice
+
+ m = Message.new("payload", "message", :type => :request, :agent => "rspec_agent", :collective => "mcollective")
+ m.encode!
+
+ m = Message.new("payload", "message", :type => :direct_request, :agent => "rspec_agent", :collective => "mcollective")
+ m.encode!
+ end
+
+ it "should retain the requestid if it was specifically set" do
+ security = mock
+ security.expects(:encoderequest).with("identity", 'payload', '123', Util.empty_filter, 'rspec_agent', 'mcollective', 60)
+ PluginManager.expects("[]").with("security_plugin").returns(security)
+
+ Config.instance.expects(:identity).returns("identity")
+
+ m = Message.new("payload", "message", :type => :request, :agent => "rspec_agent", :collective => "mcollective")
+ m.expects(:create_reqid).never
+ m.requestid = "123"
+ m.encode!
+ m.requestid.should == "123"
+ end
+
+ it "should not allow bad callerids when replying" do
+ request = mock
+ request.stubs(:agent).returns("rspec_agent")
+ request.stubs(:collective).returns("collective")
+ request.stubs(:payload).returns({:requestid => "123", :callerid => "caller/id"})
+
+ security = mock
+ security.expects(:valid_callerid?).with("caller/id").returns(false)
+ PluginManager.expects("[]").with("security_plugin").returns(security)
+
+ m = Message.new("payload", "message", :request => request, :type => :reply)
+
+ expect {
+ m.encode!
+ }.to raise_error('callerid in original request is not valid, surpressing reply to potentially forged request')
+ end
+ end
+
+ describe "#decode!" do
+ it "should check for valid types" do
+ expect {
+ m = Message.new("payload", "message", :type => :foo)
+ m.decode!
+ }.to raise_error("Cannot decode message type foo")
+ end
+
+ it "should set state based on decoded message" do
+ msg = mock
+ msg.stubs(:include?).returns(true)
+ msg.stubs("[]").with(:collective).returns("collective")
+ msg.stubs("[]").with(:agent).returns("rspecagent")
+ msg.stubs("[]").with(:filter).returns("filter")
+ msg.stubs("[]").with(:requestid).returns("1234")
+ msg.stubs("[]").with(:ttl).returns(30)
+ msg.stubs("[]").with(:msgtime).returns(1314628987)
+
+ security = mock
+ security.expects(:decodemsg).returns(msg)
+ PluginManager.expects("[]").with("security_plugin").returns(security)
+
+ m = Message.new(msg, "message", :type => :reply)
+ m.decode!
+
+ m.collective.should == "collective"
+ m.agent.should == "rspecagent"
+ m.filter.should == "filter"
+ m.requestid.should == "1234"
+ m.ttl.should == 30
+ end
+
+ it "should not allow bad callerids from the security plugin on requests" do
+ security = mock
+ security.expects(:decodemsg).returns({:callerid => "foo/bar"})
+ security.expects(:valid_callerid?).with("foo/bar").returns(false)
+
+ PluginManager.expects("[]").with("security_plugin").returns(security).twice
+
+ m = Message.new("payload", "message", :type => :request)
+
+ expect {
+ m.decode!
+ }.to raise_error('callerid in request is not valid, surpressing reply to potentially forged request')
+ end
+ end
+
+ describe "#validate" do
+ it "should only validate requests" do
+ m = Message.new("msg", "message", :type => :reply)
+ expect {
+ m.validate
+ }.to raise_error("Can only validate request messages")
+ end
+
+ it "should raise an exception for incorrect messages" do
+ sec = mock
+ sec.expects("validate_filter?").returns(false)
+ PluginManager.expects("[]").with("security_plugin").returns(sec)
+
+ payload = mock
+ payload.expects("[]").with(:filter).returns({})
+
+ m = Message.new(payload, "message", :type => :request)
+ m.instance_variable_set("@msgtime", Time.now.to_i)
+
+ expect {
+ m.validate
+ }.to raise_error(NotTargettedAtUs)
+ end
+
+ it "should pass for good messages" do
+ sec = mock
+ sec.expects(:validate_filter?).returns(true)
+ PluginManager.expects("[]").returns(sec)
+
+ payload = mock
+ payload.expects("[]").with(:filter).returns({})
+ m = Message.new(payload, "message", :type => :request)
+ m.instance_variable_set("@msgtime", Time.now.to_i)
+ m.validate
+ end
+
+ it "should set the @validated property" do
+ sec = mock
+ sec.expects(:validate_filter?).returns(true)
+ PluginManager.expects("[]").returns(sec)
+
+ payload = mock
+ payload.expects("[]").with(:filter).returns({})
+ m = Message.new(payload, "message", :type => :request)
+ m.instance_variable_set("@msgtime", Time.now.to_i)
+
+ m.validated.should == false
+ m.validate
+ m.validated.should == true
+ end
+
+ it "should not validate for messages older than TTL" do
+ stats = mock
+ stats.expects(:ttlexpired).once
+
+ MCollective::PluginManager << {:type => "global_stats", :class => stats}
+
+ m = Message.new({:callerid => "caller", :senderid => "sender"}, "message", :type => :request)
+ m.instance_variable_set("@msgtime", (Time.now.to_i - 120))
+
+ expect {
+ m.validate
+ }.to raise_error(MsgTTLExpired)
+ end
+ end
+
+ describe "#publish" do
+ it "should publish itself to the connector" do
+ m = Message.new("msg", "message", :type => :request)
+
+ connector = mock
+ connector.expects(:publish).with(m)
+ PluginManager.expects("[]").returns(connector)
+
+ m.publish
+ end
+
+ it "should support direct addressing" do
+ m = Message.new("msg", "message", :type => :request)
+ m.discovered_hosts = ["one", "two", "three"]
+
+ Config.instance.stubs(:direct_addressing).returns(true)
+ Config.instance.stubs(:direct_addressing_threshold).returns(10)
+
+ connector = mock
+ connector.expects(:publish).with(m)
+ PluginManager.expects("[]").returns(connector)
+
+ m.publish
+ m.type.should == :direct_request
+ end
+
+ it "should only direct publish below the configured threshold" do
+ m = Message.new("msg", "message", :type => :request)
+ m.discovered_hosts = ["one", "two", "three"]
+
+ Config.instance.expects(:direct_addressing).returns(true)
+ Config.instance.expects(:direct_addressing_threshold).returns(1)
+
+ connector = mock
+ connector.expects(:publish).with(m)
+ PluginManager.expects("[]").returns(connector)
+
+ m.publish
+ m.type.should == :request
+ end
+ end
+
+ describe "#create_reqid" do
+ it "should create a valid request id" do
+ m = Message.new("msg", "message", :agent => "rspec", :collective => "mc")
+
+ SSL.expects(:uuid).returns("reqid")
+
+ m.create_reqid.should == "reqid"
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Optionparser do
+ describe "#initialize" do
+ it "should store the included list as an array" do
+ parser = Optionparser.new({}, "included")
+ parser.instance_variable_get("@include").should == ["included"]
+
+ parser = Optionparser.new({}, ["included"])
+ parser.instance_variable_get("@include").should == ["included"]
+ end
+
+ it "should store the excluded list as an array" do
+ parser = Optionparser.new({}, "", "excluded")
+ parser.instance_variable_get("@exclude").should == ["excluded"]
+
+ parser = Optionparser.new({}, "", ["excluded"])
+ parser.instance_variable_get("@exclude").should == ["excluded"]
+ end
+
+ it "should gather default options" do
+ Util.expects(:default_options).returns({})
+ Optionparser.new({})
+ end
+
+ it "should merge supplied options with defaults" do
+ defaults = {}
+ supplied = {}
+
+ Util.expects(:default_options).returns(defaults)
+ defaults.expects(:merge!).with(supplied)
+
+ Optionparser.new(supplied)
+ end
+ end
+
+ describe "#parse" do
+ it "should yield to the caller" do
+ parser = Optionparser.new(defaults={:default => 1})
+
+ block_ran = false
+
+ parser.parse do |p, o|
+ p.class.should == OptionParser
+ o.should == Util.default_options.merge(defaults)
+ block_ran = true
+ end
+
+ block_ran.should == true
+ end
+
+ it "should add required options" do
+ parser = Optionparser.new(defaults={:default => 1})
+ parser.expects(:add_required_options)
+ parser.parse
+ end
+
+ it "should optionally add common options" do
+ parser = Optionparser.new(defaults={:default => 1})
+ parser.stubs(:add_required_options)
+ parser.expects(:add_common_options)
+ parser.parse
+
+ parser = Optionparser.new(defaults={:default => 1}, "", "common")
+ parser.stubs(:add_required_options)
+ parser.expects(:add_common_options).never
+ parser.parse
+ end
+
+ it "should support adding arbitrary named sections of options" do
+ parser = Optionparser.new(defaults={:default => 1}, "filter")
+ parser.stubs(:add_required_options)
+ parser.stubs(:add_common_options)
+ parser.expects(:add_filter_options)
+ parser.parse
+ end
+
+ it "should support excluding sections that was specifically included" do
+ parser = Optionparser.new(defaults={:default => 1}, "filter", "filter")
+ parser.stubs(:add_required_options)
+ parser.stubs(:add_common_options)
+ parser.expects(:add_filter_options).never
+ parser.parse
+ end
+
+ it "should parse MCOLLECTIVE_EXTRA_OPTS" do
+ ENV["MCOLLECTIVE_EXTRA_OPTS"] = "--dt 999"
+ @parser = Optionparser.new
+ @parser.parse[:disctimeout].should == 999
+ ENV.delete("MCOLLECTIVE_EXTRA_OPTS")
+ end
+
+ it "should not set the active collective from the config class if given on the cli" do
+ parser = Optionparser.new(defaults={:collective => "rspec"})
+ parser.stubs(:add_required_options)
+ parser.stubs(:add_common_options)
+ Config.instance.expects(:main_collective).never
+ parser.parse
+ end
+
+ it "should set the active collective from the config class if not given on the cli" do
+ parser = Optionparser.new(defaults={})
+ parser.stubs(:add_required_options)
+ parser.stubs(:add_common_options)
+ Config.instance.expects(:main_collective).returns(:rspec).once
+ parser.parse[:collective].should == :rspec
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe PluginManager do
+ before do
+ class MCollective::Foo; end
+
+ PluginManager.pluginlist.each {|p| PluginManager.delete p}
+ end
+
+ describe "#<<" do
+ it "should store a plugin by name" do
+ PluginManager << {:type => "foo", :class => "MCollective::Foo"}
+ PluginManager.instance_variable_get("@plugins").include?("foo").should == true
+ end
+
+ it "should store a plugin instance" do
+ f = MCollective::Foo.new
+
+ PluginManager << {:type => "foo", :class => f}
+ PluginManager.instance_variable_get("@plugins")["foo"][:instance].object_id.should == f.object_id
+ end
+
+ it "should detect duplicate plugins" do
+ PluginManager << {:type => "foo", :class => "MCollective::Foo"}
+
+ expect {
+ PluginManager << {:type => "foo", :class => "MCollective::Foo"}
+ }.to raise_error("Plugin foo already loaded")
+ end
+
+ it "should store single instance preference correctly" do
+ PluginManager << {:type => "foo", :class => "MCollective::Foo", :single_instance => false}
+ PluginManager.instance_variable_get("@plugins")["foo"][:single].should == false
+ end
+
+ it "should always set single to true when supplied an instance" do
+ PluginManager << {:type => "foo", :class => MCollective::Foo.new, :single_instance => false}
+ PluginManager.instance_variable_get("@plugins")["foo"][:single].should == true
+ end
+ end
+
+ describe "#delete" do
+ it "should remove plugins" do
+ PluginManager << {:type => "foo", :class => MCollective::Foo.new}
+ PluginManager.instance_variable_get("@plugins").include?("foo").should == true
+ PluginManager.delete("foo")
+ PluginManager.instance_variable_get("@plugins").include?("foo").should == false
+ end
+ end
+
+ describe "#include?" do
+ it "should correctly check if plugins were added" do
+ PluginManager << {:type => "foo", :class => MCollective::Foo.new}
+ PluginManager.include?("foo").should == true
+ PluginManager.include?("bar").should == false
+ end
+ end
+
+ describe "#pluginlist" do
+ it "should return the correct list of plugins" do
+ PluginManager << {:type => "foo", :class => MCollective::Foo.new}
+ PluginManager << {:type => "bar", :class => MCollective::Foo.new}
+
+ PluginManager.pluginlist.sort.should == ["bar", "foo"]
+ end
+ end
+
+ describe "#[]" do
+ it "should detect if the requested plugin does not exist" do
+ expect {
+ PluginManager["foo"]
+ }.to raise_error("No plugin foo defined")
+ end
+
+ it "should create new instances on demand" do
+ PluginManager << {:type => "foo", :class => "MCollective::Foo"}
+ PluginManager["foo"].class.should == MCollective::Foo
+ end
+
+ it "should return the cached instance" do
+ f = MCollective::Foo.new
+
+ PluginManager << {:type => "foo", :class => f}
+ PluginManager["foo"].object_id.should == f.object_id
+ end
+
+ it "should create new instances on every request if requested" do
+ PluginManager << {:type => "foo", :class => "MCollective::Foo", :single_instance => false}
+ PluginManager["foo"].object_id.should_not == PluginManager["foo"].object_id
+ end
+ end
+
+ describe "#find" do
+ before do
+ @config.stubs(:libdir).returns(["/libdir/"])
+ Config.stubs(:instance).returns(@config)
+ end
+
+ it "should find all plugins in configured libdirs" do
+ File.expects(:join).with(["/libdir/", "mcollective", "test"]).returns("/plugindir/")
+ File.expects(:directory?).with("/plugindir/").returns(true)
+ Dir.expects(:new).with("/plugindir/").returns(["plugin.rb"])
+ PluginManager.find("test").should == ["plugin"]
+ end
+
+ it "should find all plugins with a given file extension" do
+ File.expects(:join).with(["/libdir/", "mcollective", "test"]).returns("/plugindir/")
+ File.expects(:directory?).with("/plugindir/").returns(true)
+ Dir.expects(:new).with("/plugindir/").returns(["plugin.ddl"])
+ PluginManager.find("test", "ddl").should == ["plugin"]
+ end
+
+ it "should skip libdirs that do not have the plugin type directories" do
+ @config.stubs(:libdir).returns(["/plugindir/", "/tmp/"])
+ File.expects(:join).with(["/plugindir/", "mcollective", "test"]).returns("/plugindir/")
+ File.expects(:join).with(["/tmp/", "mcollective", "test"]).returns("/tmpdir/")
+ File.expects(:directory?).with("/plugindir/").returns(true)
+ File.expects(:directory?).with("/tmpdir/").returns(false)
+ Dir.expects(:new).with("/plugindir/").returns(["plugin.ddl"])
+ PluginManager.find("test", "ddl").should == ["plugin"]
+ end
+ end
+
+ describe "#find_and_load" do
+ before do
+ @config.stubs(:libdir).returns(["/libdir/"])
+ Config.stubs(:instance).returns(@config)
+ PluginManager.expects(:loadclass).with("MCollective::Test::Testplugin", true)
+ end
+
+ it "should find and load all plugins from all libdirs that match the type" do
+ PluginManager.expects(:find).with("test", ".rb").returns(["testplugin"])
+ PluginManager.find_and_load("test")
+ end
+
+ it "should exclude plugins who do not match criteria if block is given" do
+ PluginManager.expects(:find).with("test", ".rb").returns(["testplugin", "failplugin"])
+ PluginManager.find_and_load("test") {|plugin| plugin.match(/^test/)}
+ end
+ end
+
+ describe "#loadclass" do
+ it "should load the correct filename given a ruby class name" do
+ PluginManager.stubs(:load).with("mcollective/foo.rb").once
+ PluginManager.loadclass("MCollective::Foo")
+ end
+
+ it "should raise errors for load errors" do
+ PluginManager.stubs(:load).raises("load failure")
+ Log.expects(:error)
+ expect { PluginManager.loadclass("foo") }.to raise_error(/load failure/)
+ end
+
+ it "should support squashing load errors" do
+ PluginManager.stubs(:load).raises("load failure")
+ Log.expects(:error)
+ PluginManager.loadclass("foo", true)
+ end
+ end
+
+ describe "#grep" do
+ it "should return matching plugins from the list" do
+ PluginManager << {:type => "foo", :class => MCollective::Foo.new}
+ PluginManager << {:type => "bar", :class => MCollective::Foo.new}
+
+ PluginManager.grep(/oo/).should == ["foo"]
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module PluginPackager
+ describe AgentDefinition do
+ before :each do
+ PluginPackager.expects(:get_metadata).once.returns({:name => "foo", :version => 1})
+ end
+
+ describe "#initialize" do
+
+ before do
+ AgentDefinition.any_instance.expects(:common)
+ end
+
+ it "should replace spaces in the package name with dashes" do
+ agent = AgentDefinition.new(".", "test package", nil, nil, nil, nil, [], {}, "agent")
+ agent.metadata[:name].should == "test-package"
+ end
+
+ it "should set dependencies if present" do
+ agent = AgentDefinition.new(".", "test-package", nil, nil, nil, nil, [:name => "foo", :version => nil], {}, "agent")
+ agent.dependencies.should == [{:name => "foo", :version => nil}, {:name => "mcollective-common", :version => nil}]
+ end
+
+ it "should set mc name and version" do
+ agent = AgentDefinition.new(".", "test-package", nil, nil, nil, nil, [], {:mcname =>"pe-mcollective-common", :mcversion =>"1.2"}, "agent")
+ agent.mcname.should == "pe-mcollective-common"
+ agent.mcversion.should == "1.2"
+ end
+
+ it "should replace underscored with dashes in the name" do
+ agent = AgentDefinition.new(".", "test_package", nil, nil, nil, nil, [], {:mcname =>"pe-mcollective-common", :mcversion =>"1.2"}, "agent")
+ agent.metadata[:name].should == "test-package"
+ end
+
+ it "should replace whitespaces with a single dash in the name" do
+ agent = AgentDefinition.new(".", "test package", nil, nil, nil, nil, [], {:mcname =>"pe-mcollective-common", :mcversion =>"1.2"}, "agent")
+ agent.metadata[:name].should == "test-package"
+ end
+ end
+
+ describe "#identify_packages" do
+ it "should attempt to identify all agent packages" do
+ AgentDefinition.any_instance.expects(:common).once.returns(:check)
+ AgentDefinition.any_instance.expects(:agent).once.returns(:check)
+ AgentDefinition.any_instance.expects(:client).once.returns(:check)
+
+ agent = AgentDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "agent")
+ agent.packagedata[:common].should == :check
+ agent.packagedata[:agent].should == :check
+ agent.packagedata[:client].should == :check
+ end
+ end
+
+ describe "#agent" do
+ before do
+ AgentDefinition.any_instance.expects(:client).once
+ end
+
+ it "should not populate the agent files if the agent directory is empty" do
+ AgentDefinition.any_instance.expects(:common).returns(nil)
+ PluginPackager.expects(:check_dir_present).returns(false)
+ agent = AgentDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "agent")
+ agent.packagedata[:agent].should == nil
+ end
+
+ it "should add the agent file if the agent directory and implementation file is present" do
+ AgentDefinition.any_instance.expects(:common).returns(nil)
+ PluginPackager.stubs(:check_dir_present).returns(true)
+ File.stubs(:join).with(".", "agent").returns("tmpdir/agent")
+ File.stubs(:join).with("tmpdir/agent", "*.ddl").returns("tmpdir/agent/*.ddl")
+ File.stubs(:join).with("tmpdir/agent", "*").returns("tmpdir/agent/*")
+ Dir.stubs(:glob).with("tmpdir/agent/*.ddl").returns([])
+ Dir.stubs(:glob).with("tmpdir/agent/*").returns(["implementation.rb"])
+
+ agent = AgentDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "agent")
+ agent.packagedata[:agent][:files].should == ["implementation.rb"]
+ end
+
+ it "should add common package as dependency if present" do
+ AgentDefinition.any_instance.expects(:common).returns({:files=> ["test.rb"]})
+ PluginPackager.expects(:check_dir_present).with("tmpdir/agent").returns(true)
+ File.stubs(:join).returns("/tmp")
+ File.stubs(:join).with(".", "agent").returns("tmpdir/agent")
+ File.stubs(:join).with(".", "aggregate").returns("tmpdir/aggregate")
+ Dir.stubs(:glob).returns([])
+
+ agent = AgentDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "agent")
+ agent.packagedata[:agent][:dependencies].should == [{:name => "mcollective-common", :version => nil}]
+ agent.packagedata[:agent][:plugindependency].should == {:name => "mcollective-foo-common", :version =>1, :iteration => 1}
+ end
+ end
+
+ describe "#common" do
+ it "should populate the common files if there are any" do
+ AgentDefinition.any_instance.expects(:agent)
+ AgentDefinition.any_instance.expects(:client)
+ File.expects(:join).with(".", "data", "**").returns("datadir")
+ File.expects(:join).with(".", "util", "**", "**").returns("utildir")
+ File.expects(:join).with(".", "agent", "*.ddl").returns("ddldir")
+ File.expects(:join).with(".", "validator", "**").returns("validatordir")
+ Dir.expects(:glob).with("datadir").returns(["data.rb"])
+ Dir.expects(:glob).with("utildir").returns(["util.rb"])
+ Dir.expects(:glob).with("validatordir").returns(["validator.rb"])
+ Dir.expects(:glob).with("ddldir").returns(["agent.ddl"])
+
+ common = AgentDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "agent")
+ common.packagedata[:common][:files].should == ["data.rb", "util.rb", "validator.rb", "agent.ddl"]
+ end
+
+ it "should raise an exception if the ddl file isn't present" do
+ File.expects(:join).with(".", "data", "**").returns("datadir")
+ File.expects(:join).with(".", "util", "**", "**").returns("utildir")
+ File.expects(:join).with(".", "agent", "*.ddl").returns("ddldir")
+ File.expects(:join).with(".", "agent").returns("ddldir")
+ File.expects(:join).with(".", "validator", "**").returns("validatordir")
+ Dir.expects(:glob).with("datadir").returns(["data.rb"])
+ Dir.expects(:glob).with("utildir").returns(["util.rb"])
+ Dir.expects(:glob).with("validatordir").returns(["validator.rb"])
+ Dir.expects(:glob).with("ddldir").returns([])
+
+ expect{
+ common = AgentDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "agent")
+ }.to raise_error(RuntimeError, "cannot create package - No ddl file found in ddldir")
+ end
+ end
+
+ describe "#client" do
+ before do
+ AgentDefinition.any_instance.expects(:agent).returns(nil)
+ File.expects(:join).with(".", "application").returns("clientdir")
+ File.expects(:join).with(".", "aggregate").returns("aggregatedir")
+ end
+
+ it "should populate client files if all directories are present" do
+ AgentDefinition.any_instance.expects(:common).returns(nil)
+ PluginPackager.expects(:check_dir_present).times(2).returns(true)
+ File.expects(:join).with("clientdir", "*").returns("clientdir/*")
+ File.expects(:join).with("aggregatedir", "*").returns("aggregatedir/*")
+ Dir.expects(:glob).with("clientdir/*").returns(["client.rb"])
+ Dir.expects(:glob).with("aggregatedir/*").returns(["aggregate.rb"])
+
+ client = AgentDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "agent")
+ client.packagedata[:client][:files].should == ["client.rb", "aggregate.rb"]
+ end
+
+ it "should not populate client files if directories are not present" do
+ AgentDefinition.any_instance.expects(:common).returns(nil)
+ PluginPackager.expects(:check_dir_present).times(2).returns(false)
+
+ client = AgentDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "agent")
+ client.packagedata[:client].should == nil
+ end
+
+ it "should add common package as dependency if present" do
+ AgentDefinition.any_instance.expects(:common).returns("common")
+ PluginPackager.expects(:check_dir_present).times(2).returns(true)
+ File.expects(:join).with("clientdir", "*").returns("clientdir/*")
+ File.expects(:join).with("aggregatedir", "*").returns("aggregatedir/*")
+ Dir.expects(:glob).with("clientdir/*").returns(["client.rb"])
+ Dir.expects(:glob).with("aggregatedir/*").returns(["aggregate.rb"])
+
+ client = AgentDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "agent")
+ client.packagedata[:client][:dependencies].should == [{:name => "mcollective-common", :version => nil}]
+ client.packagedata[:client][:plugindependency].should == {:name => "mcollective-foo-common", :version => 1, :iteration => 1}
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module PluginPackager
+ describe StandardDefinition do
+ before :each do
+ PluginPackager.expects(:get_metadata).once.returns({:name => "foo", :version => 1})
+ end
+
+ describe "#initialize" do
+ it "should replace spaces in the package name with dashes" do
+ plugin = StandardDefinition.new(".", "test plugin", nil, nil, nil, nil, [], {}, "testplugin")
+ plugin.metadata[:name].should == "test-plugin"
+ end
+
+ it "should set dependencies if present" do
+ plugin = StandardDefinition.new(".", "test plugin", nil, nil, nil, nil, [{:name => "foo", :version => nil}], {}, "testplugin")
+ plugin.dependencies.should == [{:name => "foo", :version => nil},
+ {:name => "mcollective-common", :version => nil}]
+ end
+
+ it "should set mc name and version dependencies" do
+ plugin = StandardDefinition.new(".", "test plugin", nil, nil, nil, nil, [], {:mcname => "pe-mcollective", :mcversion => "1"}, "testplugin")
+ plugin.mcname.should == "pe-mcollective"
+ plugin.mcversion.should == "1"
+ end
+
+ it "should replace underscores with dashes in the name" do
+ plugin = StandardDefinition.new(".", "test_plugin", nil, nil, nil, nil, [], {:mcname => "pe-mcollective", :mcversion => "1"}, "testplugin")
+ plugin.metadata[:name].should == "test-plugin"
+ end
+
+ it "should replace whitespaces with a single dash in the name" do
+ plugin = StandardDefinition.new(".", "test plugin", nil, nil, nil, nil, [], {:mcname => "pe-mcollective", :mcversion => "1"}, "testplugin")
+ plugin.metadata[:name].should == "test-plugin"
+ end
+ end
+
+ describe "#identify_packages" do
+ it "should attempt to identify all packages" do
+ StandardDefinition.any_instance.expects(:common).once.returns(:check)
+ StandardDefinition.any_instance.expects(:plugin).once.returns(:check)
+
+ plugin = StandardDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "testplugin")
+ plugin.packagedata[:common].should == :check
+ plugin.packagedata["testplugin"].should == :check
+ end
+ end
+
+ describe "#plugin" do
+
+ it "should return nil if the plugin doesn't contain any files" do
+ StandardDefinition.any_instance.expects(:common).returns(nil)
+ PluginPackager.expects(:check_dir_present).returns(false)
+ plugin = StandardDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "testplugin")
+ plugin.packagedata["testplugin"].should == nil
+ end
+
+ it "should add plugin files to the file list" do
+ StandardDefinition.any_instance.expects(:common).returns(nil)
+ PluginPackager.expects(:check_dir_present).returns(true)
+ Dir.expects(:glob).with("./testplugin/*").returns(["file.rb"])
+ plugin = StandardDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "testplugin")
+ plugin.packagedata["testplugin"][:files].should == ["file.rb"]
+ end
+
+ it "should add common package as dependency if present" do
+ StandardDefinition.any_instance.expects(:common).returns(true)
+ PluginPackager.expects(:check_dir_present).returns(true)
+ Dir.expects(:glob).with("./testplugin/*").returns(["file.rb"])
+ plugin = StandardDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "testplugin")
+ plugin.packagedata["testplugin"][:files].should == ["file.rb"]
+ plugin.packagedata["testplugin"][:dependencies].should == [{:name => "mcollective-common", :version => nil}]
+ plugin.packagedata["testplugin"][:plugindependency].should == {:name => "mcollective-foo-common", :version => 1, :iteration => 1}
+ end
+ end
+
+ describe "#common" do
+ before do
+ StandardDefinition.any_instance.expects(:plugin).returns(false)
+ end
+
+ it "should return nil if common doesn't contain any files" do
+ PluginPackager.expects(:check_dir_present).returns(false)
+ plugin = StandardDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "testplugin")
+ plugin.packagedata[:common].should == nil
+ end
+
+ it "should add common files to the file list" do
+ PluginPackager.expects(:check_dir_present).returns(true)
+ Dir.expects(:glob).with("./util/*").returns(["common.rb"])
+ plugin = StandardDefinition.new(".", nil, nil, nil, nil, nil, [], {}, "testplugin")
+ plugin.packagedata[:common][:files].should == ["common.rb"]
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/enn rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe PluginPackager do
+ describe "#load_packagers" do
+ it "should load all PluginPackager plugins" do
+ PluginManager.expects(:find_and_load).with("pluginpackager")
+ PluginPackager.load_packagers
+ end
+ end
+
+ describe "#[]" do
+ it "should return the correct class" do
+ PluginPackager.expects(:const_get).with("Foo").returns(:foo)
+ result = PluginPackager["Foo"]
+ result.should == :foo
+ end
+
+ it "should do something else" do
+ expect{
+ PluginPackager["Bar"]
+ }.to raise_error(NameError, 'uninitialized constant MCollective::PluginPackager::Bar')
+ end
+ end
+
+ describe "#get_metadata" do
+ it "should raise an exception if the ddl file can't be loaded" do
+ DDL.expects(:new).with("package", :foo, false)
+ File.stubs(:join)
+ Dir.stubs(:glob).returns('')
+ expect{
+ PluginPackager.get_metadata("/tmp", "foo")
+ }.to raise_error(RuntimeError)
+ end
+
+ it "should load the ddl file and return the metadata" do
+ ddl = mock
+ DDL.expects(:new).with("package", :foo, false).returns(ddl)
+ File.stubs(:join)
+ Dir.stubs(:glob).returns(["foo.ddl"])
+ File.expects(:read).with("foo.ddl").returns("foo_ddl")
+ ddl.expects(:instance_eval).with("foo_ddl")
+ ddl.expects(:meta).returns("metadata")
+ ddl.expects(:requirements).returns({:mcollective => 1})
+
+ meta, requirements = PluginPackager.get_metadata("/tmp", "foo")
+ meta.should == "metadata"
+ requirements.should == 1
+ end
+ end
+
+ describe "#check_dir_present" do
+ it "should return true if the directory is present and not empty" do
+ File.expects(:directory?).with("/tmp").returns(true)
+ File.expects(:join).with("/tmp", "*")
+ Dir.expects(:glob).returns([1])
+ result = PluginPackager.check_dir_present("/tmp")
+ result.should == true
+ end
+
+ it "should return false if the directory is not present" do
+ File.expects(:directory?).with("/tmp").returns(false)
+ result = PluginPackager.check_dir_present("/tmp")
+ result.should == false
+ end
+
+ it "should return false if the direcotry is present but empty" do
+ File.expects(:directory?).with("/tmp").returns(true)
+ File.expects(:join).with("/tmp", "*")
+ Dir.expects(:glob).returns([])
+ result = PluginPackager.check_dir_present("/tmp")
+ result.should == false
+ end
+ end
+
+ describe "#do_quietly?" do
+ it "should call the block parameter if verbose is true" do
+ result = PluginPackager.do_quietly?(true) {:success}
+ result.should == :success
+ end
+
+ it "should call the block parameter quietly if verbose is false" do
+ std_out = Tempfile.new("mc_pluginpackager_spec")
+ File.expects(:new).with("/dev/null", "w").returns(std_out)
+ PluginPackager.do_quietly?(false) {puts "success"}
+ std_out.rewind
+ std_out.read.should == "success\n"
+ std_out.close
+ std_out.unlink
+ end
+
+ it "should raise an exception and reset stdout if the block raises an execption" do
+ expect{
+ PluginPackager.do_quietly?(false) {raise Exception, "exception"}
+ }.to raise_error(Exception, "exception")
+ end
+ end
+
+ describe "#build_tool?" do
+ it "should return true if the given build tool is present on the system" do
+ File.expects(:join).returns("foo")
+ File.expects(:exists?).with("foo").returns(true)
+ result = PluginPackager.build_tool?("foo")
+ result.should == true
+ end
+
+ it "should return false if the given build tool is not present on the system" do
+ File.stubs(:join).returns("foo")
+ File.stubs(:exists?).with("foo").returns(false)
+ result = PluginPackager.build_tool?("foo")
+ result.should == false
+ end
+ end
+
+ describe "#safe_system" do
+ it "should not raise any exceptions if a command ran" do
+ PluginPackager.expects(:system).with("foo").returns(true)
+ lambda{PluginPackager.safe_system("foo")}.should_not raise_error
+ end
+
+ it "should raise a RuntimeError if command cannot be run" do
+ PluginPackager.expects(:system).with("foo").returns(false)
+ expect{
+ PluginPackager.safe_system("foo")
+ }.to raise_error(RuntimeError, "Failed: foo")
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+require File.dirname(__FILE__) + "/../../../../../plugins/mcollective/aggregate/average.rb"
+
+module MCollective
+ class Aggregate
+ describe Average do
+ describe "#startup_hook" do
+ it "should set the correct result hash" do
+ result = Average.new(:test, [], "%d", :test_action)
+ result.result.should == {:value => 0, :type => :numeric, :output => :test}
+ result.aggregate_format.should == "%d"
+ end
+
+ it "should set a defauly aggregate_format if one isn't defined" do
+ result = Average.new(:test, [], nil, :test_action)
+ result.aggregate_format.should == "Average of test: %f"
+ end
+ end
+
+ describe "#process_result" do
+ it "should add the reply value to the result hash" do
+ average = Average.new([:test], [], "%d", :test_action)
+ average.process_result(1, {:test => 1})
+ average.result[:value].should == 1
+ end
+ end
+
+ describe "#summarize" do
+ it "should calculate the average and return a result class" do
+ result_obj = mock
+ result_obj.stubs(:new).returns(:success)
+
+ average = Average.new([:test], [], "%d", :test_action)
+ average.process_result(10, {:test => 10})
+ average.process_result(20, {:test => 20})
+ average.stubs(:result_class).returns(result_obj)
+ average.summarize.should == :success
+ average.result[:value].should == 15
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+require File.dirname(__FILE__) + "/../../../../../plugins/mcollective/aggregate/sum.rb"
+
+module MCollective
+ class Aggregate
+ describe Sum do
+ describe "#startup_hook" do
+ it "should set the correct result hash" do
+ result = Sum.new(:test, [], "%d", :test_action)
+ result.result.should == {:value => 0, :type => :numeric, :output => :test}
+ result.aggregate_format.should == "%d"
+ end
+
+ it "should set a defauly aggregate_format if one isn't defined" do
+ result = Sum.new(:test, [], nil, :test_action)
+ result.aggregate_format.should == "Sum of test: %f"
+ end
+ end
+
+ describe "#process_result" do
+ it "should add the reply value to the result hash" do
+ average = Sum.new([:test], [], "%d", :test_action)
+ average.process_result(1, {:test => 1})
+ average.result[:value].should == 1
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+require File.dirname(__FILE__) + "/../../../../../plugins/mcollective/aggregate/summary.rb"
+
+module MCollective
+ class Aggregate
+ describe Summary do
+ describe "#startup_hook" do
+ it "should set the correct result hash" do
+ result = Summary.new(:test, [], "%d", :test_action)
+ result.result.should == {:value => {}, :type => :collection, :output => :test}
+ result.aggregate_format.should == "%d"
+ end
+
+ it "should set a defauly aggregate_format if one isn't defined" do
+ result = Summary.new(:test, [], nil, :test_action)
+ result.aggregate_format.should == :calculate
+ end
+ end
+
+ describe "#process_result" do
+ it "should add the value to the result hash" do
+ sum = Summary.new(:test, [], "%d", :test_action)
+ sum.process_result(:foo, {:test => :foo})
+ sum.result[:value].should == {:foo => 1}
+ end
+
+ it "should add the reply values to the result hash if value is an array" do
+ sum = Summary.new(:test, [], "%d", :test_action)
+ sum.process_result([:foo, :foo, :bar], {:test => [:foo, :foo, :bar]})
+ sum.result[:value].should == {:foo => 2, :bar => 1}
+ end
+ end
+
+ describe "#summarize" do
+ it "should calculate an attractive format" do
+ sum = Summary.new(:test, [], nil, :test_action)
+ sum.result[:value] = {"shrt" => 1, "long key" => 1}
+ sum.summarize.aggregate_format.should == "%8s = %s"
+ end
+
+ it "should calculate an attractive format when result type is not a string" do
+ sum1 = Summary.new(:test, [], nil, :test_action)
+ sum1.result[:value] = {true => 4, false => 5}
+ sum1.summarize.aggregate_format.should == "%5s = %s"
+
+ sum2 = Summary.new(:test, [], nil, :test_action)
+ sum2.result[:value] = {1 => 1, 10 => 2}
+ sum2.summarize.aggregate_format.should == "%2s = %s"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+MCollective::PluginManager.clear
+
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/connector/activemq.rb'
+
+# create the stomp error class here as it does not always exist
+# all versions of the stomp gem and we do not want to tie tests
+# to the stomp gem
+module Stomp
+ module Error
+ class DuplicateSubscription < RuntimeError; end
+ end
+end
+
+module MCollective
+ module Connector
+ describe Activemq do
+ before do
+ unless ::Stomp::Error.constants.map{|c| c.to_s}.include?("NoCurrentConnection")
+ class ::Stomp::Error::NoCurrentConnection < RuntimeError ; end
+ end
+
+ @config = mock
+ @config.stubs(:configured).returns(true)
+ @config.stubs(:identity).returns("rspec")
+ @config.stubs(:collectives).returns(["mcollective"])
+
+ logger = mock
+ logger.stubs(:log)
+ logger.stubs(:start)
+ Log.configure(logger)
+
+ Config.stubs(:instance).returns(@config)
+
+ @msg = mock
+ @msg.stubs(:base64_encode!)
+ @msg.stubs(:payload).returns("msg")
+ @msg.stubs(:agent).returns("agent")
+ @msg.stubs(:type).returns(:reply)
+ @msg.stubs(:collective).returns("mcollective")
+
+ @subscription = mock
+ @subscription.stubs("<<").returns(true)
+ @subscription.stubs("include?").returns(false)
+ @subscription.stubs("delete").returns(false)
+
+ @connection = mock
+ @connection.stubs(:subscribe).returns(true)
+ @connection.stubs(:unsubscribe).returns(true)
+
+ @c = Activemq.new
+ @c.instance_variable_set("@subscriptions", @subscription)
+ @c.instance_variable_set("@connection", @connection)
+ end
+
+ describe "#initialize" do
+ it "should set the @config variable" do
+ c = Activemq.new
+ c.instance_variable_get("@config").should == @config
+ end
+
+ it "should set @subscriptions to an empty list" do
+ c = Activemq.new
+ c.instance_variable_get("@subscriptions").should == []
+ end
+ end
+
+ describe "#connect" do
+ it "should not try to reconnect if already connected" do
+ Log.expects(:debug).with("Already connection, not re-initializing connection").once
+ @c.connect
+ end
+
+ it "should support new style config" do
+ pluginconf = {"activemq.pool.size" => "2",
+ "activemq.pool.1.host" => "host1",
+ "activemq.pool.1.port" => "6163",
+ "activemq.pool.1.user" => "user1",
+ "activemq.pool.1.password" => "password1",
+ "activemq.pool.1.ssl" => "false",
+ "activemq.pool.2.host" => "host2",
+ "activemq.pool.2.port" => "6164",
+ "activemq.pool.2.user" => "user2",
+ "activemq.pool.2.password" => "password2",
+ "activemq.pool.2.ssl" => "true",
+ "activemq.pool.2.ssl.fallback" => "true",
+ "activemq.initial_reconnect_delay" => "0.02",
+ "activemq.max_reconnect_delay" => "40",
+ "activemq.use_exponential_back_off" => "false",
+ "activemq.back_off_multiplier" => "3",
+ "activemq.max_reconnect_attempts" => "5",
+ "activemq.randomize" => "true",
+ "activemq.backup" => "true",
+ "activemq.timeout" => "1",
+ "activemq.connect_timeout" => "5"}
+
+
+ ENV.delete("STOMP_USER")
+ ENV.delete("STOMP_PASSWORD")
+
+ @config.expects(:pluginconf).returns(pluginconf).at_least_once
+
+ Activemq::EventLogger.expects(:new).returns("logger")
+
+ connector = mock
+ connector.expects(:new).with(:backup => true,
+ :back_off_multiplier => 3,
+ :max_reconnect_delay => 40.0,
+ :timeout => 1,
+ :connect_timeout => 5,
+ :use_exponential_back_off => false,
+ :max_reconnect_attempts => 5,
+ :initial_reconnect_delay => 0.02,
+ :randomize => true,
+ :reliable => true,
+ :logger => "logger",
+ :hosts => [{:passcode => 'password1',
+ :host => 'host1',
+ :port => 6163,
+ :ssl => false,
+ :login => 'user1'},
+ {:passcode => 'password2',
+ :host => 'host2',
+ :port => 6164,
+ :ssl => true,
+ :login => 'user2'}
+ ])
+
+ @c.expects(:ssl_parameters).with(2, true).returns(true)
+
+ @c.instance_variable_set("@connection", nil)
+ @c.connect(connector)
+ end
+ end
+
+ describe "#ssl_paramaters" do
+ it "should ensure all settings are provided" do
+ pluginconf = {"activemq.pool.1.host" => "host1",
+ "activemq.pool.1.port" => "6164",
+ "activemq.pool.1.user" => "user1",
+ "activemq.pool.1.password" => "password1",
+ "activemq.pool.1.ssl" => "true",
+ "activemq.pool.1.ssl.cert" => "rspec"}
+
+ @config.expects(:pluginconf).returns(pluginconf).at_least_once
+
+ expect { @c.ssl_parameters(1, false) }.to raise_error("cert, key and ca has to be supplied for verified SSL mode")
+ end
+
+ it "should verify the ssl files exist" do
+ pluginconf = {"activemq.pool.1.host" => "host1",
+ "activemq.pool.1.port" => "6164",
+ "activemq.pool.1.user" => "user1",
+ "activemq.pool.1.password" => "password1",
+ "activemq.pool.1.ssl" => "true",
+ "activemq.pool.1.ssl.cert" => "rspec.cert",
+ "activemq.pool.1.ssl.key" => "rspec.key",
+ "activemq.pool.1.ssl.ca" => "rspec1.ca,rspec2.ca"}
+
+ @config.expects(:pluginconf).returns(pluginconf).at_least_once
+
+ File.expects(:exist?).with("rspec.cert").twice.returns(true)
+ File.expects(:exist?).with("rspec.key").twice.returns(true)
+ File.expects(:exist?).with("rspec1.ca").twice.returns(true)
+ File.expects(:exist?).with("rspec2.ca").twice.returns(false)
+
+ expect { @c.ssl_parameters(1, false) }.to raise_error("Cannot find CA file rspec2.ca")
+
+ @c.ssl_parameters(1, true).should == true
+ end
+
+ it "should support fallback mode when there are errors" do
+ pluginconf = {"activemq.pool.1.host" => "host1",
+ "activemq.pool.1.port" => "6164",
+ "activemq.pool.1.user" => "user1",
+ "activemq.pool.1.password" => "password1",
+ "activemq.pool.1.ssl" => "true"}
+
+ @config.expects(:pluginconf).returns(pluginconf).at_least_once
+
+ @c.ssl_parameters(1, true).should == true
+ end
+
+ it "should fail if fallback isnt enabled" do
+ pluginconf = {"activemq.pool.1.host" => "host1",
+ "activemq.pool.1.port" => "6164",
+ "activemq.pool.1.user" => "user1",
+ "activemq.pool.1.password" => "password1",
+ "activemq.pool.1.ssl" => "true"}
+
+ @config.expects(:pluginconf).returns(pluginconf).at_least_once
+
+ expect { @c.ssl_parameters(1, false) }.to raise_error
+ end
+ end
+
+ describe "#receive" do
+ it "should receive from the middleware" do
+ payload = mock
+ payload.stubs(:body).returns("msg")
+ payload.stubs(:headers).returns("headers")
+
+ @connection.expects(:receive).returns(payload)
+
+ Message.expects(:new).with("msg", payload, :base64 => true, :headers => "headers").returns("message")
+ @c.instance_variable_set("@base64", true)
+
+ received = @c.receive
+ received.should == "message"
+ end
+
+ it "should sleep and retry if recieving while disconnected" do
+ payload = mock
+ payload.stubs(:body).returns("msg")
+ payload.stubs(:headers).returns("headers")
+
+ Message.stubs(:new).returns("rspec")
+ @connection.expects(:receive).raises(::Stomp::Error::NoCurrentConnection).returns(payload).twice
+ @c.expects(:sleep).with(1)
+
+ @c.receive.should == "rspec"
+ end
+ end
+
+ describe "#publish" do
+ before do
+ @connection.stubs(:publish).with("test", "msg", {}).returns(true)
+ end
+
+ it "should base64 encode a message if configured to do so" do
+ @c.instance_variable_set("@base64", true)
+ @c.expects(:headers_for).returns({})
+ @c.expects(:target_for).returns({:name => "test", :headers => {}})
+ @connection.expects(:publish).with("test", "msg", {})
+ @msg.expects(:base64_encode!)
+
+ @c.publish(@msg)
+ end
+
+ it "should not base64 encode if not configured to do so" do
+ @c.instance_variable_set("@base64", false)
+ @c.expects(:headers_for).returns({})
+ @c.expects(:target_for).returns({:name => "test", :headers => {}})
+ @connection.expects(:publish).with("test", "msg", {})
+ @msg.expects(:base64_encode!).never
+
+ @c.publish(@msg)
+ end
+
+ it "should publish the correct message to the correct target with msgheaders" do
+ @connection.expects(:publish).with("test", "msg", {"test" => "test"}).once
+ @c.expects(:headers_for).returns({"test" => "test"})
+ @c.expects(:target_for).returns({:name => "test", :headers => {}})
+
+ @c.publish(@msg)
+ end
+
+ it "should publish direct messages based on discovered_hosts" do
+ msg = mock
+ msg.stubs(:base64_encode!)
+ msg.stubs(:payload).returns("msg")
+ msg.stubs(:agent).returns("agent")
+ msg.stubs(:collective).returns("mcollective")
+ msg.stubs(:type).returns(:direct_request)
+ msg.expects(:discovered_hosts).returns(["one", "two"])
+
+ @c.expects(:headers_for).with(msg, "one")
+ @c.expects(:headers_for).with(msg, "two")
+ @connection.expects(:publish).with('/queue/mcollective.nodes', 'msg', nil).twice
+
+ @c.publish(msg)
+ end
+ end
+
+ describe "#subscribe" do
+ it "should handle duplicate subscription errors" do
+ @connection.expects(:subscribe).raises(::Stomp::Error::DuplicateSubscription)
+ Log.expects(:error).with(regexp_matches(/already had a matching subscription, ignoring/))
+ @c.subscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should use the make_target correctly" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}})
+ @c.subscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should check for existing subscriptions" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}, :id => "rspec"})
+ @subscription.expects("include?").with("rspec").returns(false)
+ @connection.expects(:subscribe).never
+
+ @c.subscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should subscribe to the middleware" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}, :id => "rspec"})
+ @connection.expects(:subscribe).with("test", {}, "rspec")
+ @c.subscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should add to the list of subscriptions" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}, :id => "rspec"})
+ @subscription.expects("<<").with("rspec")
+ @c.subscribe("test", :broadcast, "mcollective")
+ end
+ end
+
+ describe "#unsubscribe" do
+ it "should use make_target correctly" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}})
+ @c.unsubscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should unsubscribe from the target" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}, :id => "rspec"})
+ @connection.expects(:unsubscribe).with("test", {}, "rspec").once
+
+ @c.unsubscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should delete the source from subscriptions" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}, :id => "rspec"})
+ @subscription.expects(:delete).with("rspec").once
+
+ @c.unsubscribe("test", :broadcast, "mcollective")
+ end
+ end
+
+ describe "#target_for" do
+ it "should create reply targets based on reply-to headers in requests" do
+ message = mock
+ message.expects(:type).returns(:reply)
+
+ request = mock
+ request.expects(:headers).returns({"reply-to" => "foo"})
+
+ message.expects(:request).returns(request)
+
+ @c.target_for(message).should == {:name => "foo", :headers => {}}
+ end
+
+ it "should create new request targets" do
+ message = mock
+ message.expects(:type).returns(:request).times(3)
+ message.expects(:agent).returns("rspecagent")
+ message.expects(:collective).returns("mcollective")
+
+ @c.expects(:make_target).with("rspecagent", :request, "mcollective")
+ @c.target_for(message)
+ end
+
+ it "should support direct requests" do
+ message = mock
+ message.expects(:type).returns(:direct_request).times(3)
+ message.expects(:agent).returns("rspecagent")
+ message.expects(:collective).returns("mcollective")
+
+ @c.expects(:make_target).with("rspecagent", :direct_request, "mcollective")
+ @c.target_for(message)
+ end
+
+ it "should fail for unknown message types" do
+ message = mock
+ message.stubs(:type).returns(:fail)
+
+ expect {
+ @c.target_for(message)
+ }.to raise_error("Don't now how to create a target for message type fail")
+ end
+ end
+
+ describe "#disconnect" do
+ it "should disconnect from the stomp connection" do
+ @connection.expects(:disconnect)
+ @c.disconnect
+ @c.connection.should == nil
+ end
+ end
+
+ describe "#headers_for" do
+ it "should return empty headers if priority is 0" do
+ message = mock
+ message.expects(:type).returns(:foo)
+
+ @c.instance_variable_set("@msgpriority", 0)
+ @c.headers_for(message).should == {}
+ end
+
+ it "should return a priority if priority is non 0" do
+ message = mock
+ message.expects(:type).returns(:foo)
+
+ @c.instance_variable_set("@msgpriority", 1)
+ @c.headers_for(message).should == {"priority" => 1}
+ end
+
+ it "should set mc_identity for direct requests" do
+ message = mock
+ message.expects(:type).returns(:direct_request).twice
+ message.expects(:agent).returns("rspecagent")
+ message.expects(:collective).returns("mcollective")
+ message.expects(:reply_to).returns(nil)
+
+ @c.instance_variable_set("@msgpriority", 0)
+ @c.expects(:make_target).with("rspecagent", :reply, "mcollective").returns({:name => "test"})
+ @c.headers_for(message, "some.node").should == {"mc_identity"=>"some.node", "reply-to"=>"test"}
+ end
+
+ it "should set a reply-to header for :request type messages" do
+ message = mock
+ message.expects(:type).returns(:request).twice
+ message.expects(:agent).returns("rspecagent")
+ message.expects(:collective).returns("mcollective")
+ message.expects(:reply_to).returns(nil)
+
+ @c.instance_variable_set("@msgpriority", 0)
+ @c.expects(:make_target).with("rspecagent", :reply, "mcollective").returns({:name => "test"})
+ @c.headers_for(message).should == {"reply-to" => "test"}
+ end
+
+ it "should set reply-to correctly if the message defines it" do
+ message = mock
+ message.expects(:type).returns(:request).twice
+ message.expects(:agent).returns("rspecagent")
+ message.expects(:collective).returns("mcollective")
+ message.expects(:reply_to).returns("rspec").twice
+
+ @c.headers_for(message).should == {"reply-to" => "rspec"}
+
+ end
+ end
+
+ describe "#make_target" do
+ it "should create correct targets" do
+ @c.make_target("test", :reply, "mcollective").should == {:name => "/queue/mcollective.reply.rspec_#{$$}", :headers => {}, :id => "/queue/mcollective.reply.rspec_#{$$}"}
+ @c.make_target("test", :broadcast, "mcollective").should == {:name => "/topic/mcollective.test.agent", :headers => {}, :id => "/topic/mcollective.test.agent"}
+ @c.make_target("test", :request, "mcollective").should == {:name => "/topic/mcollective.test.agent", :headers => {}, :id => "/topic/mcollective.test.agent"}
+ @c.make_target("test", :direct_request, "mcollective").should == {:headers=>{}, :name=>"/queue/mcollective.nodes", :id => "/queue/mcollective.nodes"}
+ @c.make_target("test", :directed, "mcollective").should == {:name => "/queue/mcollective.nodes", :headers=>{"selector"=>"mc_identity = 'rspec'"}, :id => "mcollective_directed_to_identity"}
+ end
+
+ it "should raise an error for unknown collectives" do
+ expect {
+ @c.make_target("test", :broadcast, "foo")
+ }.to raise_error("Unknown collective 'foo' known collectives are 'mcollective'")
+ end
+
+ it "should raise an error for unknown types" do
+ expect {
+ @c.make_target("test", :test, "mcollective")
+ }.to raise_error("Unknown target type test")
+ end
+ end
+
+
+ describe "#get_env_or_option" do
+ it "should return the environment variable if set" do
+ ENV["test"] = "rspec_env_test"
+
+ @c.get_env_or_option("test", nil, nil).should == "rspec_env_test"
+
+ ENV.delete("test")
+ end
+
+ it "should return the config option if set" do
+ @config.expects(:pluginconf).returns({"test" => "rspec_test"}).twice
+ @c.get_env_or_option("test", "test", "test").should == "rspec_test"
+ end
+
+ it "should return default if nothing else matched" do
+ @config.expects(:pluginconf).returns({}).once
+ @c.get_env_or_option("test", "test", "test").should == "test"
+ end
+
+ it "should raise an error if no default is supplied" do
+ @config.expects(:pluginconf).returns({}).once
+
+ expect {
+ @c.get_env_or_option("test", "test")
+ }.to raise_error("No test environment or plugin.test configuration option given")
+ end
+ end
+
+ describe "#get_option" do
+ it "should return the config option if set" do
+ @config.expects(:pluginconf).returns({"test" => "rspec_test"}).twice
+ @c.get_option("test").should == "rspec_test"
+ end
+
+ it "should return default option was not found" do
+ @config.expects(:pluginconf).returns({}).once
+ @c.get_option("test", "test").should == "test"
+ end
+
+ it "should raise an error if no default is supplied" do
+ @config.expects(:pluginconf).returns({}).once
+
+ expect {
+ @c.get_option("test")
+ }.to raise_error("No plugin.test configuration option given")
+ end
+ end
+
+ describe "#get_bool_option" do
+ it "should return the default if option isnt set" do
+ @config.expects(:pluginconf).returns({}).once
+ @c.get_bool_option("test", "default").should == "default"
+ end
+
+ ["1", "yes", "true"].each do |boolean|
+ it "should map options to true correctly" do
+ @config.expects(:pluginconf).returns({"test" => boolean}).twice
+ @c.get_bool_option("test", "default").should == true
+ end
+ end
+
+ ["0", "no", "false"].each do |boolean|
+ it "should map options to false correctly" do
+ @config.expects(:pluginconf).returns({"test" => boolean}).twice
+ @c.get_bool_option("test", "default").should == false
+ end
+ end
+
+ it "should return default for non boolean options" do
+ @config.expects(:pluginconf).returns({"test" => "foo"}).twice
+ @c.get_bool_option("test", "default").should == "default"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+MCollective::PluginManager.clear
+
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/connector/rabbitmq.rb'
+
+# create the stomp error class here as it does not always exist
+# all versions of the stomp gem and we do not want to tie tests
+# to the stomp gem
+module Stomp
+ module Error
+ class DuplicateSubscription < RuntimeError; end
+ end
+end
+
+module MCollective
+ module Connector
+ describe Rabbitmq do
+ before do
+ unless ::Stomp::Error.constants.map{|c| c.to_s}.include?("NoCurrentConnection")
+ class ::Stomp::Error::NoCurrentConnection < RuntimeError ; end
+ end
+
+ @config = mock
+ @config.stubs(:configured).returns(true)
+ @config.stubs(:identity).returns("rspec")
+ @config.stubs(:collectives).returns(["mcollective"])
+
+ logger = mock
+ logger.stubs(:log)
+ logger.stubs(:start)
+ Log.configure(logger)
+
+ Config.stubs(:instance).returns(@config)
+
+ @msg = mock
+ @msg.stubs(:base64_encode!)
+ @msg.stubs(:payload).returns("msg")
+ @msg.stubs(:agent).returns("agent")
+ @msg.stubs(:type).returns(:reply)
+ @msg.stubs(:collective).returns("mcollective")
+
+ @subscription = mock
+ @subscription.stubs("<<").returns(true)
+ @subscription.stubs("include?").returns(false)
+ @subscription.stubs("delete").returns(false)
+
+ @connection = mock
+ @connection.stubs(:subscribe).returns(true)
+ @connection.stubs(:unsubscribe).returns(true)
+
+ @c = Rabbitmq.new
+ @c.instance_variable_set("@subscriptions", @subscription)
+ @c.instance_variable_set("@connection", @connection)
+ end
+
+ describe "#initialize" do
+ it "should set the @config variable" do
+ c = Rabbitmq.new
+ c.instance_variable_get("@config").should == @config
+ end
+
+ it "should set @subscriptions to an empty list" do
+ c = Rabbitmq.new
+ c.instance_variable_get("@subscriptions").should == []
+ end
+ end
+
+ describe "#connect" do
+ it "should not try to reconnect if already connected" do
+ Log.expects(:debug).with("Already connection, not re-initializing connection").once
+ @c.connect
+ end
+
+ it "should support new style config" do
+ pluginconf = {"rabbitmq.pool.size" => "2",
+ "rabbitmq.pool.1.host" => "host1",
+ "rabbitmq.pool.1.port" => "6163",
+ "rabbitmq.pool.1.user" => "user1",
+ "rabbitmq.pool.1.password" => "password1",
+ "rabbitmq.pool.1.ssl" => "false",
+ "rabbitmq.pool.2.host" => "host2",
+ "rabbitmq.pool.2.port" => "6164",
+ "rabbitmq.pool.2.user" => "user2",
+ "rabbitmq.pool.2.password" => "password2",
+ "rabbitmq.pool.2.ssl" => "true",
+ "rabbitmq.pool.2.ssl.fallback" => "true",
+ "rabbitmq.initial_reconnect_delay" => "0.02",
+ "rabbitmq.max_reconnect_delay" => "40",
+ "rabbitmq.use_exponential_back_off" => "false",
+ "rabbitmq.back_off_multiplier" => "3",
+ "rabbitmq.max_reconnect_attempts" => "5",
+ "rabbitmq.randomize" => "true",
+ "rabbitmq.backup" => "true",
+ "rabbitmq.timeout" => "1",
+ "rabbitmq.vhost" => "mcollective",
+ "rabbitmq.connect_timeout" => "5"}
+
+
+ ENV.delete("STOMP_USER")
+ ENV.delete("STOMP_PASSWORD")
+
+ @config.expects(:pluginconf).returns(pluginconf).at_least_once
+
+ Rabbitmq::EventLogger.expects(:new).returns("logger")
+
+ connector = mock
+ connector.expects(:new).with(:backup => true,
+ :back_off_multiplier => 3,
+ :max_reconnect_delay => 40.0,
+ :timeout => 1,
+ :connect_timeout => 5,
+ :use_exponential_back_off => false,
+ :max_reconnect_attempts => 5,
+ :initial_reconnect_delay => 0.02,
+ :randomize => true,
+ :reliable => true,
+ :logger => "logger",
+ :connect_headers => {'accept-version' => '1.0', 'host' => 'mcollective'},
+ :hosts => [{:passcode => 'password1',
+ :host => 'host1',
+ :port => 6163,
+ :ssl => false,
+ :login => 'user1'},
+ {:passcode => 'password2',
+ :host => 'host2',
+ :port => 6164,
+ :ssl => true,
+ :login => 'user2'}
+ ])
+
+ @c.expects(:ssl_parameters).with(2, true).returns(true)
+
+ @c.instance_variable_set("@connection", nil)
+ @c.connect(connector)
+ end
+ end
+
+ describe "#ssl_paramaters" do
+ it "should ensure all settings are provided" do
+ pluginconf = {"rabbitmq.pool.1.host" => "host1",
+ "rabbitmq.pool.1.port" => "6164",
+ "rabbitmq.pool.1.user" => "user1",
+ "rabbitmq.pool.1.password" => "password1",
+ "rabbitmq.pool.1.ssl" => "true",
+ "rabbitmq.pool.1.ssl.cert" => "rspec"}
+
+ @config.expects(:pluginconf).returns(pluginconf).at_least_once
+
+ expect { @c.ssl_parameters(1, false) }.to raise_error("cert, key and ca has to be supplied for verified SSL mode")
+ end
+
+ it "should verify the ssl files exist" do
+ pluginconf = {"rabbitmq.pool.1.host" => "host1",
+ "rabbitmq.pool.1.port" => "6164",
+ "rabbitmq.pool.1.user" => "user1",
+ "rabbitmq.pool.1.password" => "password1",
+ "rabbitmq.pool.1.ssl" => "true",
+ "rabbitmq.pool.1.ssl.cert" => "rspec.cert",
+ "rabbitmq.pool.1.ssl.key" => "rspec.key",
+ "rabbitmq.pool.1.ssl.ca" => "rspec1.ca,rspec2.ca"}
+
+ @config.expects(:pluginconf).returns(pluginconf).at_least_once
+
+ File.expects(:exist?).with("rspec.cert").twice.returns(true)
+ File.expects(:exist?).with("rspec.key").twice.returns(true)
+ File.expects(:exist?).with("rspec1.ca").twice.returns(true)
+ File.expects(:exist?).with("rspec2.ca").twice.returns(false)
+
+ expect { @c.ssl_parameters(1, false) }.to raise_error("Cannot find CA file rspec2.ca")
+
+ @c.ssl_parameters(1, true).should == true
+ end
+
+ it "should support fallback mode when there are errors" do
+ pluginconf = {"rabbitmq.pool.1.host" => "host1",
+ "rabbitmq.pool.1.port" => "6164",
+ "rabbitmq.pool.1.user" => "user1",
+ "rabbitmq.pool.1.password" => "password1",
+ "rabbitmq.pool.1.ssl" => "true"}
+
+ @config.expects(:pluginconf).returns(pluginconf).at_least_once
+
+ @c.ssl_parameters(1, true).should == true
+ end
+
+ it "should fail if fallback isnt enabled" do
+ pluginconf = {"rabbitmq.pool.1.host" => "host1",
+ "rabbitmq.pool.1.port" => "6164",
+ "rabbitmq.pool.1.user" => "user1",
+ "rabbitmq.pool.1.password" => "password1",
+ "rabbitmq.pool.1.ssl" => "true"}
+
+ @config.expects(:pluginconf).returns(pluginconf).at_least_once
+
+ expect { @c.ssl_parameters(1, false) }.to raise_error
+ end
+ end
+
+ describe "#receive" do
+ it "should receive from the middleware" do
+ payload = mock
+ payload.stubs(:body).returns("msg")
+ payload.stubs(:headers).returns("headers")
+
+ @connection.expects(:receive).returns(payload)
+
+ Message.expects(:new).with("msg", payload, :base64 => true, :headers => "headers").returns("message")
+ @c.instance_variable_set("@base64", true)
+
+ received = @c.receive
+ received.should == "message"
+ end
+
+ it "should sleep and retry if recieving while disconnected" do
+ payload = mock
+ payload.stubs(:body).returns("msg")
+ payload.stubs(:headers).returns("headers")
+
+ Message.stubs(:new).returns("rspec")
+ @connection.expects(:receive).raises(::Stomp::Error::NoCurrentConnection).returns(payload).twice
+ @c.expects(:sleep).with(1)
+
+ @c.receive.should == "rspec"
+ end
+ end
+
+ describe "#publish" do
+ before do
+ @connection.stubs(:publish).with("test", "msg", {}).returns(true)
+ end
+
+ it "should base64 encode a message if configured to do so" do
+ @c.instance_variable_set("@base64", true)
+ @c.expects(:target_for).returns({:name => "test", :headers => {}})
+ @connection.expects(:publish).with("test", "msg", {})
+ @msg.expects(:base64_encode!)
+
+ @c.publish(@msg)
+ end
+
+ it "should not base64 encode if not configured to do so" do
+ @c.instance_variable_set("@base64", false)
+ @c.expects(:target_for).returns({:name => "test", :headers => {}})
+ @connection.expects(:publish).with("test", "msg", {})
+ @msg.expects(:base64_encode!).never
+
+ @c.publish(@msg)
+ end
+
+ it "should publish the correct message to the correct target with msgheaders" do
+ @connection.expects(:publish).with("test", "msg", {}).once
+ @c.expects(:target_for).returns({:name => "test", :headers => {}})
+
+ @c.publish(@msg)
+ end
+
+ it "should publish direct messages based on discovered_hosts" do
+ msg = mock
+ msg.stubs(:base64_encode!)
+ msg.stubs(:payload).returns("msg")
+ msg.stubs(:agent).returns("agent")
+ msg.stubs(:collective).returns("mcollective")
+ msg.stubs(:type).returns(:direct_request)
+ msg.stubs(:reply_to).returns("/topic/mcollective")
+ msg.expects(:discovered_hosts).returns(["one", "two"])
+
+ @connection.expects(:publish).with('/exchange/mcollective_directed/one', 'msg', {'reply-to' => '/temp-queue/mcollective_reply_agent'})
+ @connection.expects(:publish).with('/exchange/mcollective_directed/two', 'msg', {'reply-to' => '/temp-queue/mcollective_reply_agent'})
+
+ @c.publish(msg)
+ end
+ end
+
+ describe "#subscribe" do
+ it "should handle duplicate subscription errors" do
+ @connection.expects(:subscribe).raises(::Stomp::Error::DuplicateSubscription)
+ Log.expects(:error).with(regexp_matches(/already had a matching subscription, ignoring/))
+ @c.subscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should use the make_target correctly" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}})
+ @c.subscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should check for existing subscriptions" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}, :id => "rspec"})
+ @subscription.expects("include?").with("rspec").returns(false)
+ @connection.expects(:subscribe).never
+
+ @c.subscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should subscribe to the middleware" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}, :id => "rspec"})
+ @connection.expects(:subscribe).with("test", {}, "rspec")
+ @c.subscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should add to the list of subscriptions" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}, :id => "rspec"})
+ @subscription.expects("<<").with("rspec")
+ @c.subscribe("test", :broadcast, "mcollective")
+ end
+ end
+
+ describe "#unsubscribe" do
+ it "should use make_target correctly" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}})
+ @c.unsubscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should unsubscribe from the target" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}, :id => "rspec"})
+ @connection.expects(:unsubscribe).with("test", {}, "rspec").once
+
+ @c.unsubscribe("test", :broadcast, "mcollective")
+ end
+
+ it "should delete the source from subscriptions" do
+ @c.expects("make_target").with("test", :broadcast, "mcollective").returns({:name => "test", :headers => {}, :id => "rspec"})
+ @subscription.expects(:delete).with("rspec").once
+
+ @c.unsubscribe("test", :broadcast, "mcollective")
+ end
+ end
+
+ describe "#target_for" do
+ it "should create reply targets based on reply-to headers in requests" do
+ message = mock
+ message.expects(:type).returns(:reply)
+
+ request = mock
+ request.expects(:headers).returns({"reply-to" => "foo"})
+
+ message.expects(:request).returns(request)
+
+ @c.target_for(message).should == {:name => "foo", :headers => {}, :id => ""}
+ end
+
+ it "should create new request targets" do
+ message = mock
+ message.expects(:type).returns(:request).times(3)
+ message.expects(:agent).returns("rspecagent")
+ message.expects(:collective).returns("mcollective")
+ message.expects(:reply_to).returns("/topic/rspec")
+
+ @c.expects(:make_target).with("rspecagent", :request, "mcollective", "/topic/rspec", nil)
+ @c.target_for(message)
+ end
+
+ it "should support direct requests" do
+ message = mock
+ message.expects(:type).returns(:direct_request).times(3)
+ message.expects(:agent).returns("rspecagent")
+ message.expects(:collective).returns("mcollective")
+ message.expects(:reply_to).returns("/topic/rspec")
+
+ @c.expects(:make_target).with("rspecagent", :direct_request, "mcollective", "/topic/rspec", nil)
+ @c.target_for(message)
+ end
+
+ it "should fail for unknown message types" do
+ message = mock
+ message.stubs(:type).returns(:fail)
+
+ expect {
+ @c.target_for(message)
+ }.to raise_error("Don't now how to create a target for message type fail")
+ end
+ end
+
+ describe "#disconnect" do
+ it "should disconnect from the stomp connection" do
+ @connection.expects(:disconnect)
+ @c.disconnect
+ @c.connection.should == nil
+ end
+ end
+
+ describe "#make_target" do
+ it "should create correct targets" do
+ @c.make_target("test", :reply, "mcollective").should == {:name => "/temp-queue/mcollective_reply_test", :headers => {}, :id => "mcollective_test_replies"}
+ @c.make_target("test", :broadcast, "mcollective").should == {:name => "/exchange/mcollective_broadcast/test", :headers => {"reply-to"=>"/temp-queue/mcollective_reply_test"}, :id => "mcollective_broadcast_test"}
+ @c.make_target("test", :request, "mcollective").should == {:name => "/exchange/mcollective_broadcast/test", :headers => {"reply-to"=>"/temp-queue/mcollective_reply_test"}, :id => "mcollective_broadcast_test"}
+ @c.make_target("test", :direct_request, "mcollective", nil, "rspec").should == {:headers=>{"reply-to"=>"/temp-queue/mcollective_reply_test"}, :name=>"/exchange/mcollective_directed/rspec", :id => nil}
+ @c.make_target("test", :directed, "mcollective").should == {:name => "/exchange/mcollective_directed/rspec", :headers=>{}, :id => "rspec_directed_to_identity"}
+ @c.make_target("test", :request, "mcollective", "/topic/rspec", "rspec").should == {:headers=>{"reply-to"=>"/topic/rspec"}, :name=>"/exchange/mcollective_broadcast/test", :id => "mcollective_broadcast_test"}
+ end
+
+ it "should raise an error for unknown collectives" do
+ expect {
+ @c.make_target("test", :broadcast, "foo")
+ }.to raise_error("Unknown collective 'foo' known collectives are 'mcollective'")
+ end
+
+ it "should raise an error for unknown types" do
+ expect {
+ @c.make_target("test", :test, "mcollective")
+ }.to raise_error("Unknown target type test")
+ end
+ end
+
+
+ describe "#get_env_or_option" do
+ it "should return the environment variable if set" do
+ ENV["test"] = "rspec_env_test"
+
+ @c.get_env_or_option("test", nil, nil).should == "rspec_env_test"
+
+ ENV.delete("test")
+ end
+
+ it "should return the config option if set" do
+ @config.expects(:pluginconf).returns({"test" => "rspec_test"}).twice
+ @c.get_env_or_option("test", "test", "test").should == "rspec_test"
+ end
+
+ it "should return default if nothing else matched" do
+ @config.expects(:pluginconf).returns({}).once
+ @c.get_env_or_option("test", "test", "test").should == "test"
+ end
+
+ it "should raise an error if no default is supplied" do
+ @config.expects(:pluginconf).returns({}).once
+
+ expect {
+ @c.get_env_or_option("test", "test")
+ }.to raise_error("No test environment or plugin.test configuration option given")
+ end
+ end
+
+ describe "#get_option" do
+ it "should return the config option if set" do
+ @config.expects(:pluginconf).returns({"test" => "rspec_test"}).twice
+ @c.get_option("test").should == "rspec_test"
+ end
+
+ it "should return default option was not found" do
+ @config.expects(:pluginconf).returns({}).once
+ @c.get_option("test", "test").should == "test"
+ end
+
+ it "should raise an error if no default is supplied" do
+ @config.expects(:pluginconf).returns({}).once
+
+ expect {
+ @c.get_option("test")
+ }.to raise_error("No plugin.test configuration option given")
+ end
+ end
+
+ describe "#get_bool_option" do
+ it "should return the default if option isnt set" do
+ @config.expects(:pluginconf).returns({}).once
+ @c.get_bool_option("test", "default").should == "default"
+ end
+
+ ["1", "yes", "true"].each do |boolean|
+ it "should map options to true correctly" do
+ @config.expects(:pluginconf).returns({"test" => boolean}).twice
+ @c.get_bool_option("test", "default").should == true
+ end
+ end
+
+ ["0", "no", "false"].each do |boolean|
+ it "should map options to false correctly" do
+ @config.expects(:pluginconf).returns({"test" => boolean}).twice
+ @c.get_bool_option("test", "default").should == false
+ end
+ end
+
+ it "should return default for non boolean options" do
+ @config.expects(:pluginconf).returns({"test" => "foo"}).twice
+ @c.get_bool_option("test", "default").should == "default"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+require File.dirname(__FILE__) + "/../../../../../plugins/mcollective/data/agent_data.rb"
+
+module MCollective
+ module Data
+ describe Agent_data do
+ describe "#query_data" do
+ before do
+ @ddl = mock
+ @ddl.stubs(:dataquery_interface).returns({:output => {}})
+ @ddl.stubs(:meta).returns({:timeout => 1})
+ DDL.stubs(:new).returns(@ddl)
+ @plugin = Agent_data.new
+ end
+
+ it "should fail for unknown agents" do
+ expect { @plugin.query_data("rspec") }.to raise_error("No agent called rspec found")
+ end
+
+ it "should retrieve the correct agent and data" do
+ agent = mock
+ agent.stubs(:meta).returns({:license => "license",
+ :timeout => "timeout",
+ :description => "description",
+ :url => "url",
+ :version => "version",
+ :author => "author"})
+
+ PluginManager.stubs(:[]).with("rspec_agent").returns(agent)
+ PluginManager.expects(:include?).with("rspec_agent").returns(true)
+
+ @plugin.query_data("rspec")
+
+ [:license, :timeout, :description, :url, :version, :author].each do |item|
+ @plugin.result[item].should == item.to_s
+ end
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+require File.dirname(__FILE__) + "/../../../../../plugins/mcollective/data/fstat_data.rb"
+
+module MCollective
+ module Data
+ describe Fstat_data do
+ describe "#query_data" do
+ before do
+ @ddl = mock
+ @ddl.stubs(:meta).returns({:timeout => 1})
+ @ddl.stubs(:dataquery_interface).returns({:output => {}})
+ DDL.stubs(:new).returns(@ddl)
+ @plugin = Fstat_data.new
+
+ @time = Time.now
+
+ @stat = mock
+ @stat.stubs(:size).returns(123)
+ @stat.stubs(:uid).returns(0)
+ @stat.stubs(:gid).returns(0)
+ @stat.stubs(:mtime).returns(@time)
+ @stat.stubs(:ctime).returns(@time)
+ @stat.stubs(:atime).returns(@time)
+ @stat.stubs(:mode).returns(33188)
+ @stat.stubs(:directory?).returns(false)
+ @stat.stubs(:file?).returns(false)
+ @stat.stubs(:symlink?).returns(false)
+ @stat.stubs(:socket?).returns(false)
+ @stat.stubs(:chardev?).returns(false)
+ @stat.stubs(:blockdev?).returns(false)
+ end
+
+ it "should detect missing files" do
+ File.expects(:exists?).with("/nonexisting").returns(false)
+ @plugin.query_data("/nonexisting")
+ @plugin.result.output.should == "not present"
+ end
+
+ it "should provide correct file stats" do
+ File.expects(:exists?).with("rspec").returns(true)
+ File.expects(:symlink?).with("rspec").returns(false)
+ File.expects(:stat).with("rspec").returns(@stat)
+ File.expects(:read).with("rspec").returns("rspec")
+
+ @stat.stubs(:file?).returns(true)
+
+ @plugin.query_data("rspec")
+ @plugin.result.output.should == "present"
+ @plugin.result.size.should == 123
+ @plugin.result.uid.should == 0
+ @plugin.result.gid.should == 0
+ @plugin.result.mtime.should == @time.strftime("%F %T")
+ @plugin.result.mtime_seconds.should == @time.to_i
+ @plugin.result.mtime_age.should <= 5
+ @plugin.result.ctime.should == @time.strftime("%F %T")
+ @plugin.result.ctime_seconds.should == @time.to_i
+ @plugin.result.ctime_age.should <= 5
+ @plugin.result.atime.should == @time.strftime("%F %T")
+ @plugin.result.atime_seconds.should == @time.to_i
+ @plugin.result.atime_age.should <= 5
+ @plugin.result.mode.should == "100644"
+ @plugin.result.md5.should == "2bc84dc69b73db9383b9c6711d2011b7"
+ @plugin.result.type.should == "file"
+ end
+
+ it "should provide correct link stats" do
+ File.expects(:exists?).with("rspec").returns(true)
+ File.expects(:symlink?).with("rspec").returns(true)
+ File.expects(:lstat).with("rspec").returns(@stat)
+
+ @stat.stubs(:symlink?).returns(true)
+
+ @plugin.query_data("rspec")
+ @plugin.result.output.should == "present"
+ @plugin.result.md5.should == 0
+ @plugin.result.type.should == "symlink"
+ end
+
+ it "should provide correct directory stats" do
+ File.expects(:exists?).with("rspec").returns(true)
+ File.expects(:symlink?).with("rspec").returns(false)
+ File.expects(:stat).with("rspec").returns(@stat)
+
+ @stat.stubs(:directory?).returns(true)
+
+ @plugin.query_data("rspec")
+ @plugin.result.output.should == "present"
+ @plugin.result.md5.should == 0
+ @plugin.result.type.should == "directory"
+ end
+
+ it "should provide correct socket stats" do
+ File.expects(:exists?).with("rspec").returns(true)
+ File.expects(:symlink?).with("rspec").returns(false)
+ File.expects(:stat).with("rspec").returns(@stat)
+
+ @stat.stubs(:socket?).returns(true)
+
+ @plugin.query_data("rspec")
+ @plugin.result.output.should == "present"
+ @plugin.result.md5.should == 0
+ @plugin.result.type.should == "socket"
+ end
+
+ it "should provide correct chardev stats" do
+ File.expects(:exists?).with("rspec").returns(true)
+ File.expects(:symlink?).with("rspec").returns(false)
+ File.expects(:stat).with("rspec").returns(@stat)
+
+ @stat.stubs(:chardev?).returns(true)
+
+ @plugin.query_data("rspec")
+ @plugin.result.output.should == "present"
+ @plugin.result.md5.should == 0
+ @plugin.result.type.should == "chardev"
+ end
+
+ it "should provide correct blockdev stats" do
+ File.expects(:exists?).with("rspec").returns(true)
+ File.expects(:symlink?).with("rspec").returns(false)
+ File.expects(:stat).with("rspec").returns(@stat)
+
+ @stat.stubs(:blockdev?).returns(true)
+
+ @plugin.query_data("rspec")
+ @plugin.result.output.should == "present"
+ @plugin.result.md5.should == 0
+ @plugin.result.type.should == "blockdev"
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/discovery/flatfile.rb'
+
+module MCollective
+ class Discovery
+ describe Flatfile do
+ describe "#discover" do
+ before do
+ @client = mock
+ @client.stubs(:options).returns({})
+ @client.stubs(:options).returns({:discovery_options => ["/nonexisting"]})
+
+
+ File.stubs(:readable?).with("/nonexisting").returns(true)
+ File.stubs(:readlines).with("/nonexisting").returns(["one", "two"])
+ end
+
+ it "should use a file specified in discovery_options" do
+ File.expects(:readable?).with("/nonexisting").returns(true)
+ File.expects(:readlines).with("/nonexisting").returns(["one", "two"])
+ Flatfile.discover(Util.empty_filter, 0, 0, @client).should == ["one", "two"]
+ end
+
+ it "should fail unless a file is specified" do
+ @client.stubs(:options).returns({:discovery_options => []})
+ expect { Flatfile.discover(Util.empty_filter, 0, 0, @client) }.to raise_error("The flatfile discovery method needs a path to a text file")
+ end
+
+ it "should fail for unreadable files" do
+ File.expects(:readable?).with("/nonexisting").returns(false)
+
+ expect { Flatfile.discover(Util.empty_filter, 0, 0, @client) }.to raise_error("Cannot read the file /nonexisting specified as discovery source")
+ end
+
+ it "should regex filters" do
+ Flatfile.discover(Util.empty_filter.merge("identity" => [/one/]), 0, 0, @client).should == ["one"]
+ end
+
+ it "should filter against non regex nodes" do
+ Flatfile.discover(Util.empty_filter.merge("identity" => ["one"]), 0, 0, @client).should == ["one"]
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/discovery/mc.rb'
+
+module MCollective
+ class Discovery
+ describe Mc do
+ describe "#discover" do
+ before do
+ @reply = mock
+ @reply.stubs(:payload).returns({:senderid => "rspec"})
+
+ @client = mock
+ @client.stubs(:sendreq)
+ @client.stubs(:unsubscribe)
+ @client.stubs(:receive).returns(@reply)
+
+ Log.stubs(:debug)
+ end
+
+ it "should send the ping request via the supplied client" do
+ @client.expects(:sendreq).with("ping", "discovery", Util.empty_filter).returns("123456")
+ Mc.discover(Util.empty_filter, 1, 1, @client)
+ end
+
+ it "should stop early if a limit is supplied" do
+ @client.stubs(:receive).returns(@reply).times(10)
+ Mc.discover(Util.empty_filter, 1, 10, @client).should == ("rspec," * 10).split(",")
+ end
+
+ it "should unsubscribe from the discovery reply source" do
+ @client.expects(:unsubscribe).with("discovery", :reply)
+ Mc.discover(Util.empty_filter, 1, 10, @client).should == ("rspec," * 10).split(",")
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/pluginpackager/debpackage_packager.rb'
+
+module MCollective
+ module PluginPackager
+ describe DebpackagePackager do
+
+ let(:maketmpdir) do
+ tmpdir = Dir.mktmpdir("mc-test")
+ @tmpdirs << tmpdir
+ tmpdir
+ end
+
+ before :all do
+ @tmpdirs = []
+ end
+
+ before :each do
+ PluginPackager.stubs(:build_tool?).with("debuild").returns(true)
+ @plugin = mock()
+ @plugin.stubs(:mcname).returns("mcollective")
+ end
+
+ after :all do
+ @tmpdirs.each{|tmpdir| FileUtils.rm_rf tmpdir if File.directory? tmpdir}
+ end
+
+ describe "#initialize" do
+ it "should raise an exception if debuild isn't present" do
+ PluginPackager.expects(:build_tool?).with("debuild").returns(false)
+ expect{
+ DebpackagePackager.new("plugin")
+ }.to raise_error(RuntimeError, "package 'debuild' is not installed")
+ end
+
+ it "should set the correct libdir and verbose value" do
+ PluginPackager.expects(:build_tool?).with("debuild").returns(true)
+ packager = DebpackagePackager.new("plugin", nil, nil, true)
+ packager.libdir.should == "/usr/share/mcollective/plugins/mcollective/"
+ packager.verbose.should == true
+ end
+ end
+
+ describe "#create_packages" do
+ before :each do
+ @packager = DebpackagePackager.new(@plugin)
+ @plugin.stubs(:packagedata).returns({:test => {:files => ["test.rb"]}})
+ @plugin.stubs(:metadata).returns({:name => "test_plugin", :version => "1"})
+ @plugin.stubs(:iteration).returns("1")
+ @packager.stubs(:prepare_tmpdirs)
+ @packager.stubs(:create_package)
+ @packager.stubs(:move_packages)
+ @packager.stubs(:cleanup_tmpdirs)
+ Dir.stubs(:mktmpdir).with("mcollective_packager").returns("/tmp")
+ Dir.stubs(:mkdir)
+ end
+
+ it "should set the package instance variables" do
+ @packager.create_packages
+ @packager.current_package_type.should == :test
+ @packager.current_package_data.should == {:files => ["test.rb"]}
+ @packager.current_package_shortname.should == "mcollective-test_plugin-test"
+ @packager.current_package_fullname.should == "mcollective-test_plugin-test_1-1"
+ end
+
+ it "Should create the build dir" do
+ Dir.expects(:mkdir).with("/tmp/mcollective-test_plugin-test_1")
+ @packager.create_packages
+ end
+
+ it "should create packages" do
+ @packager.expects(:create_package)
+ @packager.create_packages
+ end
+ end
+
+ describe "#create_package" do
+ it "should raise an exception if the package cannot be created" do
+ packager = DebpackagePackager.new(@plugin)
+ packager.stubs(:create_file).raises("test exception")
+ expect{
+ packager.create_package
+ }.to raise_error(RuntimeError, "Could not build package - test exception")
+ end
+
+ it "should correctly create a package" do
+ packager = DebpackagePackager.new(@plugin, nil, nil, true)
+
+ packager.expects(:create_file).with("control")
+ packager.expects(:create_file).with("Makefile")
+ packager.expects(:create_file).with("compat")
+ packager.expects(:create_file).with("rules")
+ packager.expects(:create_file).with("copyright")
+ packager.expects(:create_file).with("changelog")
+ packager.expects(:create_tar)
+ packager.expects(:create_install)
+ packager.expects(:create_preandpost_install)
+
+ packager.build_dir = "/tmp"
+ packager.tmpdir = "/tmp"
+ packager.current_package_fullname = "test"
+ PluginPackager.expects(:safe_system).with("debuild -i -us -uc")
+ packager.expects(:puts).with("Created package test")
+
+ packager.create_package
+ end
+
+ it "should add a signature if one is given" do
+ packager = DebpackagePackager.new(@plugin, nil, "test", true)
+
+ packager.expects(:create_file).with("control")
+ packager.expects(:create_file).with("Makefile")
+ packager.expects(:create_file).with("compat")
+ packager.expects(:create_file).with("rules")
+ packager.expects(:create_file).with("copyright")
+ packager.expects(:create_file).with("changelog")
+ packager.expects(:create_tar)
+ packager.expects(:create_install)
+ packager.expects(:create_preandpost_install)
+
+ packager.build_dir = "/tmp"
+ packager.tmpdir = "/tmp"
+ packager.current_package_fullname = "test"
+ PluginPackager.expects(:safe_system).with("debuild -i -ktest")
+ packager.expects(:puts).with("Created package test")
+
+ packager.create_package
+ end
+ end
+
+ describe "#create_preandpost_install" do
+ before :each do
+ @packager = DebpackagePackager.new(@plugin)
+ end
+
+ it "should raise an exception if preinstall is not null and preinstall script isn't present" do
+ @plugin.stubs(:preinstall).returns("myscript")
+ File.expects(:exists?).with("myscript").returns(false)
+ expect{
+ @packager.create_preandpost_install
+ }.to raise_error(RuntimeError, "pre-install script 'myscript' not found")
+ end
+
+ it "should raise an exception if postinstall is not null and postinstall script isn't present" do
+ @plugin.stubs(:preinstall).returns(nil)
+ @plugin.stubs(:postinstall).returns("myscript")
+ File.expects(:exists?).with("myscript").returns(false)
+ expect{
+ @packager.create_preandpost_install
+ }.to raise_error(RuntimeError, "post-install script 'myscript' not found")
+ end
+
+ it "should copy the preinstall and postinstall scripts to the correct directory with the correct name" do
+ @plugin.stubs(:postinstall).returns("myscript")
+ @plugin.stubs(:preinstall).returns("myscript")
+ @packager.build_dir = "/tmp/"
+ @packager.current_package_shortname = "test"
+ File.expects(:exists?).with("myscript").twice.returns(true)
+ FileUtils.expects(:cp).with("myscript", "/tmp/debian/test.preinst")
+ FileUtils.expects(:cp).with("myscript", "/tmp/debian/test.postinst")
+ @packager.create_preandpost_install
+ end
+ end
+
+ describe "#create_install" do
+ before :each do
+ @packager = DebpackagePackager.new(@plugin)
+ @plugin.stubs(:path).returns("")
+ end
+
+ it "should raise an exception if the install file can't be created" do
+ File.expects(:join).raises("test error")
+ expect{
+ @packager.create_install
+ }.to raise_error(RuntimeError, "Could not create install file - test error")
+ end
+
+ it "should copy the package install file to the correct location" do
+ tmpdir = maketmpdir
+ Dir.mkdir(File.join(tmpdir, "debian"))
+ @packager.build_dir = tmpdir
+ @packager.current_package_shortname = "test"
+ @packager.current_package_data = {:files => ["foo.rb"]}
+ @packager.create_install
+ install_file = File.read("#{tmpdir}/debian/test.install")
+ install_file.should == "/usr/share/mcollective/plugins/mcollective/foo.rb /usr/share/mcollective/plugins/mcollective\n"
+ end
+ end
+
+ describe "#move_packages" do
+ before :each do
+ @plugin = mock()
+ end
+
+ it "should move the packages to the working directory" do
+ Dir.expects(:glob)
+ File.expects(:join)
+ FileUtils.expects(:cp)
+ @packager = DebpackagePackager.new(@plugin)
+ @packager.move_packages
+ end
+
+ it "should raise an error if the packages could not be moved" do
+ @packager = DebpackagePackager.new(@plugin)
+ File.expects(:join).raises("error")
+ expect{
+ @packager.move_packages
+ }.to raise_error(RuntimeError, "Could not copy packages to working directory: 'error'")
+ end
+ end
+
+ describe "#create_tar" do
+ before :each do
+ @packager = DebpackagePackager.new(@plugin, nil, true)
+ end
+
+ it "should raise an exception if the tarball can't be built" do
+ PluginPackager.expects(:do_quietly?).raises("test error")
+ expect{
+ @packager.create_tar
+ }.to raise_error(RuntimeError, "Could not create tarball - test error")
+ end
+
+ it "should create a tarball containing the package files" do
+ @packager.tmpdir = "/tmp"
+ @packager.build_dir = "/build_dir"
+ @packager.current_package_shortname = "test"
+ @plugin.stubs(:metadata).returns(@plugin)
+ @plugin.stubs(:[]).with(:version).returns("1")
+ @plugin.stubs(:iteration).returns("1")
+ PluginPackager.expects(:safe_system).with("tar -Pcvzf /tmp/test_1.orig.tar.gz test_1")
+ @packager.create_tar
+ end
+ end
+
+ describe "#create_file" do
+ before :each do
+ @packager = DebpackagePackager.new(@plugin)
+ end
+
+ it "should raise an exception if the file can't be created" do
+ File.expects(:dirname).raises("test error")
+ expect{
+ @packager.create_file("test")
+ }.to raise_error(RuntimeError, "could not create test file - test error")
+ end
+
+ it "should place a build file in the debian directory" do
+ tmpdir = maketmpdir
+ Dir.mkdir(File.join(tmpdir, "debian"))
+ @packager.build_dir = tmpdir
+ File.expects(:read).returns("testfile")
+ @packager.create_file("testfile")
+ File.unstub(:read)
+ result = File.read(File.join(tmpdir, "debian", "testfile"))
+ result.stubs(:result)
+ result.should == "testfile\n"
+ end
+ end
+
+ describe "#prepare_tmpdirs" do
+ before :each do
+ @tmpfile = Tempfile.new("mc-file").path
+ end
+
+ after :each do
+ begin
+ FileUtils.rm(@tmpfile)
+ rescue Exception
+ end
+ end
+
+ it "should create the correct tmp dirs and copy package contents to correct dir" do
+ packager = DebpackagePackager.new(@plugin)
+ tmpdir = maketmpdir
+ packager.build_dir = tmpdir
+ @plugin.stubs(:target_path).returns("")
+
+ packager.prepare_tmpdirs({:files => [@tmpfile]})
+ File.directory?(tmpdir).should == true
+ File.directory?(File.join(tmpdir, "debian")).should == true
+ File.exists?(File.join(tmpdir, packager.libdir, "tmp", File.basename(@tmpfile))).should == true
+ end
+ end
+
+ describe "#cleanup_tmpdirs" do
+ before :all do
+ @tmpdir = maketmpdir
+ end
+
+ before :each do
+ @packager = DebpackagePackager.new(@plugin)
+ end
+
+ it "should cleanup temp directories" do
+ @packager.tmpdir = @tmpdir
+ @packager.cleanup_tmpdirs
+ File.directory?(@tmpdir).should == false
+ end
+
+ it "should not delete any directories if @tmpdir isn't present" do
+ @packager = DebpackagePackager.new(@plugin)
+ @packager.tmpdir = rand.to_s
+ FileUtils.expects(:rm_r).never
+ @packager.cleanup_tmpdirs
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+MCollective::PluginManager.clear
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/pluginpackager/ospackage_packager.rb'
+
+module MCollective
+ module PluginPackager
+ describe "#initialize" do
+
+ before :all do
+ @packager = mock()
+ @packager.stubs(:new)
+ end
+
+ it "should correctly set members and create the correct packager on redhat" do
+ File.expects(:exists?).with("/etc/redhat-release").returns(true)
+ PluginPackager.expects(:[]).with("RpmpackagePackager").returns(@packager)
+ ospackager = OspackagePackager.new("package")
+ ospackager.package_type.should == "RPM"
+ end
+
+ it "should correctly set members and create the correct packager on debian" do
+ File.expects(:exists?).with("/etc/redhat-release").returns(false)
+ File.expects(:exists?).with("/etc/debian_version").returns(true)
+ PluginPackager.expects(:[]).with("DebpackagePackager").returns(@packager)
+ ospackager = OspackagePackager.new("package")
+ ospackager.package_type.should == "Deb"
+ end
+
+ it "should raise an exception if the os can't be identified" do
+ File.expects(:exists?).with("/etc/redhat-release").returns(false)
+ File.expects(:exists?).with("/etc/debian_version").returns(false)
+ expect{
+ OspackagePackager.new("package")
+ }.to raise_error(RuntimeError)
+ end
+ end
+
+ describe "#create_packages" do
+ before :all do
+ @packager = mock
+ @packager.stubs(:new).returns(@packager)
+ end
+
+ it "should call a packagers create_packages class" do
+ File.expects(:exists?).with("/etc/redhat-release").returns(true)
+ PluginPackager.expects(:[]).with("RpmpackagePackager").returns(@packager)
+ @packager.expects(:create_packages)
+ ospackager = OspackagePackager.new("package")
+ ospackager.class.should == OspackagePackager
+ ospackager.create_packages
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/pluginpackager/rpmpackage_packager.rb'
+
+module MCollective
+ module PluginPackager
+ describe RpmpackagePackager do
+ let(:maketmpdir) do
+ tmpdir = Dir.mktmpdir("mc-test")
+ @tmpdirs << tmpdir
+ tmpdir
+ end
+
+ before :all do
+ @tmpdirs = []
+ end
+
+ before :each do
+ PluginPackager.stubs(:build_tool?).with("rpmbuild-md5").returns(true)
+ PluginPackager.stubs(:build_tool?).with("rpmbuild").returns(true)
+ @plugin = mock()
+ @plugin.stubs(:iteration).returns("1")
+ @plugin.stubs(:metadata).returns({:name => "test", :version => "1"})
+ @plugin.stubs(:mcname).returns("mcollective")
+ RpmpackagePackager.any_instance.stubs(:rpmdir).returns('rpmdir')
+ RpmpackagePackager.any_instance.stubs(:srpmdir).returns('srpmdir')
+ end
+
+ after :all do
+ @tmpdirs.each{|tmpdir| FileUtils.rm_rf tmpdir if File.directory? tmpdir}
+ end
+
+ describe "#initialize" do
+
+ it "should raise and exception if neither rpmbuild or rpmbuild-md5 is installed is not present" do
+ PluginPackager.expects(:build_tool?).with("rpmbuild-md5").returns(false)
+ PluginPackager.expects(:build_tool?).with("rpmbuild").returns(false)
+ expect{
+ RpmpackagePackager.new("plugin")
+ }.to raise_exception(RuntimeError, "creating rpms require 'rpmbuild' or 'rpmbuild-md5' to be installed")
+ end
+
+ it "should set the correct libdir" do
+ packager = RpmpackagePackager.new("plugin")
+ packager.libdir.should == "/usr/libexec/mcollective/mcollective/"
+
+ packager = RpmpackagePackager.new("plugin", "/tmp/")
+ packager.libdir.should == "/tmp/"
+ end
+
+ end
+
+ describe "#create_packages" do
+ before :each do
+ @packager = RpmpackagePackager.new(@plugin)
+ @packager.tmpdir = maketmpdir
+ @packager.stubs(:create_package)
+ @packager.stubs(:cleanup_tmpdirs)
+ @plugin.stubs(:packagedata).returns(:test => {:files => ["test.rb"]})
+ @packager.stubs(:prepare_tmpdirs)
+ Dir.stubs(:mktmpdir)
+ end
+
+ it "should set the package instance variables" do
+ @packager.create_packages
+ @packager.current_package_type.should == :test
+ @packager.current_package_data.should == {:files => ["test.rb"]}
+ @packager.current_package_name.should == "mcollective-test-test"
+ end
+
+ it "should create the build dir" do
+ @packager.expects(:prepare_tmpdirs)
+ @packager.create_packages
+ end
+
+ it "should create packages" do
+ @packager.expects(:create_package)
+ @packager.create_packages
+ end
+
+ end
+
+ describe "#create_package" do
+ before :each do
+ @packager = RpmpackagePackager.new(@plugin)
+ end
+
+ it "should create the package" do
+ Dir.expects(:chdir)
+ PluginPackager.expects(:safe_system).with("rpmbuild-md5 -ta /tmp/mcollective-testplugin-test-1.tgz")
+ FileUtils.expects(:cp).times(2)
+ @packager.tmpdir = "/tmp"
+ @packager.verbose = "true"
+ @packager.expects(:make_spec_file)
+ @packager.current_package_name = "mcollective-testplugin-test"
+ @packager.expects(:puts).with('Created RPM and SRPM packages for mcollective-testplugin-test')
+ @packager.create_package(:test, {:files => ["foo.rb"]})
+ end
+
+ it "should sign the package if a signature is given" do
+ Dir.expects(:chdir)
+ PluginPackager.expects(:safe_system).with("rpmbuild-md5 -ta --sign /tmp/mcollective-testplugin-test-1.tgz")
+ FileUtils.expects(:cp).times(2)
+ @packager.signature = true
+ @packager.tmpdir = "/tmp"
+ @packager.verbose = "true"
+ @packager.expects(:make_spec_file)
+ @packager.current_package_name = "mcollective-testplugin-test"
+ @packager.expects(:puts).with('Created RPM and SRPM packages for mcollective-testplugin-test')
+ @packager.create_package(:test, {:files => ["foo.rb"]})
+ end
+
+ it "should raise an error if the package can't be built" do
+ @packager = RpmpackagePackager.new(@plugin)
+ @packager.tmpdir = "/tmp"
+ @packager.expects(:make_spec_file)
+ PluginPackager.stubs(:do_quietly?).raises("foo")
+ expect{
+ @packager.create_package("", "")
+ }.to raise_error(RuntimeError, "Could not build package. Reason - foo")
+ end
+ end
+
+ describe "#make_spec_file" do
+ before :each do
+ @plugin = mock
+ @packager = RpmpackagePackager.new(@plugin)
+ end
+
+ it "should raise an exception if specfile cannot be built" do
+ File.expects(:dirname).raises("test error")
+ expect{
+ @packager.make_spec_file
+ }.to raise_error(RuntimeError, "Could not create specfile - test error")
+ end
+
+ it "should create the specfile from the erb" do
+ File.stubs(:read).returns("specfile")
+ @plugin.stubs(:metadata).returns({:version => 2})
+ @packager.current_package_name = "test"
+ @packager.tmpdir = maketmpdir
+ Dir.mkdir(File.join(@packager.tmpdir, "test-2"))
+ @packager.make_spec_file
+ File.read(File.join(@packager.tmpdir, "test-2", "test-2.spec")).should == "specfile"
+ end
+ end
+
+ describe "#prepare_tmpdirs" do
+ it "should create the tmp dirs and cp the package files" do
+ @plugin.stubs(:target_path).returns("")
+ packager = RpmpackagePackager.new(@plugin)
+ FileUtils.expects(:mkdir_p)
+ File.stubs(:join).returns("/target")
+ FileUtils.expects(:cp_r).with("test.rb", "/target")
+ packager.prepare_tmpdirs({:files => ["test.rb"]})
+ end
+ end
+
+ describe "#cleanup_tmpdirs" do
+ it "should remove the temp directories" do
+ packager = RpmpackagePackager.new("package")
+ packager.tmpdir = maketmpdir
+ packager.cleanup_tmpdirs
+ File.directory?(packager.tmpdir).should == false
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/security/psk.rb'
+
+module MCollective::Security
+ describe Psk do
+ before do
+ @config = mock("config")
+ @config.stubs(:identity).returns("test")
+ @config.stubs(:configured).returns(true)
+ @config.stubs(:pluginconf).returns({"psk" => "12345"})
+
+ @stats = mock("stats")
+
+ @time = Time.now.to_i
+ ::Time.stubs(:now).returns(@time)
+
+ MCollective::Log.stubs(:debug).returns(true)
+
+ MCollective::PluginManager << {:type => "global_stats", :class => @stats}
+ MCollective::Config.stubs("instance").returns(@config)
+ MCollective::Util.stubs("empty_filter?").returns(false)
+
+ @plugin = Psk.new
+ end
+
+ describe "#decodemsg" do
+ it "should correctly decode a message" do
+ @plugin.stubs("validrequest?").returns(true).once
+
+ msg = mock("message")
+ msg.stubs(:payload).returns(Marshal.dump({:body => Marshal.dump("foo")}))
+ msg.stubs(:expected_msgid).returns(nil)
+
+ @plugin.decodemsg(msg).should == {:body=>"foo"}
+ end
+
+ it "should return nil on failure" do
+ @plugin.stubs("validrequest?").raises("fail").once
+
+ msg = mock("message")
+ msg.stubs(:payload).returns(Marshal.dump({:body => Marshal.dump("foo"), :requestid => "123"}))
+ msg.stubs(:expected_msgid).returns(nil)
+
+ expect { @plugin.decodemsg(msg) }.to raise_error("fail")
+ end
+
+ it "should not decode messages not addressed to us" do
+ msg = mock("message")
+ msg.stubs(:payload).returns(Marshal.dump({:body => Marshal.dump("foo"), :requestid => "456"}))
+ msg.stubs(:expected_msgid).returns("123")
+
+ expect {
+ @plugin.decodemsg(msg)
+ }.to raise_error("Got a message with id 456 but was expecting 123, ignoring message")
+
+ end
+
+ it "should only decode messages addressed to us" do
+ @plugin.stubs("validrequest?").returns(true).once
+
+ msg = mock("message")
+ msg.stubs(:payload).returns(Marshal.dump({:body => Marshal.dump("foo"), :requestid => "456"}))
+ msg.stubs(:expected_msgid).returns("456")
+
+ @plugin.decodemsg(msg).should == {:body=>"foo", :requestid=>"456"}
+ end
+ end
+
+ describe "#encodereply" do
+ it "should correctly Marshal encode the reply" do
+ @plugin.stubs("create_reply").returns({:test => "test"})
+ Marshal.stubs("dump").with("test message").returns("marshal_test_message").once
+ Marshal.stubs("dump").with({:hash => '2dbeb0d7938a08a34eacd2c1dab25602', :test => 'test'}).returns("marshal_test_reply").once
+
+ @plugin.encodereply("sender", "test message", "requestid", "callerid").should == "marshal_test_reply"
+ end
+ end
+
+ describe "#encoderequest" do
+ it "should correctly Marshal encode the request" do
+ @plugin.stubs("create_request").returns({:test => "test"})
+ Marshal.stubs("dump").with("test message").returns("marshal_test_message").once
+ Marshal.stubs("dump").with({:hash => '2dbeb0d7938a08a34eacd2c1dab25602', :test => 'test'}).returns("marshal_test_request").once
+
+ @plugin.encoderequest("sender", "test message", "requestid", "filter", "agent", "collective").should == "marshal_test_request"
+ end
+ end
+
+ describe "#validrequest?" do
+ it "should correctly validate requests" do
+ @stats.stubs(:validated).once
+ @stats.stubs(:unvalidated).never
+ @plugin.validrequest?({:body => "foo", :hash => "e83ac78027b77b659a49bccbbcfa4849"})
+ end
+
+ it "should raise an exception on failure" do
+ @stats.stubs(:validated).never
+ @stats.stubs(:unvalidated).once
+ expect { @plugin.validrequest?({:body => "foo", :hash => ""}) }.to raise_error("Received an invalid signature in message")
+ end
+ end
+
+ describe "#callerid" do
+ it "should do uid based callerid when unconfigured" do
+ @plugin.callerid.should == "uid=#{Process.uid}"
+ end
+
+ it "should support gid based callerids" do
+ @config.stubs(:pluginconf).returns({"psk.callertype" => "gid"})
+ @plugin.callerid.should == "gid=#{Process.gid}"
+ end
+
+ it "should support group based callerids", :unless => MCollective::Util.windows? do
+ @config.stubs(:pluginconf).returns({"psk.callertype" => "group"})
+ @plugin.callerid.should == "group=#{Etc.getgrgid(Process.gid).name}"
+ end
+
+ it "should raise an error if the group callerid type is used on windows" do
+ MCollective::Util.expects("windows?").returns(true)
+ @config.stubs(:pluginconf).returns({"psk.callertype" => "group"})
+ expect { @plugin.callerid }.to raise_error("Cannot use the 'group' callertype for the PSK security plugin on the Windows platform")
+ end
+
+ it "should support user based callerids" do
+ @config.stubs(:pluginconf).returns({"psk.callertype" => "user"})
+ @plugin.callerid.should == "user=#{Etc.getlogin}"
+ end
+
+ it "should support identity based callerids" do
+ @config.stubs(:pluginconf).returns({"psk.callertype" => "identity"})
+ @plugin.callerid.should == "identity=test"
+ end
+ end
+
+ describe "#makehash" do
+ it "should return the correct md5 digest" do
+ @plugin.send(:makehash, "foo").should == "e83ac78027b77b659a49bccbbcfa4849"
+ end
+
+ it "should fail if no PSK is configured" do
+ @config.stubs(:pluginconf).returns({})
+ expect { @plugin.send(:makehash, "foo") }.to raise_error("No plugin.psk configuration option specified")
+ end
+
+ it "should support reading the PSK from the environment" do
+ ENV["MCOLLECTIVE_PSK"] = "54321"
+
+ @plugin.send(:makehash, "foo").should == "d3fb63cc6b1d47cc4b2012df926c2feb"
+
+ ENV.delete("MCOLLECTIVE_PSK")
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/validator/array_validator.rb'
+
+module MCollective
+ module Validator
+ describe "#validate" do
+ it "should raise an exception if a given element is not defined in a given array" do
+ expect{
+ Validator::ArrayValidator.validate("element1",["element0", "element2"])
+ }.to raise_error(ValidatorError, "value should be one of element0, element2")
+ end
+
+ it "should not raise an exception if a given element is defined in a given array" do
+ Validator::ArrayValidator.validate("element1", ["element1", "element2"])
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/validator/ipv4address_validator.rb'
+
+module MCollective
+ module Validator
+ describe "#validate" do
+ it "should raise an exception if the supplied value is not an ipv4 address" do
+ expect{
+ Ipv4addressValidator.validate("foobar")
+ }.to raise_error(ValidatorError, "value should be an ipv4 address")
+ end
+
+ it "should not raise an exception if the supplied value is an ipv4 address" do
+ Ipv4addressValidator.validate("1.2.3.4")
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/validator/ipv6address_validator.rb'
+
+module MCollective
+ module Validator
+ describe "#validate" do
+ it "should raise an exception if the supplied value is not an ipv6 address" do
+ expect{
+ Ipv6addressValidator.validate("foobar")
+ }.to raise_error(ValidatorError, "value should be an ipv6 address")
+ end
+
+ it "should not raise an exception if the supplied value is an ipv6 address" do
+ Ipv6addressValidator.validate("2001:db8:85a3:8d3:1319:8a2e:370:7348")
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/validator/length_validator.rb'
+
+module MCollective
+ module Validator
+ describe "#validate" do
+ it "should raise an exception if the given string's length is greater than the given value" do
+ expect{
+ LengthValidator.validate("test", 3)
+ }.to raise_error(ValidatorError, "Input string is longer than 3 character(s)")
+ end
+
+ it "should not raise an exception if the given string's length is less than the given value" do
+ LengthValidator.validate("test", 4)
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/validator/regex_validator.rb'
+
+module MCollective
+ module Validator
+ describe "#validate" do
+ it "should raise an exception if the given string does not matches the given regular expression" do
+ expect{
+ RegexValidator.validate("test", "nottest")
+ }.to raise_error(ValidatorError, "value should match nottest")
+ end
+
+ it "should not raise an exception if the given string's length is less than the given value" do
+ RegexValidator.validate("test", "test")
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/validator/shellsafe_validator.rb'
+
+module MCollective
+ module Validator
+ describe "#validate" do
+ it "should raise an exception if the given string is not shellsafe" do
+ ['`', '$', ';', '|', '&&', '>', '<'].each do |chr|
+ expect{
+ ShellsafeValidator.validate("#{chr}test")
+ }.to raise_error(ValidatorError, "value should not have #{chr} in it")
+ end
+ end
+
+ it "should not raise an exception if the given string is shellsafe" do
+ ShellsafeValidator.validate("test")
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../../../../plugins/mcollective/validator/typecheck_validator.rb'
+
+module MCollective
+ module Validator
+ describe "#validate" do
+ it "should raise an exception if the given value is not of the supplied type" do
+ [[1, String], ['test', :integer], ['test', :float], ['test', :number], [1, :string], ['test', :boolean]].each do |val|
+ expect{
+ TypecheckValidator.validate(*val)
+ }.to raise_error(ValidatorError, "value should be a #{val[1].to_s}")
+ end
+ end
+
+ it "should not raise an exception if the given value is of the supplied type" do
+ [["test", String], [1, :integer], [1.2, :float], [1, :number], ["test", :string], [true, :boolean]].each do |val|
+ TypecheckValidator.validate(*val)
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module Registration
+ describe Base do
+ before do
+ @config = mock
+ @config.stubs(:identity).returns("rspec")
+ @config.stubs(:main_collective).returns("main_collective")
+ Config.stubs(:instance).returns(@config)
+
+ @reg = Base.new
+ end
+
+ describe "#config" do
+ it "should provide access the main configuration class" do
+ @reg.config.should == @config
+ end
+
+ end
+
+ describe "#identity" do
+ it "should return the correct identity" do
+ @reg.config.identity.should == "rspec"
+ end
+ end
+
+ describe "#msg_filter" do
+ it "should target the registration agent" do
+ @reg.msg_filter["agent"].should == ["registration"]
+ end
+ end
+
+ describe "#target_collective" do
+ it "should return the configured registration_collective" do
+ @config.expects(:registration_collective).returns("registration").once
+ @config.expects(:collectives).returns(["main_collective", "registration"]).once
+ @reg.target_collective.should == "registration"
+ end
+
+ it "should use the main collective if registration collective is not valid" do
+ @config.expects(:registration_collective).returns("registration").once
+ @config.expects(:collectives).returns(["main_collective"]).once
+
+ Log.expects(:warn).with("Sending registration to main_collective: registration is not a valid collective").once
+
+ @reg.target_collective.should == "main_collective"
+ end
+ end
+
+ describe "#publish" do
+ it "should skip registration for empty messages" do
+ Log.expects(:debug).with("Skipping registration due to nil body")
+ @reg.publish(nil)
+ end
+
+ it "should publish via the message object" do
+ message = mock
+ message.expects(:encode!)
+ message.expects(:publish)
+ message.expects(:requestid).returns("123")
+ message.expects(:collective).returns("mcollective")
+
+ Message.expects(:new).returns(message)
+
+ Log.expects(:debug).with("Sending registration 123 to collective mcollective")
+
+ @reg.expects(:target_collective).returns("mcollective")
+
+ @reg.publish("message")
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module RPC
+ describe ActionRunner do
+ before(:each) do
+ @req = mock
+ @req.stubs(:agent).returns("spectester")
+ @req.stubs(:action).returns("tester")
+
+ command = "/bin/echo 1"
+
+ @runner = ActionRunner.new(command, @req, :json)
+ end
+
+ describe "#initialize" do
+ it "should set command" do
+ @runner.command.should == "/bin/echo 1"
+ end
+
+ it "should set agent" do
+ @runner.agent.should == "spectester"
+ end
+
+ it "should set action" do
+ @runner.action.should == "tester"
+ end
+
+ it "should set format" do
+ @runner.format.should == :json
+ end
+
+ it "should set request" do
+ @runner.request.should == @req
+ end
+
+ it "should set stdout" do
+ @runner.stdout.should == ""
+ end
+
+ it "should set stderr" do
+ @runner.stderr.should == ""
+ end
+
+ it "should set the command via path_to_command" do
+ ActionRunner.any_instance.expects(:path_to_command).with("rspec").once
+ ActionRunner.new("rspec", @req, :json)
+ end
+ end
+
+ describe "#shell" do
+ it "should create a shell instance with correct settings" do
+ s = @runner.shell("test", "infile", "outfile")
+
+ s.command.should == "test infile outfile"
+ s.cwd.should == Dir.tmpdir
+ s.stdout.should == ""
+ s.stderr.should == ""
+ s.environment["MCOLLECTIVE_REQUEST_FILE"].should == "infile"
+ s.environment["MCOLLECTIVE_REPLY_FILE"].should == "outfile"
+ end
+ end
+
+ describe "#load_results" do
+ it "should call the correct format loader" do
+ req = mock
+ req.expects(:agent).returns("spectester")
+ req.expects(:action).returns("tester")
+
+ runner = ActionRunner.new("/bin/echo 1", req, :foo)
+ runner.expects("load_foo_results").returns({:foo => :bar})
+ runner.load_results("/dev/null").should == {:foo => :bar}
+ end
+
+ it "should set all keys to Symbol" do
+ data = {"foo" => "bar", "bar" => "baz"}
+ Tempfile.open("mcollective_test", Dir.tmpdir) do |f|
+ f.puts data.to_json
+ f.close
+
+ results = @runner.load_results(f.path)
+ results.should == {:foo => "bar", :bar => "baz"}
+ end
+ end
+ end
+
+ describe "#load_json_results" do
+ it "should load data from a file" do
+ Tempfile.open("mcollective_test", Dir.tmpdir) do |f|
+ f.puts '{"foo":"bar","bar":"baz"}'
+ f.close
+
+ @runner.load_json_results(f.path).should == {"foo" => "bar", "bar" => "baz"}
+ end
+
+ end
+
+ it "should return empty data on JSON parse error" do
+ @runner.load_json_results("/dev/null").should == {}
+ end
+
+ it "should return empty data for missing files" do
+ @runner.load_json_results("/nonexisting").should == {}
+ end
+
+ it "should load complex data correctly" do
+ data = {"foo" => "bar", "bar" => {"one" => "two"}}
+ Tempfile.open("mcollective_test", Dir.tmpdir) do |f|
+ f.puts data.to_json
+ f.close
+
+ @runner.load_json_results(f.path).should == data
+ end
+ end
+
+ end
+
+ describe "#saverequest" do
+ it "should call the correct format serializer" do
+ req = mock
+ req.expects(:agent).returns("spectester")
+ req.expects(:action).returns("tester")
+
+ runner = ActionRunner.new("/bin/echo 1", req, :foo)
+
+ runner.expects("save_foo_request").with(req).returns('{"foo":"bar"}')
+
+ runner.saverequest(req)
+ end
+
+ it "should save to a temp file" do
+ @req.expects(:to_json).returns({:foo => "bar"}.to_json)
+ fname = @runner.saverequest(@req).path
+
+ JSON.load(File.read(fname)).should == {"foo" => "bar"}
+ File.dirname(fname).should == Dir.tmpdir
+ end
+ end
+
+ describe "#save_json_request" do
+ it "should return correct json data" do
+ @req.expects(:to_json).returns({:foo => "bar"}.to_json)
+ @runner.save_json_request(@req).should == '{"foo":"bar"}'
+ end
+ end
+
+ describe "#canrun?" do
+ it "should correctly report executables" do
+ if Util.windows?
+ @runner.canrun?(File.join(ENV['SystemRoot'], "explorer.exe")).should == true
+ else
+ @runner.canrun?("/bin/true").should == true
+ end
+ end
+
+ it "should detect missing files" do
+ @runner.canrun?("/nonexisting").should == false
+ end
+ end
+
+ describe "#to_s" do
+ it "should return correct data" do
+ @runner.to_s.should == "spectester#tester command: /bin/echo 1"
+ end
+ end
+
+ describe "#tempfile" do
+ it "should return a TempFile" do
+ @runner.tempfile("foo").class.should == Tempfile
+ end
+
+ it "should contain the prefix in its name" do
+ @runner.tempfile("foo").path.should match(/foo/)
+ end
+ end
+
+ describe "#path_to_command" do
+ it "should return the command if it starts with separator" do
+ command = "#{File::SEPARATOR}rspec"
+
+ runner = ActionRunner.new(command , @req, :json)
+ runner.path_to_command(command).should == command
+ end
+
+ it "should find the first match in the libdir" do
+ Config.instance.expects(:libdir).returns(["#{File::SEPARATOR}libdir1", "#{File::SEPARATOR}libdir2"])
+
+ action_in_first_dir = File.join(File::SEPARATOR, "libdir1", "agent", "spectester", "action.sh")
+ action_in_last_dir = File.join(File::SEPARATOR, "libdir2", "agent", "spectester", "action.sh")
+
+ File.expects("exist?").with(action_in_first_dir).returns(true)
+ File.expects("exist?").with(action_in_last_dir).never
+
+ ActionRunner.new("action.sh", @req, :json).command.should == action_in_first_dir
+ end
+
+ it "should find the match even in the last libdir" do
+ Config.instance.expects(:libdir).returns(["#{File::SEPARATOR}libdir1", "#{File::SEPARATOR}libdir2"])
+
+ action_in_first_dir = File.join(File::SEPARATOR, "libdir1", "agent", "spectester", "action.sh")
+ action_in_last_dir = File.join(File::SEPARATOR, "libdir2", "agent", "spectester", "action.sh")
+
+ File.expects("exist?").with(action_in_first_dir).returns(false)
+ File.expects("exist?").with(action_in_last_dir).returns(true)
+
+ ActionRunner.new("action.sh", @req, :json).command.should == action_in_last_dir
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module RPC
+ describe Agent do
+ before do
+ ddl = stub
+ ddl.stubs(:meta).returns({})
+ ddl.stubs(:action).returns([])
+ ddl.stubs(:validate_rpc_request).returns(true)
+ DDL.stubs(:new).returns(ddl)
+
+ @agent = Agent.new
+ @agent.reply = {}
+ @agent.request = {}
+ end
+
+ describe "#handlemsg" do
+ before do
+ Reply.any_instance.stubs(:initialize_data)
+
+ @agent.stubs(:respond_to?).with("rspec_action_action").returns(true)
+ @agent.stubs(:respond_to?).with("authorization_hook").returns(false)
+ @agent.stubs(:rspec_action_action).returns(nil)
+
+ @msg = {:msgtime => 1356006671,
+ :senderid => "example.com",
+ :requestid => "55f8abe1442328321667877a08bdc586",
+ :body => {:agent => "rspec_agent",
+ :action => "rspec_action",
+ :data => {}},
+ :caller => "cert=rspec"}
+ end
+
+ it "should or validate the incoming request" do
+ exception = DDLValidationError.new(:RSPEC, "Failed to validate", :error)
+ Request.any_instance.expects(:validate!).raises(exception)
+
+ reply = @agent.handlemsg(@msg, DDL.new)
+
+ reply[:statuscode].should == 4
+ reply[:statusmsg].should == "Failed to validate"
+ end
+
+ it "should call the authorization hook if set" do
+ @agent.expects(:respond_to?).with("authorization_hook").returns(true)
+ @agent.expects(:authorization_hook).raises("authorization denied")
+ Log.stubs(:error)
+
+ reply = @agent.handlemsg(@msg, DDL.new)
+
+ reply[:statuscode].should == 5
+ reply[:statusmsg].should == "authorization denied"
+ end
+
+ it "should audit the request" do
+ @agent.expects(:audit_request)
+
+ reply = @agent.handlemsg(@msg, DDL.new)
+ reply[:statuscode].should == 0
+ end
+
+ it "should call the before_processing_hook" do
+ @agent.expects(:before_processing_hook)
+
+ reply = @agent.handlemsg(@msg, DDL.new)
+ reply[:statuscode].should == 0
+ end
+
+ it "should fail if the action does not exist" do
+ @agent.expects(:respond_to?).with("rspec_action_action").returns(false)
+ reply = @agent.handlemsg(@msg, DDL.new)
+ reply[:statuscode].should == 2
+ end
+
+ it "should call the action correctly" do
+ @agent.expects(:rspec_action_action)
+ reply = @agent.handlemsg(@msg, DDL.new)
+ reply[:statuscode].should == 0
+ end
+
+ it "should handle RPC Aborted errors" do
+ @agent.expects(:rspec_action_action).raises(RPCAborted, "rspec test")
+ reply = @agent.handlemsg(@msg, DDL.new)
+ reply[:statuscode].should == 1
+ reply[:statusmsg].should == "rspec test"
+ end
+
+ it "should handle Unknown Action errors" do
+ @agent.stubs(:respond_to?).with("rspec_action_action").returns(false)
+ reply = @agent.handlemsg(@msg, DDL.new)
+ reply[:statuscode].should == 2
+ reply[:statusmsg].should == "Unknown action 'rspec_action' for agent 'rspec_agent'"
+ end
+
+ it "should handle Missing Data errors" do
+ @agent.expects(:rspec_action_action).raises(MissingRPCData, "rspec test")
+ reply = @agent.handlemsg(@msg, DDL.new)
+ reply[:statuscode].should == 3
+ reply[:statusmsg].should == "rspec test"
+ end
+
+ it "should handle Invalid Data errors" do
+ @agent.expects(:rspec_action_action).raises(InvalidRPCData, "rspec test")
+ reply = @agent.handlemsg(@msg, DDL.new)
+ reply[:statuscode].should == 4
+ reply[:statusmsg].should == "rspec test"
+ end
+
+ it "should handle unknown errors" do
+ @agent.expects(:rspec_action_action).raises(UnknownRPCError, "rspec test")
+ Log.expects(:error).twice
+
+ reply = @agent.handlemsg(@msg, DDL.new)
+ reply[:statuscode].should == 5
+ reply[:statusmsg].should == "rspec test"
+ end
+
+ it "should handle arbitrary exceptions" do
+ @agent.expects(:rspec_action_action).raises(Exception, "rspec test")
+ Log.expects(:error).twice
+
+ reply = @agent.handlemsg(@msg, DDL.new)
+ reply[:statuscode].should == 5
+ reply[:statusmsg].should == "rspec test"
+ end
+
+ it "should call the after_processing_hook" do
+ @agent.expects(:after_processing_hook)
+ reply = @agent.handlemsg(@msg, DDL.new)
+ end
+
+ it "should respond if required" do
+ Request.any_instance.expects(:should_respond?).returns(true)
+ Reply.any_instance.expects(:to_hash).returns({})
+ @agent.handlemsg(@msg, DDL.new).should == {}
+ end
+
+ it "should not respond when not required" do
+ Request.any_instance.expects(:should_respond?).returns(false)
+ Reply.any_instance.expects(:to_hash).never
+ @agent.handlemsg(@msg, DDL.new).should == nil
+ end
+ end
+
+ describe "#meta" do
+ it "should be deprecated" do
+ Agent.expects(:log_code).with(:PLMC34, is_a(String), :warn, has_value(regexp_matches(/agent_spec.rb/)))
+ Agent.metadata("foo")
+ end
+ end
+
+ describe "#load_ddl" do
+ it "should load the correct DDL" do
+ ddl = stub
+ ddl.stubs(:meta).returns({:timeout => 5})
+
+ DDL.expects(:new).with("agent", :agent).returns(ddl)
+
+ Agent.new.timeout.should == 5
+ end
+
+ it "should fail if the DDL isn't loaded" do
+ DDL.expects(:new).raises("failed to load")
+ expect { Agent.new }.to raise_code(:PLMC24)
+ end
+
+ it "should default to 10 second timeout" do
+ ddl = stub
+ ddl.stubs(:meta).returns({})
+
+ DDL.expects(:new).with("agent", :agent).returns(ddl)
+
+ Agent.new.timeout.should == 10
+ end
+ end
+
+ describe "#run" do
+ before do
+ @status = mock
+ @status.stubs(:exitstatus).returns(0)
+ @shell = mock
+ @shell.stubs(:runcommand)
+ @shell.stubs(:status).returns(@status)
+ end
+
+ it "should accept stderr and stdout and force them to be strings" do
+ Shell.expects(:new).with("rspec", {:stderr => "", :stdout => ""}).returns(@shell)
+ @agent.send(:run, "rspec", {:stderr => :err, :stdout => :out})
+ @agent.reply[:err].should == ""
+ @agent.reply[:out].should == ""
+ end
+
+ it "should accept existing variables for stdout and stderr and fail if they dont support <<" do
+ @agent.reply[:err] = "err"
+ @agent.reply[:out] = "out"
+
+ Shell.expects(:new).with("rspec", {:stderr => "err", :stdout => "out"}).returns(@shell)
+ @agent.send(:run, "rspec", {:stderr => @agent.reply[:err], :stdout => @agent.reply[:out]})
+ @agent.reply[:err].should == "err"
+ @agent.reply[:out].should == "out"
+
+ @agent.reply.expects("fail!").with("stderr should support << while calling run(rspec)").raises("stderr fail")
+ expect { @agent.send(:run, "rspec", {:stderr => nil, :stdout => ""}) }.to raise_error("stderr fail")
+
+ @agent.reply.expects("fail!").with("stdout should support << while calling run(rspec)").raises("stdout fail")
+ expect { @agent.send(:run, "rspec", {:stderr => "", :stdout => nil}) }.to raise_error("stdout fail")
+ end
+
+ it "should set stdin, cwd and environment if supplied" do
+ Shell.expects(:new).with("rspec", {:stdin => "stdin", :cwd => "cwd", :environment => "env"}).returns(@shell)
+ @agent.send(:run, "rspec", {:stdin => "stdin", :cwd => "cwd", :environment => "env"})
+ end
+
+ it "should ignore unknown options" do
+ Shell.expects(:new).with("rspec", {}).returns(@shell)
+ @agent.send(:run, "rspec", {:rspec => "rspec"})
+ end
+
+ it "should chomp strings if configured to do so" do
+ Shell.expects(:new).with("rspec", {:stderr => 'err', :stdout => 'out'}).returns(@shell)
+
+ @agent.reply[:err] = "err"
+ @agent.reply[:out] = "out"
+
+ @agent.reply[:err].expects("chomp!")
+ @agent.reply[:out].expects("chomp!")
+
+ @agent.send(:run, "rspec", {:chomp => true, :stdout => @agent.reply[:out], :stderr => @agent.reply[:err]})
+ end
+
+ it "should return the exitstatus" do
+ Shell.expects(:new).with("rspec", {}).returns(@shell)
+ @agent.send(:run, "rspec", {}).should == 0
+ end
+
+ it "should handle nil from the shell handler" do
+ @shell.expects(:status).returns(nil)
+ Shell.expects(:new).with("rspec", {}).returns(@shell)
+ @agent.send(:run, "rspec", {}).should == -1
+ end
+ end
+
+ describe "#validate" do
+ it "should detect missing data" do
+ @agent.request = {}
+ expect { @agent.send(:validate, :foo, String) }.to raise_error(MissingRPCData, "please supply a foo argument")
+ end
+
+ it "should catch validation errors and turn them into use case specific ones" do
+ @agent.request = {:input_key => "should be a number"}
+ Validator.expects(:validate).raises(ValidatorError, "input_key should be a number")
+ expect { @agent.send(:validate, :input_key, :number) }.to raise_error("Input input_key did not pass validation: input_key should be a number")
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module RPC
+ describe Client do
+ before do
+ @coreclient = mock
+ @discoverer = mock
+
+ ddl = DDL.new("foo", "agent", false)
+ ddl.action("rspec", :description => "mock agent")
+
+ ddl.stubs(:meta).returns({:timeout => 2})
+ DDL.stubs(:new).returns(ddl)
+
+ @discoverer.stubs(:force_direct_mode?).returns(false)
+ @discoverer.stubs(:discovery_method).returns("mc")
+ @discoverer.stubs(:force_discovery_method_by_filter).returns(false)
+ @discoverer.stubs(:discovery_timeout).returns(2)
+ @discoverer.stubs(:ddl).returns(ddl)
+
+ @coreclient.stubs("options=")
+ @coreclient.stubs(:collective).returns("mcollective")
+ @coreclient.stubs(:timeout_for_compound_filter).returns(0)
+ @coreclient.stubs(:discoverer).returns(@discoverer)
+
+ Config.instance.stubs(:loadconfig).with("/nonexisting").returns(true)
+ Config.instance.stubs(:direct_addressing).returns(true)
+ Config.instance.stubs(:collectives).returns(["mcollective", "rspec"])
+ MCollective::Client.stubs(:new).returns(@coreclient)
+
+ @stderr = StringIO.new
+ @stdout = StringIO.new
+
+ @client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ @client.stubs(:ddl).returns(ddl)
+ end
+
+ describe "#initialize" do
+ it "should fail for missing DDLs" do
+ DDL.stubs(:new).raises("DDL failure")
+ expect { Client.new("foo", {:options => {:config => "/nonexisting"}}) }.to raise_error("DDL failure")
+ end
+
+ it "should set a empty filter when none is supplied" do
+ filter = Util.empty_filter
+ Util.expects(:empty_filter).once.returns(filter)
+
+ Client.new("foo", :options => {:config => "/nonexisting"})
+ end
+
+ it "should default the discovery_timeout to nil" do
+ c = Client.new("rspec", :options => {:config => "/nonexisting"})
+ c.instance_variable_get("@discovery_timeout").should == nil
+ end
+
+ it "should accept a supplied discovery_timeout" do
+ c = Client.new("rspec", :options => {:config => "/nonexisting", :disctimeout => 10})
+ c.instance_variable_get("@discovery_timeout").should == 10
+ end
+ end
+
+ describe "#validate_request" do
+ it "should fail when a DDL isn't present" do
+ @client.instance_variable_set("@ddl", nil)
+ expect { @client.validate_request("rspec", {}) }.to raise_error("No DDL found for agent foo cannot validate inputs")
+ end
+
+ it "should validate the input arguments" do
+ @client.ddl.expects(:set_default_input_arguments).with("rspec", {})
+ @client.ddl.expects(:validate_rpc_request).with("rspec", {})
+ @client.validate_request("rspec", {})
+ end
+ end
+
+ describe "#process_results_with_block" do
+ it "should inform the stats object correctly for passed requests" do
+ response = {:senderid => "rspec", :body => {:statuscode => 0}}
+
+ @client.stubs(:rpc_result_from_reply).with("foo", "rspec", response)
+ @client.stats.expects(:ok)
+ @client.stats.expects(:node_responded).with("rspec")
+ @client.stats.expects(:time_block_execution).with(:start)
+ @client.stats.expects(:time_block_execution).with(:end)
+ @client.expects(:aggregate_reply).returns("aggregate stub")
+
+ blk = Proc.new {}
+
+ @client.process_results_with_block("rspec", response, blk, "").should == "aggregate stub"
+ end
+
+ it "should inform the stats object correctly for failed requests" do
+ @client.stats.expects(:fail)
+ @client.stats.expects(:node_responded).with("rspec")
+
+ response = {:senderid => "rspec", :body => {:statuscode => 1}}
+ blk = Proc.new {}
+
+ @client.stubs(:rpc_result_from_reply).with("foo", "rspec", response)
+ @client.process_results_with_block("rspec", response, blk, nil)
+ end
+
+ it "should raise correct exceptions on failure" do
+ blk = Proc.new {}
+
+ @client.stubs(:rpc_result_from_reply)
+
+ [[2, UnknownRPCAction], [3, MissingRPCData], [4, InvalidRPCData], [5, UnknownRPCError]].each do |err|
+ response = {:senderid => "rspec", :body => {:statuscode => err[0]}}
+
+ expect { @client.process_results_with_block("rspec", response, blk, nil) }.to raise_error(err[1])
+ end
+ end
+
+ it "should pass raw results for single arity blocks" do
+ response = {:senderid => "rspec", :body => {:statuscode => 1}}
+ blk = Proc.new {|r| r.should == response}
+
+ @client.stubs(:rpc_result_from_reply).with("foo", "rspec", response)
+ @client.process_results_with_block("rspec", response, blk, nil)
+ end
+
+ it "should pass raw and rpc style results for 2 arity blocks" do
+ response = {:senderid => "rspec", :body => {:statuscode => 1}}
+ blk = Proc.new do |r, s|
+ r.should == response
+ s.should.class == RPC::Result
+ end
+
+ @client.process_results_with_block("rspec", response, blk, nil)
+ end
+ end
+
+ describe "#process_results_without_block" do
+ it "should inform the stats object correctly for passed requests" do
+ response = {:senderid => "rspec", :body => {:statuscode => 0}}
+ @client.stubs(:rpc_result_from_reply).with("foo", "rspec", response)
+ @client.stats.expects(:ok)
+ @client.stats.expects(:node_responded).with("rspec")
+ @client.process_results_without_block(response, "rspec", nil)
+ end
+
+ it "should inform the stats object correctly for failed requests" do
+ @client.stats.expects(:fail).twice
+ @client.stats.expects(:node_responded).with("rspec").twice
+
+ response = {:senderid => "rspec", :body => {:statuscode => 1}}
+ @client.stubs(:rpc_result_from_reply).with("foo", "rspec", response)
+ @client.process_results_without_block(response, "rspec", nil)
+
+ response = {:senderid => "rspec", :body => {:statuscode => 3}}
+ @client.stubs(:rpc_result_from_reply).with("foo", "rspec", response)
+ @client.process_results_without_block(response, "rspec", nil)
+ end
+
+ it "should return the result and the aggregate" do
+ @client.expects(:aggregate_reply).returns("aggregate stub")
+
+ response = {:senderid => "rspec", :body => {:statuscode => 0}}
+ result = @client.rpc_result_from_reply("foo", "rspec", response)
+
+ @client.stubs(:rpc_result_from_reply).with("foo", "rspec", response).returns(result)
+ @client.process_results_without_block(response, "rspec", "").should == [result, "aggregate stub"]
+ end
+ end
+
+ describe "#load_aggregate_functions" do
+ it "should not load if the ddl is not set" do
+ @client.load_aggregate_functions("rspec", nil).should == nil
+ end
+
+ it "should create the aggregate for the right action" do
+ @client.ddl.expects(:action_interface).with("rspec").returns({:aggregate => []}).twice
+ Aggregate.expects(:new).with(:aggregate => []).returns("rspec aggregate")
+ @client.load_aggregate_functions("rspec", @client.ddl).should == "rspec aggregate"
+ end
+
+ it "should log and return nil on failure" do
+ @client.ddl.expects(:action_interface).raises("rspec")
+ Log.expects(:error).with(regexp_matches(/Failed to load aggregate/))
+ @client.load_aggregate_functions("rspec", @client.ddl)
+ end
+ end
+
+ describe "#aggregate_reply" do
+ it "should not call anything if the aggregate isnt set" do
+ @client.aggregate_reply(nil, nil).should == nil
+ end
+
+ it "should call the aggregate functions with the right data" do
+ result = @client.rpc_result_from_reply("rspec", "rspec", {:body => {:data => "rspec"}})
+
+ aggregate = mock
+ aggregate.expects(:call_functions).with(result).returns(aggregate)
+
+ @client.aggregate_reply(result, aggregate).should == aggregate
+ end
+
+ it "should log and return nil on failure" do
+ aggregate = mock
+ aggregate.expects(:call_functions).raises
+
+ Log.expects(:error).with(regexp_matches(/Failed to calculate aggregate summaries/))
+
+ @client.aggregate_reply({}, aggregate).should == nil
+ end
+ end
+
+ describe "#collective=" do
+ it "should validate the collective" do
+ expect { @client.collective = "fail" }.to raise_error("Unknown collective fail")
+ @client.collective = "rspec"
+ end
+
+ it "should set the collective" do
+ @client.options[:collective].should == "mcollective"
+ @client.collective = "rspec"
+ @client.options[:collective].should == "rspec"
+ end
+
+ it "should reset the client" do
+ @client.expects(:reset)
+ @client.collective = "rspec"
+ end
+ end
+
+ describe "#discovery_method=" do
+ it "should set the method" do
+ @client.discovery_method = "rspec"
+ @client.discovery_method.should == "rspec"
+ end
+
+ it "should set initial options if provided" do
+ client = Client.new("rspec", {:options => {:discovery_options => ["rspec"], :filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.discovery_method = "rspec"
+ client.discovery_method.should == "rspec"
+ client.discovery_options.should == ["rspec"]
+ end
+
+ it "should clear the options if none are given initially" do
+ @client.discovery_options = ["rspec"]
+ @client.discovery_method = "rspec"
+ @client.discovery_options.should == []
+ end
+
+ it "should set the client options" do
+ @client.expects(:options).returns("rspec")
+ @client.client.expects(:options=).with("rspec")
+ @client.discovery_method = "rspec"
+ end
+
+ it "should adjust timeout for the new method" do
+ @client.expects(:discovery_timeout).once.returns(1)
+ @client.discovery_method = "rspec"
+ @client.instance_variable_get("@timeout").should == 4
+ end
+
+ it "should preserve any user supplied discovery timeout" do
+ @client.discovery_timeout = 10
+ @client.discovery_method = "rspec"
+ @client.discovery_timeout.should == 10
+ end
+
+ it "should reset the rpc client" do
+ @client.expects(:reset)
+ @client.discovery_method = "rspec"
+ end
+ end
+
+ describe "#discovery_options=" do
+ it "should flatten the options array" do
+ @client.discovery_options = "foo"
+ @client.discovery_options.should == ["foo"]
+ end
+ end
+
+ describe "#discovery_timeout" do
+ it "should favour the initial options supplied timeout" do
+ client = Client.new("rspec", {:options => {:disctimeout => 3, :filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.discovery_timeout.should == 3
+ end
+
+ it "should return the DDL data if no specific options are supplied" do
+ client = Client.new("rspec", {:options => {:disctimeout => nil, :filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.discovery_timeout.should == 2
+ end
+ end
+
+ describe "#discovery_timeout=" do
+ it "should store the discovery timeout" do
+ @client.discovery_timeout = 10
+ @client.discovery_timeout.should == 10
+ end
+
+ it "should update the overall timeout with the new discovery timeout" do
+ @client.instance_variable_get("@timeout").should == 4
+
+ @client.discovery_timeout = 10
+
+ @client.instance_variable_get("@timeout").should == 12
+ end
+ end
+
+ describe "#limit_method" do
+ it "should force strings to symbols" do
+ @client.limit_method = "first"
+ @client.limit_method.should == :first
+ end
+
+ it "should only allow valid methods" do
+ @client.limit_method = :first
+ @client.limit_method.should == :first
+ @client.limit_method = :random
+ @client.limit_method.should == :random
+
+ expect { @client.limit_method = :fail }.to raise_error(/Unknown/)
+ expect { @client.limit_method = "fail" }.to raise_error(/Unknown/)
+ end
+ end
+
+ describe "#method_missing" do
+ it "should reset the stats" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.stubs(:call_agent)
+
+ Stats.any_instance.expects(:reset).once
+ client.rspec
+ end
+
+ it "should validate the request against the ddl" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+
+ client.stubs(:call_agent)
+
+ client.expects(:validate_request).with("rspec", {:arg => :val}).raises("validation failed")
+
+ expect { client.rspec(:arg => :val) }.to raise_error("validation failed")
+ end
+
+ it "should support limited targets" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.limit_targets = 10
+
+ client.expects(:pick_nodes_from_discovered).with(10).returns(["one", "two"])
+ client.expects(:custom_request).with("rspec", {}, ["one", "two"], {"identity" => /^(one|two)$/}).once
+
+ client.rspec
+ end
+
+ describe "batch mode" do
+ before do
+ Config.instance.stubs(:direct_addressing).returns(true)
+ @client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ end
+
+ it "should support global batch_size" do
+ @client.batch_size = 10
+ @client.expects(:call_agent_batched).with("rspec", {}, @client.options, 10, 1)
+ @client.rspec
+ end
+
+ it "should support custom batch_size" do
+ @client.expects(:call_agent_batched).with("rspec", {}, @client.options, 10, 1)
+ @client.rspec :batch_size => 10
+ end
+
+ it "should allow supplied batch_size override global one" do
+ @client.batch_size = 10
+ @client.expects(:call_agent_batched).with("rspec", {}, @client.options, 20, 1)
+ @client.rspec :batch_size => 20
+ end
+
+ it "should support global batch_sleep_time" do
+ @client.batch_size = 10
+ @client.batch_sleep_time = 20
+ @client.expects(:call_agent_batched).with("rspec", {}, @client.options, 10, 20)
+ @client.rspec
+ end
+
+ it "should support custom batch_sleep_time" do
+ @client.batch_size = 10
+ @client.expects(:call_agent_batched).with("rspec", {}, @client.options, 10, 20)
+ @client.rspec :batch_sleep_time => 20
+ end
+
+ it "should allow supplied batch_sleep_time override global one" do
+ @client.batch_size = 10
+ @client.batch_sleep_time = 10
+ @client.expects(:call_agent_batched).with("rspec", {}, @client.options, 10, 20)
+ @client.rspec :batch_sleep_time => 20
+ end
+ end
+
+ it "should support normal calls" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+
+ client.expects(:call_agent).with("rspec", {}, client.options, :auto).once
+
+ client.rspec
+ end
+ end
+
+ describe "#pick_nodes_from_discovered" do
+ before do
+ client = stub
+ discoverer = stub
+ ddl = stub
+
+ ddl.stubs(:meta).returns({:timeout => 2})
+
+ discoverer.stubs(:ddl).returns(ddl)
+
+ client.stubs("options=")
+ client.stubs(:collective).returns("mcollective")
+ client.stubs(:discoverer).returns(discoverer)
+
+ Config.instance.stubs(:loadconfig).with("/nonexisting").returns(true)
+ MCollective::Client.stubs(:new).returns(client)
+ Config.instance.stubs(:direct_addressing).returns(true)
+ end
+
+ it "should return a percentage of discovered hosts" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.stubs(:discover).returns((1..10).map{|i| i.to_s})
+ client.limit_method = :first
+ client.pick_nodes_from_discovered("20%").should == ["1", "2"]
+ end
+
+ it "should return the same list when a random seed is supplied" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :limit_seed => 5}})
+ client.stubs(:discover).returns((1..10).map{|i| i.to_s})
+ client.limit_method = :random
+ client.pick_nodes_from_discovered("30%").should == ["3", "7", "8"]
+ client.pick_nodes_from_discovered("30%").should == ["3", "7", "8"]
+ client.pick_nodes_from_discovered("3").should == ["3", "7", "8"]
+ client.pick_nodes_from_discovered("3").should == ["3", "7", "8"]
+ end
+
+ it "should correctly pick a numeric amount of discovered nodes" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :limit_seed => 5}})
+ client.stubs(:discover).returns((1..10).map{|i| i.to_s})
+ client.limit_method = :first
+ client.pick_nodes_from_discovered(5).should == (1..5).map{|i| i.to_s}
+ client.pick_nodes_from_discovered(5).should == (1..5).map{|i| i.to_s}
+ end
+ end
+
+ describe "#limit_targets=" do
+ before do
+ client = stub
+ discoverer = stub
+ ddl = stub
+
+ ddl.stubs(:meta).returns({:timeout => 2})
+
+ discoverer.stubs(:force_direct_mode?).returns(false)
+ discoverer.stubs(:ddl).returns(ddl)
+ discoverer.stubs(:discovery_method).returns("mc")
+
+ client.stubs("options=")
+ client.stubs(:collective).returns("mcollective")
+ client.stubs(:discoverer).returns(discoverer)
+
+ Config.instance.stubs(:loadconfig).with("/nonexisting").returns(true)
+ MCollective::Client.expects(:new).returns(client)
+ Config.instance.stubs(:direct_addressing).returns(true)
+
+ @client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ end
+
+ it "should support percentages" do
+ @client.limit_targets = "10%"
+ @client.limit_targets.should == "10%"
+ end
+
+ it "should support integers" do
+ @client.limit_targets = 10
+ @client.limit_targets.should == 10
+ @client.limit_targets = "20"
+ @client.limit_targets.should == 20
+ @client.limit_targets = 1.1
+ @client.limit_targets.should == 1
+ @client.limit_targets = 1.7
+ @client.limit_targets.should == 1
+ end
+
+ it "should not invalid limits to be set" do
+ expect { @client.limit_targets = "a" }.to raise_error(/Invalid/)
+ expect { @client.limit_targets = "%1" }.to raise_error(/Invalid/)
+ expect { @client.limit_targets = "1.1" }.to raise_error(/Invalid/)
+ end
+ end
+
+ describe "#call_agent_batched" do
+ before do
+ @client = stub
+ @discoverer = stub
+ @ddl = stub
+
+ @ddl.stubs(:meta).returns({:timeout => 2})
+
+ @discoverer.stubs(:force_direct_mode?).returns(false)
+ @discoverer.stubs(:ddl).returns(@ddl)
+ @discoverer.stubs(:discovery_method).returns("mc")
+
+ @client.stubs("options=")
+ @client.stubs(:collective).returns("mcollective")
+ @client.stubs(:discoverer).returns(@discoverer)
+
+ Config.instance.stubs(:loadconfig).with("/nonexisting").returns(true)
+ MCollective::Client.expects(:new).returns(@client)
+ Config.instance.stubs(:direct_addressing).returns(true)
+ end
+
+ it "should require direct addressing" do
+ Config.instance.stubs(:direct_addressing).returns(false)
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+
+ expect {
+ client.send(:call_agent_batched, "foo", {}, {}, 1, 1)
+ }.to raise_error("Batched requests requires direct addressing")
+ end
+
+ it "should require that all results be processed" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+
+ expect {
+ client.send(:call_agent_batched, "foo", {:process_results => false}, {}, 1, 1)
+ }.to raise_error("Cannot bypass result processing for batched requests")
+ end
+
+ it "should only accept integer batch sizes" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+
+ expect {
+ client.send(:call_agent_batched, "foo", {}, {}, "foo", 1)
+ }.to raise_error(/invalid value for Integer/)
+ end
+
+ it "should only accept float sleep times" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+
+ expect {
+ client.send(:call_agent_batched, "foo", {}, {}, 1, "foo")
+ }.to raise_error(/invalid value for Float/)
+ end
+
+ it "should batch hosts in the correct size" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :stderr => StringIO.new}})
+
+ client.expects(:new_request).returns("req")
+
+ discovered = mock
+ discovered.stubs(:size).returns(1)
+ discovered.expects(:in_groups_of).with(10).raises("spec pass")
+
+ client.instance_variable_set("@client", @coreclient)
+ @coreclient.stubs(:discover).returns(discovered)
+ @coreclient.stubs(:timeout_for_compound_filter).returns(0)
+
+ expect { client.send(:call_agent_batched, "foo", {}, {}, 10, 1) }.to raise_error("spec pass")
+ end
+
+ it "should force direct requests" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :stderr => StringIO.new}})
+
+ Message.expects(:new).with('req', nil, {:type => :direct_request, :agent => 'foo', :filter => nil, :options => {}, :collective => 'mcollective'}).raises("spec pass")
+ client.expects(:new_request).returns("req")
+
+ client.instance_variable_set("@client", @coreclient)
+ @coreclient.stubs(:discover).returns(["test"])
+ @coreclient.stubs(:timeout_for_compound_filter).returns(0)
+
+ expect { client.send(:call_agent_batched, "foo", {}, {}, 1, 1) }.to raise_error("spec pass")
+ end
+
+ it "should process blocks correctly" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :stderr => StringIO.new}})
+
+ msg = mock
+ msg.expects(:discovered_hosts=).times(10)
+ msg.expects(:create_reqid).returns("823a3419a0975c3facbde121f72ab61f")
+ msg.expects(:requestid=).with("823a3419a0975c3facbde121f72ab61f").times(10)
+
+ stats = {:noresponsefrom => [], :responses => 0, :blocktime => 0, :totaltime => 0, :discoverytime => 0, :requestid => "823a3419a0975c3facbde121f72ab61f"}
+
+ Message.expects(:new).with('req', nil, {:type => :direct_request, :agent => 'foo', :filter => nil, :options => {}, :collective => 'mcollective'}).returns(msg).times(10)
+ client.expects(:new_request).returns("req")
+ client.expects(:sleep).with(1.0).times(9)
+
+ client.instance_variable_set("@client", @coreclient)
+ @coreclient.stubs(:discover).returns([1,2,3,4,5,6,7,8,9,0])
+ @coreclient.expects(:req).with(msg).yields("result").times(10)
+ @coreclient.stubs(:stats).returns stats
+ @coreclient.stubs(:timeout_for_compound_filter).returns(0)
+
+ client.expects(:process_results_with_block).with("foo", "result", instance_of(Proc), nil).times(10)
+
+ result = client.send(:call_agent_batched, "foo", {}, {}, 1, 1) { }
+ result[:requestid].should == "823a3419a0975c3facbde121f72ab61f"
+ result.class.should == Stats
+ end
+
+ it "should return an array of results in array mode" do
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :stderr => StringIO.new}})
+ client.instance_variable_set("@client", @coreclient)
+
+ msg = mock
+ msg.expects(:discovered_hosts=).times(10)
+ msg.expects(:create_reqid).returns("823a3419a0975c3facbde121f72ab61f")
+ msg.expects(:requestid=).with("823a3419a0975c3facbde121f72ab61f").times(10)
+
+ stats = {:noresponsefrom => [], :responses => 0, :blocktime => 0, :totaltime => 0, :discoverytime => 0, :requestid => "823a3419a0975c3facbde121f72ab61f"}
+
+ Progress.expects(:new).never
+
+ Message.expects(:new).with('req', nil, {:type => :direct_request, :agent => 'foo', :filter => nil, :options => {}, :collective => 'mcollective'}).returns(msg).times(10)
+ client.expects(:new_request).returns("req")
+ client.expects(:sleep).with(1.0).times(9)
+
+ @coreclient.stubs(:discover).returns([1,2,3,4,5,6,7,8,9,0])
+ @coreclient.expects(:req).with(msg).yields("result").times(10)
+ @coreclient.stubs(:stats).returns stats
+ @coreclient.stubs(:timeout_for_compound_filter).returns(0)
+
+ client.expects(:process_results_without_block).with("result", "foo", nil).returns("rspec").times(10)
+
+ client.send(:call_agent_batched, "foo", {}, {}, 1, 1).should == ["rspec", "rspec", "rspec", "rspec", "rspec", "rspec", "rspec", "rspec", "rspec", "rspec"]
+
+ client.stats[:requestid].should == "823a3419a0975c3facbde121f72ab61f"
+ end
+ end
+
+ describe "#batch_sleep_time=" do
+ it "should correctly set the sleep" do
+ Config.instance.stubs(:direct_addressing).returns(true)
+
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.batch_sleep_time = 5
+ client.batch_sleep_time.should == 5
+ end
+
+ it "should only allow batch sleep to be set for direct addressing capable clients" do
+ Config.instance.stubs(:direct_addressing).returns(false)
+ Config.instance.stubs(:loadconfig).with("/nonexisting").returns(true)
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+
+ expect { client.batch_sleep_time = 5 }.to raise_error("Can only set batch sleep time if direct addressing is supported")
+ end
+ end
+
+ describe "#batch_size=" do
+ it "should correctly set the size" do
+ Config.instance.stubs(:direct_addressing).returns(true)
+
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.batch_mode.should == false
+ client.batch_size = 5
+ client.batch_size.should == 5
+ client.batch_mode.should == true
+ end
+
+ it "should only allow batch size to be set for direct addressing capable clients" do
+ Config.instance.stubs(:loadconfig).with("/nonexisting").returns(true)
+ Config.instance.stubs(:direct_addressing).returns(false)
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+
+ expect { client.batch_size = 5 }.to raise_error("Can only set batch size if direct addressing is supported")
+ end
+
+ it "should support disabling batch mode when supplied a batch size of 0" do
+ Config.instance.stubs(:direct_addressing).returns(true)
+
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.batch_size = 5
+ client.batch_mode.should == true
+ client.batch_size = 0
+ client.batch_mode.should == false
+ end
+ end
+
+ describe "#discover" do
+ it "should not accept invalid flags" do
+ Config.instance.stubs(:direct_addressing).returns(true)
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+
+ expect { client.discover(:rspec => :rspec) }.to raise_error("Unknown option rspec passed to discover")
+ end
+
+ it "should reset when :json, :hosts or :nodes are provided" do
+ Config.instance.stubs(:direct_addressing).returns(true)
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.expects(:reset).times(3)
+ client.discover(:hosts => ["one"])
+ client.discover(:nodes => ["one"])
+ client.discover(:json => ["one"])
+ end
+
+ it "should only allow discovery data in direct addressing mode" do
+ Config.instance.stubs(:direct_addressing).returns(false)
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.expects(:reset).once
+
+ expect {
+ client.discover(:nodes => ["one"])
+ }.to raise_error("Can only supply discovery data if direct_addressing is enabled")
+ end
+
+ it "should parse :nodes and :hosts and force direct requests" do
+ Config.instance.stubs(:direct_addressing).returns(true)
+ Helpers.expects(:extract_hosts_from_array).with(["one"]).returns(["one"]).twice
+
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.discover(:nodes => ["one"]).should == ["one"]
+ client.discover(:hosts => ["one"]).should == ["one"]
+ client.instance_variable_get("@force_direct_request").should == true
+ client.instance_variable_get("@discovered_agents").should == ["one"]
+ end
+
+ it "should parse :json and force direct requests" do
+ Config.instance.stubs(:direct_addressing).returns(true)
+ Helpers.expects(:extract_hosts_from_json).with('["one"]').returns(["one"]).once
+
+ client = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting"}})
+ client.discover(:json => '["one"]').should == ["one"]
+ client.instance_variable_get("@force_direct_request").should == true
+ client.instance_variable_get("@discovered_agents").should == ["one"]
+ end
+
+ it "should not set direct mode for non 'mc' discovery methods" do
+ Config.instance.stubs(:direct_addressing).returns(true)
+
+ client = Client.new("foo", {:options => {:discovery_method => "rspec", :filter => {"identity" => ["foo"], "agent" => []}, :config => "/nonexisting"}})
+ @coreclient.expects(:discover).returns(["foo"])
+
+ client.discover
+ client.instance_variable_get("@discovered_agents").should == ["foo"]
+ client.instance_variable_get("@force_direct_request").should == false
+ end
+
+ it "should force direct mode for non regex identity filters" do
+ Config.instance.stubs(:direct_addressing).returns(true)
+
+ client = Client.new("foo", {:options => {:discovery_method => "mc", :filter => {"identity" => ["foo"], "agent" => []}, :config => "/nonexisting"}})
+ client.discover
+ client.instance_variable_get("@discovered_agents").should == ["foo"]
+ client.instance_variable_get("@force_direct_request").should == true
+ end
+
+ it "should not set direct mode if its disabled" do
+ Config.instance.stubs(:direct_addressing).returns(false)
+
+ client = Client.new("foo", {:options => {:discovery_method => "mc", :filter => {"identity" => ["foo"], "agent" => []}, :config => "/nonexisting"}})
+
+ client.discover
+ client.instance_variable_get("@force_direct_request").should == false
+ client.instance_variable_get("@discovered_agents").should == ["foo"]
+ end
+
+ it "should not set direct mode for regex identities" do
+ Config.instance.stubs(:direct_addressing).returns(false)
+
+ rpcclient = Client.new("foo", {:options => {:filter => {"identity" => ["/foo/"], "agent" => []}, :config => "/nonexisting"}})
+
+ rpcclient.client.expects(:discover).with({'identity' => ['/foo/'], 'agent' => ['foo']}, 2).once.returns(["foo"])
+
+ rpcclient.discover
+ rpcclient.instance_variable_get("@force_direct_request").should == false
+ rpcclient.instance_variable_get("@discovered_agents").should == ["foo"]
+ end
+
+ it "should print status to stderr if in verbose mode" do
+ @stderr.expects(:print).with("Discovering hosts using the mc method for 2 second(s) .... ")
+ @stderr.expects(:puts).with(1)
+
+ rpcclient = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :verbose => true, :disctimeout => 2, :stderr => @stderr, :stdout => @stdout}})
+
+ rpcclient.client.expects(:discover).with({'identity' => [], 'compound' => [], 'fact' => [], 'agent' => ['foo'], 'cf_class' => []}, 2).returns(["foo"])
+
+ rpcclient.discover
+ end
+
+ it "should not print status to stderr if in nonverbose mode" do
+ @stderr.expects(:print).never
+ @stderr.expects(:puts).never
+
+ rpcclient = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :verbose => false, :disctimeout => 2, :stderr => @stderr, :stdout => @stdout}})
+ rpcclient.client.expects(:discover).with({'identity' => [], 'compound' => [], 'fact' => [], 'agent' => ['foo'], 'cf_class' => []}, 2).returns(["foo"])
+
+ rpcclient.discover
+ end
+
+ it "should record the start and end times" do
+ Stats.any_instance.expects(:time_discovery).with(:start)
+ Stats.any_instance.expects(:time_discovery).with(:end)
+
+ rpcclient = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :verbose => false, :disctimeout => 2}})
+ rpcclient.client.expects(:discover).with({'identity' => [], 'compound' => [], 'fact' => [], 'agent' => ['foo'], 'cf_class' => []}, 2).returns(["foo"])
+
+ rpcclient.discover
+ end
+
+ it "should discover using limits in :first rpclimit mode given a number" do
+ Config.instance.stubs(:rpclimitmethod).returns(:first)
+ rpcclient = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :verbose => false, :disctimeout => 2}})
+ rpcclient.client.expects(:discover).with({'identity' => [], 'compound' => [], 'fact' => [], 'agent' => ['foo'], 'cf_class' => []}, 2, 1).returns(["foo"])
+
+ rpcclient.limit_targets = 1
+
+ rpcclient.discover
+ end
+
+ it "should not discover using limits in :first rpclimit mode given a string" do
+ Config.instance.stubs(:rpclimitmethod).returns(:first)
+ rpcclient = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :verbose => false, :disctimeout => 2}})
+ rpcclient.client.expects(:discover).with({'identity' => [], 'compound' => [], 'fact' => [], 'agent' => ['foo'], 'cf_class' => []}, 2).returns(["foo"])
+ rpcclient.limit_targets = "10%"
+
+ rpcclient.discover
+ end
+
+ it "should not discover using limits when not in :first mode" do
+ Config.instance.stubs(:rpclimitmethod).returns(:random)
+
+ rpcclient = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :verbose => false, :disctimeout => 2}})
+ rpcclient.client.expects(:discover).with({'identity' => [], 'compound' => [], 'fact' => [], 'agent' => ['foo'], 'cf_class' => []}, 2).returns(["foo"])
+
+ rpcclient.limit_targets = 1
+ rpcclient.discover
+ end
+
+ it "should ensure force_direct mode is false when doing traditional discovery" do
+ rpcclient = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :verbose => false, :disctimeout => 2}})
+ rpcclient.client.expects(:discover).with({'identity' => [], 'compound' => [], 'fact' => [], 'agent' => ['foo'], 'cf_class' => []}, 2).returns(["foo"])
+
+ rpcclient.instance_variable_set("@force_direct_request", true)
+ rpcclient.discover
+ rpcclient.instance_variable_get("@force_direct_request").should == false
+ end
+
+ it "should store discovered nodes in stats" do
+ rpcclient = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :verbose => false, :disctimeout => 2}})
+ rpcclient.client.expects(:discover).with({'identity' => [], 'compound' => [], 'fact' => [], 'agent' => ['foo'], 'cf_class' => []}, 2).returns(["foo"])
+
+ rpcclient.discover
+ rpcclient.stats.discovered_nodes.should == ["foo"]
+ end
+
+ it "should save discovered nodes in RPC" do
+ rpcclient = Client.new("foo", {:options => {:filter => Util.empty_filter, :config => "/nonexisting", :verbose => false, :disctimeout => 2}})
+ rpcclient.client.expects(:discover).with({'identity' => [], 'compound' => [], 'fact' => [], 'agent' => ['foo'], 'cf_class' => []}, 2).returns(["foo"])
+
+ RPC.expects(:discovered).with(["foo"]).once
+ rpcclient.discover
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module RPC
+ describe Helpers do
+ describe "#extract_hosts_from_json" do
+ it "should fail for non array data" do
+ expect {
+ Helpers.extract_hosts_from_json("{}")
+ }.to raise_error("JSON hosts list is not an array")
+ end
+
+ it "should fail for non hash array members" do
+ senders = [{"sender" => "sender1"}, {"sender" => "sender3"}, ""].to_json
+
+ expect {
+ Helpers.extract_hosts_from_json(senders)
+ }.to raise_error("JSON host list is not an array of Hashes")
+ end
+
+ it "should fail for hashes without senders" do
+ senders = [{"sender" => "sender1"}, {"sender" => "sender3"}, {}].to_json
+
+ expect {
+ Helpers.extract_hosts_from_json(senders)
+ }.to raise_error("JSON host list does not have senders in it")
+ end
+
+ it "should return all found unique senders" do
+ senders = [{"sender" => "sender1"}, {"sender" => "sender3"}, {"sender" => "sender1"}].to_json
+
+ Helpers.extract_hosts_from_json(senders).should == ["sender1", "sender3"]
+ end
+ end
+
+ describe "#extract_hosts_from_array" do
+ it "should support single string lists" do
+ Helpers.extract_hosts_from_array("foo").should == ["foo"]
+ end
+
+ it "should support arrays" do
+ Helpers.extract_hosts_from_array(["foo", "bar"]).should == ["foo", "bar"]
+ end
+
+ it "should fail for non string array members" do
+ expect {
+ Helpers.extract_hosts_from_array(["foo", 1])
+ }.to raise_error("1 should be a string")
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module RPC
+ describe Reply do
+ before(:each) do
+ Cache.delete!(:ddl) rescue nil
+
+ ddl = stub
+ ddl.stubs(:action_interface).returns({:output => {}})
+ ddl.stubs(:actions).returns(["rspec"])
+ ddl.stubs(:pluginname).returns("rspec")
+
+ @reply = Reply.new("rspec", ddl)
+ end
+
+ describe "#initialize" do
+ it "should set an empty data hash" do
+ @reply.data.should == {}
+ end
+
+ it "should set statuscode to zero" do
+ @reply.statuscode.should == 0
+ end
+
+ it "should set statusmsg to OK" do
+ @reply.statusmsg.should == "OK"
+ end
+ end
+
+ describe "#initialize_data" do
+ before do
+ Log.stubs(:warn)
+ @ddl = DDL.new("rspec", :agent, false)
+ end
+
+ it "should set defaults correctly" do
+ @ddl.action :rspec, :description => "testing rspec" do
+ @ddl.output :one, :description => "rspec test", :display_as => "rspec", :default => "default"
+ @ddl.output :three, :description => "rspec test", :display_as => "rspec", :default => []
+ @ddl.output :two, :description => "rspec test", :display_as => "rspec"
+ end
+
+ reply = Reply.new(:rspec, @ddl)
+ reply.data.should == {:one => "default", :two => nil, :three => []}
+ end
+
+ it "should detect missing actions" do
+ reply = Reply.new(:rspec, @ddl)
+ expect { reply.initialize_data }.to raise_error(/No action 'rspec' defined/)
+ end
+ end
+
+ describe "#fail" do
+ it "should set statusmsg" do
+ @reply.fail "foo"
+ @reply.statusmsg.should == "foo"
+ end
+
+ it "should set statuscode to 1 by default" do
+ @reply.fail("foo")
+ @reply.statuscode.should == 1
+ end
+
+ it "should set statuscode" do
+ @reply.fail("foo", 2)
+ @reply.statuscode.should == 2
+ end
+ end
+
+ describe "#fail!" do
+ it "should set statusmsg" do
+ expect {
+ @reply.fail! "foo"
+ }.to raise_error(RPCAborted, "foo")
+
+ @reply.statusmsg.should == "foo"
+ end
+
+ it "should set statuscode to 1 by default" do
+ expect {
+ @reply.fail! "foo"
+ }.to raise_error(RPCAborted)
+ end
+
+ it "should set statuscode" do
+ expect {
+ @reply.fail! "foo", 2
+ }.to raise_error(UnknownRPCAction)
+
+ @reply.statuscode.should == 2
+ end
+
+ it "should raise RPCAborted for code 1" do
+ expect {
+ @reply.fail! "foo", 1
+ }.to raise_error(RPCAborted)
+ end
+
+ it "should raise UnknownRPCAction for code 2" do
+ expect {
+ @reply.fail! "foo", 2
+ }.to raise_error(UnknownRPCAction)
+ end
+
+ it "should raise MissingRPCData for code 3" do
+ expect {
+ @reply.fail! "foo", 3
+ }.to raise_error(MissingRPCData)
+ end
+
+ it "should raise InvalidRPCData for code 4" do
+ expect {
+ @reply.fail! "foo", 4
+ }.to raise_error(InvalidRPCData)
+ end
+
+ it "should raise UnknownRPCError for all other codes" do
+ expect {
+ @reply.fail! "foo", 5
+ }.to raise_error(UnknownRPCError)
+
+ expect {
+ @reply.fail! "foo", "x"
+ }.to raise_error(UnknownRPCError)
+ end
+ end
+
+ describe "#[]=" do
+ it "should save the correct data to the data hash" do
+ @reply[:foo] = "foo1"
+ @reply["foo"] = "foo2"
+
+ @reply.data[:foo].should == "foo1"
+ @reply.data["foo"].should == "foo2"
+ end
+ end
+
+ describe "#[]" do
+ it "should return the correct saved data" do
+ @reply[:foo] = "foo1"
+ @reply["foo"] = "foo2"
+
+ @reply[:foo].should == "foo1"
+ @reply["foo"].should == "foo2"
+ end
+ end
+
+ describe "#to_hash" do
+ it "should have the correct keys" do
+ @reply.to_hash.keys.sort.should == [:data, :statuscode, :statusmsg]
+ end
+
+ it "should have the correct statuscode" do
+ @reply.fail "meh", 2
+ @reply.to_hash[:statuscode].should == 2
+ end
+
+ it "should have the correct statusmsg" do
+ @reply.fail "meh", 2
+ @reply.to_hash[:statusmsg].should == "meh"
+ end
+
+ it "should have the correct data" do
+ @reply[:foo] = :bar
+ @reply.to_hash[:data][:foo].should == :bar
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module RPC
+ describe Request do
+ before(:each) do
+ @req = {:msgtime => Time.now,
+ :senderid => "spec test",
+ :requestid => "12345",
+ :callerid => "rip"}
+
+ @req[:body] = {:action => "test",
+ :data => {:foo => "bar", :process_results => true},
+ :agent => "tester"}
+
+ @ddl = DDL.new("rspec", :agent, false)
+
+ @request = Request.new(@req, @ddl)
+ end
+
+ describe "#validate!" do
+ it "should validate the request using the supplied DDL" do
+ @ddl.expects(:validate_rpc_request).with("test", {:foo => "bar", :process_results => true})
+ @request.validate!
+ end
+ end
+
+ describe "#initialize" do
+ it "should set time" do
+ @request.time.should == @req[:msgtime]
+ end
+
+ it "should set action" do
+ @request.action.should == "test"
+ end
+
+ it "should set data" do
+ @request.data.should == {:foo => "bar", :process_results => true}
+ end
+
+ it "should set sender" do
+ @request.sender.should == "spec test"
+ end
+
+ it "should set agent" do
+ @request.agent.should == "tester"
+ end
+
+ it "should set uniqid" do
+ @request.uniqid.should == "12345"
+ end
+
+ it "should set caller" do
+ @request.caller.should == "rip"
+ end
+
+ it "should set unknown caller if none is supplied" do
+ @req.delete(:callerid)
+ Request.new(@req, @ddl).caller.should == "unknown"
+ end
+ end
+
+ describe "#include?" do
+ it "should correctly report on hash contents" do
+ @request.include?(:foo).should == true
+ end
+
+ it "should return false for non hash data" do
+ @req[:body][:data] = "foo"
+ Request.new(@req, @ddl).include?(:foo).should == false
+ end
+ end
+
+ describe "#should_respond?" do
+ it "should return true if the header is absent" do
+ @req[:body][:data].delete(:process_results)
+ Request.new(@req, @ddl).should_respond?.should == true
+ end
+
+ it "should return correct value" do
+ @req[:body][:data][:process_results] = false
+ Request.new(@req, @ddl).should_respond?.should == false
+ end
+ end
+
+ describe "#[]" do
+ it "should return nil for non hash data" do
+ @req[:body][:data] = "foo"
+ Request.new(@req, @ddl)["foo"].should == nil
+ end
+
+ it "should return correct data" do
+ @request[:foo].should == "bar"
+ end
+
+ it "should return nil for absent data" do
+ @request[:bar].should == nil
+ end
+ end
+
+ describe "#fetch" do
+ it "should return nil for non hash data" do
+ @req[:body][:data] = "foo"
+ Request.new(@req, @ddl)["foo"].should == nil
+ end
+
+ it "should fetch data with the correct default behavior" do
+ @request.fetch(:foo, "default").should == "bar"
+ @request.fetch(:rspec, "default").should == "default"
+ end
+ end
+
+ describe "#to_hash" do
+ it "should have the correct keys" do
+ @request.to_hash.keys.sort.should == [:action, :agent, :data]
+ end
+
+ it "should return the correct agent" do
+ @request.to_hash[:agent].should == "tester"
+ end
+
+ it "should return the correct action" do
+ @request.to_hash[:action].should == "test"
+ end
+
+ it "should return the correct data" do
+ @request.to_hash[:data].should == {:foo => "bar",
+ :process_results => true}
+ end
+ end
+ end
+ end
+end
+
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module RPC
+ describe Result do
+ before(:each) do
+ @result = Result.new("tester", "test", {:foo => "bar", :bar => "baz"})
+ end
+
+ it "should include Enumerable" do
+ Result.ancestors.include?(Enumerable).should == true
+ end
+
+ describe "#initialize" do
+ it "should set the agent" do
+ @result.agent.should == "tester"
+ end
+
+ it "should set the action" do
+ @result.action.should == "test"
+ end
+
+ it "should set the results" do
+ @result.results.should == {:foo => "bar", :bar => "baz"}
+ end
+ end
+
+ describe "#[]" do
+ it "should access the results hash and return correct data" do
+ @result[:foo].should == "bar"
+ @result[:bar].should == "baz"
+ end
+ end
+
+ describe "#[]=" do
+ it "should set the correct result data" do
+ @result[:meh] = "blah"
+
+ @result[:foo].should == "bar"
+ @result[:bar].should == "baz"
+ @result[:meh].should == "blah"
+ end
+ end
+
+ describe "#fetch" do
+ it "should fetch data with the correct default behavior" do
+ @result.fetch(:foo, "default").should == "bar"
+ @result.fetch(:rspec, "default").should == "default"
+ end
+ end
+
+ describe "#each" do
+ it "should itterate all the pairs" do
+ data = {}
+
+ @result.each {|k,v| data[k] = v}
+
+ data[:foo].should == "bar"
+ data[:bar].should == "baz"
+ end
+ end
+
+ describe "#to_json" do
+ it "should correctly json encode teh data" do
+ result = Result.new("tester", "test", {:statuscode => 0, :statusmsg => "OK", :sender => "rspec", :data => {:foo => "bar", :bar => "baz"}})
+ JSON.load(result.to_json).should == {"agent" => "tester", "action" => "test", "statuscode" => 0, "statusmsg" => "OK", "sender" => "rspec", "data" => {"foo" => "bar", "bar" => "baz"}}
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module RPC
+ describe Stats do
+ before(:each) do
+ @expected = {:discoverytime => 0,
+ :okcount => 0,
+ :blocktime => 0,
+ :failcount => 0,
+ :noresponsefrom => [],
+ :responses => 0,
+ :totaltime => 0,
+ :discovered => 0,
+ :starttime => 1300031826.0,
+ :requestid => nil,
+ :aggregate_summary => [],
+ :aggregate_failures => [],
+ :discovered_nodes => []}
+
+ @stats = Stats.new
+ end
+
+ describe "#initialize" do
+ it "should reset stats on creation" do
+ Stats.any_instance.stubs(:reset).returns(true).once
+ s = Stats.new
+ end
+ end
+
+ describe "#reset" do
+ it "should initialize data correctly" do
+ Time.stubs(:now).returns(Time.at(1300031826))
+ s = Stats.new
+
+ @expected.keys.each do |k|
+ @expected[k].should == s.send(k)
+ end
+ end
+ end
+
+ describe "#to_hash" do
+ it "should return correct stats" do
+ Time.stubs(:now).returns(Time.at(1300031826))
+ s = Stats.new
+
+ s.to_hash.should == @expected
+ end
+ end
+
+ describe "#[]" do
+ it "should return stat values" do
+ Time.stubs(:now).returns(Time.at(1300031826))
+ s = Stats.new
+
+ @expected.keys.each do |k|
+ @expected[k].should == s[k]
+ end
+ end
+
+ it "should return nil for unknown values" do
+ @stats["foo"].should == nil
+ end
+ end
+
+ describe "#ok" do
+ it "should increment stats" do
+ @stats.ok
+ @stats[:okcount].should == 1
+ end
+ end
+
+ describe "#fail" do
+ it "should increment stats" do
+ @stats.fail
+ @stats.failcount.should == 1
+ end
+ end
+
+ describe "#time_discovery" do
+ it "should set start time correctly" do
+ Time.stubs(:now).returns(Time.at(1300031826))
+
+ @stats.time_discovery(:start)
+
+ @stats.instance_variable_get("@discovery_start").should == 1300031826.0
+ end
+
+ it "should record the difference correctly" do
+ Time.stubs(:now).returns(Time.at(1300031826))
+ @stats.time_discovery(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031827))
+ @stats.time_discovery(:end)
+
+ @stats.discoverytime.should == 1.0
+ end
+
+ it "should handle unknown actions and set discovery time to 0" do
+ Time.stubs(:now).returns(Time.at(1300031826))
+ @stats.time_discovery(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031827))
+ @stats.time_discovery(:stop)
+
+ @stats.discoverytime.should == 0
+ end
+
+ end
+
+ describe "#client_stats=" do
+ it "should store stats correctly" do
+ data = {}
+ keys = [:noresponsefrom, :responses, :starttime, :blocktime, :totaltime, :discoverytime]
+ keys.each {|k| data[k] = k}
+
+ @stats.client_stats = data
+
+ keys.each do |k|
+ @stats[k].should == data[k]
+ end
+ end
+
+ it "should not store discovery time if it was already stored" do
+ data = {}
+ keys = [:noresponsefrom, :responses, :starttime, :blocktime, :totaltime, :discoverytime]
+ keys.each {|k| data[k] = k}
+
+ Time.stubs(:now).returns(Time.at(1300031826))
+ @stats.time_discovery(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031827))
+ @stats.time_discovery(:end)
+
+ dtime = @stats.discoverytime
+
+ @stats.client_stats = data
+
+ @stats.discoverytime.should == dtime
+ end
+ end
+
+ describe "#time_block_execution" do
+ it "should set start time correctly" do
+ Time.stubs(:now).returns(Time.at(1300031826))
+
+ @stats.time_block_execution(:start)
+
+ @stats.instance_variable_get("@block_start").should == 1300031826.0
+ end
+
+ it "should record the difference correctly" do
+ Time.stubs(:now).returns(Time.at(1300031826))
+ @stats.time_block_execution(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031827))
+ @stats.time_block_execution(:end)
+
+ @stats.blocktime.should == 1
+ end
+
+ it "should handle unknown actions and set discovery time to 0" do
+ Time.stubs(:now).returns(Time.at(1300031826))
+ @stats.time_block_execution(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031827))
+ @stats.time_block_execution(:stop)
+
+ @stats.blocktime.should == 0
+ end
+ end
+
+ describe "#discovered_agents" do
+ it "should set discovered_nodes" do
+ nodes = ["one", "two"]
+ @stats.discovered_agents(nodes)
+ @stats.discovered_nodes.should == nodes
+ end
+
+ it "should set discovered count" do
+ nodes = ["one", "two"]
+ @stats.discovered_agents(nodes)
+ @stats.discovered.should == 2
+ end
+ end
+
+ describe "#finish_request" do
+ it "should calculate totaltime correctly" do
+ Time.stubs(:now).returns(Time.at(1300031824))
+ @stats.time_discovery(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031825))
+ @stats.time_discovery(:end)
+
+ Time.stubs(:now).returns(Time.at(1300031826))
+ @stats.time_block_execution(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031827))
+ @stats.time_block_execution(:end)
+
+ @stats.discovered_agents(["one", "two", "three"])
+ @stats.node_responded("one")
+ @stats.node_responded("two")
+
+ @stats.finish_request
+
+ @stats.totaltime.should == 2
+ end
+
+ it "should calculate no responses correctly" do
+ Time.stubs(:now).returns(Time.at(1300031824))
+ @stats.time_discovery(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031825))
+ @stats.time_discovery(:end)
+
+ Time.stubs(:now).returns(Time.at(1300031826))
+ @stats.time_block_execution(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031827))
+ @stats.time_block_execution(:end)
+
+ @stats.discovered_agents(["one", "two", "three"])
+ @stats.node_responded("one")
+ @stats.node_responded("two")
+
+ @stats.finish_request
+
+ @stats.noresponsefrom.should == ["three"]
+ end
+
+ it "should recover from failure correctly" do
+ Time.stubs(:now).returns(Time.at(1300031824))
+ @stats.time_discovery(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031825))
+ @stats.time_discovery(:end)
+
+ Time.stubs(:now).returns(Time.at(1300031826))
+ @stats.time_block_execution(:start)
+
+ Time.stubs(:now).returns(Time.at(1300031827))
+ @stats.time_block_execution(:end)
+
+ # cause the .each to raise an exception
+ @stats.instance_variable_set("@responsesfrom", nil)
+ @stats.finish_request
+
+ @stats.noresponsefrom.should == []
+ @stats.totaltime.should == 0
+ end
+ end
+
+ describe "#node_responded" do
+ it "should append to the list of nodes" do
+ @stats.node_responded "foo"
+ @stats.responsesfrom.should == ["foo"]
+ end
+
+ it "should create a new array if adding fails" do
+ # cause the << to fail
+ @stats.instance_variable_set("@responsesfrom", nil)
+
+ @stats.node_responded "foo"
+ @stats.responsesfrom.should == ["foo"]
+ end
+ end
+
+ describe "#no_response_report" do
+ it "should create an empty report if all nodes responded" do
+ @stats.discovered_agents ["foo"]
+ @stats.node_responded "foo"
+ @stats.finish_request
+
+ @stats.no_response_report.should == ""
+ end
+
+ it "should list all nodes that did not respond" do
+ @stats.discovered_agents ["foo", "bar"]
+ @stats.finish_request
+
+ @stats.no_response_report.should match(Regexp.new(/No response from.+bar\s+foo/m))
+ end
+ end
+
+ describe "#text_for_aggregates" do
+ let(:aggregate){mock()}
+
+ before :each do
+ aggregate.stubs(:result).returns({:output => "success"})
+ aggregate.stubs(:action).returns("action")
+ end
+
+ it "should create the correct output text for aggregate functions" do
+ @stats.aggregate_summary = [aggregate]
+ aggregate.stubs(:is_a?).returns(true)
+ @stats.text_for_aggregates.should =~ /Summary of.*/
+ end
+
+ it "should display an error message for a failed statup hook" do
+ @stats.aggregate_failures = [{:name => "rspec", :type => :startup}]
+ @stats.text_for_aggregates.should =~ /exception raised while processing startup hook/
+ end
+
+ it "should display an error message for an unspecified output" do
+ @stats.aggregate_failures = [{:name => "rspec", :type => :create}]
+ @stats.text_for_aggregates.should =~ /unspecified output 'rspec' for the action/
+ end
+
+ it "should display an error message for a failed process_result" do
+ @stats.aggregate_failures = [{:name => "rspec", :type => :process_result}]
+ @stats.text_for_aggregates.should =~ /exception raised while processing result data/
+ end
+
+ it "should display an error message for a failed summarize" do
+ @stats.aggregate_failures = [{:name => "rspec", :type => :summarize}]
+ @stats.text_for_aggregates.should =~ /exception raised while summarizing/
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe RPC do
+ describe "#const_missing" do
+ it "should deprecate only the DDL class" do
+ Log.expects(:warn).with("MCollective::RPC::DDL is deprecatd, please use MCollective::DDL instead")
+ MCollective::RPC::DDL.should == MCollective::DDL
+
+ expect { MCollective::RPC::Foo }.to raise_error(NameError)
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe RunnerStats do
+ before do
+ Agents.stubs(:agentlist).returns("agents")
+ Time.stubs(:now).returns(Time.at(0))
+
+ @stats = RunnerStats.new
+
+ logger = mock
+ logger.stubs(:log)
+ logger.stubs(:start)
+ Log.configure(logger)
+ end
+
+ describe "#to_hash" do
+ it "should return the correct data" do
+ @stats.to_hash.keys.sort.should == [:stats, :threads, :pid, :times, :agents].sort
+
+ @stats.to_hash[:stats].should == {:validated => 0, :unvalidated => 0, :passed => 0, :filtered => 0,
+ :starttime => 0, :total => 0, :ttlexpired => 0, :replies => 0}
+
+ @stats.to_hash[:agents].should == "agents"
+ end
+ end
+
+ [[:ttlexpired, :ttlexpired], [:passed, :passed], [:filtered, :filtered],
+ [:validated, :validated], [:received, :total], [:sent, :replies]].each do |tst|
+ describe "##{tst.first}" do
+ it "should increment #{tst.first}" do
+ @stats.send(tst.first)
+ @stats.to_hash[:stats][tst.last].should == 1
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ module Security
+ describe Base do
+ before do
+ @config = mock("config")
+ @config.stubs(:identity).returns("test")
+ @config.stubs(:configured).returns(true)
+ @config.stubs(:topicsep).returns(".")
+
+ @stats = mock("stats")
+
+ @time = Time.now
+ ::Time.stubs(:now).returns(@time)
+
+ MCollective::Log.stubs(:debug).returns(true)
+
+ MCollective::PluginManager << {:type => "global_stats", :class => @stats}
+ MCollective::Config.stubs("instance").returns(@config)
+ MCollective::Util.stubs("empty_filter?").returns(false)
+
+ @plugin = Base.new
+ end
+
+ describe "#should_process_msg?" do
+ it "should correctly validate messages" do
+ m = mock
+ m.stubs(:expected_msgid).returns("rspec")
+
+ @plugin.should_process_msg?(m, "rspec").should == true
+
+ expect {
+ @plugin.should_process_msg?(m, "fail").should == true
+ }.to raise_error(MsgDoesNotMatchRequestID)
+ end
+
+ it "should not test messages without expected_msgid" do
+ m = mock
+ m.stubs(:expected_msgid).returns(nil)
+
+ @plugin.should_process_msg?(m, "rspec").should == true
+ end
+ end
+
+ describe "#validate_filter?" do
+ it "should pass on empty filter" do
+ MCollective::Util.stubs("empty_filter?").returns(true)
+
+ @stats.stubs(:passed).once
+ @stats.stubs(:filtered).never
+ @stats.stubs(:passed).never
+
+ MCollective::Log.expects(:debug).with("Message passed the filter checks").once
+
+ @plugin.validate_filter?({}).should == true
+ end
+
+ it "should pass for known classes" do
+ MCollective::Util.stubs("has_cf_class?").with("foo").returns(true)
+
+ @stats.stubs(:passed).once
+ @stats.stubs(:filtered).never
+
+ MCollective::Log.expects(:debug).with("Message passed the filter checks").once
+ MCollective::Log.expects(:debug).with("Passing based on configuration management class foo").once
+
+ @plugin.validate_filter?({"cf_class" => ["foo"]}).should == true
+ end
+
+ it "should fail for unknown classes" do
+ MCollective::Util.stubs("has_cf_class?").with("foo").returns(false)
+
+ @stats.stubs(:filtered).once
+ @stats.stubs(:passed).never
+
+ MCollective::Log.expects(:debug).with("Message failed the filter checks").once
+ MCollective::Log.expects(:debug).with("Failing based on configuration management class foo").once
+
+ @plugin.validate_filter?({"cf_class" => ["foo"]}).should == false
+ end
+
+ it "should pass for known agents" do
+ MCollective::Util.stubs("has_agent?").with("foo").returns(true)
+
+ @stats.stubs(:passed).once
+ @stats.stubs(:filtered).never
+
+ MCollective::Log.expects(:debug).with("Message passed the filter checks").once
+ MCollective::Log.expects(:debug).with("Passing based on agent foo").once
+
+ @plugin.validate_filter?({"agent" => ["foo"]}).should == true
+ end
+
+ it "should fail for unknown agents" do
+ MCollective::Util.stubs("has_agent?").with("foo").returns(false)
+
+ @stats.stubs(:filtered).once
+ @stats.stubs(:passed).never
+
+ MCollective::Log.expects(:debug).with("Message failed the filter checks").once
+ MCollective::Log.expects(:debug).with("Failing based on agent foo").once
+
+ @plugin.validate_filter?({"agent" => ["foo"]}).should == false
+ end
+
+ it "should pass for known facts" do
+ MCollective::Util.stubs("has_fact?").with("fact", "value", "operator").returns(true)
+
+ @stats.stubs(:passed).once
+ @stats.stubs(:filtered).never
+
+ MCollective::Log.expects(:debug).with("Message passed the filter checks").once
+ MCollective::Log.expects(:debug).with("Passing based on fact fact operator value").once
+
+ @plugin.validate_filter?({"fact" => [{:fact => "fact", :operator => "operator", :value => "value"}]}).should == true
+ end
+
+ it "should fail for unknown facts" do
+ MCollective::Util.stubs("has_fact?").with("fact", "value", "operator").returns(false)
+
+ @stats.stubs(:filtered).once
+ @stats.stubs(:passed).never
+
+ MCollective::Log.expects(:debug).with("Message failed the filter checks").once
+ MCollective::Log.expects(:debug).with("Failing based on fact fact operator value").once
+
+ @plugin.validate_filter?({"fact" => [{:fact => "fact", :operator => "operator", :value => "value"}]}).should == false
+ end
+
+ it "should pass for known identity" do
+ MCollective::Util.stubs("has_identity?").with("test").returns(true)
+
+ @stats.stubs(:passed).once
+ @stats.stubs(:filtered).never
+
+ MCollective::Log.expects(:debug).with("Message passed the filter checks").once
+ MCollective::Log.expects(:debug).with("Passing based on identity").once
+
+ @plugin.validate_filter?({"identity" => ["test"]}).should == true
+ end
+
+ it "should fail for known identity" do
+ MCollective::Util.stubs("has_identity?").with("test").returns(false)
+
+ @stats.stubs(:passed).never
+ @stats.stubs(:filtered).once
+
+ MCollective::Log.expects(:debug).with("Message failed the filter checks").once
+ MCollective::Log.expects(:debug).with("Failed based on identity").once
+
+ @plugin.validate_filter?({"identity" => ["test"]}).should == false
+ end
+
+ it "should treat multiple identity filters correctly" do
+ MCollective::Util.stubs("has_identity?").with("foo").returns(false)
+ MCollective::Util.stubs("has_identity?").with("bar").returns(true)
+
+ @stats.stubs(:passed).once
+ @stats.stubs(:filtered).never
+
+ MCollective::Log.expects(:debug).with("Message passed the filter checks").once
+ MCollective::Log.expects(:debug).with("Passing based on identity").once
+
+ @plugin.validate_filter?({"identity" => ["foo", "bar"]}).should == true
+ end
+
+ it "should fail if no identity matches are found" do
+ MCollective::Util.stubs("has_identity?").with("foo").returns(false)
+ MCollective::Util.stubs("has_identity?").with("bar").returns(false)
+
+ @stats.stubs(:passed).never
+ @stats.stubs(:filtered).once
+
+ MCollective::Log.expects(:debug).with("Message failed the filter checks").once
+ MCollective::Log.expects(:debug).with("Failed based on identity").once
+
+ @plugin.validate_filter?({"identity" => ["foo", "bar"]}).should == false
+ end
+ end
+
+ describe "#create_reply" do
+ it "should return correct data" do
+ expected = {:senderid => "test",
+ :requestid => "reqid",
+ :senderagent => "agent",
+ :msgtime => @time.to_i,
+ :body => "body"}
+
+ @plugin.create_reply("reqid", "agent", "body").should == expected
+ end
+ end
+
+ describe "#create_request" do
+ it "should return correct data" do
+ expected = {:body => "body",
+ :senderid => "test",
+ :requestid => "reqid",
+ :callerid => "uid=#{Process.uid}",
+ :agent => "discovery",
+ :collective => "mcollective",
+ :filter => "filter",
+ :ttl => 20,
+ :msgtime => @time.to_i}
+
+ @plugin.create_request("reqid", "filter", "body", :server, "discovery", "mcollective", 20).should == expected
+ end
+
+ it "should set the callerid when appropriate" do
+ expected = {:body => "body",
+ :senderid => "test",
+ :requestid => "reqid",
+ :agent => "discovery",
+ :collective => "mcollective",
+ :filter => "filter",
+ :callerid => "callerid",
+ :ttl => 60,
+ :msgtime => @time.to_i}
+
+ @plugin.stubs(:callerid).returns("callerid")
+ @plugin.create_request("reqid", "filter", "body", :client, "discovery", "mcollective").should == expected
+ end
+ end
+
+ describe "#valid_callerid?" do
+ it "should not pass invalid callerids" do
+ @plugin.valid_callerid?("foo-bar").should == false
+ @plugin.valid_callerid?("foo=bar=baz").should == false
+ @plugin.valid_callerid?('foo=bar\baz').should == false
+ @plugin.valid_callerid?("foo=bar/baz").should == false
+ @plugin.valid_callerid?("foo=bar|baz").should == false
+ end
+
+ it "should pass valid callerids" do
+ @plugin.valid_callerid?("cert=foo-bar").should == true
+ @plugin.valid_callerid?("uid=foo.bar").should == true
+ @plugin.valid_callerid?("uid=foo.bar.123").should == true
+ end
+ end
+
+ describe "#callerid" do
+ it "should return a unix UID based callerid" do
+ @plugin.callerid.should == "uid=#{Process.uid}"
+ end
+ end
+
+ describe "#validrequest?" do
+ it "should log an error when not implemented" do
+ MCollective::Log.expects(:error).with("validrequest? is not implemented in MCollective::Security::Base")
+ @plugin.validrequest?(nil)
+ end
+ end
+
+ describe "#encoderequest" do
+ it "should log an error when not implemented" do
+ MCollective::Log.expects(:error).with("encoderequest is not implemented in MCollective::Security::Base")
+ @plugin.encoderequest(nil, nil, nil)
+ end
+ end
+
+ describe "#encodereply" do
+ it "should log an error when not implemented" do
+ MCollective::Log.expects(:error).with("encodereply is not implemented in MCollective::Security::Base")
+ @plugin.encodereply(nil, nil, nil)
+ end
+ end
+
+ describe "#decodemsg" do
+ it "should log an error when not implemented" do
+ MCollective::Log.expects(:error).with("decodemsg is not implemented in MCollective::Security::Base")
+ @plugin.decodemsg(nil)
+ end
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Shell do
+ describe "#initialize" do
+ it "should set locale by default" do
+ s = Shell.new("date")
+ s.environment.should == {"LC_ALL" => "C"}
+ end
+
+ it "should merge environment and keep locale" do
+ s = Shell.new("date", :environment => {"foo" => "bar"})
+ s.environment.should == {"LC_ALL" => "C", "foo" => "bar"}
+ end
+
+ it "should allow locale to be overridden" do
+ s = Shell.new("date", :environment => {"LC_ALL" => "TEST", "foo" => "bar"})
+ s.environment.should == {"LC_ALL" => "TEST", "foo" => "bar"}
+ end
+
+ it "should set no environment when given nil" do
+ s = Shell.new("date", :environment => nil)
+ s.environment.should == {}
+ end
+
+ it "should save the command" do
+ s = Shell.new("date")
+ s.command.should == "date"
+ end
+
+ it "should check the cwd exist" do
+ expect {
+ s = Shell.new("date", :cwd => "/nonexistant")
+ }.to raise_error("Directory /nonexistant does not exist")
+ end
+
+ it "should warn of illegal stdin" do
+ expect {
+ s = Shell.new("date", :stdin => nil)
+ }.to raise_error("stdin should be a String")
+ end
+
+ it "should warn of illegal stdout" do
+ expect {
+ s = Shell.new("date", :stdout => nil)
+ }.to raise_error("stdout should support <<")
+ end
+
+ it "should warn of illegal stderr" do
+ expect {
+ s = Shell.new("date", :stderr => nil)
+ }.to raise_error("stderr should support <<")
+ end
+
+ it "should set stdout" do
+ s = Shell.new("date", :stdout => "stdout")
+ s.stdout.should == "stdout"
+ end
+
+ it "should set stderr" do
+ s = Shell.new("date", :stderr => "stderr")
+ s.stderr.should == "stderr"
+ end
+
+ it "should set stdin" do
+ s = Shell.new("date", :stdin => "hello world")
+ s.stdin.should == "hello world"
+ end
+ end
+
+ describe "#runcommand" do
+ it "should run the command" do
+ Shell.any_instance.stubs("systemu").returns(true).once.with("date", "stdout" => '', "stderr" => '', "env" => {"LC_ALL" => "C"}, 'cwd' => Dir.tmpdir)
+ s = Shell.new("date")
+ s.runcommand
+ end
+
+ it "should set stdin, stdout and status" do
+ s = Shell.new('ruby -e "STDERR.puts \"stderr\"; STDOUT.puts \"stdout\""')
+ s.runcommand
+ s.stdout.should == "stdout\n"
+ s.stderr.should == "stderr\n"
+ s.status.exitstatus.should == 0
+ end
+
+ it "should report correct exitcode" do
+ s = Shell.new('ruby -e "exit 1"')
+ s.runcommand
+
+ s.status.exitstatus.should == 1
+ end
+
+ it "shold have correct environment" do
+ s = Shell.new('ruby -e "puts ENV[\'LC_ALL\'];puts ENV[\'foo\'];"', :environment => {"foo" => "bar"})
+ s.runcommand
+ s.stdout.should == "C\nbar\n"
+ end
+
+ it "should save stdout in custom stdout variable" do
+ out = "STDOUT"
+
+ s = Shell.new('echo foo', :stdout => out)
+ s.runcommand
+
+ s.stdout.should == "STDOUTfoo\n"
+ out.should == "STDOUTfoo\n"
+ end
+
+ it "should save stderr in custom stderr variable" do
+ out = "STDERR"
+
+ s = Shell.new('ruby -e "STDERR.puts \"foo\""', :stderr => out)
+ s.runcommand
+
+ s.stderr.should == "STDERRfoo\n"
+ out.should == "STDERRfoo\n"
+ end
+
+ it "should run in the correct cwd" do
+ s = Shell.new('ruby -e "puts Dir.pwd"', :cwd => Dir.tmpdir)
+
+ s.runcommand
+
+ s.stdout.should == "#{Dir.tmpdir}\n"
+ end
+
+ it "should send the stdin" do
+ s = Shell.new('ruby -e "puts STDIN.gets"', :stdin => "hello world")
+ s.runcommand
+
+ s.stdout.should == "hello world\n"
+ end
+
+ it "should support multiple lines of stdin" do
+ s = Shell.new('ruby -e "puts STDIN.gets;puts;puts STDIN.gets"', :stdin => "first line\n2nd line")
+ s.runcommand
+
+ s.stdout.should == "first line\n\n2nd line\n"
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe SSL do
+ before do
+ @rootdir = File.dirname(__FILE__)
+ @ssl = SSL.new("#{@rootdir}/../fixtures/test-public.pem", "#{@rootdir}/../fixtures/test-private.pem")
+ end
+
+ it "should be able to decode base64 text it encoded" do
+ @ssl.base64_decode(@ssl.base64_encode("foo")).should == "foo"
+ end
+
+ it "should decrypt what it encrypted with RSA" do
+ crypted = @ssl.aes_encrypt("foo")
+ decrypted = @ssl.aes_decrypt(crypted[:key], crypted[:data])
+
+ decrypted.should == "foo"
+ end
+
+ it "should be able to decrypt using RSA private key what it encrypted with RSA public key" do
+ crypted = @ssl.rsa_encrypt_with_public("foo")
+ decrypted = @ssl.rsa_decrypt_with_private(crypted)
+
+ decrypted.should == "foo"
+ end
+
+ it "should be able to decrypt using RSA public key what it encrypted with RSA private key" do
+ crypted = @ssl.rsa_encrypt_with_private("foo")
+ decrypted = @ssl.rsa_decrypt_with_public(crypted)
+
+ decrypted.should == "foo"
+ end
+
+ it "using a helper it should be able to decrypt with private key what it encrypted using the public key" do
+ @ssl.decrypt_with_private(@ssl.encrypt_with_public("foo")).should == "foo"
+ @ssl.decrypt_with_private(@ssl.encrypt_with_public("foo", false), false).should == "foo"
+ end
+
+ it "using a helper it should be able to decrypt with public key what it encrypted using the private key" do
+ @ssl.decrypt_with_public(@ssl.encrypt_with_private("foo")).should == "foo"
+ @ssl.decrypt_with_public(@ssl.encrypt_with_private("foo", false), false).should == "foo"
+ end
+
+ describe "#initialize" do
+ it "should default to aes-256-cbc" do
+ @ssl.ssl_cipher.should == "aes-256-cbc"
+ end
+
+ it "should take the configured value when present" do
+ Config.instance.stubs("ssl_cipher").returns("aes-128-cbc")
+ @ssl = SSL.new("#{@rootdir}/../fixtures/test-public.pem", "#{@rootdir}/../fixtures/test-private.pem")
+
+ @ssl.ssl_cipher.should == "aes-128-cbc"
+ end
+
+ it "should set the supplied ssl cipher" do
+ @ssl = SSL.new("#{@rootdir}/../fixtures/test-public.pem", "#{@rootdir}/../fixtures/test-private.pem", nil, "aes-128-cbc")
+ @ssl.ssl_cipher.should == "aes-128-cbc"
+ end
+
+ it "should prefer the supplied cipher over configured cipher" do
+ Config.instance.stubs("aes_key_size").returns("foo-foo-foo")
+ @ssl = SSL.new("#{@rootdir}/../fixtures/test-public.pem", "#{@rootdir}/../fixtures/test-private.pem", nil, "aes-128-cbc")
+
+ @ssl.ssl_cipher.should == "aes-128-cbc"
+ end
+
+ it "should fail on invalid ciphers" do
+ expect {
+ @ssl = SSL.new("#{@rootdir}/../fixtures/test-public.pem", "#{@rootdir}/../fixtures/test-private.pem", nil, "foo-foo-foo")
+ }.to raise_error("The supplied cipher 'foo-foo-foo' is not supported")
+ end
+ end
+
+ describe "#read_key" do
+ it "should fail on non exiting files" do
+ expect {
+ @ssl.read_key(:public, "/nonexisting")
+ }.to raise_error("Could not find key /nonexisting")
+ end
+
+ it "should fail on existing, empty files" do
+ File.expects(:exist?).with('key').returns(true)
+ File.expects(:zero?).with('key').returns(true)
+ expect{
+ @ssl.read_key(:public, 'key')
+ }.to raise_error("public key file 'key' is empty")
+ end
+
+ it "should fail on unknown key types" do
+ expect {
+ @ssl.read_key(:unknown, @ssl.public_key_file)
+ }.to raise_error("Can only load :public or :private keys")
+ end
+
+ it "should read a public key" do
+ @ssl.read_key(:public, "#{@rootdir}/../fixtures/test-public.pem")
+ end
+
+ it "should read the public key from a certificate" do
+ @ssl.read_key(:public, "#{@rootdir}/../fixtures/test-cert.pem").to_s.should match(/.+BEGIN.+PUBLIC KEY.+END.+PUBLIC KEY.+/m)
+ end
+
+ it "should return nil if no key was given" do
+ @ssl.read_key(:public).should == nil
+ end
+
+ it "should return nil if nil key was given" do
+ @ssl.read_key(:public, nil).should == nil
+ end
+
+ it "should clear the OpenSSL error queue on ruby 1.8" do
+ Util.expects(:ruby_version).returns("1.8.7")
+ OpenSSL.expects(:errors)
+ @ssl.read_key(:public, "#{@rootdir}/../fixtures/test-public.pem")
+ @ssl.read_key(:private, "#{@rootdir}/../fixtures/test-private.pem")
+ end
+
+ it "should not clear the OpenSSL error queue on ruby > 1.8" do
+ Util.expects(:ruby_version).returns("1.9.3")
+ OpenSSL.expects(:errors).never
+ @ssl.read_key(:public, "#{@rootdir}/../fixtures/test-public.pem")
+ @ssl.read_key(:private, "#{@rootdir}/../fixtures/test-private.pem")
+ end
+ end
+
+ describe "#base64_encode" do
+ it "should correctly encode" do
+ @ssl.base64_encode("foo").should == "Zm9v\n"
+ SSL.base64_encode("foo").should == "Zm9v\n"
+ end
+ end
+
+ describe "#base64_decode" do
+ it "should correctly decode" do
+ @ssl.base64_decode("Zm9v").should == "foo"
+ SSL.base64_decode("Zm9v").should == "foo"
+ end
+ end
+
+ describe "#aes_encrypt" do
+ it "should create a key and data" do
+ crypted = @ssl.aes_encrypt("foo")
+
+ crypted.include?(:key).should == true
+ crypted.include?(:data).should == true
+ end
+ end
+
+ describe "#aes_decrypt" do
+ it "should decrypt correctly given key and data" do
+ key = @ssl.base64_decode("rAaCyW6qB0XqZNa9hji0qHwrI3P47t8diLNXoemW9ss=")
+ data = @ssl.base64_decode("mSthvO/wSl0ArNOcgysTVw==")
+
+ @ssl.aes_decrypt(key, data).should == "foo"
+ end
+
+ it "should decrypt correctly given key, data and cipher" do
+ key = @ssl.base64_decode("VEma3a/R7fjw2M4d0NIctA==")
+ data = @ssl.base64_decode("FkH6qLvKTn7a+uNPe8ciHA==")
+
+ # the default aes-256-cbc should fail here, the key above is 128 bit
+ # the exception classes changed mid-1.9.2 :(
+ if OpenSSL.constants.include?("CipherError")
+ expect { @ssl.aes_decrypt(key, data) }.to raise_error(OpenSSL::CipherError)
+ else
+ expect { @ssl.aes_decrypt(key, data) }.to raise_error(OpenSSL::Cipher::CipherError)
+ end
+
+ # new ssl instance configured for aes-128-cbc, should work
+ @ssl = SSL.new("#{@rootdir}/../fixtures/test-public.pem", "#{@rootdir}/../fixtures/test-private.pem", nil, "aes-128-cbc")
+ @ssl.aes_decrypt(key, data).should == "foo"
+ end
+ end
+
+ describe "#md5" do
+ it "should produce correct md5 sums" do
+ # echo -n 'hello world'|md5sum
+ @ssl.md5("hello world").should == "5eb63bbbe01eeed093cb22bb8f5acdc3"
+ end
+ end
+ describe "#sign" do
+ it "should sign the message without base64 by default" do
+ SSL.md5(@ssl.sign("hello world")).should == "8269b23f55945aaa82efbff857c845a6"
+ end
+
+ it "should support base64 encoding messages" do
+ SSL.md5(@ssl.sign("hello world", true)).should == "8a4eb3c3d44d22c46dc36a7e441d8db0"
+ end
+ end
+
+ describe "#verify_signature" do
+ it "should correctly verify a message signed using the same keypair" do
+ @ssl.verify_signature(@ssl.sign("hello world"), "hello world").should == true
+ @ssl.verify_signature(@ssl.sign("hello world", true), "hello world", true).should == true
+ end
+
+ it "should fail to verify messages not signed by the key" do
+ @ssl.verify_signature("evil fake signature", "hello world").should == false
+ end
+ end
+
+ describe "#decrypt_with_public" do
+ it "should decrypt correctly given key and data in base64 format" do
+ crypted = {:key=> "YaRcSDdcKgnRZ4Eu2eirl/+lzDgVkPZ41kXAQQNOi+6AfjdbbOW7Zblibx9r\n3TzZAi0ulA94gqNAXPvPC8LaO8W9TtJwlto/RHwDM7ZdfqEImSYoVACFNq28\n+0MLr3K3hIBsB1pyxgFTQul+MrCq+3Fik7Nj7ZKkJUT2veyqbg8=",
+ :data=>"TLVw1EYeOaGDmEC/R2I/cA=="}
+
+ @ssl.decrypt_with_public(crypted).should == "foo"
+ end
+ end
+
+ describe "#decrypt_with_private" do
+ it "should decrypt correctly given key and data in base64 format" do
+ crypted = {:key=> "kO1kUgJBiEBdoajN4OHp9BOie6dCznf1YKbBnp3LOyBxcDDQtjxEBlPmjQve\npXrQJ5xpLX6oNBxzU18Pf2SKYUZSbzIkDUb97GQY0WoBQsdM2OwPXH+HtF2A\no5N8iIx9srPAEAFa6hZAdqvcmRT/SzhP1kH+Gyy8fyvW8HGBjNY=",
+ :data=>"gDTaHCmes/Yua4jtjmgukQ=="}
+
+ @ssl.decrypt_with_private(crypted).should == "foo"
+ end
+ end
+
+ describe "#decrypt_with_private" do
+ it "should fail if not given a key" do
+ expect {
+ @ssl.decrypt_with_private({:iv => "x", :data => "x"})
+ }.to raise_error("Crypted data should include a key")
+ end
+
+ it "should fail if not given data" do
+ expect {
+ @ssl.decrypt_with_private({:iv => "x", :key => "x"})
+ }.to raise_error("Crypted data should include data")
+ end
+ end
+
+ describe "#decrypt_with_public" do
+ it "should fail if not given a key" do
+ expect {
+ @ssl.decrypt_with_public({:iv => "x", :data => "x"})
+ }.to raise_error("Crypted data should include a key")
+ end
+
+ it "should fail if not given data" do
+ expect {
+ @ssl.decrypt_with_public({:iv => "x", :key => "x"})
+ }.to raise_error("Crypted data should include data")
+ end
+ end
+
+ describe "#uuid" do
+ it "should produce repeatable uuids" do
+ SSL.uuid("hello world").should == SSL.uuid("hello world")
+ end
+
+ it "should not always produce the same uuid" do
+ SSL.uuid.should_not == SSL.uuid
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+class String
+ describe "#start_with?" do
+ it "should return true for matches" do
+ "hello world".start_with?("hello").should == true
+ end
+
+ it "should return false for non matches" do
+ "hello world".start_with?("world").should == false
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+class Symbol
+ describe "#<=>" do
+ it "should be sortable" do
+ [:foo, :bar].sort.should == [:bar, :foo]
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+require 'mcollective/unix_daemon'
+
+module MCollective
+ describe UnixDaemon do
+ describe "#daemonize_runner" do
+ it "should not run on the windows platform" do
+ Util.expects("windows?").returns(true)
+ expect { UnixDaemon.daemonize_runner }.to raise_error("The Unix Daemonizer can not be used on the Windows Platform")
+ end
+
+ it "should write the pid file if requested" do
+ f = mock
+ f.expects(:write).with(Process.pid)
+
+ File.expects(:open).with("/nonexisting", "w").yields(f)
+
+ r = mock
+ r.expects(:run)
+
+ Runner.expects(:new).returns(r)
+ UnixDaemon.expects(:daemonize).yields
+
+ UnixDaemon.daemonize_runner("/nonexisting")
+ end
+
+ it "should not write a pid file unless requested" do
+ r = mock
+ r.expects(:run)
+
+ UnixDaemon.expects(:daemonize).yields
+ Runner.expects(:new).returns(r)
+ File.expects(:open).never
+
+ UnixDaemon.daemonize_runner(nil)
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Util do
+ before do
+ class MCollective::Connector::Stomp<MCollective::Connector::Base; end
+
+ PluginManager.clear
+ PluginManager << {:type => "connector_plugin", :class => MCollective::Connector::Stomp.new}
+ end
+
+ describe "#t" do
+ it "should correctly translate the message" do
+ I18n.expects(:t).with("PLMC1.pattern", {:rspec => "test"})
+ I18n.expects(:t).with("PLMC1.expanded", {:rspec => "test"})
+ Util.t(:PLMC1, :rspec => "test")
+ Util.t("PLMC1.expanded", :rspec => "test")
+ end
+ end
+
+ describe "#windows?" do
+ it "should correctly detect windows on unix platforms" do
+ RbConfig::CONFIG.expects("[]").returns("linux")
+ Util.windows?.should == false
+ end
+
+ it "should correctly detect windows on windows platforms" do
+ RbConfig::CONFIG.expects("[]").returns("win32")
+ Util.windows?.should == true
+ end
+ end
+
+ describe "#setup_windows_sleeper" do
+ it "should set up a thread on the windows platform" do
+ Thread.expects(:new)
+ Util.expects("windows?").returns(true).once
+ Util.setup_windows_sleeper
+ end
+
+ it "should not set up a thread on other platforms" do
+ Thread.expects(:new).never
+ Util.expects("windows?").returns(false).once
+ Util.setup_windows_sleeper
+ end
+ end
+
+ describe "#has_cf_class?" do
+ before do
+ logger = mock
+ logger.stubs(:log)
+ logger.stubs(:start)
+ Log.configure(logger)
+
+ config = mock
+ config.stubs(:classesfile).returns("/some/file")
+ Config.expects(:instance).returns(config)
+ end
+
+ it "should read the classes lines from the correct file" do
+ File.expects(:readlines).with("/some/file")
+
+ Util.has_cf_class?("test")
+ end
+
+ it "should support regular expression searches" do
+ File.stubs(:readlines).returns(["test_class_test"])
+ String.any_instance.expects(:match).with("^/").returns(true)
+ String.any_instance.expects(:match).with(Regexp.new("class")).returns(true)
+
+ Util.has_cf_class?("/class/").should == true
+ end
+
+ it "should support exact string matches" do
+ File.stubs(:readlines).returns(["test_class_test"])
+ String.any_instance.expects(:match).with("^/").returns(false)
+ String.any_instance.expects(:match).with(Regexp.new("test_class_test")).never
+
+ Util.has_cf_class?("test_class_test").should == true
+ end
+
+ it "should report a warning when the classes file cannot be parsed" do
+ File.stubs(:readlines).returns(nil)
+ Log.expects(:warn).with("Parsing classes file '/some/file' failed: NoMethodError: undefined method `each' for nil:NilClass")
+
+ Util.has_cf_class?("test_class_test").should == false
+ end
+ end
+
+ describe "#shellescape" do
+ it "should return '' for empty strings" do
+ Util.shellescape("").should == "''"
+ end
+
+ it "should quote newlines" do
+ Util.shellescape("\n").should == "'\n'"
+ end
+
+ it "should escape unwanted characters" do
+ Util.shellescape("foo;bar").should == 'foo\;bar'
+ Util.shellescape('foo`bar').should == 'foo\`bar'
+ Util.shellescape('foo$bar').should == 'foo\$bar'
+ Util.shellescape('foo|bar').should == 'foo\|bar'
+ Util.shellescape('foo&&bar').should == 'foo\&\&bar'
+ Util.shellescape('foo||bar').should == 'foo\|\|bar'
+ Util.shellescape('foo>bar').should == 'foo\>bar'
+ Util.shellescape('foo<bar').should == 'foo\<bar'
+ Util.shellescape('foobar').should == 'foobar'
+ end
+ end
+
+ describe "#make_subscription" do
+ it "should validate target types" do
+ expect {
+ Util.make_subscriptions("test", "test", "test")
+ }.to raise_error("Unknown target type test")
+
+ Config.instance.stubs(:collectives).returns(["test"])
+ Util.make_subscriptions("test", :broadcast, "test")
+ end
+
+ it "should return a subscription for each collective" do
+ Config.instance.stubs(:collectives).returns(["collective1", "collective2"])
+ Util.make_subscriptions("test", :broadcast).should == [{:type=>:broadcast,
+ :agent=>"test",
+ :collective=>"collective1"},
+ {:type=>:broadcast,
+ :agent=>"test",
+ :collective=>"collective2"}]
+ end
+
+ it "should validate given collective" do
+ Config.instance.stubs(:collectives).returns(["collective1", "collective2"])
+
+ expect {
+ Util.make_subscriptions("test", :broadcast, "test")
+ }.to raise_error("Unknown collective 'test' known collectives are 'collective1, collective2'")
+ end
+
+ it "should return a single subscription array given a collective" do
+ Config.instance.stubs(:collectives).returns(["collective1", "collective2"])
+ Util.make_subscriptions("test", :broadcast, "collective1").should == [{:type=>:broadcast, :agent=>"test", :collective=>"collective1"}]
+ end
+ end
+
+ describe "#subscribe" do
+ it "should subscribe to multiple topics given an Array" do
+ subs1 = {:agent => "test_agent", :type => "test_type", :collective => "test_collective"}
+ subs2 = {:agent => "test_agent2", :type => "test_type2", :collective => "test_collective2"}
+
+ MCollective::Connector::Stomp.any_instance.expects(:subscribe).with("test_agent", "test_type", "test_collective").once
+ MCollective::Connector::Stomp.any_instance.expects(:subscribe).with("test_agent2", "test_type2", "test_collective2").once
+
+ Util.subscribe([subs1, subs2])
+ end
+
+ it "should subscribe to a single topic given a hash" do
+ MCollective::Connector::Stomp.any_instance.expects(:subscribe).with("test_agent", "test_type", "test_collective").once
+ Util.subscribe({:agent => "test_agent", :type => "test_type", :collective => "test_collective"})
+ end
+ end
+
+ describe "#unsubscribe" do
+ it "should unsubscribe to multiple topics given an Array" do
+ subs1 = {:agent => "test_agent", :type => "test_type", :collective => "test_collective"}
+ subs2 = {:agent => "test_agent2", :type => "test_type2", :collective => "test_collective2"}
+ MCollective::Connector::Stomp.any_instance.expects(:unsubscribe).with("test_agent", "test_type", "test_collective").once
+ MCollective::Connector::Stomp.any_instance.expects(:unsubscribe).with("test_agent2", "test_type2", "test_collective2").once
+
+ Util.unsubscribe([subs1, subs2])
+ end
+
+ it "should subscribe to a single topic given a hash" do
+ MCollective::Connector::Stomp.any_instance.expects(:unsubscribe).with("test_agent", "test_type", "test_collective").once
+ Util.unsubscribe({:agent => "test_agent", :type => "test_type", :collective => "test_collective"})
+ end
+ end
+
+ describe "#empty_filter?" do
+ it "should correctly compare empty filters" do
+ Util.empty_filter?(Util.empty_filter).should == true
+ end
+
+ it "should treat an empty hash as an empty filter" do
+ Util.empty_filter?({}).should == true
+ end
+
+ it "should detect non empty filters correctly" do
+ filter = Util.empty_filter
+ filter["cf_class"] << "meh"
+
+
+ Util.empty_filter?(filter).should == false
+ end
+ end
+
+ describe "#empty_filter" do
+ it "should create correct empty filters" do
+ Util.empty_filter.should == {"fact" => [], "cf_class" => [], "agent" => [], "identity" => [], "compound" => []}
+ end
+ end
+
+ describe "#default_options" do
+ it "should supply correct default options" do
+ Config.instance.stubs(:default_discovery_options).returns([])
+ empty_filter = Util.empty_filter
+ config_file = Util.config_file_for_user
+
+ Util.default_options.should == {:verbose => false, :disctimeout => nil, :timeout => 5, :config => config_file, :filter => empty_filter, :collective => nil, :discovery_method => nil, :discovery_options => []}
+ end
+ end
+
+ describe "#has_fact?" do
+ it "should handle missing facts correctly" do
+ MCollective::Facts.expects("[]").with("foo").returns(nil).once
+ Util.has_fact?("foo", "1", "==").should == false
+ end
+
+ it "should handle regex in a backward compatible way" do
+ MCollective::Facts.expects("[]").with("foo").returns("foo").times(6)
+ Util.has_fact?("foo", "foo", "=~").should == true
+ Util.has_fact?("foo", "/foo/", "=~").should == true
+ Util.has_fact?("foo", "foo", "=~").should == true
+ Util.has_fact?("foo", "bar", "=~").should == false
+ Util.has_fact?("foo", "/bar/", "=~").should == false
+ Util.has_fact?("foo", "bar", "=~").should == false
+ end
+
+ it "should evaluate equality" do
+ MCollective::Facts.expects("[]").with("foo").returns("foo").twice
+ Util.has_fact?("foo", "foo", "==").should == true
+ Util.has_fact?("foo", "bar", "==").should == false
+ end
+
+ it "should handle numeric comparisons correctly" do
+ MCollective::Facts.expects("[]").with("foo").returns("1").times(8)
+ Util.has_fact?("foo", "2", ">=").should == false
+ Util.has_fact?("foo", "1", ">=").should == true
+ Util.has_fact?("foo", "2", "<=").should == true
+ Util.has_fact?("foo", "1", "<=").should == true
+ Util.has_fact?("foo", "1", "<").should == false
+ Util.has_fact?("foo", "1", ">").should == false
+ Util.has_fact?("foo", "1", "!=").should == false
+ Util.has_fact?("foo", "2", "!=").should == true
+ end
+
+ it "should handle alphabetic comparisons correctly" do
+ MCollective::Facts.expects("[]").with("foo").returns("b").times(8)
+ Util.has_fact?("foo", "c", ">=").should == false
+ Util.has_fact?("foo", "a", ">=").should == true
+ Util.has_fact?("foo", "a", "<=").should == false
+ Util.has_fact?("foo", "b", "<=").should == true
+ Util.has_fact?("foo", "b", "<").should == false
+ Util.has_fact?("foo", "b", ">").should == false
+ Util.has_fact?("foo", "b", "!=").should == false
+ Util.has_fact?("foo", "a", "!=").should == true
+ end
+ end
+
+ describe "#parse_fact_string" do
+ it "should parse old style regex fact matches" do
+ Util.parse_fact_string("foo=/bar/").should == {:fact => "foo", :value => "/bar/", :operator => "=~"}
+ Util.parse_fact_string("foo = /bar/").should == {:fact => "foo", :value => "/bar/", :operator => "=~"}
+ end
+
+ it "should parse old style equality" do
+ Util.parse_fact_string("foo=bar").should == {:fact => "foo", :value => "bar", :operator => "=="}
+ Util.parse_fact_string("foo = bar").should == {:fact => "foo", :value => "bar", :operator => "=="}
+ end
+
+ it "should parse regex fact matches" do
+ Util.parse_fact_string("foo=~bar").should == {:fact => "foo", :value => "bar", :operator => "=~"}
+ Util.parse_fact_string("foo =~ bar").should == {:fact => "foo", :value => "bar", :operator => "=~"}
+ end
+
+ it "should treat => like >=" do
+ Util.parse_fact_string("foo=>bar").should == {:fact => "foo", :value => "bar", :operator => ">="}
+ Util.parse_fact_string("foo => bar").should == {:fact => "foo", :value => "bar", :operator => ">="}
+ end
+
+ it "should treat =< like <=" do
+ Util.parse_fact_string("foo=<bar").should == {:fact => "foo", :value => "bar", :operator => "<="}
+ Util.parse_fact_string("foo =< bar").should == {:fact => "foo", :value => "bar", :operator => "<="}
+ end
+
+ it "should parse less than or equal" do
+ Util.parse_fact_string("foo<=bar").should == {:fact => "foo", :value => "bar", :operator => "<="}
+ Util.parse_fact_string("foo <= bar").should == {:fact => "foo", :value => "bar", :operator => "<="}
+ end
+
+ it "should parse greater than or equal" do
+ Util.parse_fact_string("foo>=bar").should == {:fact => "foo", :value => "bar", :operator => ">="}
+ Util.parse_fact_string("foo >= bar").should == {:fact => "foo", :value => "bar", :operator => ">="}
+ end
+
+ it "should parse less than" do
+ Util.parse_fact_string("foo<bar").should == {:fact => "foo", :value => "bar", :operator => "<"}
+ Util.parse_fact_string("foo < bar").should == {:fact => "foo", :value => "bar", :operator => "<"}
+ end
+
+ it "should parse greater than" do
+ Util.parse_fact_string("foo>bar").should == {:fact => "foo", :value => "bar", :operator => ">"}
+ Util.parse_fact_string("foo > bar").should == {:fact => "foo", :value => "bar", :operator => ">"}
+ end
+
+ it "should parse greater than" do
+ Util.parse_fact_string("foo>bar").should == {:fact => "foo", :value => "bar", :operator => ">"}
+ Util.parse_fact_string("foo > bar").should == {:fact => "foo", :value => "bar", :operator => ">"}
+ end
+
+ it "should parse not equal" do
+ Util.parse_fact_string("foo!=bar").should == {:fact => "foo", :value => "bar", :operator => "!="}
+ Util.parse_fact_string("foo != bar").should == {:fact => "foo", :value => "bar", :operator => "!="}
+ end
+
+ it "should parse equal to" do
+ Util.parse_fact_string("foo==bar").should == {:fact => "foo", :value => "bar", :operator => "=="}
+ Util.parse_fact_string("foo == bar").should == {:fact => "foo", :value => "bar", :operator => "=="}
+ end
+
+ it "should fail for facts in the wrong format" do
+ expect {
+ Util.parse_fact_string("foo")
+ }.to raise_error("Could not parse fact foo it does not appear to be in a valid format")
+ end
+ end
+
+ describe "#colorize" do
+ it "should not add color codes when color is disabled" do
+ Config.instance.stubs(:color).returns(false)
+ Util.colorize(:red, "hello world").should == "hello world"
+ end
+
+ it "should add color when color is enabled" do
+ Config.instance.stubs(:color).returns(true)
+ Util.colorize(:red, "hello world").should == "\e[31mhello world\e[0m"
+ end
+ end
+
+ describe "#align_text" do
+ it "should default to 80 if the terminal dimensions are unknown" do
+ Util.stubs(:terminal_dimensions).returns([0,0])
+
+ rootdir = File.dirname(__FILE__)
+ input = File.read("#{rootdir}/../fixtures/util/4.in")
+ output = File.read("#{rootdir}/../fixtures/util/4.out")
+
+ (Util.align_text(input, nil, 3) + "\n").should == output
+ end
+
+ it "should return the origional string if console lines are 0" do
+ result = Util.align_text("test", 0)
+ result.should == "test"
+ end
+
+ it "should return the origional string if preamble is greater than console lines" do
+ result = Util.align_text("test", 5, 6)
+ result.should == "test"
+ end
+
+ it "should return a string prefixed by the preamble" do
+ result = Util.align_text("test")
+ result.should == " test"
+ end
+
+ it "should correctly align strings" do
+ rootdir = File.dirname(__FILE__)
+ (1..2).each do |i|
+ input = File.read("#{rootdir}/../fixtures/util/#{i}.in")
+ output = File.read("#{rootdir}/../fixtures/util/#{i}.out")
+
+ (Util.align_text(input, 158 , 5) + "\n").should == output
+ end
+
+ input = File.read("#{rootdir}/../fixtures/util/3.in")
+ output = File.read("#{rootdir}/../fixtures/util/3.out")
+
+ (Util.align_text(input, 30, 0) + "\n").should == output
+ end
+ end
+
+ describe "#terminal_dimensions" do
+ it "should return 0 if there is no tty" do
+ stdout = mock()
+ stdout.expects(:tty?).returns(false)
+ result = Util.terminal_dimensions(stdout)
+ result.should == [0,0]
+ end
+
+ it "should return the default dimensions for a windows terminal" do
+ stdout = mock()
+ stdout.expects(:tty?).returns(true)
+ Util.expects(:windows?).returns(true)
+ result = Util.terminal_dimensions(stdout)
+ result.should == [80, 40]
+ end
+
+ it "should return 0 if an exception was raised" do
+ stdout = mock()
+ stdout.expects(:tty?).raises("error")
+ result = Util.terminal_dimensions(stdout)
+ result.should == [0, 0]
+ end
+
+ it "should return the correct dimensions if ENV columns and lines are set" do
+ stdout = mock()
+ stdout.expects(:tty?).returns(true)
+ environment = mock()
+ environment.expects(:[]).with("COLUMNS").returns(5).twice
+ environment.expects(:[]).with("LINES").returns(5).twice
+ result = Util.terminal_dimensions(stdout, environment)
+ result.should == [5,5]
+ end
+
+ it "should return the correct dimensions if ENV term is set and tput is present" do
+ stdout = mock()
+ stdout.expects(:tty?).returns(true)
+ environment = mock()
+ environment.expects(:[]).with("COLUMNS").returns(false)
+ environment.expects(:[]).with("TERM").returns(true)
+
+ Util.expects(:command_in_path?).with("tput").returns(true)
+ Util.stubs(:`).returns("5")
+
+ result = Util.terminal_dimensions(stdout, environment)
+ result.should == [5,5]
+ end
+
+ it "should return the correct dimensions if stty is present" do
+ stdout = mock()
+ stdout.expects(:tty?).returns(true)
+
+ environment = mock()
+ environment.expects(:[]).with("COLUMNS").returns(false)
+ environment.expects(:[]).with("TERM").returns(false)
+
+ Util.expects(:command_in_path?).with("stty").returns(true)
+ Util.stubs(:`).returns("5 5")
+
+ result = Util.terminal_dimensions(stdout, environment)
+ result.should == [5,5]
+ end
+ end
+
+ describe "#command_in_path?" do
+ it "should return true if the command is found" do
+ File.stubs(:exist?).returns(true)
+ result = Util.command_in_path?("test")
+ result.should == true
+ end
+
+ it "should return false if the command cannot be found" do
+ File.stubs(:exist?).returns(false)
+ result = Util.command_in_path?("test")
+ result.should == false
+ end
+ end
+
+ describe "#absolute_path?" do
+ it "should work correctly validate the path" do
+ Util.absolute_path?('.', '/', '\\').should == false
+ Util.absolute_path?('foo/foo', '/', '\\').should == false
+ Util.absolute_path?('foo\\bar', '/', '\\').should == false
+ Util.absolute_path?('../foo/bar', '/', '\\').should == false
+
+ Util.absolute_path?('\\foo/foo', '/', '\\').should == true
+ Util.absolute_path?('\\', '/', '\\').should == true
+ Util.absolute_path?('/foo', '/', '\\').should == true
+ Util.absolute_path?('/foo/foo', '/', '\\').should == true
+
+ Util.absolute_path?('.', '/', nil).should == false
+ Util.absolute_path?('foo/foo', '/', nil).should == false
+ Util.absolute_path?('foo\\bar', '/', nil).should == false
+ Util.absolute_path?('../foo/bar', '/', nil).should == false
+
+ Util.absolute_path?('\\foo/foo', '/', nil).should == false
+ Util.absolute_path?('\\', '/', nil).should == false
+ Util.absolute_path?('/foo', '/', nil).should == true
+ Util.absolute_path?('/foo/foo', '/', nil).should == true
+ end
+ end
+
+ describe "#versioncmp" do
+ it "should be able to sort a long set of various unordered versions" do
+ ary = %w{ 1.1.6 2.3 1.1a 3.0 1.5 1 2.4 1.1-4 2.3.1 1.2 2.3.0 1.1-3 2.4b 2.4 2.40.2 2.3a.1 3.1 0002 1.1-5 1.1.a 1.06}
+
+ newary = ary.sort {|a, b| Util.versioncmp(a,b) }
+
+ newary.should == ["0002", "1", "1.06", "1.1-3", "1.1-4", "1.1-5", "1.1.6", "1.1.a", "1.1a", "1.2", "1.5", "2.3", "2.3.0", "2.3.1", "2.3a.1", "2.4", "2.4", "2.4b", "2.40.2", "3.0", "3.1"]
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+require File.dirname(__FILE__) + '/../../plugins/mcollective/validator/array_validator.rb'
+
+module MCollective
+ module Validator
+ describe "#load_validators" do
+ it "should not reload the plugins if the plugin cache has not expired" do
+ Validator.instance_variable_set(:@last_load, nil)
+ PluginManager.expects(:find_and_load).with("validator").once
+ Validator.load_validators
+ Validator.load_validators
+ end
+ end
+
+ describe "#[]" do
+ before do
+ Validator.load_validators
+ end
+
+ it "should return the correct class if klass is given as klass" do
+ result = Validator["array"]
+ result.should == ArrayValidator
+ end
+ it "should return the correct class if klass is given as KlassValidator" do
+ result = Validator["ArrayValidator"]
+ result.should == ArrayValidator
+ end
+ it "should return the correct class if klass is given as :klass" do
+ result = Validator[:array]
+ result.should == ArrayValidator
+ end
+ end
+
+ describe "#method_missing" do
+ it "should load a plugin if a validator method is called and the plugin exists" do
+ ArrayValidator.expects(:validate).with(2, [1,2,3])
+ result = Validator.array(2,[1,2,3])
+ end
+
+ it "should call super if a validator method is called and the plugin does not exist" do
+ expect{
+ Validator.rspec(1,2,3)
+ }.to raise_error(ValidatorError)
+ end
+ end
+
+ describe "#validator_class" do
+ it "should return the correct string for a given validator plugin name" do
+ result = Validator.validator_class("test")
+ result.should == "TestValidator"
+ end
+ end
+
+ describe "#has_validator?" do
+ it "should return true if the validator has been loaded" do
+ Validator.const_set(:TestValidator, Class)
+ Validator.has_validator?("test").should == true
+ end
+
+ it "should return false if the validator has not been loaded" do
+ Validator.has_validator?("test2").should == false
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ describe Vendor do
+ describe "#vendor_dir" do
+ it "should return correct vendor directory" do
+ specdir = File.dirname(__FILE__)
+ expected_dir = File.expand_path("#{specdir}/../../lib/mcollective/vendor")
+ Vendor.vendor_dir.should == expected_dir
+ end
+ end
+
+ describe "#load_entry" do
+ it "should attempt to load the correct path" do
+ specdir = File.dirname(__FILE__)
+ expected_dir = File.expand_path("#{specdir}/../../lib/mcollective/vendor")
+
+ Class.any_instance.stubs("load").with("#{expected_dir}/foo").once
+
+ Vendor.load_entry("foo")
+ end
+ end
+
+ describe "#require_libs" do
+ it "should require the vendor loader" do
+ Class.any_instance.stubs("require").with("mcollective/vendor/require_vendored").once
+
+ Vendor.require_libs
+ end
+ end
+ end
+end
--- /dev/null
+#!/usr/bin/env rspec
+
+require 'spec_helper'
+
+module MCollective
+ if Util.windows?
+ require 'mcollective/windows_daemon'
+
+ describe WindowsDaemon do
+ describe "#daemonize_runner" do
+ it "should only run on the windows platform" do
+ Util.expects("windows?").returns(false)
+ expect { WindowsDaemon.daemonize_runner }.to raise_error("The Windows Daemonizer should only be used on the Windows Platform")
+ end
+
+ it "should not support writing pid files" do
+ expect { WindowsDaemon.daemonize_runner(true) }.to raise_error("Writing pid files are not supported on the Windows Platform")
+ end
+
+ it "should start the mainloop" do
+ WindowsDaemon.expects(:mainloop)
+ WindowsDaemon.daemonize_runner
+ end
+ end
+
+ describe "#service_stop" do
+ it "should disconnect and exit" do
+ Log.expects(:info)
+
+ connector = mock
+ connector.expects(:disconnect).once
+
+ PluginManager.expects("[]").with("connector_plugin").returns(connector)
+
+ d = WindowsDaemon.new
+ d.expects("exit!").once
+
+ d.service_stop
+ end
+ end
+ end
+ end
+end
--- /dev/null
+--format s --backtrace
--- /dev/null
+ <div class="span-3">
+
+ </div>
+ <div class="span-3">
+ <a href="/">Home</a>
+ </div>
+ <div class="span-3">
+ <a href="http://projects.puppetlabs.com/projects/mcollective/issues">Bugs</a>
+ </div>
+ <div class="span-3">
+ <a href="https://github.com/puppetlabs/marionette-collective">GitHub</a>
+ </div>
+ <div class="span-3">
+ <a href="http://code.google.com/p/mcollective/downloads/list">Files</a>
+ </div>
+ <div class="span-3 last">
+ <a href="http://groups.google.com/group/mcollective-users">User List</a>
+ </div>
+
--- /dev/null
+/* -----------------------------------------------------------------------
+
+
+ Blueprint CSS Framework 0.9
+ http://blueprintcss.org
+
+ * Copyright (c) 2007-Present. See LICENSE for more info.
+ * See README for instructions on how to use Blueprint.
+ * For credits and origins, see AUTHORS.
+ * This is a compressed file. See the sources in the 'src' directory.
+
+----------------------------------------------------------------------- */
+
+/* ie.css */
+body {text-align:center;}
+.container {text-align:left;}
+* html .column, * html .span-1, * html .span-2, * html .span-3, * html .span-4, * html .span-5, * html .span-6, * html .span-7, * html .span-8, * html .span-9, * html .span-10, * html .span-11, * html .span-12, * html .span-13, * html .span-14, * html .span-15, * html .span-16, * html .span-17, * html .span-18, * html .span-19, * html .span-20, * html .span-21, * html .span-22, * html .span-23, * html .span-24 {display:inline;overflow-x:hidden;}
+* html legend {margin:0px -8px 16px 0;padding:0;}
+sup {vertical-align:text-top;}
+sub {vertical-align:text-bottom;}
+html>body p code {*white-space:normal;}
+hr {margin:-8px auto 11px;}
+img {-ms-interpolation-mode:bicubic;}
+.clearfix, .container {display:inline-block;}
+* html .clearfix, * html .container {height:1%;}
+fieldset {padding-top:0;}
+textarea {overflow:auto;}
+input.text, input.title, textarea {background-color:#fff;border:1px solid #bbb;}
+input.text:focus, input.title:focus {border-color:#666;}
+input.text, input.title, textarea, select {margin:0.5em 0;}
+input.checkbox, input.radio {position:relative;top:.25em;}
+form.inline div, form.inline p {vertical-align:middle;}
+form.inline label {position:relative;top:-0.25em;}
+form.inline input.checkbox, form.inline input.radio, form.inline input.button, form.inline button {margin:0.5em 0;}
+button, input.button {position:relative;top:0.25em;}
\ No newline at end of file
--- /dev/null
+Buttons
+
+* Gives you great looking CSS buttons, for both <a> and <button>.
+* Demo: particletree.com/features/rediscovering-the-button-element
+
+
+Credits
+----------------------------------------------------------------
+
+* Created by Kevin Hale [particletree.com]
+* Adapted for Blueprint by Olav Bjorkoy [bjorkoy.com]
+
+
+Usage
+----------------------------------------------------------------
+
+1) Add this plugin to lib/settings.yml.
+ See compress.rb for instructions.
+
+2) Use the following HTML code to place the buttons on your site:
+
+ <button type="submit" class="button positive">
+ <img src="css/blueprint/plugins/buttons/icons/tick.png" alt=""/> Save
+ </button>
+
+ <a class="button" href="/password/reset/">
+ <img src="css/blueprint/plugins/buttons/icons/key.png" alt=""/> Change Password
+ </a>
+
+ <a href="#" class="button negative">
+ <img src="css/blueprint/plugins/buttons/icons/cross.png" alt=""/> Cancel
+ </a>
--- /dev/null
+/* --------------------------------------------------------------
+
+ buttons.css
+ * Gives you some great CSS-only buttons.
+
+ Created by Kevin Hale [particletree.com]
+ * particletree.com/features/rediscovering-the-button-element
+
+ See Readme.txt in this folder for instructions.
+
+-------------------------------------------------------------- */
+
+a.button, button {
+ display:block;
+ float:left;
+ margin: 0.7em 0.5em 0.7em 0;
+ padding:5px 10px 5px 7px; /* Links */
+
+ border:1px solid #dedede;
+ border-top:1px solid #eee;
+ border-left:1px solid #eee;
+
+ background-color:#f5f5f5;
+ font-family:"Lucida Grande", Tahoma, Arial, Verdana, sans-serif;
+ font-size:100%;
+ line-height:130%;
+ text-decoration:none;
+ font-weight:bold;
+ color:#565656;
+ cursor:pointer;
+}
+button {
+ width:auto;
+ overflow:visible;
+ padding:4px 10px 3px 7px; /* IE6 */
+}
+button[type] {
+ padding:4px 10px 4px 7px; /* Firefox */
+ line-height:17px; /* Safari */
+}
+*:first-child+html button[type] {
+ padding:4px 10px 3px 7px; /* IE7 */
+}
+button img, a.button img{
+ margin:0 3px -3px 0 !important;
+ padding:0;
+ border:none;
+ width:16px;
+ height:16px;
+ float:none;
+}
+
+
+/* Button colors
+-------------------------------------------------------------- */
+
+/* Standard */
+button:hover, a.button:hover{
+ background-color:#dff4ff;
+ border:1px solid #c2e1ef;
+ color:#336699;
+}
+a.button:active{
+ background-color:#6299c5;
+ border:1px solid #6299c5;
+ color:#fff;
+}
+
+/* Positive */
+body .positive {
+ color:#529214;
+}
+a.positive:hover, button.positive:hover {
+ background-color:#E6EFC2;
+ border:1px solid #C6D880;
+ color:#529214;
+}
+a.positive:active {
+ background-color:#529214;
+ border:1px solid #529214;
+ color:#fff;
+}
+
+/* Negative */
+body .negative {
+ color:#d12f19;
+}
+a.negative:hover, button.negative:hover {
+ background-color:#fbe3e4;
+ border:1px solid #fbc2c4;
+ color:#d12f19;
+}
+a.negative:active {
+ background-color:#d12f19;
+ border:1px solid #d12f19;
+ color:#fff;
+}
--- /dev/null
+Fancy Type
+
+* Gives you classes to use if you'd like some
+ extra fancy typography.
+
+Credits and instructions are specified above each class
+in the fancy-type.css file in this directory.
+
+
+Usage
+----------------------------------------------------------------
+
+1) Add this plugin to lib/settings.yml.
+ See compress.rb for instructions.
--- /dev/null
+/* --------------------------------------------------------------
+
+ fancy-type.css
+ * Lots of pretty advanced classes for manipulating text.
+
+ See the Readme file in this folder for additional instructions.
+
+-------------------------------------------------------------- */
+
+/* Indentation instead of line shifts for sibling paragraphs. */
+ p + p { text-indent:2em; margin-top:-1.5em; }
+ form p + p { text-indent: 0; } /* Don't want this in forms. */
+
+
+/* For great looking type, use this code instead of asdf:
+ <span class="alt">asdf</span>
+ Best used on prepositions and ampersands. */
+
+.alt {
+ color: #666;
+ font-family: "Warnock Pro", "Goudy Old Style","Palatino","Book Antiqua", Georgia, serif;
+ font-style: italic;
+ font-weight: normal;
+}
+
+
+/* For great looking quote marks in titles, replace "asdf" with:
+ <span class="dquo">“</span>asdf”
+ (That is, when the title starts with a quote mark).
+ (You may have to change this value depending on your font size). */
+
+.dquo { margin-left: -.5em; }
+
+
+/* Reduced size type with incremental leading
+ (http://www.markboulton.co.uk/journal/comments/incremental_leading/)
+
+ This could be used for side notes. For smaller type, you don't necessarily want to
+ follow the 1.5x vertical rhythm -- the line-height is too much.
+
+ Using this class, it reduces your font size and line-height so that for
+ every four lines of normal sized type, there is five lines of the sidenote. eg:
+
+ New type size in em's:
+ 10px (wanted side note size) / 12px (existing base size) = 0.8333 (new type size in ems)
+
+ New line-height value:
+ 12px x 1.5 = 18px (old line-height)
+ 18px x 4 = 72px
+ 72px / 5 = 14.4px (new line height)
+ 14.4px / 10px = 1.44 (new line height in em's) */
+
+p.incr, .incr p {
+ font-size: 10px;
+ line-height: 1.44em;
+ margin-bottom: 1.5em;
+}
+
+
+/* Surround uppercase words and abbreviations with this class.
+ Based on work by Jørgen Arnor Gårdsø Lom [http://twistedintellect.com/] */
+
+.caps {
+ font-variant: small-caps;
+ letter-spacing: 1px;
+ text-transform: lowercase;
+ font-size:1.2em;
+ line-height:1%;
+ font-weight:bold;
+ padding:0 2px;
+}
--- /dev/null
+Link Icons
+* Icons for links based on protocol or file type.
+
+This is not supported in IE versions < 7.
+
+
+Credits
+----------------------------------------------------------------
+
+* Marc Morgan
+* Olav Bjorkoy [bjorkoy.com]
+
+
+Usage
+----------------------------------------------------------------
+
+1) Add this line to your HTML:
+ <link rel="stylesheet" href="css/blueprint/plugins/link-icons/screen.css" type="text/css" media="screen, projection">
--- /dev/null
+/* --------------------------------------------------------------
+
+ link-icons.css
+ * Icons for links based on protocol or file type.
+
+ See the Readme file in this folder for additional instructions.
+
+-------------------------------------------------------------- */
+
+/* Use this class if a link gets an icon when it shouldn't. */
+body a.noicon {
+ background:transparent none !important;
+ padding:0 !important;
+ margin:0 !important;
+}
+
+/* Make sure the icons are not cut */
+a[href^="http:"], a[href^="mailto:"], a[href^="http:"]:visited,
+a[href$=".pdf"], a[href$=".doc"], a[href$=".xls"], a[href$=".rss"],
+a[href$=".rdf"], a[href^="aim:"] {
+ padding:2px 22px 2px 0;
+ margin:-2px 0;
+ background-repeat: no-repeat;
+ background-position: right center;
+}
+
+/* External links */
+a[href^="http:"] { background-image: url(icons/external.png); }
+a[href^="mailto:"] { background-image: url(icons/email.png); }
+a[href^="http:"]:visited { background-image: url(icons/visited.png); }
+
+/* Files */
+a[href$=".pdf"] { background-image: url(icons/pdf.png); }
+a[href$=".doc"] { background-image: url(icons/doc.png); }
+a[href$=".xls"] { background-image: url(icons/xls.png); }
+
+/* Misc */
+a[href$=".rss"],
+a[href$=".rdf"] { background-image: url(icons/feed.png); }
+a[href^="aim:"] { background-image: url(icons/im.png); }
--- /dev/null
+RTL
+* Mirrors Blueprint, so it can be used with Right-to-Left languages.
+
+By Ran Yaniv Hartstein, ranh.co.il
+
+Usage
+----------------------------------------------------------------
+
+1) Add this line to your HTML:
+ <link rel="stylesheet" href="css/blueprint/plugins/rtl/screen.css" type="text/css" media="screen, projection">
--- /dev/null
+/* --------------------------------------------------------------
+
+ rtl.css
+ * Mirrors Blueprint for left-to-right languages
+
+ By Ran Yaniv Hartstein [ranh.co.il]
+
+-------------------------------------------------------------- */
+
+body .container { direction: rtl; }
+body .column, body .span-1, body .span-2, body .span-3, body .span-4, body .span-5, body .span-6, body .span-7, body .span-8, body .span-9, body .span-10, body .span-11, body .span-12, body .span-13, body .span-14, body .span-15, body .span-16, body .span-17, body .span-18, body .span-19, body .span-20, body .span-21, body .span-22, body .span-23, body .span-24 {
+ float: right;
+ margin-right: 0;
+ margin-left: 10px;
+ text-align:right;
+}
+
+body div.last { margin-left: 0; }
+body table .last { padding-left: 0; }
+
+body .append-1 { padding-right: 0; padding-left: 40px; }
+body .append-2 { padding-right: 0; padding-left: 80px; }
+body .append-3 { padding-right: 0; padding-left: 120px; }
+body .append-4 { padding-right: 0; padding-left: 160px; }
+body .append-5 { padding-right: 0; padding-left: 200px; }
+body .append-6 { padding-right: 0; padding-left: 240px; }
+body .append-7 { padding-right: 0; padding-left: 280px; }
+body .append-8 { padding-right: 0; padding-left: 320px; }
+body .append-9 { padding-right: 0; padding-left: 360px; }
+body .append-10 { padding-right: 0; padding-left: 400px; }
+body .append-11 { padding-right: 0; padding-left: 440px; }
+body .append-12 { padding-right: 0; padding-left: 480px; }
+body .append-13 { padding-right: 0; padding-left: 520px; }
+body .append-14 { padding-right: 0; padding-left: 560px; }
+body .append-15 { padding-right: 0; padding-left: 600px; }
+body .append-16 { padding-right: 0; padding-left: 640px; }
+body .append-17 { padding-right: 0; padding-left: 680px; }
+body .append-18 { padding-right: 0; padding-left: 720px; }
+body .append-19 { padding-right: 0; padding-left: 760px; }
+body .append-20 { padding-right: 0; padding-left: 800px; }
+body .append-21 { padding-right: 0; padding-left: 840px; }
+body .append-22 { padding-right: 0; padding-left: 880px; }
+body .append-23 { padding-right: 0; padding-left: 920px; }
+
+body .prepend-1 { padding-left: 0; padding-right: 40px; }
+body .prepend-2 { padding-left: 0; padding-right: 80px; }
+body .prepend-3 { padding-left: 0; padding-right: 120px; }
+body .prepend-4 { padding-left: 0; padding-right: 160px; }
+body .prepend-5 { padding-left: 0; padding-right: 200px; }
+body .prepend-6 { padding-left: 0; padding-right: 240px; }
+body .prepend-7 { padding-left: 0; padding-right: 280px; }
+body .prepend-8 { padding-left: 0; padding-right: 320px; }
+body .prepend-9 { padding-left: 0; padding-right: 360px; }
+body .prepend-10 { padding-left: 0; padding-right: 400px; }
+body .prepend-11 { padding-left: 0; padding-right: 440px; }
+body .prepend-12 { padding-left: 0; padding-right: 480px; }
+body .prepend-13 { padding-left: 0; padding-right: 520px; }
+body .prepend-14 { padding-left: 0; padding-right: 560px; }
+body .prepend-15 { padding-left: 0; padding-right: 600px; }
+body .prepend-16 { padding-left: 0; padding-right: 640px; }
+body .prepend-17 { padding-left: 0; padding-right: 680px; }
+body .prepend-18 { padding-left: 0; padding-right: 720px; }
+body .prepend-19 { padding-left: 0; padding-right: 760px; }
+body .prepend-20 { padding-left: 0; padding-right: 800px; }
+body .prepend-21 { padding-left: 0; padding-right: 840px; }
+body .prepend-22 { padding-left: 0; padding-right: 880px; }
+body .prepend-23 { padding-left: 0; padding-right: 920px; }
+
+body .border {
+ padding-right: 0;
+ padding-left: 4px;
+ margin-right: 0;
+ margin-left: 5px;
+ border-right: none;
+ border-left: 1px solid #eee;
+}
+
+body .colborder {
+ padding-right: 0;
+ padding-left: 24px;
+ margin-right: 0;
+ margin-left: 25px;
+ border-right: none;
+ border-left: 1px solid #eee;
+}
+
+body .pull-1 { margin-left: 0; margin-right: -40px; }
+body .pull-2 { margin-left: 0; margin-right: -80px; }
+body .pull-3 { margin-left: 0; margin-right: -120px; }
+body .pull-4 { margin-left: 0; margin-right: -160px; }
+
+body .push-0 { margin: 0 18px 0 0; }
+body .push-1 { margin: 0 18px 0 -40px; }
+body .push-2 { margin: 0 18px 0 -80px; }
+body .push-3 { margin: 0 18px 0 -120px; }
+body .push-4 { margin: 0 18px 0 -160px; }
+body .push-0, body .push-1, body .push-2,
+body .push-3, body .push-4 { float: left; }
+
+
+/* Typography with RTL support */
+body h1,body h2,body h3,
+body h4,body h5,body h6 { font-family: Arial, sans-serif; }
+html body { font-family: Arial, sans-serif; }
+body pre,body code,body tt { font-family: monospace; }
+
+/* Mirror floats and margins on typographic elements */
+body p img { float: right; margin: 1.5em 0 1.5em 1.5em; }
+body dd, body ul, body ol { margin-left: 0; margin-right: 1.5em;}
+body td, body th { text-align:right; }
--- /dev/null
+/* -----------------------------------------------------------------------
+
+
+ Blueprint CSS Framework 0.9
+ http://blueprintcss.org
+
+ * Copyright (c) 2007-Present. See LICENSE for more info.
+ * See README for instructions on how to use Blueprint.
+ * For credits and origins, see AUTHORS.
+ * This is a compressed file. See the sources in the 'src' directory.
+
+----------------------------------------------------------------------- */
+
+/* print.css */
+body {line-height:1.5;font-family:"Helvetica Neue", Arial, Helvetica, sans-serif;color:#000;background:none;font-size:10pt;}
+.container {background:none;}
+hr {background:#ccc;color:#ccc;width:100%;height:2px;margin:2em 0;padding:0;border:none;}
+hr.space {background:#fff;color:#fff;visibility:hidden;}
+h1, h2, h3, h4, h5, h6 {font-family:"Helvetica Neue", Arial, "Lucida Grande", sans-serif;}
+code {font:.9em "Courier New", Monaco, Courier, monospace;}
+a img {border:none;}
+p img.top {margin-top:0;}
+blockquote {margin:1.5em;padding:1em;font-style:italic;font-size:.9em;}
+.small {font-size:.9em;}
+.large {font-size:1.1em;}
+.quiet {color:#999;}
+.hide {display:none;}
+a:link, a:visited {background:transparent;font-weight:700;text-decoration:underline;}
+a:link:after, a:visited:after {content:" (" attr(href) ")";font-size:90%;}
\ No newline at end of file
--- /dev/null
+/* -----------------------------------------------------------------------
+
+
+ Blueprint CSS Framework 0.9
+ http://blueprintcss.org
+
+ * Copyright (c) 2007-Present. See LICENSE for more info.
+ * See README for instructions on how to use Blueprint.
+ * For credits and origins, see AUTHORS.
+ * This is a compressed file. See the sources in the 'src' directory.
+
+----------------------------------------------------------------------- */
+
+/* reset.css */
+html, body, div, span, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, acronym, address, code, del, dfn, em, img, q, dl, dt, dd, ol, ul, li, fieldset, form, label, legend, table, caption, tbody, tfoot, thead, tr, th, td, article, aside, dialog, figure, footer, header, hgroup, nav, section {margin:0;padding:0;border:0;font-weight:inherit;font-style:inherit;font-size:100%;font-family:inherit;vertical-align:baseline;}
+article, aside, dialog, figure, footer, header, hgroup, nav, section {display:block;}
+body {line-height:1.5;}
+table {border-collapse:separate;border-spacing:0;}
+caption, th, td {text-align:left;font-weight:normal;}
+table, td, th {vertical-align:middle;}
+blockquote:before, blockquote:after, q:before, q:after {content:"";}
+blockquote, q {quotes:"" "";}
+a img {border:none;}
+
+/* typography.css */
+html {font-size:100.01%;}
+body {font-size:75%;color:#222;background:#fff;font-family:"Helvetica Neue", Arial, Helvetica, sans-serif;}
+h1, h2, h3, h4, h5, h6 {font-weight:normal;color:#111;}
+h1 {font-size:3em;line-height:1;margin-bottom:0.5em;}
+h2 {font-size:2em;margin-bottom:0.75em;}
+h3 {font-size:1.5em;line-height:1;margin-bottom:1em;}
+h4 {font-size:1.2em;line-height:1.25;margin-bottom:1.25em;}
+h5 {font-size:1em;font-weight:bold;margin-bottom:1.5em;}
+h6 {font-size:1em;font-weight:bold;}
+h1 img, h2 img, h3 img, h4 img, h5 img, h6 img {margin:0;}
+p {margin:0 0 1.5em;}
+p img.left {float:left;margin:1.5em 1.5em 1.5em 0;padding:0;}
+p img.right {float:right;margin:1.5em 0 1.5em 1.5em;}
+a:focus, a:hover {color:#000;}
+a {color:#009;text-decoration:underline;}
+blockquote {margin:1.5em;color:#666;font-style:italic;}
+strong {font-weight:bold;}
+em, dfn {font-style:italic;}
+dfn {font-weight:bold;}
+sup, sub {line-height:0;}
+abbr, acronym {border-bottom:1px dotted #666;}
+address {margin:0 0 1.5em;font-style:italic;}
+del {color:#666;}
+pre {margin:1.5em 0;white-space:pre;}
+pre, code, tt {font:1em 'andale mono', 'lucida console', monospace;line-height:1.5;}
+li ul, li ol {margin:0;}
+ul, ol {margin:0 1.5em 1.5em 0;padding-left:3.333em;}
+ul {list-style-type:disc;}
+ol {list-style-type:decimal;}
+dl {margin:0 0 1.5em 0;}
+dl dt {font-weight:bold;}
+dd {margin-left:1.5em;}
+table {margin-bottom:1.4em;width:100%;}
+th {font-weight:bold;}
+thead th {background:#c3d9ff;}
+th, td, caption {padding:4px 10px 4px 5px;}
+tr.even td {background:#e5ecf9;}
+tfoot {font-style:italic;}
+caption {background:#eee;}
+.small {font-size:.8em;margin-bottom:1.875em;line-height:1.875em;}
+.large {font-size:1.2em;line-height:2.5em;margin-bottom:1.25em;}
+.hide {display:none;}
+.quiet {color:#666;}
+.loud {color:#000;}
+.highlight {background:#ff0;}
+.added {background:#060;color:#fff;}
+.removed {background:#900;color:#fff;}
+.first {margin-left:0;padding-left:0;}
+.last {margin-right:0;padding-right:0;}
+.top {margin-top:0;padding-top:0;}
+.bottom {margin-bottom:0;padding-bottom:0;}
+
+/* forms.css */
+label {font-weight:bold;}
+fieldset {padding:1.4em;margin:0 0 1.5em 0;border:1px solid #ccc;}
+legend {font-weight:bold;font-size:1.2em;}
+input[type=text], input[type=password], input.text, input.title, textarea, select {background-color:#fff;border:1px solid #bbb;}
+input[type=text]:focus, input[type=password]:focus, input.text:focus, input.title:focus, textarea:focus, select:focus {border-color:#666;}
+input[type=text], input[type=password], input.text, input.title, textarea, select {margin:0.5em 0;}
+input.text, input.title {width:300px;padding:5px;}
+input.title {font-size:1.5em;}
+textarea {width:390px;height:250px;padding:5px;}
+input[type=checkbox], input[type=radio], input.checkbox, input.radio {position:relative;top:.25em;}
+form.inline {line-height:3;}
+form.inline p {margin-bottom:0;}
+.error, .notice, .success {padding:.8em;margin-bottom:1em;border:2px solid #ddd;}
+.error {background:#FBE3E4;color:#8a1f11;border-color:#FBC2C4;}
+.notice {background:#FFF6BF;color:#514721;border-color:#FFD324;}
+.success {background:#E6EFC2;color:#264409;border-color:#C6D880;}
+.error a {color:#8a1f11;}
+.notice a {color:#514721;}
+.success a {color:#264409;}
+
+/* grid.css */
+.container {width:950px;margin:0 auto;}
+.showgrid {background:url(src/grid.png);}
+.column, .span-1, .span-2, .span-3, .span-4, .span-5, .span-6, .span-7, .span-8, .span-9, .span-10, .span-11, .span-12, .span-13, .span-14, .span-15, .span-16, .span-17, .span-18, .span-19, .span-20, .span-21, .span-22, .span-23, .span-24 {float:left;margin-right:10px;}
+.last {margin-right:0;}
+.span-1 {width:30px;}
+.span-2 {width:70px;}
+.span-3 {width:110px;}
+.span-4 {width:150px;}
+.span-5 {width:190px;}
+.span-6 {width:230px;}
+.span-7 {width:270px;}
+.span-8 {width:310px;}
+.span-9 {width:350px;}
+.span-10 {width:390px;}
+.span-11 {width:430px;}
+.span-12 {width:470px;}
+.span-13 {width:510px;}
+.span-14 {width:550px;}
+.span-15 {width:590px;}
+.span-16 {width:630px;}
+.span-17 {width:670px;}
+.span-18 {width:710px;}
+.span-19 {width:750px;}
+.span-20 {width:790px;}
+.span-21 {width:830px;}
+.span-22 {width:870px;}
+.span-23 {width:910px;}
+.span-24 {width:950px;margin-right:0;}
+input.span-1, textarea.span-1, input.span-2, textarea.span-2, input.span-3, textarea.span-3, input.span-4, textarea.span-4, input.span-5, textarea.span-5, input.span-6, textarea.span-6, input.span-7, textarea.span-7, input.span-8, textarea.span-8, input.span-9, textarea.span-9, input.span-10, textarea.span-10, input.span-11, textarea.span-11, input.span-12, textarea.span-12, input.span-13, textarea.span-13, input.span-14, textarea.span-14, input.span-15, textarea.span-15, input.span-16, textarea.span-16, input.span-17, textarea.span-17, input.span-18, textarea.span-18, input.span-19, textarea.span-19, input.span-20, textarea.span-20, input.span-21, textarea.span-21, input.span-22, textarea.span-22, input.span-23, textarea.span-23, input.span-24, textarea.span-24 {border-left-width:1px;border-right-width:1px;padding-left:5px;padding-right:5px;}
+input.span-1, textarea.span-1 {width:18px;}
+input.span-2, textarea.span-2 {width:58px;}
+input.span-3, textarea.span-3 {width:98px;}
+input.span-4, textarea.span-4 {width:138px;}
+input.span-5, textarea.span-5 {width:178px;}
+input.span-6, textarea.span-6 {width:218px;}
+input.span-7, textarea.span-7 {width:258px;}
+input.span-8, textarea.span-8 {width:298px;}
+input.span-9, textarea.span-9 {width:338px;}
+input.span-10, textarea.span-10 {width:378px;}
+input.span-11, textarea.span-11 {width:418px;}
+input.span-12, textarea.span-12 {width:458px;}
+input.span-13, textarea.span-13 {width:498px;}
+input.span-14, textarea.span-14 {width:538px;}
+input.span-15, textarea.span-15 {width:578px;}
+input.span-16, textarea.span-16 {width:618px;}
+input.span-17, textarea.span-17 {width:658px;}
+input.span-18, textarea.span-18 {width:698px;}
+input.span-19, textarea.span-19 {width:738px;}
+input.span-20, textarea.span-20 {width:778px;}
+input.span-21, textarea.span-21 {width:818px;}
+input.span-22, textarea.span-22 {width:858px;}
+input.span-23, textarea.span-23 {width:898px;}
+input.span-24, textarea.span-24 {width:938px;}
+.append-1 {padding-right:40px;}
+.append-2 {padding-right:80px;}
+.append-3 {padding-right:120px;}
+.append-4 {padding-right:160px;}
+.append-5 {padding-right:200px;}
+.append-6 {padding-right:240px;}
+.append-7 {padding-right:280px;}
+.append-8 {padding-right:320px;}
+.append-9 {padding-right:360px;}
+.append-10 {padding-right:400px;}
+.append-11 {padding-right:440px;}
+.append-12 {padding-right:480px;}
+.append-13 {padding-right:520px;}
+.append-14 {padding-right:560px;}
+.append-15 {padding-right:600px;}
+.append-16 {padding-right:640px;}
+.append-17 {padding-right:680px;}
+.append-18 {padding-right:720px;}
+.append-19 {padding-right:760px;}
+.append-20 {padding-right:800px;}
+.append-21 {padding-right:840px;}
+.append-22 {padding-right:880px;}
+.append-23 {padding-right:920px;}
+.prepend-1 {padding-left:40px;}
+.prepend-2 {padding-left:80px;}
+.prepend-3 {padding-left:120px;}
+.prepend-4 {padding-left:160px;}
+.prepend-5 {padding-left:200px;}
+.prepend-6 {padding-left:240px;}
+.prepend-7 {padding-left:280px;}
+.prepend-8 {padding-left:320px;}
+.prepend-9 {padding-left:360px;}
+.prepend-10 {padding-left:400px;}
+.prepend-11 {padding-left:440px;}
+.prepend-12 {padding-left:480px;}
+.prepend-13 {padding-left:520px;}
+.prepend-14 {padding-left:560px;}
+.prepend-15 {padding-left:600px;}
+.prepend-16 {padding-left:640px;}
+.prepend-17 {padding-left:680px;}
+.prepend-18 {padding-left:720px;}
+.prepend-19 {padding-left:760px;}
+.prepend-20 {padding-left:800px;}
+.prepend-21 {padding-left:840px;}
+.prepend-22 {padding-left:880px;}
+.prepend-23 {padding-left:920px;}
+.border {padding-right:4px;margin-right:5px;border-right:1px solid #eee;}
+.colborder {padding-right:24px;margin-right:25px;border-right:1px solid #eee;}
+.pull-1 {margin-left:-40px;}
+.pull-2 {margin-left:-80px;}
+.pull-3 {margin-left:-120px;}
+.pull-4 {margin-left:-160px;}
+.pull-5 {margin-left:-200px;}
+.pull-6 {margin-left:-240px;}
+.pull-7 {margin-left:-280px;}
+.pull-8 {margin-left:-320px;}
+.pull-9 {margin-left:-360px;}
+.pull-10 {margin-left:-400px;}
+.pull-11 {margin-left:-440px;}
+.pull-12 {margin-left:-480px;}
+.pull-13 {margin-left:-520px;}
+.pull-14 {margin-left:-560px;}
+.pull-15 {margin-left:-600px;}
+.pull-16 {margin-left:-640px;}
+.pull-17 {margin-left:-680px;}
+.pull-18 {margin-left:-720px;}
+.pull-19 {margin-left:-760px;}
+.pull-20 {margin-left:-800px;}
+.pull-21 {margin-left:-840px;}
+.pull-22 {margin-left:-880px;}
+.pull-23 {margin-left:-920px;}
+.pull-24 {margin-left:-960px;}
+.pull-1, .pull-2, .pull-3, .pull-4, .pull-5, .pull-6, .pull-7, .pull-8, .pull-9, .pull-10, .pull-11, .pull-12, .pull-13, .pull-14, .pull-15, .pull-16, .pull-17, .pull-18, .pull-19, .pull-20, .pull-21, .pull-22, .pull-23, .pull-24 {float:left;position:relative;}
+.push-1 {margin:0 -40px 1.5em 40px;}
+.push-2 {margin:0 -80px 1.5em 80px;}
+.push-3 {margin:0 -120px 1.5em 120px;}
+.push-4 {margin:0 -160px 1.5em 160px;}
+.push-5 {margin:0 -200px 1.5em 200px;}
+.push-6 {margin:0 -240px 1.5em 240px;}
+.push-7 {margin:0 -280px 1.5em 280px;}
+.push-8 {margin:0 -320px 1.5em 320px;}
+.push-9 {margin:0 -360px 1.5em 360px;}
+.push-10 {margin:0 -400px 1.5em 400px;}
+.push-11 {margin:0 -440px 1.5em 440px;}
+.push-12 {margin:0 -480px 1.5em 480px;}
+.push-13 {margin:0 -520px 1.5em 520px;}
+.push-14 {margin:0 -560px 1.5em 560px;}
+.push-15 {margin:0 -600px 1.5em 600px;}
+.push-16 {margin:0 -640px 1.5em 640px;}
+.push-17 {margin:0 -680px 1.5em 680px;}
+.push-18 {margin:0 -720px 1.5em 720px;}
+.push-19 {margin:0 -760px 1.5em 760px;}
+.push-20 {margin:0 -800px 1.5em 800px;}
+.push-21 {margin:0 -840px 1.5em 840px;}
+.push-22 {margin:0 -880px 1.5em 880px;}
+.push-23 {margin:0 -920px 1.5em 920px;}
+.push-24 {margin:0 -960px 1.5em 960px;}
+.push-1, .push-2, .push-3, .push-4, .push-5, .push-6, .push-7, .push-8, .push-9, .push-10, .push-11, .push-12, .push-13, .push-14, .push-15, .push-16, .push-17, .push-18, .push-19, .push-20, .push-21, .push-22, .push-23, .push-24 {float:right;position:relative;}
+.prepend-top {margin-top:1.5em;}
+.append-bottom {margin-bottom:1.5em;}
+.box {padding:1.5em;margin-bottom:1.5em;background:#E5ECF9;}
+hr {background:#ddd;color:#ddd;clear:both;float:none;width:100%;height:.1em;margin:0 0 1.45em;border:none;}
+hr.space {background:#fff;color:#fff;visibility:hidden;}
+.clearfix:after, .container:after {content:"\0020";display:block;height:0;clear:both;visibility:hidden;overflow:hidden;}
+.clearfix, .container {display:block;}
+.clear {clear:both;}
--- /dev/null
+/* --------------------------------------------------------------
+
+ forms.css
+ * Sets up some default styling for forms
+ * Gives you classes to enhance your forms
+
+ Usage:
+ * For text fields, use class .title or .text
+ * For inline forms, use .inline (even when using columns)
+
+-------------------------------------------------------------- */
+
+label { font-weight: bold; }
+fieldset { padding:1.4em; margin: 0 0 1.5em 0; border: 1px solid #ccc; }
+legend { font-weight: bold; font-size:1.2em; }
+
+
+/* Form fields
+-------------------------------------------------------------- */
+
+input[type=text], input[type=password],
+input.text, input.title,
+textarea, select {
+ background-color:#fff;
+ border:1px solid #bbb;
+}
+input[type=text]:focus, input[type=password]:focus,
+input.text:focus, input.title:focus,
+textarea:focus, select:focus {
+ border-color:#666;
+}
+
+input[type=text], input[type=password],
+input.text, input.title,
+textarea, select {
+ margin:0.5em 0;
+}
+
+input.text,
+input.title { width: 300px; padding:5px; }
+input.title { font-size:1.5em; }
+textarea { width: 390px; height: 250px; padding:5px; }
+
+input[type=checkbox], input[type=radio],
+input.checkbox, input.radio {
+ position:relative; top:.25em;
+}
+
+form.inline { line-height:3; }
+form.inline p { margin-bottom:0; }
+
+
+/* Success, notice and error boxes
+-------------------------------------------------------------- */
+
+.error,
+.notice,
+.success { padding: .8em; margin-bottom: 1em; border: 2px solid #ddd; }
+
+.error { background: #FBE3E4; color: #8a1f11; border-color: #FBC2C4; }
+.notice { background: #FFF6BF; color: #514721; border-color: #FFD324; }
+.success { background: #E6EFC2; color: #264409; border-color: #C6D880; }
+.error a { color: #8a1f11; }
+.notice a { color: #514721; }
+.success a { color: #264409; }
--- /dev/null
+/* --------------------------------------------------------------
+
+ grid.css
+ * Sets up an easy-to-use grid of 24 columns.
+
+ By default, the grid is 950px wide, with 24 columns
+ spanning 30px, and a 10px margin between columns.
+
+ If you need fewer or more columns, namespaces or semantic
+ element names, use the compressor script (lib/compress.rb)
+
+-------------------------------------------------------------- */
+
+/* A container should group all your columns. */
+.container {
+ width: 950px;
+ margin: 0 auto;
+}
+
+/* Use this class on any .span / container to see the grid. */
+.showgrid {
+ background: url(src/grid.png);
+}
+
+
+/* Columns
+-------------------------------------------------------------- */
+
+/* Sets up basic grid floating and margin. */
+.column, .span-1, .span-2, .span-3, .span-4, .span-5, .span-6, .span-7, .span-8, .span-9, .span-10, .span-11, .span-12, .span-13, .span-14, .span-15, .span-16, .span-17, .span-18, .span-19, .span-20, .span-21, .span-22, .span-23, .span-24 {
+ float: left;
+ margin-right: 10px;
+}
+
+/* The last column in a row needs this class. */
+.last { margin-right: 0; }
+
+/* Use these classes to set the width of a column. */
+.span-1 {width: 30px;}
+
+.span-2 {width: 70px;}
+.span-3 {width: 110px;}
+.span-4 {width: 150px;}
+.span-5 {width: 190px;}
+.span-6 {width: 230px;}
+.span-7 {width: 270px;}
+.span-8 {width: 310px;}
+.span-9 {width: 350px;}
+.span-10 {width: 390px;}
+.span-11 {width: 430px;}
+.span-12 {width: 470px;}
+.span-13 {width: 510px;}
+.span-14 {width: 550px;}
+.span-15 {width: 590px;}
+.span-16 {width: 630px;}
+.span-17 {width: 670px;}
+.span-18 {width: 710px;}
+.span-19 {width: 750px;}
+.span-20 {width: 790px;}
+.span-21 {width: 830px;}
+.span-22 {width: 870px;}
+.span-23 {width: 910px;}
+.span-24 {width:950px; margin-right:0;}
+
+/* Use these classes to set the width of an input. */
+input.span-1, textarea.span-1, input.span-2, textarea.span-2, input.span-3, textarea.span-3, input.span-4, textarea.span-4, input.span-5, textarea.span-5, input.span-6, textarea.span-6, input.span-7, textarea.span-7, input.span-8, textarea.span-8, input.span-9, textarea.span-9, input.span-10, textarea.span-10, input.span-11, textarea.span-11, input.span-12, textarea.span-12, input.span-13, textarea.span-13, input.span-14, textarea.span-14, input.span-15, textarea.span-15, input.span-16, textarea.span-16, input.span-17, textarea.span-17, input.span-18, textarea.span-18, input.span-19, textarea.span-19, input.span-20, textarea.span-20, input.span-21, textarea.span-21, input.span-22, textarea.span-22, input.span-23, textarea.span-23, input.span-24, textarea.span-24 {
+ border-left-width: 1px;
+ border-right-width: 1px;
+ padding-left: 5px;
+ padding-right: 5px;
+}
+
+input.span-1, textarea.span-1 { width: 18px; }
+input.span-2, textarea.span-2 { width: 58px; }
+input.span-3, textarea.span-3 { width: 98px; }
+input.span-4, textarea.span-4 { width: 138px; }
+input.span-5, textarea.span-5 { width: 178px; }
+input.span-6, textarea.span-6 { width: 218px; }
+input.span-7, textarea.span-7 { width: 258px; }
+input.span-8, textarea.span-8 { width: 298px; }
+input.span-9, textarea.span-9 { width: 338px; }
+input.span-10, textarea.span-10 { width: 378px; }
+input.span-11, textarea.span-11 { width: 418px; }
+input.span-12, textarea.span-12 { width: 458px; }
+input.span-13, textarea.span-13 { width: 498px; }
+input.span-14, textarea.span-14 { width: 538px; }
+input.span-15, textarea.span-15 { width: 578px; }
+input.span-16, textarea.span-16 { width: 618px; }
+input.span-17, textarea.span-17 { width: 658px; }
+input.span-18, textarea.span-18 { width: 698px; }
+input.span-19, textarea.span-19 { width: 738px; }
+input.span-20, textarea.span-20 { width: 778px; }
+input.span-21, textarea.span-21 { width: 818px; }
+input.span-22, textarea.span-22 { width: 858px; }
+input.span-23, textarea.span-23 { width: 898px; }
+input.span-24, textarea.span-24 { width: 938px; }
+
+/* Add these to a column to append empty cols. */
+
+.append-1 { padding-right: 40px;}
+.append-2 { padding-right: 80px;}
+.append-3 { padding-right: 120px;}
+.append-4 { padding-right: 160px;}
+.append-5 { padding-right: 200px;}
+.append-6 { padding-right: 240px;}
+.append-7 { padding-right: 280px;}
+.append-8 { padding-right: 320px;}
+.append-9 { padding-right: 360px;}
+.append-10 { padding-right: 400px;}
+.append-11 { padding-right: 440px;}
+.append-12 { padding-right: 480px;}
+.append-13 { padding-right: 520px;}
+.append-14 { padding-right: 560px;}
+.append-15 { padding-right: 600px;}
+.append-16 { padding-right: 640px;}
+.append-17 { padding-right: 680px;}
+.append-18 { padding-right: 720px;}
+.append-19 { padding-right: 760px;}
+.append-20 { padding-right: 800px;}
+.append-21 { padding-right: 840px;}
+.append-22 { padding-right: 880px;}
+.append-23 { padding-right: 920px;}
+
+/* Add these to a column to prepend empty cols. */
+
+.prepend-1 { padding-left: 40px;}
+.prepend-2 { padding-left: 80px;}
+.prepend-3 { padding-left: 120px;}
+.prepend-4 { padding-left: 160px;}
+.prepend-5 { padding-left: 200px;}
+.prepend-6 { padding-left: 240px;}
+.prepend-7 { padding-left: 280px;}
+.prepend-8 { padding-left: 320px;}
+.prepend-9 { padding-left: 360px;}
+.prepend-10 { padding-left: 400px;}
+.prepend-11 { padding-left: 440px;}
+.prepend-12 { padding-left: 480px;}
+.prepend-13 { padding-left: 520px;}
+.prepend-14 { padding-left: 560px;}
+.prepend-15 { padding-left: 600px;}
+.prepend-16 { padding-left: 640px;}
+.prepend-17 { padding-left: 680px;}
+.prepend-18 { padding-left: 720px;}
+.prepend-19 { padding-left: 760px;}
+.prepend-20 { padding-left: 800px;}
+.prepend-21 { padding-left: 840px;}
+.prepend-22 { padding-left: 880px;}
+.prepend-23 { padding-left: 920px;}
+
+
+/* Border on right hand side of a column. */
+.border {
+ padding-right: 4px;
+ margin-right: 5px;
+ border-right: 1px solid #eee;
+}
+
+/* Border with more whitespace, spans one column. */
+.colborder {
+ padding-right: 24px;
+ margin-right: 25px;
+ border-right: 1px solid #eee;
+}
+
+
+/* Use these classes on an element to push it into the
+next column, or to pull it into the previous column. */
+
+
+.pull-1 { margin-left: -40px; }
+.pull-2 { margin-left: -80px; }
+.pull-3 { margin-left: -120px; }
+.pull-4 { margin-left: -160px; }
+.pull-5 { margin-left: -200px; }
+.pull-6 { margin-left: -240px; }
+.pull-7 { margin-left: -280px; }
+.pull-8 { margin-left: -320px; }
+.pull-9 { margin-left: -360px; }
+.pull-10 { margin-left: -400px; }
+.pull-11 { margin-left: -440px; }
+.pull-12 { margin-left: -480px; }
+.pull-13 { margin-left: -520px; }
+.pull-14 { margin-left: -560px; }
+.pull-15 { margin-left: -600px; }
+.pull-16 { margin-left: -640px; }
+.pull-17 { margin-left: -680px; }
+.pull-18 { margin-left: -720px; }
+.pull-19 { margin-left: -760px; }
+.pull-20 { margin-left: -800px; }
+.pull-21 { margin-left: -840px; }
+.pull-22 { margin-left: -880px; }
+.pull-23 { margin-left: -920px; }
+.pull-24 { margin-left: -960px; }
+
+.pull-1, .pull-2, .pull-3, .pull-4, .pull-5, .pull-6, .pull-7, .pull-8, .pull-9, .pull-10, .pull-11, .pull-12, .pull-13, .pull-14, .pull-15, .pull-16, .pull-17, .pull-18, .pull-19, .pull-20, .pull-21, .pull-22, .pull-23, .pull-24 {float: left; position:relative;}
+
+
+.push-1 { margin: 0 -40px 1.5em 40px; }
+.push-2 { margin: 0 -80px 1.5em 80px; }
+.push-3 { margin: 0 -120px 1.5em 120px; }
+.push-4 { margin: 0 -160px 1.5em 160px; }
+.push-5 { margin: 0 -200px 1.5em 200px; }
+.push-6 { margin: 0 -240px 1.5em 240px; }
+.push-7 { margin: 0 -280px 1.5em 280px; }
+.push-8 { margin: 0 -320px 1.5em 320px; }
+.push-9 { margin: 0 -360px 1.5em 360px; }
+.push-10 { margin: 0 -400px 1.5em 400px; }
+.push-11 { margin: 0 -440px 1.5em 440px; }
+.push-12 { margin: 0 -480px 1.5em 480px; }
+.push-13 { margin: 0 -520px 1.5em 520px; }
+.push-14 { margin: 0 -560px 1.5em 560px; }
+.push-15 { margin: 0 -600px 1.5em 600px; }
+.push-16 { margin: 0 -640px 1.5em 640px; }
+.push-17 { margin: 0 -680px 1.5em 680px; }
+.push-18 { margin: 0 -720px 1.5em 720px; }
+.push-19 { margin: 0 -760px 1.5em 760px; }
+.push-20 { margin: 0 -800px 1.5em 800px; }
+.push-21 { margin: 0 -840px 1.5em 840px; }
+.push-22 { margin: 0 -880px 1.5em 880px; }
+.push-23 { margin: 0 -920px 1.5em 920px; }
+.push-24 { margin: 0 -960px 1.5em 960px; }
+
+.push-1, .push-2, .push-3, .push-4, .push-5, .push-6, .push-7, .push-8, .push-9, .push-10, .push-11, .push-12, .push-13, .push-14, .push-15, .push-16, .push-17, .push-18, .push-19, .push-20, .push-21, .push-22, .push-23, .push-24 {float: right; position:relative;}
+
+
+/* Misc classes and elements
+-------------------------------------------------------------- */
+
+/* In case you need to add a gutter above/below an element */
+.prepend-top {
+ margin-top:1.5em;
+}
+.append-bottom {
+ margin-bottom:1.5em;
+}
+
+/* Use a .box to create a padded box inside a column. */
+.box {
+ padding: 1.5em;
+ margin-bottom: 1.5em;
+ background: #E5ECF9;
+}
+
+/* Use this to create a horizontal ruler across a column. */
+hr {
+ background: #ddd;
+ color: #ddd;
+ clear: both;
+ float: none;
+ width: 100%;
+ height: .1em;
+ margin: 0 0 1.45em;
+ border: none;
+}
+
+hr.space {
+ background: #fff;
+ color: #fff;
+ visibility: hidden;
+}
+
+
+/* Clearing floats without extra markup
+ Based on How To Clear Floats Without Structural Markup by PiE
+ [http://www.positioniseverything.net/easyclearing.html] */
+
+.clearfix:after, .container:after {
+ content: "\0020";
+ display: block;
+ height: 0;
+ clear: both;
+ visibility: hidden;
+ overflow:hidden;
+}
+.clearfix, .container {display: block;}
+
+/* Regular clearing
+ apply to column that should drop below previous ones. */
+
+.clear { clear:both; }
--- /dev/null
+/* --------------------------------------------------------------
+
+ ie.css
+
+ Contains every hack for Internet Explorer,
+ so that our core files stay sweet and nimble.
+
+-------------------------------------------------------------- */
+
+/* Make sure the layout is centered in IE5 */
+body { text-align: center; }
+.container { text-align: left; }
+
+/* Fixes IE margin bugs */
+* html .column, * html .span-1, * html .span-2,
+* html .span-3, * html .span-4, * html .span-5,
+* html .span-6, * html .span-7, * html .span-8,
+* html .span-9, * html .span-10, * html .span-11,
+* html .span-12, * html .span-13, * html .span-14,
+* html .span-15, * html .span-16, * html .span-17,
+* html .span-18, * html .span-19, * html .span-20,
+* html .span-21, * html .span-22, * html .span-23,
+* html .span-24 { display:inline; overflow-x: hidden; }
+
+
+/* Elements
+-------------------------------------------------------------- */
+
+/* Fixes incorrect styling of legend in IE6. */
+* html legend { margin:0px -8px 16px 0; padding:0; }
+
+/* Fixes wrong line-height on sup/sub in IE. */
+sup { vertical-align:text-top; }
+sub { vertical-align:text-bottom; }
+
+/* Fixes IE7 missing wrapping of code elements. */
+html>body p code { *white-space: normal; }
+
+/* IE 6&7 has problems with setting proper <hr> margins. */
+hr { margin:-8px auto 11px; }
+
+/* Explicitly set interpolation, allowing dynamically resized images to not look horrible */
+img { -ms-interpolation-mode:bicubic; }
+
+/* Clearing
+-------------------------------------------------------------- */
+
+/* Makes clearfix actually work in IE */
+.clearfix, .container { display:inline-block; }
+* html .clearfix,
+* html .container { height:1%; }
+
+
+/* Forms
+-------------------------------------------------------------- */
+
+/* Fixes padding on fieldset */
+fieldset { padding-top:0; }
+
+/* Makes classic textareas in IE 6 resemble other browsers */
+textarea { overflow:auto; }
+
+/* Fixes rule that IE 6 ignores */
+input.text, input.title, textarea { background-color:#fff; border:1px solid #bbb; }
+input.text:focus, input.title:focus { border-color:#666; }
+input.text, input.title, textarea, select { margin:0.5em 0; }
+input.checkbox, input.radio { position:relative; top:.25em; }
+
+/* Fixes alignment of inline form elements */
+form.inline div, form.inline p { vertical-align:middle; }
+form.inline label { position:relative;top:-0.25em; }
+form.inline input.checkbox, form.inline input.radio,
+form.inline input.button, form.inline button {
+ margin:0.5em 0;
+}
+button, input.button { position:relative;top:0.25em; }
--- /dev/null
+/* --------------------------------------------------------------
+
+ print.css
+ * Gives you some sensible styles for printing pages.
+ * See Readme file in this directory for further instructions.
+
+ Some additions you'll want to make, customized to your markup:
+ #header, #footer, #navigation { display:none; }
+
+-------------------------------------------------------------- */
+
+body {
+ line-height: 1.5;
+ font-family: "Helvetica Neue", Arial, Helvetica, sans-serif;
+ color:#000;
+ background: none;
+ font-size: 10pt;
+}
+
+
+/* Layout
+-------------------------------------------------------------- */
+
+.container {
+ background: none;
+}
+
+hr {
+ background:#ccc;
+ color:#ccc;
+ width:100%;
+ height:2px;
+ margin:2em 0;
+ padding:0;
+ border:none;
+}
+hr.space {
+ background: #fff;
+ color: #fff;
+ visibility: hidden;
+}
+
+
+/* Text
+-------------------------------------------------------------- */
+
+h1,h2,h3,h4,h5,h6 { font-family: "Helvetica Neue", Arial, "Lucida Grande", sans-serif; }
+code { font:.9em "Courier New", Monaco, Courier, monospace; }
+
+a img { border:none; }
+p img.top { margin-top: 0; }
+
+blockquote {
+ margin:1.5em;
+ padding:1em;
+ font-style:italic;
+ font-size:.9em;
+}
+
+.small { font-size: .9em; }
+.large { font-size: 1.1em; }
+.quiet { color: #999; }
+.hide { display:none; }
+
+
+/* Links
+-------------------------------------------------------------- */
+
+a:link, a:visited {
+ background: transparent;
+ font-weight:700;
+ text-decoration: underline;
+}
+
+a:link:after, a:visited:after {
+ content: " (" attr(href) ")";
+ font-size: 90%;
+}
+
+/* If you're having trouble printing relative links, uncomment and customize this:
+ (note: This is valid CSS3, but it still won't go through the W3C CSS Validator) */
+
+/* a[href^="/"]:after {
+ content: " (http://www.yourdomain.com" attr(href) ") ";
+} */
--- /dev/null
+/* --------------------------------------------------------------
+
+ reset.css
+ * Resets default browser CSS.
+
+-------------------------------------------------------------- */
+
+html, body, div, span, object, iframe,
+h1, h2, h3, h4, h5, h6, p, blockquote, pre,
+a, abbr, acronym, address, code,
+del, dfn, em, img, q, dl, dt, dd, ol, ul, li,
+fieldset, form, label, legend,
+table, caption, tbody, tfoot, thead, tr, th, td,
+article, aside, dialog, figure, footer, header,
+hgroup, nav, section {
+ margin: 0;
+ padding: 0;
+ border: 0;
+ font-weight: inherit;
+ font-style: inherit;
+ font-size: 100%;
+ font-family: inherit;
+ vertical-align: baseline;
+}
+
+article, aside, dialog, figure, footer, header,
+hgroup, nav, section {
+ display:block;
+}
+
+body {
+ line-height: 1.5;
+}
+
+/* Tables still need 'cellspacing="0"' in the markup. */
+table { border-collapse: separate; border-spacing: 0; }
+caption, th, td { text-align: left; font-weight: normal; }
+table, td, th { vertical-align: middle; }
+
+/* Remove possible quote marks (") from <q>, <blockquote>. */
+blockquote:before, blockquote:after, q:before, q:after { content: ""; }
+blockquote, q { quotes: "" ""; }
+
+/* Remove annoying border on linked images. */
+a img { border: none; }
--- /dev/null
+/* --------------------------------------------------------------
+
+ typography.css
+ * Sets up some sensible default typography.
+
+-------------------------------------------------------------- */
+
+/* Default font settings.
+ The font-size percentage is of 16px. (0.75 * 16px = 12px) */
+html { font-size:100.01%; }
+body {
+ font-size: 75%;
+ color: #222;
+ background: #fff;
+ font-family: "Helvetica Neue", Arial, Helvetica, sans-serif;
+}
+
+
+/* Headings
+-------------------------------------------------------------- */
+
+h1,h2,h3,h4,h5,h6 { font-weight: normal; color: #111; }
+
+h1 { font-size: 3em; line-height: 1; margin-bottom: 0.5em; }
+h2 { font-size: 2em; margin-bottom: 0.75em; }
+h3 { font-size: 1.5em; line-height: 1; margin-bottom: 1em; }
+h4 { font-size: 1.2em; line-height: 1.25; margin-bottom: 1.25em; }
+h5 { font-size: 1em; font-weight: bold; margin-bottom: 1.5em; }
+h6 { font-size: 1em; font-weight: bold; }
+
+h1 img, h2 img, h3 img,
+h4 img, h5 img, h6 img {
+ margin: 0;
+}
+
+
+/* Text elements
+-------------------------------------------------------------- */
+
+p { margin: 0 0 1.5em; }
+p img.left { float: left; margin: 1.5em 1.5em 1.5em 0; padding: 0; }
+p img.right { float: right; margin: 1.5em 0 1.5em 1.5em; }
+
+a:focus,
+a:hover { color: #000; }
+a { color: #009; text-decoration: underline; }
+
+blockquote { margin: 1.5em; color: #666; font-style: italic; }
+strong { font-weight: bold; }
+em,dfn { font-style: italic; }
+dfn { font-weight: bold; }
+sup, sub { line-height: 0; }
+
+abbr,
+acronym { border-bottom: 1px dotted #666; }
+address { margin: 0 0 1.5em; font-style: italic; }
+del { color:#666; }
+
+pre { margin: 1.5em 0; white-space: pre; }
+pre,code,tt { font: 1em 'andale mono', 'lucida console', monospace; line-height: 1.5; }
+
+
+/* Lists
+-------------------------------------------------------------- */
+
+li ul,
+li ol { margin: 0; }
+ul, ol { margin: 0 1.5em 1.5em 0; padding-left: 3.333em; }
+
+ul { list-style-type: disc; }
+ol { list-style-type: decimal; }
+
+dl { margin: 0 0 1.5em 0; }
+dl dt { font-weight: bold; }
+dd { margin-left: 1.5em;}
+
+
+/* Tables
+-------------------------------------------------------------- */
+
+table { margin-bottom: 1.4em; width:100%; }
+th { font-weight: bold; }
+thead th { background: #c3d9ff; }
+th,td,caption { padding: 4px 10px 4px 5px; }
+tr.even td { background: #e5ecf9; }
+tfoot { font-style: italic; }
+caption { background: #eee; }
+
+
+/* Misc classes
+-------------------------------------------------------------- */
+
+.small { font-size: .8em; margin-bottom: 1.875em; line-height: 1.875em; }
+.large { font-size: 1.2em; line-height: 2.5em; margin-bottom: 1.25em; }
+.hide { display: none; }
+
+.quiet { color: #666; }
+.loud { color: #000; }
+.highlight { background:#ff0; }
+.added { background:#060; color: #fff; }
+.removed { background:#900; color: #fff; }
+
+.first { margin-left:0; padding-left:0; }
+.last { margin-right:0; padding-right:0; }
+.top { margin-top:0; padding-top:0; }
+.bottom { margin-bottom:0; padding-bottom:0; }
--- /dev/null
+---
+layout: default
+title: Changelog
+toc: false
+---
+
+## Version 2.3.x
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2013/03/22|Remove the topicprefix, queueprefix and topicsep options|19673|
+|2013/03/21|Remove the plugin.discovery.timeout setting as it's not relevant anymore|19694|
+|2013/03/21|Improve error reporting from the rpc application in the light of direct_addressing|19827|
+|2013/03/20|Fail with a friendly error message when no libdir is set|19752|
+|2013/03/14|Change default RabbitMQ and ActiveMQ ports to 61613|19734|
+|2013/03/13|Set correct reply-to headers in the RabbitMQ connector|17034|
+|2013/03/12|Pre-populate the data from data plugins like agent replies|19564|
+|2013/03/12|Explicitly include StringIO|19367|
+|2013/03/12|Enable direct addressing by default|19665|
+|2013/02/20|Fix error code collision on PLMC18|19366|
+|2013/02/15|Validate arguments supplied to the RPC application and raise errors sooner|19181|
+|*2013/02/14*|*Release 2.3.1*|19265|
+|2013/02/14|Initial work towards internationalization and online help|18663|
+|2013/02/14|Update vendored JSON gem for CVE-2013-0269|19265|
+|2013/02/13|Restore the ability to set a discovery timeout on a RPC client|19238|
+|2013/02/12|Replace underscores in plugin names with dashes to keep Debian happy|19200|
+|2013/02/12|Fix package building on certain Debian systems|19141|
+|2013/02/12|Remove the stomp connector|19146|
+|2013/02/07|Read the client config before trying to use any configuration options|19105|
+|2013/01/22|When an argument fails to parse in the rpc application fail rather than continue with unintended consequences|18773|
+|2013/01/22|The fix the *--no-response* argument to the rpc application that broke due to 18438|18513|
+|2013/01/22|Set *=* dependencies on the various packages that form a plugin rather than *>=*|18758|
+|2013/01/21|Improve presentation of the --help output for applications|18447|
+|2013/01/21|When a request failed via *reply.fail*, only show the message and not the half built data|18434|
+|*2013/01/10*|*Release 2.3.0*|18259|
+|2013/01/10|Raise the correct exception when trying to access unknown data items in a Data results|18466|
+|2013/01/10|Fix failing documentation generation for data plugins|18437|
+|2013/01/09|Correctly support negative boolean flags declared as --[no]-foo|18438|
+|2013/01/03|Add the package iteration number as a dependency for the common packages|18273|
+|2012/12/21|The libdirs supplied in the config file now has to be absolute paths to avoid issues when daemonising|16018|
+|2012/12/20|Logs the error and backtrace when an action fails|16414|
+|2012/12/20|Display the values of :optional and :default in DDL generated help|16616|
+|2012/12/20|Allow the query string for the get_data action in rpcutil to be 200 chars|18200|
+|2012/12/19|Do not fail when packaging non-agent packages using custom paths|17281|
+|2012/12/19|Require Ruby > 1.8 in the RPM specs for Ruby 1.9|17149|
+|2012/12/18|Allow required inputs to specify default data in DDLs|17615|
+|2012/11/12|When disconnecting set the connection to nil|17384|
+|2012/11/08|Define a specific buildroot to support RHEL5 systems correctly|17516|
+|2012/11/08|Use the correct rpmbuild commands on systems with rpmbuild-md5|17515|
+|2012/10/22|Correctly show help for data plugins without any input queries|17137|
+|2012/10/22|Allow the rpcutil#get_data action to work with data queries that takes no input|17138|
+|2012/10/03|Improve text output when providing custom formats for aggregations|16735|
+|2012/10/03|Correctly process supplied formats when displaying aggregate results|16415|
+|2012/10/03|Prevent one failing aggregate function from impacting others|16411|
+|2012/10/03|When validation fails indicate which input key has the problem|16617|
+|2012/09/26|Data queries can be written without any input queries meaning they take no input|16424|
+|2012/09/26|Use correct timeout for agent requests when using direct addressing|16569|
+|2012/09/26|Allow BigNum data to be used in data plugin replies|16503|
+|2012/09/26|Support non string data in the summary aggregate function|16410|
+
+## Version 2.2.x
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|*2013/02/14*|*Release 2.2.3*|19265|
+|2013/02/14|Update vendored JSON gem for CVE-2013-0269|19265|
+|2013/02/13|Restore the ability to set a discovery timeout on a RPC client|19238|
+|2013/02/12|Replace underscores in plugin names with dashes to keep Debian happy|19200|
+|2013/02/12|Fix package building on certain Debian systems|19141|
+|2013/02/12|Deprecate the stomp connector|19146|
+|2013/02/07|Read the client config before trying to use any configuration options|19105|
+|2013/01/22|Set *=* dependencies on the various packages that form a plugin rather than *>=*|18758|
+|*2013/01/17*|*Release 2.2.2*|18258|
+|2013/01/03|Add the package iteration number as a dependency for the common packages|18273|
+|2012/12/24|Restore the :any validator|18265|
+|2012/12/19|Do not fail when packaging non-agent packages using custom paths|17281|
+|2012/12/19|Require Ruby > 1.8 in the RPM specs for Ruby 1.9|17149|
+|2012/11/08|Define a specific buildroot to support RHEL5 systems correctly|17516|
+|2012/11/08|Use the correct rpmbuild commands on systems with rpmbuild-md5|17515|
+|2012/10/22|Correctly show help for data plugins without any input queries|17137|
+|2012/10/22|Allow the rpcutil#get_data action to work with data queries that takes no input|17138|
+|*2012/10/17*|*Release 2.2.1*|16965|
+|2012/10/03|Improve text output when providing custom formats for aggregations|16735|
+|2012/10/03|Correctly process supplied formats when displaying aggregate results|16415|
+|2012/10/03|Prevent one failing aggregate function from impacting others|16411|
+|2012/10/03|When validation fails indicate which input key has the problem|16617|
+|2012/09/26|Data queries can be written without any input queries meaning they take no input|16424|
+|2012/09/26|Use correct timeout for agent requests when using direct addressing|16569|
+|2012/09/26|Allow BigNum data to be used in data plugin replies|16503|
+|2012/09/26|Support non string data in the summary aggregate function|16410|
+|2012/09/14|Package discovery plugins that was left out for debian|16413|
+|*2012/09/13*|*Release 2.2.0*|16323|
+
+## Version 2.1.x
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2012/09/10|Update the vendored systemu gem|16289|
+|2012/09/06|Improve error reporting for empty certificate files|15924|
+|2012/09/05|Restore the verbose behavior while building packages|16216|
+|2012/09/04|Add a fetch method that mimic Hash#fetch to RPC Results and Requests|16222|
+|2012/09/04|Include the required mcollective version in packages that include the requirement|16173|
+|2012/08/29|Add a RabbitMQ specific connector plugin|16168|
+|2012/08/22|DDL files can now specify which is the minimal version of mcollective they require|15850|
+|2012/08/22|Fix a bug when specifying a custom target directory for packages|15956|
+|2012/08/22|When producing plugin packages keep the source deb and rpm|15917|
+|2012/08/09|Improve error handling in the plugin application|15848|
+|2012/08/08|Add the ability to store general usage information in the DDL|15633|
+|2012/08/02|Restore the formatting of the progress bar that was broken in 14255|15805|
+|2012/08/01|Display an error when no aggregate results could be computed|15793|
+|2012/08/01|Create a plugin system for validators|5078|
+|2012/07/19|Create a thread safe caching layer and use it to optimize loading of DDL files|15582|
+|2012/07/19|Correctly calculate discovery timeout in all cases and simplify logic around this|15602|
+|2012/07/17|Update the *name* field in the rpcutil DDL for consistency|15558
+|2012/07/17|Validate requests against the DDL in the agents prior to authorization or calling actions|15557|
+|2012/07/17|Refactor the single big DDL class into a class per type of plugin|15109|
+|2012/07/16|Default to the configured default discovery method in the RPC client when nothing is supplied|15506|
+|2012/07/16|Improve error handling in generate application|15473|
+|*2012/07/12*|*Release 2.1.1*|15379|
+|2012/07/11|Add a --display option to RPC clients that overrides the DDL display mode|15273|
+|2012/07/10|Do not add a metadata to agents created with the generator as they are now deprecated|15445|
+|2012/07/03|Correctly parse numeric and boolean data on the CLI in the rpc application|15344|
+|2012/07/03|Fix a bug related to parsing regular expressions in compound statements|15323|
+|2012/07/02|Update vim snippets in ext for new DDL features|15273|
+|2012/06/29|Create a common package for agent packages containing the DDL for servers and clients|15268|
+|2012/06/28|Improve parsing of compound filters where the first argument is a class|15271|
+|2012/06/28|Add the ability to declare automatic result summarization in the DDL files for agents|15031|
+|2012/06/26|Suppress subscribing to reply queues when no reply is expected|15226|
+|2012/06/25|Batched RPC requests will now all have the same requestid|15195|
+|2012/06/25|Record the request id on M::Client and in the RPC client stats|15194|
+|2012/06/24|Use UUIDs for the request id rather than our own weak implementation|15191|
+|2012/06/18|The DDL can now define defaults for outputs and the RPC replies are pre-populated|15087|
+|2012/06/18|Remove unused agent help code|15084|
+|2012/06/18|Remove unused code from the *discovery* agent related to inventory and facts|15083|
+|2012/06/18|Nodes will now refuse to load RPC agents without DDL files|15082|
+|2012/06/18|The Plugin Name and Type is now available to DDL objects|15076|
+|2012/06/15|Add a get_data action to the rpcutil agent that can retrieve data from data plugins|15057|
+|2012/06/14|Allow the random selection of nodes to be deterministic|14960|
+|2012/06/12|Remove the Client#discovered_req method and add warnings to the documentation about its use|14777|
+|2012/06/11|Add a discovery source capable of doing introspection on running agents|14945|
+|2012/06/11|Only do identity filter optimisations for the *mc* discovery source|14942|
+|*2012/06/08*|*Release 2.1.0*|14846|
+|2012/06/07|Force discovery state to be reset when changing collectives in the RPC client|14874|
+|2012/06/07|Create code generators for agents and data plugins|14717|
+|2012/06/07|Fix the _No response from_ report to be correctly formatted|14868|
+|2012/06/07|Sub collectives and direct addressing mode now works correctly|14668|
+|2012/06/07|The discovery method is now pluggable, included is one supporting flat files|14255|
+|2012/05/28|Add an application to assist shell completion systems with bash and zsh completion plugins|14196|
+|2012/05/22|Improve error messages from the packager when a DDL file cannot be found|14595|
+|2012/05/17|Add a dependency on stomp to the rubygem|14300|
+|2012/05/17|Adjust the ActiveMQ and Stomp connect_timeout to allow IPv4 fall back to happen in dual homed hosts|14496|
+|2012/05/16|Add a plugable data source usable in discovery and other plugins|14254|
+|2012/05/04|Improve version dependencies and upgrade experience of debian packages|14277|
+|2012/05/03|Add the ability for the DDL to load DDL files from any plugin type|14293|
+|2012/05/03|Rename the MCollective::RPC::DDL to MCollective::DDL to match its larger role in all plugins|14254|
+
+## Version 2.0.x
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|*2013/02/14*|*Release 2.0.1*|19265|
+|2013/02/14|Update vendored JSON gem for CVE-2013-0269|19265|
+|2012/05/04|Improve version dependencies and upgrade experience of debian packages|14277|
+|*2012/04/30*|*Release 2.0.0*|13900|
+
+## Version 1.3.x
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2012/04/30|Compound filters when set from the RPC client were not working|14239|
+|2012/04/25|Various improvements to the RPM spec file wrt licencing, dependencies etc|9451|
+|2012/04/25|Support using rpmbuild-md5 to create RPMs and support Fedora|14159|
+|2012/04/25|Improve LSB compliance in the Red Hat and Debian RC scripts|14151|
+|2012/04/19|Fix reference to _topicnamesep_ and remove _topicprefix_ from examples|13873|
+|2012/04/19|Remove dependency on FPM for building RPM and Deb packages|13573|
+|2012/04/18|Improve default output format from the mco script|14056|
+|2012/04/17|Remove unintended requirement that only newest stomp gems be used|13978|
+|2012/04/12|New init script for Debian that uses LSB functions to start and stop the daemon|13043|
+|2012/04/12|Use sed -i in the Rakefile to improve compatibility with OS X|13324|
+|2012/04/11|Fix compatibility with Ruby 1.9.1 by specifically loading rbconfig early on|13872|
+|*2012/04/05*|*Release 1.3.3*|13599|
+|2012/04/04|Use the MCollective::SSL utility class for crypto functions in the SSL security plugin|13615|
+|2012/04/02|Support reading public keys from SSL Certificates as well as keys|13534|
+|2012/04/02|Move the help template to the common package for both Debian and RedHat|13434|
+|2012/03/30|Support Stomp 1.2.2 CA verified connection to ActiveMQ|10596|
+|2012/03/27|_mco help rpc_ now shows the help for the rpc application|13350|
+|2012/03/22|Add a mco command that creates native OS packaging for plugins|12597|
+|2012/03/21|Default to console based logging at warning level for clients|13285|
+|2012/03/20|Work around SSL_read errors when using SSL or AES plugins and Stomp+SSL in Ruby < 1.9.3|13207|
+|2012/03/16|Improve logging for SSL connections when using Stomp Gem newer than 1.2.0|13165|
+|2012/03/14|Simplify handling of signals like TERM and INT and remove pid file on exit|13105|
+|2012/03/13|Create a conventional place to store implemented_by scripts|13064|
+|2012/03/09|Handle exceptions added to the Stomp 1.1 compliant versions of the Stomp gem|13020|
+|2012/03/09|Specifically enable reliable communications while using the pool style syntax|13040|
+|2012/03/06|Initial support for the Windows Platform|12555|
+|2012/03/05|Application plugins can now disable any of 3 sections of the standard CLI argument parsers|12859|
+|2012/03/05|Fix base 64 encoding and decoding of message payloads that would previous raise unexpected exceptions|12950|
+|2012/03/02|Treat :hosts and :nodes as equivalents when supplying discovery data, be more strict about flags discover will accept|12852|
+|2012/03/02|Allow exit() to be used everywhere in application plugins, not just in the main method|12927|
+|2012/03/02|Allow batch mode to be enabled and disabled on demand during the life of a client|12854|
+|2012/02/29|Show the progress bar before sending any requests to give users feedback as soon as possible rather than after first result only|12865|
+|2012/02/23|Do not log exceptions in the RPC application when a non existing action is called with request parameters|12719|
+|2012/02/17|Log miscellaneous Stomp errors at error level rather than debug|12705|
+|2012/02/17|Improve subscription tracking by using the subID feature of the Stomp gem and handle duplicate exceptions|12703|
+|2012/02/15|Improve error handling in the inventory application for non responsive nodes|12638|
+|2012/02/14|Comply to Red Hat guideline by not setting mcollective to start by default after RPM install|9453|
+|2012/02/14|Allow building the client libraries as a gem|9383|
+|2012/02/13|On Red Hat like systems read /etc/sysconfig/mcollective in the init script to allow modification of the environment|7441|
+|2012/02/13|Make the handling of symlinks to the mco script more robust to handle directories with mc- in their name|6275|
+|2012/02/01|systemu and therefore MC::Shell can sometimes return nil exit code, the run() method now handles this better by returning -1 exit status|12082|
+|2012/01/27|Improve handling of discovery data on STDIN to avoid failures when run without a TTY and without supplying discovery data|12084|
+|2012/01/25|Allow the syslog facility to be configured|12109|
+|2012/01/13|Add a RPC agent validator to ensure input is one of list of known good values|11935|
+|2012/01/09|The printrpc helper did not correctly display empty strings in received output|11012|
+|2012/01/09|Add a halt method to the Application framework and standardize exit codes|11280|
+|2011/11/21|Remove unintended dependency on _pp_ in the ActiveMQ plugin|10992|
+|2011/11/17|Allow reply to destinations to be supplied on the command line or API|9847|
+|*2011/11/17*|*Release 1.3.2*|*10830*|
+|2011/11/16|Improve error reporting for code errors in application plugins|10883|
+|2011/11/15|The limit method is now configurable on each RPC client as well as the config file|7772|
+|2011/11/15|Add a --graph option to the ping application that shows response distribution|10864|
+|2011/11/14|An ActiveMQ specific connector was added that supports direct connections|7899|
+|2011/11/11|SimpleRPC clients now support native batching with --batch|5939|
+|2011/11/11|The client now unsubscribes from topics when it's idle minimising the risk of receiving misdirected messages|10670|
+|2011/11/09|Security plugins now ignore miss directed messages early thus using fewer resources|10671|
+|2011/10/28|Support ruby-1.9.2-p290 and ruby-1.9.3-rc1|10352|
+|2011/10/27|callerid, certificate names, and identity names can now only have \w . and - in them|10327|
+|2011/10/25|When discovery information is provided always accept it without requiring reset first|10265|
+|2011/10/24|Add :number, :integer and :float to the DDL and rpc application|9902|
+|2011/10/22|Speed up discovery when limit targets are set|10133|
+|2011/10/22|Do not attempt to validate TTL and Message Times on replies in the SSL plugin|10226|
+|2011/10/03|Allow the RPC client to raise an exception rather than exit on failure|9360|
+|2011/10/03|Allow the TTL of requests to be set in the config file and the SimpleRPC API|9399|
+|2011/09/26|Cryptographically secure the TTL and Message Time of requests when using AES and SSL plugins|9400|
+|2011/09/20|Update default shipped configurations to provide a better out of the box experience|9452|
+|2011/09/20|Remove deprecated mc- scripts|9402|
+|2011/09/20|Keep track of messages that has expired and expose the stat in rpcutil and inventory application|9456|
+|*2011/09/16*|*Release 1.3.1*|*9133*|
+|2011/09/9|Use direct messaging where possible for identity filters and make the rpc application direct aware|8466|
+|2011/08/29|Enforce a 60 second TTL on all messages by default|8325|
+|2011/08/29|Change the default classes.txt file to be in line with Puppet defaults|9133|
+|2011/08/06|Add reload-agents and reload-loglevel commands to the redhat RC script|7730|
+|2011/08/06|Avoid reloading the authorization class over and over from disk on each request|8703|
+|2011/08/06|Add a boolean validator to SimpleRPC agents|8799|
+|2011/08/06|Justify text results better when using printrpc|8807|
+|2011/07/22|Add --version to the mco utility|7822|
+|2011/07/22|Add missing meta data to the discovery agent|8497|
+|2011/07/18|Raise an error if invalid format fact filters are supplied|8419|
+|2011/07/14|Add a rich discovery query language|8181|
+|2011/07/08|Do not set RUBYLIB in the RC scripts, the OS should do the right thing|8063|
+|2011/07/07|Add a -j argument to all SimpleRPC clients that causes printrpc to produce JSON data|8280|
+|2011/06/30|Add the ability to do point to point comms for requests affecting small numbers of hosts|7988|
+|2011/06/21|Add support for Stomp Gem version 1.1.9 callback based logging|7960|
+|2011/06/21|On the server side log missing DDL files at debug and not warning level|7961|
+|2011/06/16|Add the ability for nodes to subscribe to per-node queues, off by default|7225|
+|2011/06/12|Remove assumptions about middleware structure from the core and move it to the connector plugins|7619|
+|*2011/06/08*|*Release 1.3.0*|7796|
+|2011/06/07|Exceptions raised during option parsing were not handled and resulted in stack traces|7796|
+|2011/06/06|Remove the sshkey, it's being moved to the plugin repository|7794|
+|2011/06/02|Correct parsing of MCOLLECTIVE_EXTRA_OPTS in cases where no config related settings were set|7755|
+|2011/05/31|Disconnect from the middleware when an application calls exit|7712|
+|2011/05/29|Validations failure in RPC agents will now raise the correct exceptions as documented|7711|
+|2011/05/25|Make the target collective for registration messages configurable|7650|
+|2011/05/24|Rename the connector plugins send method to publish to avoid issues ruby Object#send|7623|
+|2011/05/23|Log a warning when the CF file parsing fails rather than raise a whole ruby exception|7627|
+|2011/05/23|Allow applications to use the exit method as would normally be expected|7626|
+|2011/05/22|Refactor subscribe and unsubscribe so that middleware structure is entirely contained in the connectors|7620|
+|2011/05/21|Add the ability for agents to programatically declare if they should work on a node|7583|
+|2011/05/20|Improve error reporting in the single application framework|7574|
+|2011/05/16|Allow _._ in fact names|7532|
+|2011/05/16|Fix compatability issues with RH4 init system|7448|
+|2011/05/15|Handle failures from remote nodes better in the inventory app|7524|
+|2011/05/06|Revert unintended changes to the Debian rc script|7420|
+|2011/05/06|Remove the _test_ agent that was accidentally checked in|7425|
+
+## Version 1.2.x
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|*2011/06/30*|*Release 1.2.1*|8117|
+|2011/06/02|Correct parsing of MCOLLECTIVE_EXTRA_OPTS in cases where no config related settings were set|7755|
+|2011/05/23|Allow applications to use the exit method as would normally be expected|7626|
+|2011/05/16|Allow _._ in fact names|7532|
+|2011/05/16|Fix compatability issues with RH4 init system|7448|
+|2011/05/15|Handle failures from remote nodes better in the inventory app|7524|
+|2011/05/06|Revert unintended changes to the Debian rc script|7420|
+|2011/05/06|Remove the _test_ agent that was accidentally checked in|7425|
+|*2011/05/04*|*Release 1.2.0*|7227|
+
+## Version 1.1.x
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/05/03|Improve Red Hat RC script by using distro builtin functions|7340|
+|2011/05/01|Support setting a priority on Stomp messages|7246|
+|2011/04/30|Handle broken and incomplete DDLs better and improve the format of DDL output|7191|
+|2011/04/23|Encode the target agent and collective in requests|7223|
+|2011/04/20|Make the SSL Cipher used a config option|7191|
+|2011/04/20|Add a clear method to the PluginManager that deletes all plugins, improve test isolation|7176|
+|2011/04/19|Abstract the creation of request and reply hashes to simplify connector plugin development|5701|
+|2011/04/15|Improve the shellsafe validator and add a Util method to do shell escaping|7066|
+|2011/04/14|Update Rakefile to have a mail_patches task|6874|
+|2011/04/13|Update vendored systemu library for Ruby 1.9.2 compatability|7067|
+|2011/04/12|Fix failing tests on Ruby 1.9.2|7067|
+|2011/04/11|Update the DDL documentation to reflect the _mco help_ command|7042|
+|2011/04/11|Document the use filters on the CLI|5917|
+|2011/04/11|Improve handling of unknown facts in Util#has_fact? to avoid exceptions about nil#clone|6956|
+|2011/04/11|Correctly set timeout on the discovery agent to 5 seconds as default|7045|
+|2011/04/11|Let rpcutil#agent_inventory supply _unknown_ for missing values in agent meta data|7044|
+|*2011/04/07*|*Release 1.1.4*|6952|
+|2011/03/28|Correct loading of vendored JSON gem|6877|
+|2011/03/28|Show collective and sub collective info in the inventory application|6872|
+|2011/03/23|Disconnect from the middleware when mcollectived disconnects|6821|
+|2011/03/21|Update rpcutil ddl file to be less strict about valid fact names|6764|
+|2011/03/22|Support reading configuration from configfir/plugin.d for plugins|6623|
+|2011/03/21|Update default configuration files for subcollectives|6741|
+|2011/03/16|Add the ability to implement actions using external scripts|6705|
+|2011/03/15|Port mc-controller to the Application framework and deprecate the exit command|6637|
+|2011/03/13|Only cache registration and discovery agents, handle the rest as new instances|6692|
+|2011/03/08|PluginManager can now create new instances on demand for a plugin type|6622|
+|*2011/03/07*|*Release 1.1.3*|6609|
+|2011/03/04|Rename /usr/sbin/mc to /usr/bin/mco|6578|
+|2011/03/01|Wrap rpcclient in applications ensuring that options is always set|6308|
+|2011/02/28|Make communicating with the middleware more robust by including send calls in timeouts|6505|
+|2011/02/28|Create a wrapper to safely run shell commands avoiding zombies|6392|
+|2011/02/19|Introduce Subcollectives for network partitioning|5967|
+|2011/02/19|Improve error handling when parsing arguments in the rpc application|6388|
+|2011/02/19|Fix error logging when file_logger creation fails|6387|
+|2011/02/17|Correctly parse MCOLLECTIVE_EXTRA_OPTS in the new unified binary framework|6354|
+|2011/02/15|Allow the signing key and Debian distribution to be customized|6321|
+|2011/02/14|Remove inadvertently included package.ddl|6313|
+|2011/02/14|Handle missing libdirs without crashing|6306|
+|*2011/02/14*|*Release 1.1.2*|6303|
+|2011/02/13|Surpress replies to SimpleRPC clients who did not request results|6305|
+|2011/02/11|Fix Debian packaging error due to the same file in multiple packages|6276|
+|2011/02/11|The application framework will now disconnect from the middleware for consistancy|6292|
+|2011/02/11|Returning _nil_ from a registration plugin will skip registration|6289|
+|2011/02/11|Set loglevel to warn by default if not specified in the config file|6287|
+|2011/02/10|Fix backward compatability with empty fact strings|6278|
+|*2011/02/07*|*Release 1.1.1*|6080|
+|2011/02/02|Load the DDL from disk once per printrpc call and not for every result|5958|
+|2011/02/02|Include full Apache 2 license text|6113|
+|2011/01/31|Create a new single executable application framework|5897|
+|2011/01/30|Fix backward compatibility with old foo=/bar/ style fact searches|5985|
+|2011/01/30|Documentation update to reflect correct default identity behavior|6073|
+|2011/01/29|Let the YAML file force fact reloads when the files update|6057|
+|2011/01/29|Add the ability for fact plugins to force fact invalidation|6057|
+|2011/01/29|Document an approach to disable type validation in the DDL|6066|
+|2011/01/19|Add basic filters to the mc-ping command|5933|
+|2011/01/19|Add a ping action to the rpcutil agent|5937|
+|2011/01/17|Allow MC::RPC#printrpc to print single results|5918|
+|2011/01/16|Provide SimpleRPC style results when accessing the MC::Client results directly|5912|
+|2011/01/11|Add an option to Base64 encode the STOMP payload|5815|
+|2011/01/11|Fix a bug with forcing all facts to be strings|5832|
+|2011/01/08|When using reload_agents or USR1 signal no agents would be reloaded|5808|
+|2011/01/04|Use the LSB based init script on SUSE|5762|
+|2011/01/04|Remove the use of a Singleton in the logging class|5749|
+|2011/01/02|Add AES+RSA security plugin|5696|
+|2010/12/31|Security plugins now have access to the callerid of the message they are replying to|5745|
+|2010/12/30|Allow - in fact names|5727|
+|2010/12/29|Treat machines that fail security validation like ones that did not respond|5700|
+|*2010/12/29*|*Release 1.1.0*|5695|
+|2010/12/28|Remove trailing whitespace from all source files|5702|
+|2010/12/28|Adjust the logfile audit format to include local time and all on one line|5694|
+|2010/12/26|Improve the SimpleRPC fact_filter helper to support new fact operators|5678|
+|2010/12/25|Increase the rpcutil timeout to allow for slow facts|5679|
+|2010/12/25|Allow for network and fact source latency when calculating client timeout|5676|
+|2010/12/25|Remove MCOLLECTIVE_TIMEOUT and MCOLLECTIVE_DTIMEOUT environment vars in favour of MCOLLECTIVE_EXTRA_OPTS|5675|
+|2010/12/25|Refactor the creation of the options hash so other tools don't need to know the internal formats|5672|
+|2010/12/21|The fact plugin format has been changed and simplified, the base now provides caching and thread safety|5083|
+|2010/12/20|Add parameters <=, >=, <, >, !=, == and =~ to fact selection|5084|
+|2010/12/14|Add experimental sshkey security plugin|5085|
+|2010/12/13|Log a startup message showing version and log level|5538|
+|2010/12/13|Add a console logger|5537|
+|2010/12/13|Logging is now plugable and a syslog plugin was provided|5082|
+|2010/12/13|Allow libdir to be an array of directories for agents and ddl files|5253|
+|2010/12/13|The progress bar will now intelligently figure out the terminal dimentions|5524|
+
+## Version 1.0.x
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|*2011/02/16*|*Release 1.0.1*|6342|
+|2011/02/02|Include full Apache 2 license text|6113|
+|2011/01/29|The YAML fact plugin kept deleted facts in memory|6056|
+|2011/01/04|Use the LSB based init script on SUSE|5762|
+|2010/12/30|Allow - in fact names|5727|
+|2010/12/29|Treat machines that fail security validation like ones that did not respond|5700|
+|2010/12/25|Allow for network and fact source latency when calculating client timeout|5676|
+|2010/12/25|Increase the rpcutil timeout to allow for slow facts|5679|
+|*2010/12/13*|*Release 1.0.0*|5453|
+
+## Version 0.4.x
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2010/12/04|Remove the LSB requirements for RedHat systems|5451|
+|2010/11/23|Make the YAML fact source thread safe and force all facts to strings|5377|
+|2010/11/23|Add get_config_item to rpcutil to retrieve a running config value from a server|5376|
+|2010/11/20|Convert mc-facts into a SimpleRPC client|5371|
+|2010/11/18|Added GPG signing to Rake packaging tasks (SIGNED=1)|5355|
+|2010/11/17|Improve error messages from clients in the case of failure|5329|
+|2010/11/17|Add helpers to disconnect from the middleware and update all bundled clients|5328|
+|2010/11/16|Correct LSB provides and requires in default init script|5222|
+|2010/11/16|Input validation on rpcutil has been improved to match all valid facts|5320|
+|2010/11/16|Add the ability to limit the results to a subset of hosts|5306|
+|2010/11/15|Add fire and forget mode to SimpleRPC custom_request|5305|
+|2010/11/09|General connection settings to the Stomp connector was ignored|5245|
+|*2010/10/18*|*Release version 0.4.10*| |
+|2010/10/18|Document exit command to mc-controller|152|
+|2010/10/13|Log messages that don't pass the filters at debug level|149|
+|2010/10/03|Preserve options in cases where RPC::Client instances exist in the same program|148|
+|2010/09/30|Add the ability to set different types of callerid in the PSK plugin|145|
+|2010/09/30|Improve Ruby 1.9.x compatibility|142|
+|2010/09/29|Improve error handling in registration to avoid high CPU usage loops|143|
+|*2010/09/21*|*Release version 0.4.9*| |
+|2010/09/20|Improve Debian packaging task|140|
+|2010/09/20|Add :boolean type support to the DDL|138|
+|2010/09/19|Refactor MCollective::RPC to add less unneeded stuff to Object|137|
+|2010/09/18|Prevent duplicate config loading with multiple clients active|136|
+|2010/09/18|Rotate the log file by default, keeping 5 x 2MB files|135|
+|2010/09/18|Write a overview document detailing security of the collective|131|
+|2010/09/18|Add MCollective.version, set it during packaging and include it in the rpcutil agent|134|
+|2010/09/13|mc-inventory now use SimpleRPC and the rpcutil agent and display server stats|133|
+|2010/09/13|Make the path to the rpc-help.erb configurable and set sane default|130|
+|2010/09/13|Make the configfile used available in the Config class and add to rpcutil|132|
+|2010/09/12|Rework internal statistics and add a rpcutil agent|129|
+|2010/09/12|Fix internal memory structures related to agent meta data|128|
+|2010/08/24|Update the OpenBSD port for changes in 0.4.8 tarball|125|
+|2010/08/23|Fix indention/block error in M:R:Stats|124|
+|2010/08/23|Fix permissions in the RPM for files in /etc|123|
+|2010/08/23|Fix language in two error messages|122|
+|*2010/08/20*|*Release version 0.4.8*| |
+|2010/08/19|Fix missing help template in debian packages|90|
+|2010/08/18|Clean up some hardlink warnings in the Rakefile|117|
+|2010/08/18|Include the website in the main repo and add a simple Rake task|118|
+|2010/08/17|Handle exceptions for missing plugins better|115|
+|2010/08/17|Add support for ~/.mcollective as a config file|114|
+|2010/08/07|SSL security plugin can use either YAML or Marshal|94|
+|2010/08/06|Multiple YAML files can now be used as fact source|112|
+|2010/08/06|Allow log level to be adjusted at run time with USR2|113|
+|2010/07/31|Add basic report scripting support to mc-inventory|111|
+|2010/07/06|Removed 'rpm' from the default rake task|109|
+|2010/07/06|Add redhat-lsb to the server RPM dependencies|108|
+|*2010/06/29*|*Release version 0.4.7*| |
+|2010/06/27|Change default factsource to Yaml|106|
+|2010/06/27|Added VIM snippets to create DDLs and Agents|102|
+|2010/06/26|DDL based help now works better with Symbols in in/output|105|
+|2010/06/23|Whitespace at the end of config lines are now stripped|100|
+|2010/06/22|printrpc will now inject some colors into results|99|
+|2010/06/22|Recover from syntax and other errors in agents|98|
+|2010/06/17|The agent a MC::RPC::Client is working on is now available|97|
+|2010/06/17|Integrate the DDL with data display helpers like printrpc|92|
+|2010/06/15|Avoid duplicate topic subscribes in complex clients|95|
+|2010/06/15|Catch some unhandled exceptions in RPC Agents|96|
+|2010/06/15|Fix missing help template file from RPM|90|
+|*2010/06/14*|*Release version 0.4.6* | |
+|2010/06/12|Qualify the Process class to avoid clashes in the discovery agent|88|
+|2010/06/12|Add mc-inventory which shows agents, classes and facts for a node|87|
+|2010/06/11|mc-facts now supports standard filters|86|
+|2010/06/11|Add connection pool retry options and ssl for connection|85|
+|2010/06/11|Add support for specifying multiple stomp hosts for failover|84|
+|2010/06/10|Tighten up handling of filters to avoid nil's getting into them|83|
+|2010/06/09|Sort the mc-facts output to be more readable|82|
+|2010/06/08|Fix deprecation warnings in newer Stomp gems|81|
+|*2010/06/03*|*Release version 0.4.5* | |
+|2010/06/01|Improve the main discovery agent by adding facts and classes to its inventory action|79|
+|2010/05/30|Add various helpers to get reports as text instead of printing them|43|
+|2010/05/30|Add a custom_request method to call SimpleRPC agents with your own discovery|75|
+|2010/05/30|Refactor RPC::Client to be more generic and easier to maintain|75|
+|2010/05/29|Fix a small scoping issue in Security::Base|76|
+|2010/05/25|Add option --no-progress to disable progress bar for SimpleRPC|74|
+|2010/05/23|Add some missing dependencies to the RPMs|72 |
+|2010/05/22|Add an option _:process_results_ to the client|71|
+|2010/05/13|Fix help output that still shows old branding|70|
+|2010/04/27|The supplied generic stompclient now accepts STOMP_PORT in the environment|68 |
+|2010/04/26|Add a SimpleRPC Client helper to reset filters|64 |
+|2010/04/26|Listen for signal USR1 and reload all agents from disk|65 |
+|2010/04/12|Add SimpleRPC Authorization support|63|
+|*2010/04/03*|*Release version 0.4.4* | |
+|2010/03/27|Make it easier to construct SimpleRPC requests to use with the standard client library|60 |
+|2010/03/27|Manipulating the filters via the helper methods will force rediscovery|59 |
+|2010/03/23|Prevent Activesupport when brought in by Facter from breaking our logs|57 |
+|2010/03/23|Clean up logging for messages not targeted at us|56 |
+|2010/03/19|Add exception handling to the registration base class|55 |
+|2010/03/03|Use /usr/bin/env ruby instead of hardcoded paths|54|
+|2010/02/17|Improve mc-controller and document it|46|
+|2010/02/08|Remove some close coupling with Stomp to easy creating of other connectors|49|
+|2010/02/01|Made the discovery agent timeout configurable using plugin.discovery.timeout|48|
+|2010/01/25|mc-controller now correctly loads/reloads agents.|45|
+|2010/01/25|Building packages has been improved to ensure rdocs are always included|44 |
+|*2010/01/24*|*Release version 0.4.3* | |
+|2010/01/23|Handle ctrl-c during discovery without showing exceptions to users|34 |
+|2010/01/21|Force all facts in the YAML fact source to be strings|41 |
+|2010/01/19|Add auditing to SimpleRPC clients and Agents | |
+|2010/01/18|The SRPM we provide will now build outside of the Rake environment|40|
+|2010/01/18|Add a _fail!_ method to RPC::Agent|37|
+|2010/01/18|mc-rpc can now be used without supplying arguments|38 |
+|2010/01/18|Don't raise an error if no user/pass is given to the stomp connector, try unauthenticated mode|35|
+|2010/01/17|Improve error message when Regex validation failed on SimpleRPC input|36|
+|*2010/01/13*|*Release version 0.4.2* | |
+|2010/01/13|New packaging for Debian provided by Riccardo Setti|29|
+|2010/01/07|Improved LSB compliance of the init script - thanks Riccardo Setti|32|
+|2010/01/07|Multiple calls to SimpleRPC client would reset discovered hosts|31|
+|2010/01/04|Timeouts can now be changed with MCOLLECTIVE_TIMEOUT and MCOLLECTIVE_DTIMEOUT environment vars|25|
+|2010/01/04|Specify class and fact filters easier with the new -W or --with option|27 |
+|2010/01/04|Added COPYING file to RPMs and tarball|28|
+|2010/01/04|Make shorter filter options -C, -I, -A and -F|26|
+|*2010/01/02*|*Release version 0.4.1* | |
+|2010/01/02|Added hooks to plug into the processing of requests, also enabled setting meta data and timeouts|14|
+|2010/01/02|Created readers for @config and @logger in the SimpleRPC agent|23|
+|2009/12/30|Don't send out any requests if no nodes were discovered|17|
+|2009/12/30|Added :discovered and :discovered_nodes to client stats|20|
+|2009/12/30|Add a empty_filter? helper to the RPC mixin|18|
+|2009/12/30|Fix formatting bug with progress bar|21|
+|2009/12/29|Simplify mc-rpc command line|16|
+|2009/12/29|Fix layout issue when printing hosts that did not respond|15|
+|*2009/12/29*|*Release version 0.4.0* | |
+|2009/12/28|Add support for other configuration management systems like chef in the --with-class filters|13|
+|2009/12/28|Add a <em>Util.empty_filter?</em> to test for an empty filter| |
+|2009/12/27|Create a new client framework - SimpleRPC|6|
+|2009/12/27|Add support for multiple filters of the same type|3|
+
+## Version 0.3.x
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|*2009/12/17*|*Release version 0.3.0* | |
+|2009/12/16|Improvements for newer versions of Ruby where TERM signal was not handled|7|
+|2009/12/07|MCollective::Util is now a module and plugins can drop in util classes in the plugin dir| |
+|2009/12/07|The Rakefile now works with rake provided on Debian 4 systems|2|
+|2009/12/07|Improvements in the RC script for Debian and older Ubuntu systems|5|
+
+## Version 0.2.x
+
+|Date|Description|Ticket|
+|2009/12/01|Release version 0.2.0| |
--- /dev/null
+---
+title: "MCollective » Configure » Servers"
+layout: default
+---
+
+
+<!-- TODO Got to change this middleware link as soon as enough of the deploy docs come up. -->
+[middleware]: /mcollective/deploy/middleware/activemq.html
+[filters]: /mcollective/reference/ui/filters.html
+[plugin_directory]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki
+[facter_plugin]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsFacter
+[ohai_plugin]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsOhai
+[chef_classfile]: /mcollective/reference/integration/chef.html#class-filters
+[fact]: #facts-identity-and-classes
+[connector_plugin]: #connector-settings
+[subcollectives]: /mcollective/reference/basic/subcollectives.html
+[registration]: /mcollective/reference/plugins/registration.html
+[puppetdb]: /puppetdb/
+[security_plugin]: #security-plugin-settings
+[auditing]: /mcollective/simplerpc/auditing.html
+[authorization]: /mcollective/simplerpc/authorization.html
+[actionpolicy]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AuthorizationActionPolicy
+[security_aes]: /mcollective/reference/plugins/security_aes.html
+[security_overview]: /mcollective/security.html
+[ssl_plugin]: /mcollective/reference/plugins/security_ssl.html
+[activemq_tls_verified]: /mcollective/reference/integration/activemq_ssl.html#ca-verified-tls
+[activemq_connector]: /mcollective/reference/plugins/connector_activemq.html
+[rabbitmq_connector]: /mcollective/reference/plugins/connector_rabbitmq.html
+[stdlib]: http://forge.puppetlabs.com/puppetlabs/stdlib
+
+{% capture badbool %}**Note:** Use these exact values only; do not use "true" or "false."{% endcapture %}
+
+{% capture pluginname %}**Note:** Capitalization of plugin names doesn't matter; MCollective normalizes it before loading the plugin.{% endcapture %}
+
+{% capture path_separator %}system path separator (colon \[`:`\] on \*nix, semicolon \[`;`\] on Windows){% endcapture %}
+
+
+
+This document describes MCollective server configuration in MCollective 2.0.0 and higher. Older versions may lack certain fetaures.
+
+
+Example / Index
+-----
+
+The following is an example MCollective server config file showing all of the major groups of settings. All of the setting names styled as links can be clicked, and will take you down the page to a full description of that setting.
+
+[See below the example for a full description of the config file location and format.](#the-server-config-files)
+
+<pre><code># /etc/mcollective/server.cfg
+
+# <a href="#connector-settings">Connector settings (required):</a>
+# -----------------------------
+
+<a href="#connector">connector</a> = activemq
+<a href="#directaddressing">direct_addressing</a> = 1
+
+# <a href="#activemq-connector-settings">ActiveMQ connector settings:</a>
+plugin.activemq.pool.size = 1
+plugin.activemq.pool.1.host = middleware.example.net
+plugin.activemq.pool.1.port = 61614
+plugin.activemq.pool.1.user = mcollective
+plugin.activemq.pool.1.password = secret
+plugin.activemq.pool.1.ssl = 1
+plugin.activemq.pool.1.ssl.ca = /var/lib/puppet/ssl/certs/ca.pem
+plugin.activemq.pool.1.ssl.cert = /var/lib/puppet/ssl/certs/web01.example.com.pem
+plugin.activemq.pool.1.ssl.key = /var/lib/puppet/ssl/private_keys/web01.example.com.pem
+plugin.activemq.pool.1.ssl.fallback = 0
+
+# <a href="#rabbitmq-connector-settings">RabbitMQ connector settings:</a>
+plugin.rabbitmq.vhost = /mcollective
+plugin.rabbitmq.pool.size = 1
+plugin.rabbitmq.pool.1.host = middleware.example.net
+# ... etc., similar to activemq settings
+
+# <a href="#security-plugin-settings">Security plugin settings (required):</a>
+# -----------------------------------
+
+<a href="#securityprovider">securityprovider</a> = ssl
+
+# <a href="#ssl-plugin-settings">SSL plugin settings:</a>
+plugin.ssl_client_cert_dir = /etc/mcollective.d/clients
+plugin.ssl_server_private = /etc/mcollective.d/server_private.pem
+plugin.ssl_server_public = /etc/mcollective.d/server_public.pem
+
+# <a href="#psk-plugin-settings">PSK plugin settings:</a>
+plugin.psk = j9q8kx7fnuied9e
+
+# <a href="#facts-identity-and-classes">Facts, identity, and classes (recommended):</a>
+# ------------------------------------------
+
+<a href="#factsource">factsource</a> = yaml
+<a href="#pluginyaml">plugin.yaml</a> = /etc/mcollective/facts.yaml
+<a href="#factcachetime">fact_cache_time</a> = 300
+
+<a href="#identity">identity</a> = web01.example.com
+
+<a href="#classesfile">classesfile</a> = /var/lib/puppet/state/classes.txt
+
+# <a href="#subcollectives">Subcollectives (optional):</a>
+# -------------------------
+
+<a href="#collectives">collectives</a> = mcollective,uk_collective
+<a href="#maincollective">main_collective</a> = mcollective
+
+# <a href="#node-registration">Registration (optional):</a>
+# -----------------------
+
+<a href="#registerinterval">registerinterval</a> = 300
+<a href="#registration">registration</a> = agentlist
+<a href="#registrationcollective">registration_collective</a> = mcollective
+
+# <a href="#auditing">Auditing (optional):</a>
+# -------------------
+
+<a href="#rpcaudit">rpcaudit</a> = 1
+<a href="#rpcauditprovider">rpcauditprovider</a> = logfile
+<a href="#pluginrpcauditlogfile">plugin.rpcaudit.logfile</a> = /var/log/mcollective-audit.log
+
+# <a href="#authorization">Authorization (optional):</a>
+# ------------------------
+
+<a href="#rpcauthorization">rpcauthorization</a> = 1
+<a href="#rpcauthprovider">rpcauthprovider</a> = action_policy
+
+# <a href="#logging">Logging:</a>
+# -------
+
+<a href="#loggertype">logger_type</a> = file
+<a href="#loglevel">loglevel</a> = info
+<a href="#keeplogs">keeplogs</a> = 5
+<a href="#maxlogsize">max_log_size</a> = 2097152
+<a href="#logfacility">logfacility</a> = user
+
+# <a href="#platform-defaults">Platform defaults:</a>
+# -----------------
+
+<a href="#daemonize">daemonize</a> = 1
+<a href="#libdir">libdir</a> = /usr/libexec/mcollective
+<a href="#sslcipher">ssl_cipher</a> = aes-256-cbc
+</code>
+</pre>
+
+
+([↑ Back to top](#content))
+
+
+
+
+
+
+The Server Config File(s)
+-----
+
+### Main Config File
+
+MCollective servers are configured with the `/etc/mcollective/server.cfg` file. It contains MCollective's core settings, as well as settings for the various plugins.
+
+> **Warning:** This file contains sensitive credentials, and should only be readable by the root user, or whatever user the MCollective daemon runs as.
+
+### File Format
+
+Each line consists of a setting, an equals sign, and a value:
+
+ # setting = value
+ connector = activemq
+
+The spaces on either side of the equals sign are optional. Lines starting with a `#` are comments.
+
+> **Note on Boolean Values:** MCollective's config code does not have consistent handling of boolean values. Many of the core settings will accept values of `1/0` and `y/n`, but will fail to handle `true/false`; additionally, each plugin can handle boolean values differently, and some of them do not properly handle the `y/n` values accepted by the core settings.
+>
+> Nearly all known plugins and core settings accept `1` and `0`. Until further notice, you should always use these for all boolean settings, as no other values are universally safe.
+
+### Plugin Config Directory (Optional)
+
+Many of MCollective's settings are named with the format `plugin.<NAME>.<SETTING_NAME>`. These settings can optionally be put in separate files, in the `/etc/mcollective/plugin.d/` directory.
+
+To move a plugin setting to an external file, put it in `/etc/mcollective/plugin.d/<NAME>.cfg`, and use only the `<SETTING_NAME>` segment of the setting. So this:
+
+ # /etc/mcollective/server.cfg
+ plugin.puppet.splay = true
+
+...is equivalent to:
+
+ # /etc/mcollective/plugin.d/puppet.cfg
+ splay = true
+
+Note that this doesn't work for settings like `plugin.psk`, since they have no `<SETTING_NAME>` segment; a setting must have at least three segments to go in a plugin.cfg file.
+
+### Best Practices
+
+You should manage your MCollective servers' config files with config management software (such as Puppet). While most settings in a deployment are the same, several should be different for each server, and managing these differences manually is impractical.
+
+If your deployment is fairly simple and there is little division of responsibility (e.g. having one group in charge of MCollective core and another group in charge of several agent plugins), then you can manage the config file with a simple template.
+
+If your deployment is large or complex, or you expect it to become so, you should manage MCollective settings as individual resources, as this is the only practical way to divide responsibilities within a single file.
+
+Below is an example of how to do this using the `file_line` type from the [puppetlabs/stdlib module][stdlib]:
+
+{% highlight ruby %}
+ # /etc/puppet/modules/mcollective/manifests/setting.pp
+ define mcollective::setting ($setting = $title, $target = '/etc/mcollective/server.cfg', $value) {
+ validate_re($target, '\/(plugin\.d\/[a-z]+|server)\.cfg\Z')
+ $regex_escaped_setting = regsubst($setting, '\.', '\\.', 'G') # assume dots are the only regex-unsafe chars in a setting name.
+
+ file_line {"mco_setting_${title}":
+ path => $target,
+ line => "${setting} = ${value}",
+ match => "^ *${regex_escaped_setting} *=.*$",
+ }
+ }
+
+ # /etc/puppet/modules/mcollective_core/manifests/server/connector.pp
+ # ...
+ # Connector settings:
+ mcollective::setting {
+ 'connector':
+ value => 'activemq';
+ 'direct_addressing':
+ value => '1';
+ 'plugin.activemq.pool.size':
+ value => '1';
+ 'plugin.activemq.pool.1.host':
+ value => $activemq_server;
+ 'plugin.activemq.pool.1.port':
+ value => '61614';
+ 'plugin.activemq.pool.1.user':
+ value => $activemq_user;
+ 'plugin.activemq.pool.1.password':
+ value => $activemq_password;
+ 'plugin.activemq.pool.1.ssl':
+ value => '1';
+ 'plugin.activemq.pool.1.ssl.fallback':
+ value => '1';
+ }
+ # ...
+{% endhighlight %}
+
+([↑ Back to top](#content))
+
+
+Required Settings
+-----
+
+### Connector Settings
+
+
+<pre><code><a href="#connector">connector</a> = activemq
+<a href="#directaddressing">direct_addressing</a> = 1
+
+# <a href="#activemq-connector-settings">ActiveMQ connector settings:</a>
+plugin.activemq.pool.size = 1
+plugin.activemq.pool.1.host = middleware.example.net
+plugin.activemq.pool.1.port = 61614
+plugin.activemq.pool.1.user = mcollective
+plugin.activemq.pool.1.password = secret
+plugin.activemq.pool.1.ssl = 1
+plugin.activemq.pool.1.ssl.ca = /var/lib/puppet/ssl/certs/ca.pem
+plugin.activemq.pool.1.ssl.cert = /var/lib/puppet/ssl/certs/web01.example.com.pem
+plugin.activemq.pool.1.ssl.key = /var/lib/puppet/ssl/private_keys/web01.example.com.pem
+plugin.activemq.pool.1.ssl.fallback = 0
+
+# <a href="#rabbitmq-connector-settings">RabbitMQ connector settings:</a>
+plugin.rabbitmq.vhost = /mcollective
+plugin.rabbitmq.pool.size = 1
+plugin.rabbitmq.pool.1.host = middleware.example.net
+plugin.rabbitmq.pool.1.port = 61613
+# ... etc., similar to activemq settings
+</code>
+</pre>
+
+
+MCollective always requires a connector plugin. The connector plugin is determined by the [middleware][] you chose for your deployment. Each connector plugin will have additional settings it requires.
+
+> #### Shared Configuration
+>
+> * All servers and clients must use the same connector plugin, and its settings must be configured compatibly.
+> * You must use the right connector plugin for your [choice of middleware][middleware].
+> * The hostname and port must match what the middleware is using. The username and password must be valid login accounts on the middleware. If you are using [CA-verified TLS][activemq_tls_verified], the certificate must be signed by the same CA the middleware is using.
+
+#### `connector`
+
+Which connector plugin to use. This is determined by your [choice of middleware][middleware].
+
+- _Default:_ `activemq`
+- _Allowed values:_ `activemq`, `rabbitmq`, or the name of a third-party connector plugin. {{ pluginname }}
+
+
+#### `direct_addressing`
+
+Whether your middleware supports direct point-to-point messages. **This should usually be turned on,** although it is off by default. The built-in `activemq` and `rabbitmq` connectors both support direct addressing, as does the external `redis` connector. (The older `stomp` connector, however, does not.)
+
+- _Default:_ `0`
+- _Allowed values:_ `1`, `0`, `y`, `n` --- {{ badbool }}
+
+#### ActiveMQ Connector Settings
+
+ActiveMQ is the main middleware we recommend for MCollective. The ActiveMQ connector can use multiple servers as a failover pool; if you have only one server, just use a pool size of `1`.
+
+> **Note:** This is only a summary of the most commonly used ActiveMQ settings; there are about ten more settings that can be used to tune the connector's performance. [See the ActiveMQ connector documentation][activemq_connector] for more complete details.
+
+- **`plugin.activemq.pool.size`** --- How many ActiveMQ servers to attempt to use. _Default:_ (nothing)
+- **`plugin.activemq.pool.1.host`** --- The hostname of the first ActiveMQ server. (Note that additional servers use the same settings as the first, incrementing the number.) _Default:_ (nothing)
+- **`plugin.activemq.pool.1.port`** --- The Stomp port of the first ActiveMQ server. _Default:_ `61613` or `6163`, depending on the MCollective version.
+- **`plugin.activemq.pool.1.user`** --- The ActiveMQ user account to connect as. If the `STOMP_USER` environment variable is set, MCollective will use its value instead of this setting.
+- **`plugin.activemq.pool.1.password`** --- The password for the user account being used. If the `STOMP_PASSWORD` environment variable is set, MCollective will use its value instead of this setting.
+- **`plugin.activemq.pool.1.ssl`** --- Whether to use TLS when connecting to ActiveMQ. _Default:_ `0`; _allowed:_ `1/0`, `true/false`, `yes/no`
+- **`plugin.activemq.pool.1.ssl.fallback`** --- _(When `ssl == 1`)_ Whether to allow unverified TLS if the ca/cert/key settings aren't set. _Default:_ `0`; _allowed:_ `1/0`, `true/false`, `yes/no`
+- **`plugin.activemq.pool.1.ssl.ca`** --- _(When `ssl == 1`)_ The CA certificate to use when verifying ActiveMQ's certificate. Must be the path to a `.pem` file. _Default:_ (nothing)
+- **`plugin.activemq.pool.1.ssl.cert`** --- _(When `ssl == 1`)_ The certificate to present when connecting to ActiveMQ. Must be the path to a `.pem` file. _Default:_ (nothing)
+- **`plugin.activemq.pool.1.ssl.key`** --- _(When `ssl == 1`)_ The private key corresponding to this node's certificate. Must be the path to a `.pem` file. _Default:_ (nothing)
+
+#### RabbitMQ Connector Settings
+
+The RabbitMQ connector uses very similar settings to the ActiveMQ connector, with the same `.pool.1.host` style of setting names.
+
+[See the RabbitMQ connector documentation][rabbitmq_connector] for more complete details.
+
+
+([↑ Back to top](#content))
+
+
+### Security Plugin Settings
+
+<pre><code><a href="#securityprovider">securityprovider</a> = ssl
+
+# <a href="#ssl-plugin-settings">SSL plugin settings:</a>
+plugin.ssl_client_cert_dir = /etc/mcollective.d/clients
+plugin.ssl_server_private = /etc/mcollective.d/server_private.pem
+plugin.ssl_server_public = /etc/mcollective.d/server_public.pem
+
+# <a href="#psk-plugin-settings">PSK plugin settings:</a>
+plugin.psk = j9q8kx7fnuied9e
+</code>
+</pre>
+
+MCollective always requires a security plugin. (Although they're called security plugins, they actually handle more, including message serialization.) Each security plugin will have additional settings it requires.
+
+> #### Shared Configuration
+>
+> All servers and clients must use the same security plugin, and its settings must be configured compatibly.
+
+It's possible to write new security plugins, but most people use one of the three included in MCollective:
+
+- **SSL:** The best choice for most users. Provides very good security when combined with TLS on the connector plugin (see above).
+- **PSK:** Poor security, but easy to configure; fine for proof-of-concept deployments.
+- **AES:** Complex to configure, and carries a noticable performance cost in large networks. Only suitable for certain use cases, like where TLS on the middleware is impossible.
+
+For a full-system look at how security works in MCollective, see [Security Overview][security_overview].
+
+
+#### `securityprovider`
+
+Which security plugin to use.
+
+- _Default:_ `psk`
+- _Allowed values:_ `ssl`, `psk`, `aes_security`, or the name of a third-party security plugin. {{ pluginname }}
+
+#### SSL Plugin Settings
+
+> **Note:** This security plugin requires you to manage and distribute SSL credentials. [See the SSL security plugin page][ssl_plugin] for full details. In summary:
+>
+> * All servers share **one** "server" keypair. They must all have a copy of the public key and private key.
+> * Every admin user must have a copy of the server public key.
+> * Every admin user has their own "client" keypair.
+> * Every server must have a copy of **every** authorized client public key.
+
+All of these settings have **no default,** and must be set for the SSL plugin to work.
+
+- **`plugin.ssl_server_private`** --- The path to the server private key file, which must be in `.pem` format.
+- **`plugin.ssl_server_public`** --- The path to the server public key file, which must be in `.pem` format.
+- **`plugin.ssl_client_cert_dir`** --- A directory containing every authorized client public key.
+
+
+#### PSK Plugin Settings
+
+> **Note:** The only credential used by this plugin is a single shared password, which all servers and admin users have a copy of.
+
+- **`plugin.psk`** --- The shared passphrase. If the `MCOLLECTIVE_PSK` environment variable is set, MCollective will use its value instead of this setting.
+
+
+([↑ Back to top](#content))
+
+
+Facts, Identity, and Classes
+-----
+
+<pre><code><a href="#factsource">factsource</a> = yaml
+<a href="#pluginyaml">plugin.yaml</a> = /etc/mcollective/facts.yaml
+<a href="#factcachetime">fact_cache_time</a> = 300
+
+<a href="#identity">identity</a> = web01.example.com
+
+<a href="#classesfile">classesfile</a> = /var/lib/puppet/state/classes.txt
+</code>
+</pre>
+
+MCollective clients use filters to discover nodes and limit commands. (See [Discovery Filters][filters] for more details.) These filters can use a variety of per-server metadata, including **facts, identity,** and **classes.**
+
+* **Facts:** A collection of key/value data about a server's hardware and software. (E.g. `memorytotal = 8.00 GB`, `kernel = Darwin`, etc.)
+* **Identity:** The name of the node.
+* **Classes:** The Puppet classes (or Chef cookbooks, etc.) applied to this node. Classes are useful as metadata because they describe what _roles_ a server fills at your site.
+
+None of these settings are mandatory, but MCollective is less useful without them.
+
+#### `identity`
+
+The node's name or identity. This **should** be unique for each node, but does not **need** to be.
+
+- _Default:_ The value of Ruby's `Socket.gethostname` method, which is usually the server's FQDN.
+- _Sample value:_ `web01.example.com`
+- _Allowed values:_ Any string containing only alphanumeric characters, hyphens, and dots --- i.e. matching the regular expression `/\A[\w\.\-]+\Z/`
+
+#### `classesfile`
+
+A file with a list of classes applied by your configuration management system. This should be a plain text file containing one class name per line.
+
+Puppet automatically writes a class file, which can be found by running `sudo puppet apply --configprint classfile`. [Chef can be made to write a class file][chef_classfile].
+
+- _Default:_ `/var/lib/puppet/state/classes.txt`
+
+
+#### `factsource`
+
+Which fact plugin to use.
+
+MCollective includes a fact plugin called `yaml`. Most users should use this default, set [the `plugin.yaml` setting (see below)](#pluginyaml), and arrange to fill the file it relies on.
+
+Other fact plugins (including [Facter][facter_plugin] and [Ohai][ohai_plugin] ones) are available in the [plugin directory][plugin_directory]. These may require settings of their own.
+
+- _Default:_ `yaml`
+- _Allowed values:_ The name of any installed fact plugin, with the `_facts` suffix trimmed off. {{ pluginname }}
+
+#### `plugin.yaml`
+
+_When `factsource == yaml`_
+
+The fact file(s) to load for [the default `yaml` fact plugin](#factsource).
+
+- _Default:_ (nothing)
+- _Sample value:_ `/etc/mcollective/facts.yaml`
+- _Valid values:_ A single path, or a list of paths separated by the {{ path_separator }}.
+
+**Notes:**
+
+The default `yaml` fact plugin reads cached facts from a file, which should be a simple YAML hash. If multiple files are specified, they will be merged. (Later files override prior ones if there are conflicting values.)
+
+**The user is responsible for populating the fact file;** by default it is empty, and MCollective has no facts.
+
+If you are using Puppet and Facter, you can populate it by putting something like the following in your puppet master's `/etc/puppet/manifests/site.pp`, outside any node definition:
+
+{% highlight ruby %}
+ # /etc/puppet/manifests/site.pp
+ file{"/etc/mcollective/facts.yaml":
+ owner => root,
+ group => root,
+ mode => 400,
+ loglevel => debug, # reduce noise in Puppet reports
+ content => inline_template("<%= scope.to_hash.reject { |k,v| k.to_s =~ /(uptime_seconds|timestamp|free)/ }.to_yaml %>"), # exclude rapidly changing facts
+ }
+{% endhighlight %}
+
+#### `fact_cache_time`
+
+How long (in seconds) to cache fact results before refreshing from source. This can be ignored unless you're using a non-default `factsource`.
+
+- _Default:_ `300`
+
+([↑ Back to top](#content))
+
+
+Optional Features
+-----
+
+### Subcollectives
+
+<pre><code><a href="#collectives">collectives</a> = mcollective,uk_collective
+<a href="#maincollective">main_collective</a> = mcollective
+</code>
+</pre>
+
+Subcollectives provide an alternate way of dividing up the servers in a deployment. They are mostly useful because the middleware can be made aware of them, which enables traffic flow and access restrictions. In multi-datacenter deployments, this can save bandwidth costs and give some extra security.
+
+* [See the Subcollectives and Partitioning page][subcollectives] for more details and an example of site partitioning.
+
+Subcollective membership is managed **on the server side,** by each server's config file. A given server can join any number of collectives, and will respond to commands from any of them.
+
+> #### Shared Configuration
+>
+> If you are using any additional collectives (besides the default `mcollective` collective), your middleware must be configured to permit traffic on those collectives. See the middleware deployment guide for your specific middleware to see how to do this:
+>
+> * ActiveMQ: [Subcollective topic/queue names](/mcollective/deploy/middleware/activemq.html#topic-and-queue-names) --- [Multi-subcollective authorization example](/mcollective/deploy/middleware/activemq.html#detailed-restrictions-with-multiple-subcollectives)
+
+
+#### `collectives`
+
+A comma-separated list (spaces OK) of [subcollectives][] this server should join.
+
+- _Default:_ `mcollective`
+- _Sample value:_ `mcollective,uk_collective`
+
+#### `main_collective`
+
+The main collective for this server. Currently, this is only used as the default value for the [`registration_collective`](#registrationcollective) setting.
+
+- _Default:_ (the first value of [the `collectives` setting](#collectives), usually `mcollective`)
+
+
+([↑ Back to top](#content))
+
+
+### Node Registration
+
+<pre><code><a href="#registerinterval">registerinterval</a> = 300
+<a href="#registration">registration</a> = agentlist
+<a href="#registrationcollective">registration_collective</a> = mcollective
+</code>
+</pre>
+
+By default, registration is disabled, due to [`registerinterval`](#registerinterval) being set to 0.
+
+Optionally, MCollective servers can [send periodic heartbeat messages][registration] containing some inventory information. This can provide a central inventory at sites that don't already use something like [PuppetDB][], and can also be used as a simple passive monitoring system.
+
+The default registration plugin, `agentlist`, sends a standard SimpleRPC command over the MCollective middleware, to be processed by some server with an agent called `registration` installed. You would need to ensure that the `registration` agent is extremely performant (due to the volume of message it will receive) and installed on a limited number of servers. If your [middleware][] supports detailed permissions, you must also ensure that it allows servers to send commands to the registration agent ([ActiveMQ instructions](/mcollective/deploy/middleware/activemq.html#detailed-restrictions)).
+
+Some registration plugins (e.g. `redis`) can insert data directly into the inventory instead of sending an RPC message. This is a flexible system, and the user is in charge of deciding what to build with it, if anything. If all you need is a searchable inventory, [PuppetDB][] is probably closer to your needs.
+
+#### `registerinterval`
+
+How long (in seconds) to wait between registration messages. Setting this to 0 disables registration.
+
+- _Default:_ `0`
+
+#### `registration`
+
+The [registration plugin][registration] to use.
+
+This plugin must be installed on the server sending the message, and will dictate what the message contains. The default `agentlist` plugin will only send a list of the installed agents. See [Registration Plugins][registration] for more details.
+
+- _Default:_ `agentlist`
+- _Allowed values:_ The name of any installed registration plugin. {{ pluginname }}
+
+#### `registration_collective`
+
+Which subcollective to send registration messages to, when using a SimpleRPC-based registration plugin.
+
+- _Default:_ (the value of [`main_collective`](#maincollective), usually `mcollective`)
+
+
+([↑ Back to top](#content))
+
+
+### Auditing
+
+<pre><code><a href="#rpcaudit">rpcaudit</a> = 1
+<a href="#rpcauditprovider">rpcauditprovider</a> = logfile
+<a href="#pluginrpcauditlogfile">plugin.rpcaudit.logfile</a> = /var/log/mcollective-audit.log
+</code>
+</pre>
+
+Optionally, MCollective can log the SimpleRPC agent commands it receives from admin users, recording both the command itself and some identifying information about the user who issued it. The caller ID of a command is determined by the [security plugin][security_plugin] being used.
+
+MCollective ships with a local logging audit plugin, called `Logfile`, which saves audit info to a local file (different from the daemon log file). Log lines look like this:
+
+ 2010-12-28T17:09:03.889113+0000: reqid=319719cc475f57fda3f734136a31e19b: reqtime=1293556143 caller=cert=nagios@monitor1 agent=nrpe action=runcommand data={:process_results=>true, :command=>"check_mailq"}
+
+Tthere are central loggers available from [the plugin directory][plugin_directory], and you can also write your own audit plugins; see [Writing Auditing Plugins][auditing] for more information.
+
+
+#### `rpcaudit`
+
+Whether to enable [SimpleRPC auditing][Auditing] for all SimpleRPC agent commands.
+
+- _Default:_ `0`
+- _Allowed values:_ `1`, `0`, `y`, `n` --- {{ badbool }}
+
+#### `rpcauditprovider`
+
+The name of the audit plugin to use whenever SimpleRPC commands are received.
+
+- _Default:_ (nothing)
+- _Sample value:_ `logfile`
+- _Allowed values:_ The name of any installed audit plugin. {{ pluginname }}
+
+
+#### `plugin.rpcaudit.logfile`
+
+_When `rpcauditprovider == logfile`_
+
+The file to write to when using the `logfile` audit plugin. **Note:** this file is not automatically rotated.
+
+- _Default:_ `/var/log/mcollective-audit.log`
+
+
+([↑ Back to top](#content))
+
+
+### Authorization
+
+<pre><code><a href="#rpcauthorization">rpcauthorization</a> = 1
+<a href="#rpcauthprovider">rpcauthprovider</a> = action_policy
+</code>
+</pre>
+
+Optionally, MCollective can refuse to execute agent commands unless they meet some requirement. The exact requirement is defined by an [authorization plugin][authorization].
+
+See [SimpleRPC Authorization][authorization] for more details, including how to enable authorization for only certain agents.
+
+The [actionpolicy][] plugin, which is provided in the [plugin directory][plugin_directory], is fairly popular and seems to meet many users' needs, when combined with a [security plugin][security_plugin] that provides a verified caller ID (such as the SSL plugin). [See its documentation][actionpolicy] for details.
+
+#### `rpcauthorization`
+
+Whether to enable [SimpleRPC authorization][Authorization] globally.
+
+- _Default:_ `0`
+- _Allowed values:_ `1`, `0`, `y`, `n` --- {{ badbool }}
+
+#### `rpcauthprovider`
+
+The plugin to use when globally managing authorization. See [SimpleRPC Authorization][authorization] for more details.
+
+- _Default:_ (nothing)
+- _Sample value:_ `action_policy`
+- _Allowed values:_ The name of any installed authorization plugin. This uses different capitalization/formatting rules from the other plugin settings: if the name of the plugin (in the code) has any interior capital letters (e.g. `ActionPolicy`), you should use a lowercase value for the setting but insert an underscore before the place where the interior capital letter(s) would have gone (e.g. `action_policy`). If the name contains no interior capital letters, simply use a lowercase value with no other changes.
+
+
+([↑ Back to top](#content))
+
+
+Advanced Settings
+-----
+
+### Logging
+
+<pre><code><a href="#loggertype">logger_type</a> = file
+<a href="#loglevel">loglevel</a> = info
+<a href="#logfile">logfile</a> = /var/log/mcollective.log
+<a href="#keeplogs">keeplogs</a> = 5
+<a href="#maxlogsize">max_log_size</a> = 2097152
+<a href="#logfacility">logfacility</a> = user
+</code>
+</pre>
+
+The MCollective server daemon can log to its own log file (which it will automatically rotate), or to the syslog. It can also log directly to the console, if you are running it in the foreground instead of daemonized.
+
+Some of the settings below only apply if you are using log files, and some only apply if you are using syslog.
+
+#### `logger_type`
+
+How the MCollective server daemon should log. You generally want to use a file or syslog.
+
+- _Default:_ `file`
+- _Allowed values:_ `file`, `syslog`, `console`
+
+#### `loglevel`
+
+How verbosely to log.
+
+- _Default:_ `info`
+- _Allowed values:_ In increasing order of verbosity: `fatal`, `error` , `warn`, `info`, `debug`
+
+#### `logfile`
+
+_When `logger_type == file`_
+
+Where the log file should be stored.
+
+- _Default:_ (nothing; the package's default config file usually sets a platform-appropriate value)
+- _Sample value:_ `/var/log/mcollective.log`
+
+#### `keeplogs`
+
+_When `logger_type == file`_
+
+The number of log files to keep when rotating.
+
+- _Default:_ `5`
+
+#### `max_log_size`
+
+_When `logger_type == file`_
+
+Max size in bytes for log files before rotation happens.
+
+- _Default:_ `2097152`
+
+#### `logfacility`
+
+_When `logger_type == syslog`_
+
+The syslog facility to use.
+
+- _Default:_ `user`
+
+
+([↑ Back to top](#content))
+
+
+### Platform Defaults
+
+<pre><code><a href="#daemonize">daemonize</a> = 1
+<a href="#libdir">libdir</a> = /usr/libexec/mcollective
+<a href="#sslcipher">ssl_cipher</a> = aes-256-cbc
+</code>
+</pre>
+
+These settings generally shouldn't be changed by the user, but their values may vary by platform. The package you used to install MCollective should have created a config file with platform-appropriate values for these settings.
+
+#### `daemonize`
+
+Whether to fork and run the MCollective server daemon in the background.
+
+This depends on your platform's init system. For example, newer Ubuntu releases require this to be false, while RHEL-derived systems require it to be true.
+
+- _Default:_ `0` <!-- Actually nil but acts like false -->
+- _Allowed values:_ `1`, `0`, `y`, `n` --- {{ badbool }}
+
+#### `libdir`
+
+Where to look for plugins. Should be a single path or a list of paths separated by the {{ path_separator }}.
+
+By default, this setting is blank, but the package you installed MCollective with should supply a default server.cfg file with a platform-appropriate value for this setting. **If server.cfg has no value for this setting, MCollective will not work.**
+
+- _Default:_ (nothing; the package's default config file usually sets a platform-appropriate value)
+- _Sample value:_ `/usr/libexec/mcollective:/opt/mcollective`
+
+#### `ssl_cipher`
+
+The cipher to use for encryption. This is currently only relevant if you are using the [AES security plugin][security_aes].
+
+This setting should be a standard OpenSSL cipher string. See `man enc` for a list.
+
+- _Default:_ `aes-256-cbc`
--- /dev/null
+---
+title: "MCollective » Deploy » Middleware » ActiveMQ Config"
+subtitle: "ActiveMQ Config Reference for MCollective Users"
+layout: default
+---
+
+[Wildcard]: http://activemq.apache.org/wildcards.html
+[minimal_example]: http://github.com/puppetlabs/marionette-collective/raw/master/ext/activemq/examples/single-broker/activemq.xml
+[maximal_example]: https://github.com/puppetlabs/marionette-collective/tree/master/ext/activemq/examples/multi-broker
+[template_example]: TODO
+[apache_activemq_config_docs]: http://activemq.apache.org/version-5-xml-configuration.html
+
+[subcollectives]: /reference/basic/subcollectives.html
+[activemq_connector]: /reference/plugins/connector_activemq.html
+[mcollective_connector_tls]: /mcollective/reference/integration/activemq_ssl.html
+
+
+Summary
+-----
+
+Apache ActiveMQ is the primary middleware we recommend with MCollective. It's good software, but its XML config file is large and unwieldy, and you may need to edit many sections of it in a complex MCollective deployment. This reference guide attempts to describe every major ActiveMQ setting that matters to MCollective.
+
+### How to Use This Page
+
+* This page **doesn't** describe the complete format of the activemq.xml config file, and will sometimes use incomplete shorthand to describe elements of it.
+* You should definitely refer to an example config file while reading, so you can see each element's full syntax in context.
+* You don't need to read this entire page when setting up a new deployment. We recommend that you:
+ * Start with an example config file (see directly below).
+ * Make heavy use of the table of contents above.
+ * Skim the sections of this page you currently care about, and edit your config as needed.
+ * Refer back to this page later when you need to expand your broker infrastructure.
+* If you are a new user, we recommend that you:
+ * Start with the [single-broker example config][minimal_example].
+ * Change the [user account passwords](#authentication-users-and-groups).
+ * [Set up TLS](#tls-credentials).
+
+### Example Config Files
+
+We have several.
+
+* [Minimal config example][minimal_example] --- single broker <!-- , minimal authorization. -->
+* [Maximal config example][maximal_example] --- multi-broker <!-- with extensive authorization. -->
+
+<!--
+* [Template-based example][template_example] --- reduces configuration down to a handful of variables; shows how those few decisions ramify into many config edits.
+ -->
+
+
+> **Note:** Some config data needs to be set in both MCollective and ActiveMQ; your configuration of one will affect the other. In this page, we call out that information with headers labeled "Shared Configuration."
+
+### Version Limits
+
+This document is about the "new" MCollective/ActiveMQ interface, which means it requires the following:
+
+* MCollective 2.0.0 or newer
+* ActiveMQ 5.5.0 or newer
+* Stomp gem 1.2.2 or newer
+* The [activemq connector][activemq_connector] plugin (included with MCollective 2.0.0 and newer)
+
+([↑ Back to top](#content))
+
+How MCollective Uses ActiveMQ
+-----
+
+MCollective connects to ActiveMQ over the Stomp protocol, and presents certain credentials:
+
+* It provides a username and password, with which ActiveMQ can do what it pleases.
+* If TLS is in use, it will also present a certificate (and verify the ActiveMQ server's certificate).
+
+Once allowed to connect, MCollective will use the Stomp protocol to create subscriptions. It will then produce and consume a lot of traffic on queues and topics whose names begin with `mcollective`. (See "Topic and Queue Names" directly below.)
+
+### Absolute Minimum Requirements
+
+ActiveMQ defaults to believing that it is routing traffic between processes in a single JVM instance: it doesn't assume it is connected to the network, and it uses a loose-to-nonexistent security model.
+
+This means that if you do nothing but enable Stomp traffic, MCollective will work fine. (Albeit with terrible security and eventual out-of-control memory usage.)
+
+### Topic and Queue Names
+
+MCollective uses the following destination names. This list uses standard [ActiveMQ destination wildcards][wildcard]. "COLLECTIVE" is the name of the collective being used; by default, this is `mcollective`, but if you are using [subcollectives][], each one is implemented as an equal peer of the default collective.
+
+Topics:
+
+- `ActiveMQ.Advisory.>` (built-in topics that all ActiveMQ producers and consumers need all permissions on)
+- `COLLECTIVE.*.agent` (for each agent plugin, where the `*` is the name of the plugin)
+
+Queues:
+
+- `COLLECTIVE.nodes` (used for direct addressing; this is a single destination that uses JMS selectors, rather than a group of destinations)
+- `COLLECTIVE.reply.>` (where the continued portion is a request ID)
+
+> #### Shared Configuration
+>
+> Subcollectives must also be configured in the MCollective client and server config files. ActiveMQ must allow traffic on any subcollective that MCollective servers and clients expect to use.
+
+([↑ Back to top](#content))
+
+Config File Location and Format
+-----
+
+ActiveMQ's config is usually called activemq.xml, and is kept in ActiveMQ's configuration directory (`/etc/activemq` with Puppet Labs's Red Hat-like packages, or a subdirectory of `/etc/activemq/instances-enabled` with the standard Debian or Ubuntu packages). Any other files referenced in activemq.xml will be looked for in the same directory.
+
+The config file is in Java's Beans XML format. Note that all of the settings relevant to MCollective are located inside activemq.xml's `<broker>` element.
+
+This document won't describe the complete format of the activemq.xml config file, and will sometimes use incomplete shorthand to describe elements of it. You should definitely [refer to an example config file](#example-config-files) while reading, so you can see each element's full syntax in context.
+
+You can also read external documentation for a more complete understanding.
+
+> **Bug Warning:** In ActiveMQ 5.5, the first-level children of the `<broker>` element must be arranged in alphabetical order. There is no good reason for this behavior, and it was fixed in ActiveMQ 5.6.
+
+### External ActiveMQ Documentation
+
+The Apache ActiveMQ documentation contains important information, but it is often incomplete, badly organized, and confusing. The Fuse documentation (a commercially supported release of ActiveMQ) is significantly better written and better organized, although it requires signing up for an email newsletter, but it may be out of sync with the most recent ActiveMQ releases.
+
+* [Apache ActiveMQ Documentation][apache_activemq_config_docs]
+* [Fuse Documentation](http://fusesource.com/documentation/fuse-message-broker-documentation/)
+
+### Wildcards
+
+You'll see a lot of [ActiveMQ destination wildcards][wildcard] below. In short:
+
+* Segments in a destination name are separated with dots (`.`)
+* A `*` represents _one segment_ (i.e. any sequence of non-dot characters)
+* A `>` represents _the whole rest of the name_ after a prefix
+
+
+([↑ Back to top](#content))
+
+Required Settings
+-----
+
+One way or another, you must set all of the following.
+
+### Transport Connector(s)
+
+It's generally best to only enable the transport connectors you need. For example, if you're using Stomp over TLS, don't leave a bare Stomp transport open. If you're not using a network of brokers, close the OpenWire transport.
+
+The `name` attribute of a transport connector doesn't seem to matter as long as it's locally unique.
+
+#### Stomp
+
+ActiveMQ must listen over the network for Stomp connections; otherwise, MCollective can't reach it. Enable this with a `<transportConnector>` element inside the `<transportConnectors>` element. We generally recommend using TLS.
+
+{% highlight xml %}
+ <transportConnectors>
+ <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613"/>
+ </transportConnectors>
+{% endhighlight %}
+
+* Note that the protocol/port/arguments for Stomp URIs can differ:
+ * Unencrypted: `stomp+nio://0.0.0.0:61613`
+ * CA-verified TLS: `stomp+ssl://0.0.0.0:61614?needClientAuth=true`
+ * Anonymous TLS: `stomp+ssl://0.0.0.0:61614`
+* You can choose to restrict the interface/hostname to use instead of listening on `0.0.0.0`.
+
+> **If you are using TLS,** note that you must also:
+>
+> * [Configure ActiveMQ's TLS credentials](#tls-credentials) (see below)
+> * [Configure MCollective to use TLS credentials][mcollective_connector_tls]
+
+#### OpenWire
+
+If you are using a network of brokers instead of just one ActiveMQ server, they talk to each other over OpenWire, and will all need a transport connector for that protocol too:
+
+{% highlight xml %}
+ <transportConnector name="openwire+ssl" uri="ssl://0.0.0.0:61617?needClientAuth=true"/>
+{% endhighlight %}
+
+* Note that the protocol/port/arguments for OpenWire URIs can differ:
+ * Unencrypted: `tcp://0.0.0.0:61616`
+ * CA-verified TLS: `ssl://0.0.0.0:61617?needClientAuth=true`
+ * Anonymous TLS: `ssl://0.0.0.0:61617`
+* You can choose to restrict the interface/hostname to use instead of listening on `0.0.0.0`.
+
+
+
+#### Standard Ports for Stomp and OpenWire
+
+Alas, there aren't any; just a rough consensus.
+
+* 61613 for unencrypted Stomp
+* 61614 for Stomp with TLS
+* 61616 for unencrypted OpenWire
+* 61617 for OpenWire with TLS
+
+All of our documentation assumes these ports.
+
+> #### Shared Configuration
+>
+> MCollective needs to know the following:
+>
+> * The port to use for Stomp traffic
+> * The hostname or IP address to reach ActiveMQ at
+> * Whether to use TLS
+>
+> In a network of brokers, the other ActiveMQ servers need to know the following:
+>
+> * The port to use for OpenWire traffic
+> * The hostname or IP address to reach peer ActiveMQ servers at
+> * Whether to use TLS
+
+
+### Reply Queue Pruning
+
+MCollective sends replies on uniquely-named, single-use queues with names like `mcollective.reply.<UNIQUE ID>`. These have to be deleted after about five minutes, lest they clog up ActiveMQ's available memory. By default, queues live forever, so you have to configure this.
+
+Use a `<policyEntry>` element for `*.reply.>` queues, with `gcInactiveDestinations` set to true and `inactiveTimoutBeforeGC` set to 300000 ms (five minutes).
+
+{% highlight xml %}
+ <destinationPolicy>
+ <policyMap>
+ <policyEntries>
+ <policyEntry queue="*.reply.>" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" />
+ <policyEntry topic=">" producerFlowControl="false"/>
+ </policyEntries>
+ </policyMap>
+ </destinationPolicy>
+{% endhighlight %}
+
+### Disable Producer Flow Control on Topics
+
+In the example above, you can also see that `producerFlowControl` is set to false for all topics. This is highly recommended; setting it to true can cause MCollective servers to appear blocked when there's heavy traffic.
+
+
+([↑ Back to top](#content))
+
+Recommended Settings
+-----
+
+### TLS Credentials
+
+If you are using TLS in either your Stomp or OpenWire [transport connectors](#transport-connectors), ActiveMQ needs a keystore file, a truststore file, and a password for each:
+
+{% highlight xml %}
+ <sslContext>
+ <sslContext
+ keyStore="keystore.jks" keyStorePassword="secret"
+ trustStore="truststore.jks" trustStorePassword="secret"
+ />
+ </sslContext>
+{% endhighlight %}
+
+**Note:** This example is for CA-verified TLS. If you are using anonymous TLS, you may optionally skip the truststore attributes.
+
+The redundant nested `<sslContext>` elements are not a typo; for some reason ActiveMQ actually needs this.
+
+ActiveMQ will expect to find these files in the same directory as activemq.xml.
+
+> #### Creating a Keystore and Truststore
+>
+> There is a [separate guide that covers how to create keystores.][activemq_keystores]
+
+[activemq_keystores]: ./activemq_keystores.html
+
+
+### Authentication (Users and Groups)
+
+When they connect, MCollective clients and servers provide a username, password, and optionally an SSL certificate. ActiveMQ can use any of these to authenticate them.
+
+By default, ActiveMQ ignores all of these and has no particular concept of "users." Enabling authentication means ActiveMQ will only allow users with proper credentials to connect. It also gives you the option of setting up per-destination authorization (see below).
+
+You set up authentication by adding the appropriate element to the `<plugins>` element. [The Fuse documentation has a more complete description of ActiveMQ's authentication capabilities;][fuse_security] the [ActiveMQ docs version][activemq_security] is less organized and less complete. In summary:
+
+- `simpleAuthenticationPlugin` defines users directly in activemq.xml. It's well-tested and easy. It also requires you to edit activemq.xml and restart the broker every time you add a new user. The activemq.xml file will contain sensitive passwords and must be protected.
+- `jaasAuthenticationPlugin` lets you use external text files (or even an LDAP database) to define users and groups. You need to make a `login.config` file in the ActiveMQ config directory, and possibly two more files. You can add users and groups without restarting the broker. The external users file will contain sensitive passwords and must be protected.
+- `jaasCertificateAuthenticationPlugin` ignores the username and password that MCollective presents; instead, it reads the distinguished name of the certificate and maps that to a username. It requires TLS, a `login.config` file, and two other external files. It is also impractical unless your servers are all using the same SSL certificate to connect to ActiveMQ; the currently recommended approach of re-using Puppet certificates makes this problematic, but you can probably ship credentials around and figure out a way to make it work. This is not currently well-tested with MCollective.
+
+[fuse_security]: http://fusesource.com/docs/broker/5.5/security/front.html
+[activemq_security]: http://activemq.apache.org/security.html
+
+The example below uses `simpleAuthenticationPlugin`.
+
+{% highlight xml %}
+ <plugins>
+ <simpleAuthenticationPlugin>
+ <users>
+ <authenticationUser username="mcollective" password="marionette" groups="mcollective,everyone"/>
+ <authenticationUser username="admin" password="secret" groups="mcollective,admins,everyone"/>
+ </users>
+ </simpleAuthenticationPlugin>
+ <!-- ... authorization goes below... -->
+ </plugins>
+{% endhighlight %}
+
+This creates two users, with the expectation that MCollective servers will log in as `mcollective` and admin users issuing commands will log in as `admin`.
+
+Note that unless you set up authorization (see below), these users have the exact same capabilities.
+
+> #### Shared Configuration
+>
+> MCollective servers and clients both need a username and password to use when connecting. That user **must** have appropriate permissions (see "Authorization," directly below) for that server or client's role.
+
+
+
+### Authorization (Group Permissions)
+
+By default, ActiveMQ allows everyone to **read** from any topic or queue, **write** to any topic or queue, and create (**admin**) any topic or queue.
+
+By setting rules in an `<authorizationPlugin>` element, you can regulate things a bit. Some notes:
+
+* Authorization is done **by group.**
+* The exact behavior of authorization doesn't seem to be documented anywhere. Going by observation, it appears that ActiveMQ first tries the most specific rule available, then retreats to less specific rules. This means if a given group isn't allowed an action by a more specific rule but is allowed it by a more general rule, it still gets authorized to take that action. If you have any solid information about how this works, please email us at <faq@puppetlabs.com>.
+* MCollective creates subscriptions before it knows whether there will be any content coming. That means any role able to **read** from or **write** to a destination must also be able to **admin** that destination. Think of "admin" as a superset of both read and write.
+
+#### Simple Restrictions
+
+The following example grants all permissions on destinations beginning with `mcollective` to everyone in group `mcollective`:
+
+{% highlight xml %}
+ <plugins>
+ <!-- ...authentication stuff... -->
+ <authorizationPlugin>
+ <map>
+ <authorizationMap>
+ <authorizationEntries>
+ <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <authorizationEntry queue="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
+ </authorizationEntries>
+ </authorizationMap>
+ </map>
+ </authorizationPlugin>
+ </plugins>
+{% endhighlight %}
+
+This means admins can issue commands and MCollective servers can read those commands and reply. However, it also means that servers can issue commands, which you probably don't want.
+
+Note that the `everyone` group (as seen in the `ActiveMQ.Advisory.>` topics) **isn't special.** You need to manually make sure all users are members of it. ActiveMQ does not appear to have any kind of real wildcard "everyone" group, unfortunately.
+
+#### Detailed Restrictions
+
+The following example splits permissions along a simple user/server model:
+
+{% highlight xml %}
+ <plugins>
+ <!-- ...authentication stuff... -->
+ <authorizationPlugin>
+ <map>
+ <authorizationMap>
+ <authorizationEntries>
+ <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry queue="mcollective.>" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic="mcollective.>" write="admins" read="admins" admin="admins" />
+ <authorizationEntry queue="mcollective.nodes" read="mcollective" admin="mcollective" />
+ <authorizationEntry queue="mcollective.reply.>" write="mcollective" admin="mcollective" />
+ <authorizationEntry topic="mcollective.*.agent" read="mcollective" admin="mcollective" />
+ <authorizationEntry topic="mcollective.registration.agent" write="mcollective" read="mcollective" admin="mcollective" />
+ <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
+ </authorizationEntries>
+ </authorizationMap>
+ </map>
+ </authorizationPlugin>
+ </plugins>
+{% endhighlight %}
+
+This means admins can issue commands and MCollective servers can read those commands and reply. This time, though, servers can't issue commands. The exception is the `mcollective.registration.agent` destination, which servers DO need the ability to write to if you've turned on registration.
+
+Admins, of course, can also read commands and reply, since they have power over the entire `mcollective.>` destination set. This isn't considered much of an additional security risk, considering that admins can already control your entire infrastructure.
+
+#### Detailed Restrictions with Multiple Subcollectives
+
+Both of the above examples assume only a single `mcollective` collective. If you are using additional [subcollectives][] (e.g. `uk_collective`, `us_collective`, etc.), their destinations will start with their name instead of `mcollective`. If you need to separately control authorization for each collective, it's best to use a template to do so, so you can avoid repeating yourself.
+
+{% highlight xml %}
+ <plugins>
+ <!-- ...authentication stuff... -->
+ <authorizationPlugin>
+ <map>
+ <authorizationMap>
+ <authorizationEntries>
+ <!-- "admins" group can do anything. -->
+ <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
+
+ <%- @collectives.each do |collective| -%>
+ <authorizationEntry queue="<%= collective %>.>" write="admins,<%= collective %>-admins" read="admins,<%= collective %>-admins" admin="admins,<%= collective %>-admins" />
+ <authorizationEntry topic="<%= collective %>.>" write="admins,<%= collective %>-admins" read="admins,<%= collective %>-admins" admin="admins,<%= collective %>-admins" />
+ <authorizationEntry queue="<%= collective %>.nodes" read="servers,<%= collective %>-servers" admin="servers,<%= collective %>-servers" />
+ <authorizationEntry queue="<%= collective %>.reply.>" write="servers,<%= collective %>-servers" admin="servers,<%= collective %>-servers" />
+ <authorizationEntry topic="<%= collective %>.*.agent" read="servers,<%= collective %>-servers" admin="servers,<%= collective %>-servers" />
+ <authorizationEntry topic="<%= collective %>.registration.agent" write="servers,<%= collective %>-servers" read="servers,<%= collective %>-servers" admin="servers,<%= collective %>-servers" />
+ <%- end -%>
+
+ <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
+ </authorizationEntries>
+ </authorizationMap>
+ </map>
+ </authorizationPlugin>
+ </plugins>
+{% endhighlight %}
+
+This example divides your users into several groups:
+
+* `admins` is the "super-admins" group, who can command all servers.
+* `servers` is the "super-servers" group, who can receive and respond to commands on any collective they believe themselves to be members of.
+* `COLLECTIVE-admins` can only command servers on their specific collective. (Since all servers are probably members of the default `mcollective` collective, the `mcollective-admins` group are sort of the "almost-super-admins.")
+* `COLLECTIVE-servers` can only receive and respond to commands on their specific collective.
+
+Thus, when you define your users in the [authentication setup](#authentication-users-and-groups), you could allow a certain user to command both the EU and UK collectives (but not the US collective) with `groups="eu_collective-admins,uk_collective-admins"`. You would probably want most servers to be "super-servers," since each server already gets to choose which collectives to ignore.
+
+#### MCollective's Exact Authorization Requirements
+
+As described above, any user able to read OR write on a destination must also be able to admin that destination.
+
+Topics:
+
+- `ActiveMQ.Advisory.>` --- Everyone must be able to read and write.
+- `COLLECTIVE.*.agent` --- Admin users must be able to write. Servers must be able to read.
+- `COLLECTIVE.registration.agent` --- If you're using registration, servers must be able to read and write. Otherwise, it can be ignored.
+
+Queues:
+
+- `COLLECTIVE.nodes` --- Admin users must be able to write. Servers must be able to read.
+- `COLLECTIVE.reply.>` --- Servers must be able to write. Admin users must be able to read.
+
+
+> #### Shared Configuration
+>
+> Subcollectives must also be configured in the MCollective client and server config files. If you're setting up authorization per-subcollective, ActiveMQ must allow traffic on any subcollective that MCollective servers and clients expect to use.
+
+
+([↑ Back to top](#content))
+
+Settings for Networks of Brokers
+-----
+
+You can group multiple ActiveMQ servers into networks of brokers, and they can route local MCollective traffic amongst themselves. There are a lot of reasons to do this:
+
+* Scale --- we recommend a maximum of about 800 MCollective servers per ActiveMQ broker, and multiple brokers let you expand past this.
+* High availability --- MCollective servers and clients can attempt to connect to mupltiple brokers in a failover pool.
+* Partition resilience --- if an inter-datacenter link goes down, each half's local MCollective system will still work fine.
+* Network isolation and traffic limiting --- if your clients default to only sending messages to local machines, you can get better performance in the most common case while still being able to command a global collective when you need to.
+* Security --- destination filtering can prevent certain users from sending requests to certain datacenters.
+
+This is naturally more complicated than configuring a single broker.
+
+Designing your broker network's topology is beyond the scope of this reference. The [ActiveMQ Clusters guide](/mcollective/reference/integration/activemq_clusters.html) has a brief description of an example network; see [the ActiveMQ docs][NetworksOfBrokers] or [the Fuse docs][fuse_cluster] for more detailed info. For our purposes, we assume you have already decided:
+
+* Which ActiveMQ brokers can communicate with each other.
+* What kinds of traffic should be excluded from other brokers.
+
+[NetworksOfBrokers]: http://activemq.apache.org/networks-of-brokers.html
+[fuse_cluster]: http://fusesource.com/docs/broker/5.5/clustering/index.html
+
+
+### Broker Name
+
+_Required._
+
+The main `<broker>` element has a `brokerName` attribute. In single-broker deployments, this can be anything and defaults to `localhost`. In a network of brokers, each broker's name must be globally unique across the deployment; duplicates can cause message loops.
+
+{% highlight xml %}
+ <broker xmlns="http://activemq.apache.org/schema/core" brokerName="uk-datacenter-broker" dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true">
+{% endhighlight %}
+
+### OpenWire Transports
+
+_Required._
+
+All participants in a network of brokers need OpenWire network transports enabled. [See "Transport Connectors" above](#transport-connectors) for details.
+
+### Network Connectors
+
+_Required._
+
+If you are using a network of brokers, you need to configure which brokers can talk to each other.
+
+On **one** broker in each linked pair, set up **two** bi-directional network connectors: one for topics, and one for queues. (The only difference between them is the `conduitSubscriptions` attribute, which must be false for the queue connector.)
+
+This is done with a pair of `<networkConnector>` elements inside the `<networkConnectors>` element. Note that the queues connector excludes topics and vice-versa.
+
+{% highlight xml %}
+ <networkConnectors>
+ <networkConnector
+ name="stomp1-stomp2-topics"
+ uri="static:(tcp://stomp2.example.com:61616)"
+ userName="amq"
+ password="secret"
+ duplex="true"
+ decreaseNetworkConsumerPriority="true"
+ networkTTL="2"
+ dynamicOnly="true">
+ <excludedDestinations>
+ <queue physicalName=">" />
+ </excludedDestinations>
+ </networkConnector>
+ <networkConnector
+ name="stomp1-stomp2-queues"
+ uri="static:(tcp://stomp2.example.com:61616)"
+ userName="amq"
+ password="secret"
+ duplex="true"
+ decreaseNetworkConsumerPriority="true"
+ networkTTL="2"
+ dynamicOnly="true"
+ conduitSubscriptions="false">
+ <excludedDestinations>
+ <topic physicalName=">" />
+ </excludedDestinations>
+ </networkConnector>
+ </networkConnectors>
+{% endhighlight %}
+
+Notes:
+
+* If you're using TLS for OpenWire, you'll need to change the URIs to something like `static:(ssl://stomp2.example.com:61617)` --- note the change of both protocol and port.
+* You will need to adjust the TTL for your network's conditions.
+* A username and password are required. The broker with the `<networkConnector>` connects to the other broker as this user. This user should have **full rights** on **all** queues and topics, unless you really know what you're doing. (See [authentication](#authentication-users-and-groups) and [authorization](#authorization-group-permissions) above.)
+* The `name` attribute on each connector must be globally unique. Easiest way to do that is to combine the pair of hostnames involved with the word "queues" or "topics."
+* Alternately, you can set up two uni-directional connectors on both brokers; see the Fuse or ActiveMQ documentation linked above for more details.
+
+
+### Destination Filtering
+
+[fuse_filtering]: http://fusesource.com/docs/broker/5.5/clustering/Networks-Filtering.html
+
+_Optional._
+
+Relevant external docs:
+
+* [Fuse filtering guide][fuse_filtering]
+
+If you want to prevent certain traffic from leaving a given datacenter, you can do so with `<excludedDestinations>` or `<dynamicallyIncludedDestinations>` elements **inside each `<networkConnector>` element.** This is mostly useful for noise reduction, by blocking traffic that other datacenters don't care about, but it can also serve security purposes. Generally, you'll be filtering on [subcollectives][], which, as described above, begin their destination names with the name of the collective.
+
+Both types of filter element can contain `<queue>` and `<topic>` elements, with their `physicalName` attributes defining a destination name with the normal wildcards.
+
+**Remember to retain the all-queues/all-topics exclusions as [shown above](#network-connectors).**
+
+#### Examples
+
+Assume a star network topology.
+
+This topology can be achieved by either having each edge broker connect to the central broker, or having the central broker connect to each edge broker. You can achieve the same filtering in both situations, but with slightly different configuration. The two examples below have similar but not identical effects; the ramifications are subtle, and we _really_ recommend reading the external ActiveMQ and Fuse documentation if you've come this far in your deployment scale.
+
+If your central broker is connecting to the UK broker, and you want it to only pass on traffic for the global `mcollective` collective and the UK-specific `uk_collective` collective:
+
+{% highlight xml %}
+ <dynamicallyIncludedDestinations>
+ <topic physicalName="mcollective.>" />
+ <queue physicalName="mcollective.>" />
+ <topic physicalName="uk_collective.>" />
+ <queue physicalName="uk_collective.>" />
+ </dynamicallyIncludedDestinations>
+{% endhighlight %}
+
+In this case, admin users connected to the central broker can command nodes on the `uk_collective`, but admin users connected to the UK broker can't command nodes on the `us_collective`, etc.
+
+Alternately, if your UK broker is connecting to your central broker and you want it to refrain from passing on UK-specific traffic that no one outside that datacenter cares about:
+
+{% highlight xml %}
+ <excludedDestinations>
+ <topic physicalName="uk_collective.>" />
+ <queue physicalName="uk_collective.>" />
+ </excludedDestinations>
+{% endhighlight %}
+
+In this case, admin users connected to the central broker **cannot** command nodes on the `uk_collective`; it's expected that they'll be issuing commands to the main `mcollective` collective if they need to (and are authorized to) cross outside their borders.
+
+([↑ Back to top](#content))
+
+Tuning
+-----
+
+The minor adjustments listed below (turn off dedicated task runner, increase heap, and increase memory and temp usage in activemq.xml) will generally let you reach about 800 MCollective nodes connected to a single ActiveMQ server, depending on traffic and usage patterns, number of topics and queues, etc. Any more detailed tuning is beyond the scope of this reference, and is likely to be unnecessary for your deployment.
+
+### Don't Use Dedicated Task Runner
+
+Set `-Dorg.apache.activemq.UseDedicatedTaskRunner=false` when starting ActiveMQ. MCollective creates a lot of queues and topics, so _not_ using a thread per destination will save you a lot of memory usage.
+
+This setting is **not** configured in activemq.xml; it's an extra argument to the JVM, which should be provided when ActiveMQ starts up. The place to put this varies, depending on the package you installed ActiveMQ with; it usually goes in the wrapper config file. Check your init script for clues about this file's location. With the common TanukiWrapper scripts, it would look something like this:
+
+ wrapper.java.additional.4=-Dorg.apache.activemq.UseDedicatedTaskRunner=false
+
+### Increase JVM Heap if Necessary
+
+Likewise, the max heap is usually configured in the wrapper config file (`wrapper.java.maxmemory=512`) or on the command line (`-Xmx512m`).
+
+### Memory and Temp Usage for Messages (`systemUsage`)
+
+Since ActiveMQ expects to be embedded in another JVM application, it won't automatically fill up the heap with messages; it has extra limitations on how much space to take up with message contents.
+
+As your deployment gets bigger, you may need to increase the `<memaryUsage>` and `<tempUsage>` elements in the `<systemUsage>` element. Unfortunately, we lack a lot of solid data for what to actually set these to. Most users leave the defaults for memory and temp until they have problems, then double the defaults and see whether their problems go away. This isn't perfectly effecient, but anecdotally it appears to work.
+
+The many redundant nested elements are not a typo; for some reason, ActiveMQ seems to require this.
+
+{% highlight xml %}
+ <!--
+ The systemUsage controls the maximum amount of space the broker will
+ use for messages.
+
+ - memoryUsage is the amount of memory ActiveMQ will take up with
+ *actual messages;* it doesn't include things like thread management.
+ - tempUsage is the amount of disk space it will use for stashing
+ non-persisted messages if the memoryUsage is exceeded (e.g. in the
+ event of a sudden flood of messages).
+ - storeUsage is the amount of disk space dedicated to persistent
+ messages (which MCollective doesn't use directly, but which may be
+ used in networks of brokers to avoid duplicates).
+
+ If producer flow control is on, ActiveMQ will slow down producers
+ when any limits are reached; otherwise, it will use up the
+ memoryUsage, overflow into tempUsage (assuming the default cursor
+ settings aren't changed), and start failing message deliveries after
+ tempUsage is spent. In MCollective, the latter behavior is generally
+ preferable. For more information, see:
+
+ http://activemq.apache.org/producer-flow-control.html
+ http://activemq.apache.org/javalangoutofmemory.html
+ -->
+ <systemUsage>
+ <systemUsage>
+ <memoryUsage>
+ <memoryUsage limit="20 mb"/>
+ </memoryUsage>
+ <storeUsage>
+ <storeUsage limit="1 gb"/>
+ </storeUsage>
+ <tempUsage>
+ <tempUsage limit="100 mb"/>
+ </tempUsage>
+ </systemUsage>
+ </systemUsage>
+{% endhighlight %}
+
+
+([↑ Back to top](#content))
+
+Boilerplate
+-----
+
+There's little reason to care about these settings in most conditions, but they're in the example config files anyway.
+
+### Persistence
+
+MCollective rarely uses this. It's only necessary in networks of brokers, where it is used to prevent routing loops. Leave it enabled; it has no notable performance penalty and its disk usage is limited by the `<storeUsage>` element described above.
+
+{% highlight xml %}
+ <persistenceAdapter>
+ <kahaDB directory="${activemq.base}/data/kahadb"/>
+ </persistenceAdapter>
+{% endhighlight %}
+
+### Management Context
+
+This is for monitoring. MCollective doesn't use this and the examples have it turned off, but you may want it for your own purposes.
+
+{% highlight xml %}
+ <!--
+ The managementContext is used to configure how ActiveMQ is exposed in
+ JMX. By default, ActiveMQ uses the MBean server that is started by
+ the JVM. For more information, see:
+
+ http://activemq.apache.org/jmx.html
+ -->
+
+ <managementContext>
+ <managementContext createConnector="false"/>
+ </managementContext>
+{% endhighlight %}
+
+### Statistics Broker
+
+MCollective doesn't use this.
+
+{% highlight xml %}
+ <plugins>
+ <!--
+ Enable the statisticsBrokerPlugin to allow ActiveMQ to collect
+ statistics.
+ -->
+ <statisticsBrokerPlugin/>
+ <!-- ...auth, etc... -->
+ </plugins>
+{% endhighlight %}
+
+
+### Jetty (Web Consoles, APIs, etc.)
+
+The activemq.xml file will often either contain Jetty settings or import them from another file. MCollective doesn't use this. If you're not using it to manage ActiveMQ, leaving it enabled may be a security risk. Note that this configuration is **outside** the `<broker>` element.
+
+{% highlight xml %}
+ <!--
+ Enable web consoles, REST and Ajax APIs and demos
+ It also includes Camel (with its web console), see ${ACTIVEMQ_HOME}/conf/camel.xml for more info
+
+ Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
+ -->
+ <import resource="jetty.xml"/>
+{% endhighlight %}
+
+([↑ Back to top](#content))
--- /dev/null
+---
+title: "MCollective » Deploy » Middleware » ActiveMQ Keystores"
+subtitle: "Setting Up Keystores For ActiveMQ"
+layout: default
+---
+
+[tls]: ./activemq.html#tls-credentials
+
+Since ActiveMQ runs on the JVM, [configuring it to use TLS encryption/authentication][tls] requires a pair of Java keystores; it can't just use the normal PEM format certificates and keys used by Puppet and MCollective.
+
+Java keystores require some non-obvious steps to set up, so this guide provides full instructions, including both a [manual method](#manually-creating-keystores) and a [Puppet method](#creating-keystores-with-puppet).
+
+
+## Step 0: Obtain Certificates and Keys
+
+ActiveMQ needs the following credentials:
+
+* A copy of the site's CA certificate
+* A certificate signed by the site's CA
+* A private key to match its certificate
+
+These can come from anywhere, but the CA has to match the one used by MCollective.
+
+The easiest approach is to re-use your site's Puppet cert infrastructure, since it's already everywhere and has tools for issuing and signing arbitrary certificates.
+
+As ever, remember to **protect the private key.**
+
+### Option A: Re-Use the Node's Puppet Agent Certificate
+
+On your ActiveMQ server:
+
+* Locate the ssldir by running `sudo puppet agent --configprint ssldir`.
+* Copy the following files to your working directory, making sure to give unique names to the cert and private key:
+ * `<ssldir>/certs/ca.pem`
+ * `<ssldir>/certs/<node name>.pem`
+ * `<ssldir>/private_keys/<node name>.pem`
+
+### Option B: Get a New Certificate from the Puppet CA
+
+On your CA puppet master:
+
+* Run `sudo puppet cert generate activemq.example.com`, substituting some name for your ActiveMQ server. Unlike with a puppet master, the cert's common name can be anything; it doesn't have to be the node's hostname or FQDN.
+* Locate the ssldir by running `sudo puppet master --configprint ssldir`.
+* Retrieve the following files and copy them to a working directory on your ActiveMQ server, making sure to give unique names to the cert and private key:
+ * `<ssldir>/certs/ca.pem`
+ * `<ssldir>/certs/activemq.example.com.pem`
+ * `<ssldir>/private_keys/activemq.example.com.pem`
+
+### Option C: Do Whatever You Want
+
+If you have some other CA infrastructure, you can use that instead.
+
+You can now:
+
+* [Manually create the keystores](#manually-creating-keystores)
+* [Use Puppet to create the keystores](#creating-keystores-with-puppet)
+
+## Manually Creating Keystores
+
+We need a **"truststore"** and a **"keystore."** We also need a **password** for each. (You can use the same password for both stores.)
+
+Remember the password(s) for later, because it needs to [go in the activemq.xml file][tls].
+
+### Step 1: Truststore
+
+> **Note:** The truststore is only required for CA-verified TLS. If you are using anonymous TLS, you may skip it.
+
+The truststore determines which certificates are allowed to connect to ActiveMQ. If you import a CA cert into it, ActiveMQ will trust any certificate signed by that CA.
+
+> You could also _not_ import a CA, and instead import every individual certificate you want to allow. If you do that, you're on your own, but the commands should be similar.
+
+In the working directory with your PEM-format credentials, run the following command. Replace `ca.pem` with whatever you named your copy of the CA cert, and use the password when requested.
+
+{% highlight console %}
+$ sudo keytool -import -alias "My CA" -file ca.pem -keystore truststore.jks
+Enter keystore password:
+Re-enter new password:
+.
+.
+.
+Trust this certificate? [no]: y
+Certificate was added to keystore
+{% endhighlight %}
+
+The truststore is now done.
+
+If you want, you can compare fingerprints:
+
+{% highlight console %}
+$ sudo keytool -list -keystore truststore.jks
+Enter keystore password:
+
+Keystore type: JKS
+Keystore provider: SUN
+
+Your keystore contains 1 entry
+
+my ca, Mar 30, 2012, trustedCertEntry,
+Certificate fingerprint (MD5): 99:D3:28:6B:37:13:7A:A2:B8:73:75:4A:31:78:0B:68
+
+$ sudo openssl x509 -in ca.pem -fingerprint -md5
+MD5 Fingerprint=99:D3:28:6B:37:13:7A:A2:B8:73:75:4A:31:78:0B:68
+{% endhighlight %}
+
+
+### Step 2: Keystore
+
+The keystore contains the ActiveMQ broker's certificate and private key, which it uses to identify itself to the applications that connect to it.
+
+In the working directory with your PEM-format credentials, run the following commands. Substitute the names of your key and certificate files where necessary, and the common name of your ActiveMQ server's certificate for `activemq.example.com`.
+
+These commands use both an "export/source" password and a "destination" password. The export/source password is never used again after this series of commands.
+
+{% highlight console %}
+$ sudo cat activemq_private.pem activemq_cert.pem > temp.pem
+$ sudo openssl pkcs12 -export -in temp.pem -out activemq.p12 -name activemq.example.com
+Enter Export Password:
+Verifying - Enter Export Password:
+$sudo keytool -importkeystore -destkeystore keystore.jks -srckeystore activemq.p12 -srcstoretype PKCS12 -alias activemq.example.com
+Enter destination keystore password:
+Re-enter new password:
+Enter source keystore password:
+{% endhighlight %}
+
+The keystore is now done.
+
+If you want, you can compare fingerprints:
+
+{% highlight console %}
+$ sudo keytool -list -keystore keystore.jks
+Enter keystore password:
+
+Keystore type: JKS
+Keystore provider: SUN
+
+Your keystore contains 1 entry
+
+activemq.example.com, Mar 30, 2012, PrivateKeyEntry,
+Certificate fingerprint (MD5): 7E:2A:B4:4D:1E:6D:D1:70:A9:E7:20:0D:9D:41:F3:B9
+
+$ sudo openssl x509 -in activemq_cert.pem -fingerprint -md5
+MD5 Fingerprint=7E:2A:B4:4D:1E:6D:D1:70:A9:E7:20:0D:9D:41:F3:B9
+{% endhighlight %}
+
+### Step 3: Finish
+
+Move the keystore and truststore to ActiveMQ's config directory. Make sure they are owned by the ActiveMQ user and unreadable to any other users. [Configure ActiveMQ to use them in its `sslContext`.][tls] **Double-check** that you've made activemq.xml world-unreadable, since it now contains sensitive credentials.
+
+## Creating Keystores with Puppet
+
+If you're managing your ActiveMQ server with Puppet anyway, you can use the [puppetlabs/java_ks module](http://forge.puppetlabs.com/puppetlabs/java_ks) to handle the format conversion.
+
+This approach is more work for a single permanent ActiveMQ server, but less work if you intend to deploy multiple ActiveMQ servers or eventually change the credentials.
+
+### Step 1: Install the `java_ks` Module
+
+On your puppet master, run `sudo puppet module install puppetlabs/java_ks`.
+
+### Step 2: Create a Puppet Class
+
+The class to manage the keystores should do the following:
+
+* Make sure the PEM cert and key files are present and protected.
+* Declare a pair of `java_ks` resources.
+* Manage the mode and ownership of the keystore files.
+
+The code below is an example, but it will work fine if you put it in a module (example file location in the first comment) and set its parameters when you declare it. The name of the class (and its home module) can be changed as needed.
+
+
+{% highlight ruby %}
+ # /etc/puppet/modules/activemq/manifests/keystores.pp
+ class activemq::keystores (
+ $keystore_password, # required
+
+ # User must put these files in the module, or provide other URLs
+ $ca = 'puppet:///modules/activemq/ca.pem',
+ $cert = 'puppet:///modules/activemq/cert.pem',
+ $private_key = 'puppet:///modules/activemq/private_key.pem',
+
+ $activemq_confdir = '/etc/activemq',
+ $activemq_user = 'activemq',
+ ) {
+
+ # ----- Restart ActiveMQ if the SSL credentials ever change -----
+ # ----- Uncomment if you are fully managing ActiveMQ with Puppet. -----
+
+ # Package['activemq'] -> Class[$title]
+ # Java_ks['activemq_cert:keystore'] ~> Service['activemq']
+ # Java_ks['activemq_ca:truststore'] ~> Service['activemq']
+
+
+ # ----- Manage PEM files -----
+
+ File {
+ owner => root,
+ group => root,
+ mode => 0600,
+ }
+ file {"${activemq_confdir}/ssl_credentials":
+ ensure => directory,
+ mode => 0700,
+ }
+ file {"${activemq_confdir}/ssl_credentials/activemq_certificate.pem":
+ ensure => file,
+ source => $cert,
+ }
+ file {"${activemq_confdir}/ssl_credentials/activemq_private.pem":
+ ensure => file,
+ source => $private_key,
+ }
+ file {"${activemq_confdir}/ssl_credentials/ca.pem":
+ ensure => file,
+ source => $ca,
+ }
+
+
+ # ----- Manage Keystore Contents -----
+
+ # Each keystore should have a dependency on the PEM files it relies on.
+
+ # Truststore with copy of CA cert
+ java_ks { 'activemq_ca:truststore':
+ ensure => latest,
+ certificate => "${activemq_confdir}/ssl_credentials/ca.pem",
+ target => "${activemq_confdir}/truststore.jks",
+ password => $keystore_password,
+ trustcacerts => true,
+ require => File["${activemq_confdir}/ssl_credentials/ca.pem"],
+ }
+
+ # Keystore with ActiveMQ cert and private key
+ java_ks { 'activemq_cert:keystore':
+ ensure => latest,
+ certificate => "${activemq_confdir}/ssl_credentials/activemq_certificate.pem",
+ private_key => "${activemq_confdir}/ssl_credentials/activemq_private.pem",
+ target => "${activemq_confdir}/keystore.jks",
+ password => $keystore_password,
+ require => [
+ File["${activemq_confdir}/ssl_credentials/activemq_private.pem"],
+ File["${activemq_confdir}/ssl_credentials/activemq_certificate.pem"]
+ ],
+ }
+
+
+ # ----- Manage Keystore Files -----
+
+ # Permissions only.
+ # No ensure, source, or content.
+
+ file {"${activemq_confdir}/keystore.jks":
+ owner => $activemq_user,
+ group => $activemq_user,
+ mode => 0600,
+ require => Java_ks['activemq_cert:keystore'],
+ }
+ file {"${activemq_confdir}/truststore.jks":
+ owner => $activemq_user,
+ group => $activemq_user,
+ mode => 0600,
+ require => Java_ks['activemq_ca:truststore'],
+ }
+
+ }
+{% endhighlight %}
+
+
+### Step 3: Assign the Class to the ActiveMQ Server
+
+...using your standard Puppet site tools.
+
+### Step 4: Finish
+
+[Configure ActiveMQ to use the keystores in its `sslContext`][tls], probably with the Puppet template you're using to manage activemq.xml. **Double-check** that you've made activemq.xml world-unreadable, since it now contains sensitive credentials.
--- /dev/null
+---
+layout: default
+title: EC2 Demo
+toc: false
+---
+[Amazon Console]: https://console.aws.amazon.com/ec2/
+[Puppet based Service]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AgentService
+[Puppet based Package]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AgentPackage
+[NRPE]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AgentNRPE
+[Meta Registration]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/RegistrationMetaData
+[URL Tester]: https://github.com/ripienaar/mc-plugins/tree/master/agent/urltest
+[Registration]: /mcollective/reference/plugins/registration.html
+[Registration Monitor]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AgentRegistrationMonitor
+
+We've created an Amazon hosted demo of mcollective that can give you a quick feel
+for how things work without all the hassle of setting up from scratch.
+
+The demo uses the new Amazon CloudFormation technology that you can access using the [Amazon Console].
+To access the AMI you must select the 'EU - West' Region. Also, prior to following the steps in the demo
+please create a SSH keypair and register it under the EC2 tab in the console for that region.
+
+The video below shows how to get going with the demo and runs through a few of the availbable options.
+For best experience please maximise the video.
+
+The two passwords requested during creation is completely arbitrary you can provide anything there and
+past entering them on the creation page you don't need to know them again later. They are used internally
+to the demo without you being aware of them.
+
+You'll need to enter the url _http://mcollective-120-demo.s3.amazonaws.com/cloudfront`_`demo.json_ into the
+creation step.
+
+<iframe width="640" height="360" src="http://www.youtube-nocookie.com/embed/Hw0Z1xfg050" frameborder="0" allowfullscreen></iframe>
+
+## Agents
+The images all have the basic agents going as well as some community ones:
+
+ * [Puppet based Service]
+ * [Puppet based Package]
+ * [NRPE]
+ * [Meta Registration]
+ * [URL Tester]
+
+## Registration
+The main node will have [Registration] setup and the community [Registration Monitor] agent,
+look in */var/tmp/mcollective* for meta data from all your nodes.
--- /dev/null
+---
+layout: default
+title: Marionette Collective
+toc: false
+---
+[Func]: https://fedorahosted.org/func/
+[Fabric]: http://fabfile.org/
+[Capistrano]: http://www.capify.org
+[Publish Subscribe Middleware]: http://en.wikipedia.org/wiki/Publish/subscribe
+[Screencasts]: /mcollective/screencasts.html
+[Amazon EC2 based demo]: /mcollective/ec2demo.html
+[broadcast paradigm]: /mcollective/reference/basic/messageflow.html
+[UsingWithPuppet]: /mcollective/reference/integration/puppet.html
+[UsingWithChef]: /mcollective/reference/integration/chef.html
+[Facter]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsFacterYAML
+[Ohai]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsOhai
+[WritingFactsPlugins]: /mcollective/reference/plugins/facts.html
+[NodeReports]: /mcollective/reference/ui/nodereports.html
+[PluginsSite]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki
+[SimpleRPCIntroduction]: /mcollective/simplerpc/
+[SecurityOverview]: /mcollective/security.html
+[SecurityWithActiveMQ]: /mcollective/reference/integration/activemq_security.html
+[SSLSecurityPlugin]: /mcollective/reference/plugins/security_ssl.html
+[AESSecurityPlugin]: /mcollective/reference/plugins/security_aes.html
+[SimpleRPCAuthorization]: /mcollective/simplerpc/authorization.html
+[SimpleRPCAuditing]: /mcollective/simplerpc/auditing.html
+[ActiveMQClusters]: /mcollective/reference/integration/activemq_clusters.html
+[JSONSchema]: http://json-schema.org/
+[Registration]: /mcollective/reference/plugins/registration.html
+[GettingStarted]: /mcollective/reference/basic/gettingstarted.html
+[Configuration]: /mcollective/reference/basic/configuration.html
+[Terminology]: /mcollective/terminology.html
+[devco]: http://www.devco.net/archives/tag/mcollective
+[mcollective-users]: http://groups.google.com/group/mcollective-users
+[WritingAgents]: /mcollective/reference/basic/basic_agent_and_client.html
+[ActiveMQ]: /mcollective/reference/integration/activemq_security.html
+[MessageFormat]: /mcollective/reference/basic/messageformat.html
+[ChangeLog]: /mcollective/changelog.html
+[server_config]: /mcollective/configure/server.html
+
+The Marionette Collective AKA mcollective is a framework to build server
+orchestration or parallel job execution systems.
+
+Primarily we'll use it as a means of programmatic execution of Systems Administration
+actions on clusters of servers. In this regard we operate in the same space as tools
+like [Func], [Fabric] or [Capistrano].
+
+We've attempted to think out of the box a bit designing this system by not relying on
+central inventories and tools like SSH, we're not simply a fancy SSH "for loop". MCollective use modern tools like
+[Publish Subscribe Middleware] and modern philosophies like real time discovery of network resources using meta data
+and not hostnames. Delivering a very scalable and very fast parallel execution environment.
+
+To get an immediate feel for what I am on about you can look at some of the videos on the
+[Screencasts] page and then keep reading below for further info and links. We've also created an [Amazon EC2 based demo]
+where you can launch as many instances as you want to see how it behaves first hand.
+
+## What is MCollective and what does it allow you to do
+
+ * Interact with small to very large clusters of servers
+ * Use a [broadcast paradigm] for request distribution. All servers get all requests at the same time, requests have
+ filters attached and only servers matching the filter will act on requests. There is no central asset database to
+ go out of sync, the network is the only source of truth.
+ * Break free from ever more complex naming conventions for hostnames as a means of identity. Use a very
+ rich set of meta data provided by each machine to address them. Meta data comes from
+ [Puppet][UsingWithPuppet], [Chef][UsingWithChef], [Facter], [Ohai] or [plugins][WritingFactsPlugins] you provide yourself.
+ * Comes with simple to use command line tools to call remote agents.
+ * Ability to write [custom reports][NodeReports] about your infrastructure.
+ * A number of agents to manage packages, services and other common components are [available from
+ the community][PluginsSite].
+ * Allows you to write [simple RPC style agents, clients][SimpleRPCIntroduction] and Web UIs in an easy to understand language - Ruby
+ * Extremely pluggable and adaptable to local needs
+ * Middleware systems already have rich [authentication and authorization models][SecurityWithActiveMQ], leverage these as a first
+ line of control. Include fine grained Authentication using [SSL][SSLSecurityPlugin] or [RSA][AESSecurityPlugin], [Authorization][SimpleRPCAuthorization] and
+ [Auditing][SimpleRPCAuditing] of requests. You can see more details in the [Security Overview][SecurityOverview].
+ * Re-use the ability of middleware to do [clustering, routing and network isolation][ActiveMQClusters]
+ to realize secure and scalable setups.
+
+## Pluggable Core
+We aim to provide a stable core framework that allows you to build it out into a system that meets
+your own needs, we are pluggable in the following areas:
+
+ * Replace our choice of middleware - STOMP compliant middleware - with your own like AMQP based.
+ * Replace our authorization system with one that suits your local needs
+ * Replace our serialization - Ruby Marshal and YAML based - with your own like [JSONSchema] that is cross language.
+ * Add sources of data, we support [Chef][UsingWithChef] and [Puppet][UsingWithPuppet]. You can
+ [provide a plugin to give us access to your tools data][WritingFactsPlugins].
+ The community have ones for [Facter and Ohai already][PluginsSite]
+ * Create a central inventory of services [leveraging MCollective as transport][Registration]
+ that can run and distribute inventory data on a regular basis.
+
+MCollective is licensed under the Apache 2 license.
+
+## Next Steps and Further Reading
+
+### Introductory and Tutorial Pages
+ * See it in action in our [Screencasts]
+ * Read the [GettingStarted], [Server Configuration Reference][server_config], and [Configuration] guides for installation instructions
+ * Read the [Terminology] page if you see any words where the meaning in the context of MCollective is not clear
+ * Read the [ChangeLog] page to see how MCollective has developed
+ * Learn how to write basic reports for your servers - [NodeReports]
+ * Learn how to write simple Agents and Clients using our [Simple RPC Framework][SimpleRPCIntroduction]
+ * The author maintains some agents and clients on another project [here][PluginsSite].
+ * The author has written [several blog posts][devco] about mcollective.
+ * Subscribe and post questions to the [mailing list][mcollective-users].
+
+### Internal References and Developer Docs
+ * Finding it hard to do something complex with Simple RPC? See [WritingAgents] for what lies underneath
+ * Role based security, authentication and authorization using [ActiveMQ]
+ * Structure of [Request and Reply][MessageFormat] messages
+
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC1
+toc: false
+---
+
+Example Message
+---------------
+
+ The Marionette Collective version 2.2.2 started by /usr/bin/mco using config file /etc/mcollective/client.cfg
+
+Additional Information
+----------------------
+
+This message gets logged each time MCollective reads it's config file. Generally this only happens once per process. It shows the version, process name and config file as a simple debugging aid
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC10
+toc: false
+---
+
+Example Message
+---------------
+
+ Failed to handle message: RuntimeError: none.rb:15:in `decodemsg': Could not decrypt message
+
+Additional Information
+----------------------
+
+When a message arrives from the middleware it gets decoded, security validated and then dispatched to the agent code.
+
+There exist a number of errors that can happen here, some are handled specifically others will be logged by this "catch all" handler.
+
+Generally there should not be many messages logged here but we include a stack trace to assist with debugging these.
+
+The messages here do not tend to originate from your Agents unless they are syntax error related but more likely to be situations like security failures due to incorrect SSL keys and so forth
+
+Should you come across one that is a regular accorance in your logs please open a ticket including your backtrace and we will improve the handling of that situation
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC11
+toc: false
+---
+
+Example Message
+---------------
+
+ Cache expired on 'ddl' key 'agent/nrpe'
+
+Additional Information
+----------------------
+
+MCollective has an internal Cache used to speed up operations like parsing of DDL files. The cache is also usable from the agents and other plugins you write.
+
+Each entry in the cache has an associated TTL or maximum life time, once the maximum time on an item is reached it is considered expired. Next time anything attempts to read this entry from the cache this log line will be logged.
+
+This is part of the normal operations of MCollective and doesn't indicate any problem. We log this debug message to help you debug your own use of the cache.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC12
+toc: false
+---
+
+Example Message
+---------------
+
+ Cache hit on 'ddl' key 'agent/nrpe'
+
+Additional Information
+----------------------
+
+MCollective has an internal Cache used to speed up operations like parsing of DDL files. The cache is also usable from the agents and other plugins you write.
+
+Each entry in the cache has an associated TTL or maximum life time, once the maximum time on an item is reached it is considered expired.
+
+This log line indicates that a request for a cache entry was made, the entry had not yet expired and so the cached data is being returned.
+
+It does not indicate a problem, it's just a debugging aid
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC13
+toc: false
+---
+
+Example Message
+---------------
+
+ Could not find a cache called 'my_cache'
+
+Additional Information
+----------------------
+
+MCollective has an internal Cache used to speed up operations like parsing of DDL files. The cache is also usable from the agents and other plugins you write.
+
+The cache is made up of many named caches, this error indicates that a cache has not yet been setup prior to trying to use it.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC14
+toc: false
+---
+
+Example Message
+---------------
+
+ No block supplied to synchronize on cache 'my_cache'
+
+Additional Information
+----------------------
+
+When using the Cache to synchronize your own code across agents or other plugins you have to supply a block to synchronise.
+
+Correct usage would be:
+
+ Cache.setup(:customer, 600)
+ Cache(:customer).synchronize do
+ # update customer record
+ end
+
+This error is raise when the logic to update the customer record is not in a block as in the example
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC15
+toc: false
+---
+
+Example Message
+---------------
+
+ No item called 'nrpe_agent' for cache 'ddl'
+
+Additional Information
+----------------------
+
+MCollective has an internal Cache used to speed up operations like parsing of DDL files. The cache is also usable from the agents and other plugins you write.
+
+The cache stored items using a key, this error will be logged and raised when you try to access a item that does not yet exist in the cache.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC16
+toc: false
+---
+
+Example Message
+---------------
+
+ 'hello' does not look like a numeric value
+
+Additional Information
+----------------------
+
+When MCollective receives an argument from the command line like port=fello it consults the DDL file to determine the desired type of this parameter, it then tries to convert the input string into the correct numeric value.
+
+This error indicates the input you provided could not be converted into the desired format.
+
+You'll usually see this when using the 'mco rpc' command to interact with an agent. This is usually a fatal error, the request will not be sent if it does not validate against the DDL.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC17
+toc: false
+---
+
+Example Message
+---------------
+
+ 'flase' does not look like a boolean value
+
+Additional Information
+----------------------
+
+When MCollective receives an argument from the command line like force=true it consults the DDL file to determine the desired type of this parameter, it then tries to convert the input string into the correct boolean value.
+
+This error indicates the input you provided could not be converted into the desired boolean format. It wil accept "true", "t", "yes", "y" and "1" as being true. It will accept "false", "f", "no", "n", "0" as being false.
+
+You'll usually see this when using the 'mco rpc' command to interact with an agent. This is usually a fatal error, the request will not be sent if it does not validate against the DDL.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC18
+toc: false
+---
+
+Example Message
+---------------
+
+ Found 'rpcutil' ddl at '/usr/libexec/mcollective/mcollective/agent/rpcutil.ddl'
+
+Additional Information
+----------------------
+
+Most plugin types have a DDL file that needs to be correctly installed for the plugin to function. There can be multiple libdirs that can provide the DDL file for a specific plugin.
+
+This message is a message designed to help you debug issues when you think the wrong DDL file is being used.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC19
+toc: false
+---
+
+Example Message
+---------------
+
+ DDL requirements validation being skipped in development
+
+Additional Information
+----------------------
+
+Usually when MCollective run it validates all requirements are met before publishing a request or processing a request against the DDL file for the agent.
+
+This can be difficult to satisfy in development perhaps because you are still writing your DDL files or debugging issues.
+
+This message indicates that when MCollective detects it's running against an unreleased version of MCollective - like directly out of a Git clone - it will skip the DDL validation steps. It is logged at a warning level as this significantly changes the behaviour of the client and server.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC2
+toc: false
+---
+
+Example Message
+---------------
+
+ Reloading all agents after receiving USR1 signal
+
+Additional Information
+----------------------
+
+When sending the mcollectived process the USR1 signal on a Unix based machine this message will indicate that the signal got received and all agents are being reloaded from disk
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC20
+toc: false
+---
+
+Example Message
+---------------
+
+ Agent plugin 'example' requires MCollective version 2.2.1 or newer
+
+Additional Information
+----------------------
+
+MCollective plugins can declare a minimum version of MCollective that they require to be functional.
+
+MCollective validates this when it loads the plugin and this error will be logged or shown to the user when this requirement cannot be satisfied.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC21
+toc: false
+---
+
+Example Message
+---------------
+
+ Cannot validate input 'service': Input string is longer than 40 character(s)
+
+Additional Information
+----------------------
+
+Every input you provide to a RPC request is validated against it's DDL file. This error will be shown when the supplied data does not pass validation against the DDL.
+
+Consult the 'mco plugin doc' command to view the DDL file for your action and input.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC22
+toc: false
+---
+
+Example Message
+---------------
+
+ Cannot determine what entity input 'port' belongs to
+
+Additional Information
+----------------------
+
+When writing a DDL you declare inputs into plugins using the input keyword. Each input has to belong to a wrapping entity like in the example below:
+
+ action "get_data", :description => "Get data from a data plugin" do
+ input :source,
+ :prompt => "Data Source",
+ :description => "The data plugin to retrieve information from",
+ :type => :string,
+ :validation => '^\w+$',
+ :optional => false,
+ :maxlength => 50
+ end
+
+Here the input belongs to the action entity "get_data", this error indicates that an input were found without it belonging to any specific entity
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC23
+toc: false
+---
+
+Example Message
+---------------
+
+ Input needs a :prompt property
+
+Additional Information
+----------------------
+
+When writing a DDL you declare inputs for all data that you pass into the plugin. Inputs have a minimally required field set and this error indicate that you did not provide some key field while describing the input.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC24
+toc: false
+---
+
+Example Message
+---------------
+
+ Failed to load DDL for the 'rpcutil' agent, DDLs are required: RuntimeError: failed to parse DDL file
+
+Additional Information
+----------------------
+
+As of version 2.0.0 DDL files are required by the MCollective server. This server indicates that it either could not find the DDL for the rpcutil agent or that there was a syntax error.
+
+Confirm that the DDL file is installed in the agent directory and that it parses correctly using the 'mco plugin doc' command.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC25
+toc: false
+---
+
+Example Message
+---------------
+
+ aggregate method for action 'rpcutil' is missing a function parameter
+
+Additional Information
+----------------------
+
+DDL files can declare aggregation rules for the data returned by actions as seen below:
+
+ summarize do
+ aggregate summary(:collectives)
+ end
+
+This error indicates that you did not supply the argument - :collectives in this example.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC26
+toc: false
+---
+
+Example Message
+---------------
+
+ Functions supplied to aggregate should be a hash
+
+Additional Information
+----------------------
+
+Internally when MCollective parse the DDL it converts the aggregate functions into a hash with the function name and any arguments.
+
+This error indicates that the internal translation failed, this is a critical error and would probably indicate a structure problem in your DDL or a bug in MCollective
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC27
+toc: false
+---
+
+Example Message
+---------------
+
+ Formats supplied to aggregation functions must have a :format key
+
+Additional Information
+----------------------
+
+DDL aggregate functions can take a custom format as in the example below:
+
+ aggregate average(:total_resources, :format => "Average: %d")
+
+This error indicate that the :format above could not be correctly parsed.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC28
+toc: false
+---
+
+Example Message
+---------------
+
+ Formats supplied to aggregation functions should be a hash
+
+Additional Information
+----------------------
+
+DDL aggregate functions can take a custom format as in the example below:
+
+ aggregate average(:total_resources, :format => "Average: %d")
+
+This error indicate that the :format above was not supplied as a hash as in the above example
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC29
+toc: false
+---
+
+Example Message
+---------------
+
+ Attempted to call action 'yum_clean' for 'package' but it's not declared in the DDL
+
+Additional Information
+----------------------
+
+Every agent has a DDL file that describes valid actions that they can perform
+
+This error indicates you attempted to perform an action that does not exist. Review the plugin documentation using 'mco plugin doc' for correct usage
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC3
+toc: false
+---
+
+Example Message
+---------------
+
+ Cycling logging level due to USR2 signal
+
+Additional Information
+----------------------
+
+When sending the mcollectived process the USR2 signal on a Unix based machine this message indicates that the signal for received and the logging level is being adjusted to the next higher level or back down to debug if it was already at it's highest level.
+
+This message will be followed by another message similar to the one below:
+
+ Logging level is now WARN
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC30
+toc: false
+---
+
+Example Message
+---------------
+
+ Action 'get_fact' needs a 'fact' argument
+
+Additional Information
+----------------------
+
+Actions can declare that they expect a required input, this error indicates that you did not supply the required input
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC31
+toc: false
+---
+
+Example Message
+---------------
+
+ No dataquery has been defined in the DDL for data plugin 'package'
+
+Additional Information
+----------------------
+
+Each data plugin requires a DDL that has a 'dataquery' block in it.
+
+ dataquery :description => "Agent Meta Data" do
+ .
+ .
+ end
+
+This error indicates that the DDL file for a specific data plugin did not contain this block.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC32
+toc: false
+---
+
+Example Message
+---------------
+
+ No output has been defined in the DDL for data plugin '%{plugin}'
+
+Additional Information
+----------------------
+
+Data plugins must return data. The DDL files for a data plugin must declare at least one output parameter else you will see this error.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC33
+toc: false
+---
+
+Example Message
+---------------
+
+ No data plugin argument was declared in the 'puppet' DDL but an input was supplied
+
+Additional Information
+----------------------
+
+Data plugins can take an optional input argument. This error indicate that you supplied an input argument but the plugin does not actually expect any input.
+
+Review the documentation for the data plugin using 'mco plugin doc'
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC34
+toc: false
+---
+
+Example Message
+---------------
+
+ setting meta data in agents have been deprecated, DDL files are now being used for this information. Please update the 'package' agent
+
+Additional Information
+----------------------
+
+In the past each MCollective agent had a metadata section containing information like the timeout.
+
+As of 2.2.0 the agents will now consult the DDL files that ship with each agent for this purpose so the metadata in agents are not used at all.
+
+In order to remove confusing duplication setting metadata in agents have been deprecated and from version 2.4.0 will not be supported at all.
+
+Please update your agent by removing the metadata section from it and make sure the DDL file has the correct information instead.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC35
+toc: false
+---
+
+Example Message
+---------------
+
+ Client did not request a response, surpressing reply
+
+Additional Information
+----------------------
+
+The MCollective client can ask that the agent just performs the action and never respond. Like when supplying the --no-results option to the 'mco rpc' application.
+
+This log line indicates that the request was received and interpreted as such and no reply will be sent. This does not indicate a problem generally it's just there to assist with debugging of problems.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC36
+toc: false
+---
+
+Example Message
+---------------
+
+ Unknown action 'png' for agent 'rpcutil'
+
+Additional Information
+----------------------
+
+Agents are made up of a number of named actions. When the MCollective Server receives a request it double checks if the agent in question actually implements the logic for a specific action.
+
+When it cannot find the implementation this error will be raised. This is an unusual situation since at this point on both the client and the server the DDL will already have been used to validate the request and the DDL would have indicated that the action is valid.
+
+Check your agent code and make sure you have code like:
+
+ action "the_action" do
+ .
+ .
+ end
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC37
+toc: false
+---
+
+Example Message
+---------------
+
+ Starting default activation checks for the 'rpcutil' agent
+
+Additional Information
+----------------------
+
+Each time the MCollective daemon starts it loads each agent from disk. It then tries to determine if the agent should start on this node by using the activate_when method or per-agent configuration.
+
+This is a debug statement that shows you it is about to start interacting with the logic in the agent to determine if it should be made available or not.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC38
+toc: false
+---
+
+Example Message
+---------------
+
+ Found plugin configuration 'exim.activate_agent' with value 'y'
+
+Additional Information
+----------------------
+
+The administrator can choose that a certain agent that is deployed on this machine should not be made available to the network.
+
+To do this you would add a configuration value like this example to the mcollective server.cfg:
+
+ plugin.exim.activate_agent = n
+
+If this value is "1", "y" or "true" the agent will be activated else it will be disabled.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC39
+toc: false
+---
+
+Example Message
+---------------
+
+ Audit failed with an error, processing the request will continue.
+
+Additional Information
+----------------------
+
+Every MCollective request can be audited. In the event where the auditing fails the processing continues.
+
+At present the failure handling is not configurable, in future the administrator might elect to not run unaudited requests via configuration.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC4
+toc: false
+---
+
+Example Message
+---------------
+
+ Failed to start registration plugin: ArgumentError: meta.rb:6:in `gsub': wrong number of arguments (0 for 2)
+
+Additional Information
+----------------------
+
+Registration plugins are loaded into the MCollective daemon at startup and ran on a regular interval.
+
+This message indicate that on first start this plugin failed to run, it will show the exception class, line and exception message to assist with debugging
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC5
+toc: false
+---
+
+Example Message
+---------------
+
+ Received a control message, possibly via 'mco controller' but this has been deprecated
+
+Additional Information
+----------------------
+
+In previous versions of MCollective a application called 'controller' were included that had the ability to request agent reloads and other commands that would control the runner.
+
+Unfortunately this method of controlling the daemon was never considered stable or reliable and has since been deprecate for removal during the 2.3.x development series
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC6
+toc: false
+---
+
+Example Message
+---------------
+
+ Message does not pass filters, ignoring
+
+Additional Information
+----------------------
+
+When a specific MCollective daemon receives a message from a network it validates the filter before processing the message. The filter is the list of classes, facts or other conditions that are associated with any message.
+
+In the case where the filter does not match the specific host this line gets logged.
+
+It's often the case for broadcast messages that all MCollective nodes will receive a message but only a subset of nodes are targetted using the filters, in that situation the nodes that received the request but should not respond to it will see this log line.
+
+It does not indicate anything to be concerned about but if you find debugging a problem and expect a node to have responded when it shouldn't this log line will give you a hint that some condition specified in the filter did not match the local host
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC7
+toc: false
+---
+
+Example Message
+---------------
+
+ Exiting after signal: SignalException: runner.rb:6:in `run': Interrupt
+
+Additional Information
+----------------------
+
+When the MCollective daemon gets a signal from the Operating System that it does not specifically handle it will log this line before exiting.
+
+You would see this whenever the daemon is stopped by init script or when sending it a kill signal, it will then proceed to disconnect from the middleware and exit its main loop
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC8
+toc: false
+---
+
+Example Message
+---------------
+
+ Handling message for agent 'rpcutil' on collective 'eu_collective' with requestid 'a8a78d0ff555c931f045b6f448129846'
+
+Additional Information
+----------------------
+
+After receiving a message from the middleware, decoding it, validating it's security credentials and doing other checks on it the MCollective daemon will pass it off to the actual agent code for processing.
+
+Prior to doing so it will log this line indicating the agent name and sub-collective and other information that will assist in correlating the message sent from the client with those in the server logs being processed.
--- /dev/null
+---
+layout: default
+title: Message detail for PLMC9
+toc: false
+---
+
+Example Message
+---------------
+
+ Expired Message: message 8b4fe522f0d0541dabe83ec10b7fa446 from cert=client@node created at 1358840888 is 653 seconds old, TTL is 60
+
+Additional Information
+----------------------
+
+Requests sent from clients to servers all have a creation time and a maximum validity time called a TTL.
+
+This message indicates that a message was received from the network but that it was determined to be too based on the TTL settings.
+
+Usually this happens because your clocks are not in sync - something that can be fixed by rolling out a tool like ntp across your server estate.
+
+It might also happen during very slow network conditions or when the TTL is set too low for your general network latency.
--- /dev/null
+---
+layout: default
+title: Basic Agents and Clients
+---
+[SimpleRPCIntroduction]: /mcollective/simplerpc/
+
+## Deprecation Warning
+
+You must use [SimpleRPC][SimpleRPCIntroduction] for writing agents and clients. SimpleRPC is a framework that wraps the core MCollective client and agent structures in a lot of helpful convention and tools.
+
+**The approach to developing agents and clients documented below is not supported post version 2.0.0 and the functionality will be removed for the next major release.**
+
+## Overview
+
+Writing agents for mcollective is simple, we'll write a simple _echo_ agent as well as a cli tool to communicate with it that supports discovery, filtering and more.
+
+The agent will send back everything that got sent to it, not overly useful but enough to demonstrate the concepts.
+
+## The Agent
+
+Agents go into a directory configured in the _server.cfg_ using the _libdir_ directive. You should have _mcollective/agent_ directory under that, restart the daemon when you've put it there.
+
+Create a file echo.rb with the following, I'll walk through each part:
+
+{% highlight ruby linenos %}
+module MCollective
+ module Agent
+ # Simple echo agent
+ class Echo
+ attr_reader :timeout, :meta
+
+ def initialize
+ @timeout = 5
+ @meta = {:license => "Apache License, Version 2",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :version => "1.0"}
+ end
+
+ def handlemsg(msg, stomp)
+ msg[:body]
+ end
+
+ def help
+ <<-EOH
+ Echo Agent
+ ==========
+
+ Simple agent that just sends the body of any request back
+ EOH
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+Once you have it running you can test your agent works as below, we'll send the word _hello_ to the agent and we'll see if we get it back:
+
+{% highlight console %}
+% /usr/sbin/mc-call-agent --config etc/client.cfg --agent echo --arg hello
+Determining the amount of hosts matching filter for 2 seconds .... 1
+
+devel.your.com>
+"hello"
+
+---- stomp call summary ----
+ Nodes: 1 / 1
+ Start Time: Tue Nov 03 23:18:40 +0000 2009
+ Discovery Time: 2001.65ms
+ Agent Time: 44.17ms
+ Total Time: 2045.82ms
+{% endhighlight %}
+
+
+### Agent name
+Each agent should be wrapped in a module and class as below, this will create an agent called _echo_ and should live in a file called _agents/echo.rb_.
+
+{% highlight ruby %}
+module MCollective
+ module Agent
+ class Echo
+ end
+ end
+end
+{% endhighlight %}
+
+### Initialization
+Every agent needs to specify a timeout and meta data. The timeout gets used by the app server to kill off agents that is taking too long to finish.
+
+Meta data contains some information about the licence, author and version of the agent. Right now the information is free-form but I suggest supplying at the very least the details below as we'll start enforcing the existence of it in future.
+
+{% highlight ruby %}
+ attr_reader :timeout, :meta
+
+ def initialize
+ @timeout = 1
+ @meta = {:license => "Apache License, Version 2",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :version => "1.0"}
+ end
+{% endhighlight %}
+
+### Handling incoming messages
+You do not need to be concerned with filtering, authentication, authorization etc when writing agents - the app server takes care of all of that for you.
+
+Whenever a message for your agent pass all the checks it will be passed to you via the _handlemsg_ method.
+
+{% highlight ruby %}
+ def handlemsg(msg, stomp)
+ msg[:body]
+ end
+{% endhighlight %}
+
+The msg will be a hash with several keys giving you information about sender, hashes, time it was sent and so forth, in our case we just want to know about the body and we're sending it right back as a return value.
+
+### Online Help
+We keep help information, not used right now but future version of the code will have a simple way to get help for each agent, the intent is that inputs, outputs and behavior to all agents would be described in this.
+
+{% highlight ruby %}
+ def help
+ <<-EOH
+ Echo Agent
+ ==========
+
+ Simple agent that just sends the body of any request back
+ EOH
+ end
+{% endhighlight %}
+
+## More complex agents
+As you write more complex agents and clients you might find the need to have a few separate files make up your agent, you can drop these files into a directory named _util_ under the plugins (that is, at the same level of the agent folder, so as _/usr/libexec/mcollective/mcollective/util_ at the time of writing).
+
+Create the _util_ folder if needed.
+
+The classes should be _MCollective::Util::Yourclass_ and you should use the following construct to require them from disk:
+
+{% highlight ruby %}
+MCollective::Util.loadclass("MCollective::Util::Yourclass")
+{% endhighlight %}
+
+Create _util/yourclass.rb_ with this content :
+
+{% highlight ruby %}
+module MCollective
+ module Util
+ class Yourclass
+ # The class definition here
+ end
+ end
+end
+{% endhighlight %}
+
+_loadclass_ on _Yourclass_ will automatically search for a _yourclass.rb_ file (lowercase).
+
+At present simply requiring them will work and we'll hope to maintain that but to be 100% future safe use this method.
+
+
+It also loads modules in exactly the same way.
+
+## The Client
+We provide a client library that abstracts away a lot of the work in writing simple attractive cli frontends to your agents that supports discovery, filtering, generates help etc. The client lib is functional but we will improve/refactor the options parsing a bit in future.
+
+{% highlight ruby linenos %}
+#!/usr/bin/ruby
+
+require 'mcollective'
+
+oparser = MCollective::Optionparser.new({}, "filter")
+
+options = oparser.parse{|parser, options|
+ parser.define_head "Tester for the echo agent"
+ parser.banner = "Usage: mc-echo [options] msg"
+}
+
+if ARGV.length > 0
+ msg = ARGV.shift
+else
+ puts("Please specify a message to send")
+ exit 1
+end
+
+begin
+ options[:filter]["agent"] = "echo"
+
+ client = MCollective::Client.new(options[:config])
+
+ stats = client.discovered_req(msg, "echo", options) do |resp|
+ next if resp == nil
+
+ printf("%30s> %s\n", resp[:senderid], resp[:body])
+ end
+rescue Exception => e
+ puts("Failed to contact any agents: #{e}")
+ exit 1
+end
+
+client.display_stats(stats, options)
+{% endhighlight %}
+
+We can test it works as below:
+
+{% highlight console %}
+% ./mc-echo --config etc/client.cfg "hello world"
+Determining the amount of hosts matching filter for 2 seconds .... 1
+
+ devel.your.com> hello world
+
+Finished processing 1 / 1 hosts in 45.02 ms
+{% endhighlight %}
+
+Verbose statistics:
+
+{% highlight console %}
+% ./mc-echo --config etc/client.cfg "hello world" -v
+Determining the amount of hosts matching filter for 2 seconds .... 1
+
+ devel.your.com> hello world
+
+---- stomp call summary ----
+ Nodes: 1 / 1
+ Start Time: Tue Nov 03 23:28:34 +0000 2009
+ Discovery Time: 2002.27ms
+ Agent Time: 45.84ms
+ Total Time: 2048.11ms
+{% endhighlight %}
+
+Discovery and filtering:
+
+{% highlight console %}
+% ./mc-echo --config etc/client.cfg "hello world" --with-fact country=au
+
+Failed to contact any agents: No matching clients found
+
+% ./mc-echo --config etc/client.cfg "hello world" --with-fact country=uk
+Determining the amount of hosts matching filter for 2 seconds .... 1
+
+ devel.your.com> hello world
+
+Finished processing 1 / 1 hosts in 38.77 ms
+{% endhighlight %}
+
+Standard Help:
+
+{% highlight console %}
+% ./mc-echo --help
+Usage: mc-echo [options] msg
+Tester for the echo agent
+
+Common Options
+ -c, --config FILE Load configuratuion from file rather than default
+ --dt SECONDS Timeout for doing discovery
+ --discovery-timeout
+ -t, --timeout SECONDS Timeout for calling remote agents
+ -q, --quiet Do not be verbose
+ -v, --verbose Be verbose
+ -h, --help Display this screen
+
+Host Filters
+ --wf, --with-fact fact=val Match hosts with a certain fact
+ --wc, --with-class CLASS Match hosts with a certain configuration management class
+ --wa, --with-agent AGENT Match hosts with a certain agent
+ --wi, --with-identity IDENT Match hosts with a certain configured identity
+{% endhighlight %}
--- /dev/null
+---
+layout: default
+title: Using MCollective Command Line Applications
+---
+MCollective is designed first and foremost for the CLI. You will mostly
+interact with a single executable called *mco* which has a number of
+sub-commands, arguments and flags.
+
+## Basic Usage of the *mco* Command
+
+A simple example of a *mco* command can be seen below:
+
+{% highlight console %}
+% mco ping
+dev8 time=126.19 ms
+dev6 time=132.79 ms
+dev10 time=133.57 ms
+.
+.
+
+---- ping statistics ----
+25 replies max: 305.58 min: 57.50 avg: 113.16
+{% endhighlight %}
+
+In this example the *ping* sub-command is referred to as an
+*application*. Mcollective provides many applications, for a list of
+them, type *mco help*. You can also create your own application to plug
+into the framework. The *help* sub-command will show you something like
+this:
+
+
+{% highlight console %}
+% mco help
+The Marionette Collective version 2.0.0
+
+ controller Control the mcollective daemon
+ facts Reports on usage for a specific fact
+ find Find hosts matching criteria
+ help Application list and help
+ inventory General reporting tool for nodes, collectives and subcollectives
+ ping Ping all nodes
+ plugin MCollective Plugin Application
+ rpc Generic RPC agent client application
+{% endhighlight %}
+
+You can request help for a specific application using either *mco help
+application* or *mco application ---help*. Shown below is part of the
+help for the *rpc* application:
+
+{% highlight console %}
+% mco help rpc
+Generic RPC agent client application
+
+Usage: mco rpc [options] [filters] --agent <agent> --action <action> [--argument <key=val> --argument ...]
+Usage: mco rpc [options] [filters] <agent> <action> [<key=val> <key=val> ...]
+
+ --no-results, --nr Do not process results, just send request
+ -a, --agent AGENT Agent to call
+ --action ACTION Action to call
+ --arg, --argument ARGUMENT Arguments to pass to agent
+
+ --np, --no-progress Do not show the progress bar
+ -1, --one Send request to only one discovered nodes
+ --batch SIZE Do requests in batches
+ --batch-sleep SECONDS Sleep time between batches
+ --limit-nodes, --ln COUNT Send request to only a subset of nodes, can be a percentage
+ -j, --json Produce JSON output
+ -c, --config FILE Load configuratuion from file rather than default
+ -v, --verbose Be verbose
+ -h, --help Display this screen
+
+Common Options
+ -T, --target COLLECTIVE Target messages to a specific sub collective
+ --dt, --discovery-timeout SECONDS
+ Timeout for doing discovery
+ -t, --timeout SECONDS Timeout for calling remote agents
+ -q, --quiet Do not be verbose
+ --ttl TTL Set the message validity period
+ --reply-to TARGET Set a custom target for replies
+
+Host Filters
+ -W, --with FILTER Combined classes and facts filter
+ -S, --select FILTER Compound filter combining facts and classes
+ -F, --wf, --with-fact fact=val Match hosts with a certain fact
+ -C, --wc, --with-class CLASS Match hosts with a certain config management class
+ -A, --wa, --with-agent AGENT Match hosts with a certain agent
+ -I, --wi, --with-identity IDENT Match hosts with a certain configured identity
+
+The Marionette Collective 2.0.0
+{% endhighlight %}
+
+The *help* first shows a basic overview of the command line syntax
+followed by options specific to this command. Following that you will
+see some *Common Options* and *Host Filters* that generally apply to
+most applications.
+
+## Making RPC Requests
+
+### Overview of a Request
+
+The *rpc* application is the main application used to make requests to
+your servers. It is capable of interacting with any standard Remote
+Procedure Call (RPC) agent. Below is an example that shows an attempt to
+start a webserver on several machines:
+
+{% highlight console %}
+% mco rpc service start service=httpd
+Determining the amount of hosts matching filter for 2 seconds .... 10
+
+ * [ ============================================================> ] 10 / 10
+
+dev4 Request Aborted
+ Could not start Service[httpd]: Execution of '/sbin/service httpd start' returned 1:
+
+
+Finished processing 10 / 10 hosts in 1323.61 ms
+{% endhighlight %}
+
+The order of events in this process are:
+
+ * Perform discovery against the network and discover 10 servers
+ * Send the request and then show a progress bar of the replies
+ * Show any results that were out of the ordinary
+ * Show some statistics
+
+Mcollective client applications aim to only provide the most relevant
+information. In this case, the application is not showing verbose
+information about the nine *OK* results, since the most important issue
+is the one *Failure*. Keep this in mind when viewing the results of
+commands.
+
+### Anatomy of a Request
+
+MCollective agents are broken up into actions and each action can take
+input arguments.
+
+{% highlight console %}
+% mco rpc service stop service=httpd
+{% endhighlight %}
+
+This shows the basic make-up of an RPC command. In this case we are:
+
+ * using the *rpc* application - a generic application that can interact with any agent
+ * directing our request to machines with the *service* agent
+ * sending a request to the *stop* action of the service agent
+ * supplying a value, *httpd*, to the *service* argument of the *stop* action
+
+The same command has a longer form as well:
+
+{% highlight console %}
+% mco rpc --agent service --action stop --argument service=httpd
+{% endhighlight %}
+
+These two commands are functionally identical.
+
+### Discovering Available *Agents*
+
+The above command showed you how to interact with the *service* agent,
+but how can you find out that this agent even exists? On a correctly
+installed MCollective system you can use the *plugin* application to get
+a list:
+
+{% highlight console %}
+% mco plugin doc
+Please specify a plugin. Available plugins are:
+
+Agents:
+ package Install and uninstall software packages
+ puppetd Run puppet agent, get its status, and enable/disable it
+ rpcutil General helpful actions that expose stats and internals to SimpleRPC clients
+ service Agent to manage services
+{% endhighlight %}
+
+The first part of this list shows all the agents this computer is aware
+of. In order to show up on this list, an agent must have a *DDL* file
+and be installed locally.
+
+To find out the *actions*, *inputs* and *outputs* for a specific agent
+use the plugin application again:
+
+{% highlight console %}
+% mco plugin doc service
+SimpleRPC Service Agent
+=======================
+
+Agent to manage services
+
+ Author: R.I.Pienaar
+ Version: 1.2
+ License: GPLv2
+ Timeout: 60
+ Home Page: http://mcollective-plugins.googlecode.com/
+
+
+
+ACTIONS:
+========
+ restart, start, status, stop
+
+ status action:
+ --------------
+ Gets the status of a service
+
+ INPUT:
+ service:
+ Description: The service to get the status for
+ Prompt: Service Name
+ Type: string
+ Validation: ^[a-zA-Z\-_\d]+$
+ Length: 30
+
+
+ OUTPUT:
+ status:
+ Description: The status of service
+ Display As: Service Status
+{% endhighlight %}
+
+This shows a truncated example of the auto-generated help for the
+*service* agent. First shown is metadata such as version, author and
+license. This is followed by the list of actions available, in this case
+the *restart*, *start*, *status* and *stop* actions.
+
+Further information is shown about each action. For example, you can see
+that the *status* action requires an input called *service* which is a
+string, has a maximum length of 30, etc. You can also see you will
+receive one output called *status*
+
+With this information, you can request the status for a specific
+service:
+
+{% highlight console %}
+% mco rpc service status service=httpd
+Determining the amount of hosts matching filter for 2 seconds .... 10
+
+ * [ ============================================================> ] 10 / 10
+
+
+dev1
+ Service Status: stopped
+
+dev4
+ Service Status: stopped
+
+.
+.
+.
+
+Finished processing 10 / 10 hosts in 326.01 ms
+{% endhighlight %}
+
+Unlike the previous example, in this case specific information is
+returned on the success of the action. This is because this specific
+action is meant to retrieve information and so mcollective assumes you
+would like to see complete, thorough data regardless of success or
+failure.
+
+Note that this output displays *Service Status* as shown in the *mco
+plugin doc service* help page. Any time you need more information about
+a display name, the doc for the associated agent will have a
+*Description* section for every input and output.
+
+## Selecting Request Targets Using *Filters*
+
+### Basic Filters
+
+A key capability of mcollective is fast discovery of network resources.
+Discovery rules are written using *filters*. For example:
+
+{% highlight console %}
+% mco rpc service status service=httpd -S "environment=development or customer=acme"
+{% endhighlight %}
+
+This shows a filter rule that limits the RPC request to being run on
+machines that are either in the Puppet environment *development* or
+belong to the Customer *acme*.
+
+Filtering can be based on *facts*, the presence of a *Configuration
+Management Class* on the node, the node's *Identity*, or installed
+*Agents* on the node.
+
+Here are a number of examples of this with short descriptions of each
+filter:
+
+{% highlight console %}
+# all machines with the service agent
+% mco ping -A service
+% mco ping --with-agent service
+
+# all machines with the apache class on them
+% mco ping -C apache
+% mco ping --with-class apache
+
+# all machines with a class that match the regular expression
+% mco ping -C /service/
+
+# all machines in the UK
+% mco ping -F country=uk
+% mco ping --with-fact country=uk
+
+# all machines in either UK or USA
+% mco ping -F "country=/uk|us/"
+
+# just the machines called dev1 or dev2
+% mco ping -I dev1 -I dev2
+
+# all machines in the domain foo.com
+% mco ping -I /foo.com$/
+{% endhighlight %}
+
+As you can see, you can filter by Agent, Class and/or Fact, and you can
+use regular expressions almost anywhere. You can also combine filters
+additively in a command so that all the criteria have to be matched.
+
+Note: You can use a shortcut to combine Class and Fact filters:
+
+{% highlight console %}
+# all machines with classes matching /apache/ in the UK
+% mco ping -W "/apache/ location=uk"
+{% endhighlight %}
+
+### Complex *Compound* or *Select* Queries
+
+While the above examples are easy to enter, they are limited in that
+they can only combine filters additively. If you want to create searches
+with more complex boolean logic use the *-S* switch. For example:
+
+{% highlight console %}
+% mco ping -S "((customer=acme and environment=staging) or environment=development) and /apache/"
+{% endhighlight %}
+
+The above example shows a scenario where the development environment is
+usually labeled *development* but one customer has chosen to use
+*staging*. You want to find all machines in those customer's
+environments that match the class *apache*. This search would be
+impossible using the previously shown methods, but the above command
+uses *-S* to allow the use of boolean operators such as *and* and *or*
+so you can easily build the logic of the search.
+
+The *-S* switch also allows for negative matches using *not* or *!*:
+
+{% highlight console %}
+% mco ping -S "environment=development and !customer=acme"
+% mco ping -S "environment=development and not customer=acme"
+{% endhighlight %}
+
+### Filtering Using Data Plugins
+
+As of version 2.1.0, custom data plugins can also be used to create
+complex filters:
+
+{% highlight console %}
+% mco ping -S "fstat('/etc/hosts').md5=/baa3772104/ and environment=development"
+{% endhighlight %}
+
+This will search for the md5 hash of a specific file with matches
+restricted to the *development* environment. Note that as before,
+regular expressions can also be used.
+
+As with agents, you can also discover which plugins are available for
+use:
+
+{% highlight console %}
+% mco plugin doc
+
+Please specify a plugin. Available plugins are:
+
+Agents:
+ .
+ .
+
+Data Queries:
+ agent Meta data about installed MColletive Agents
+ augeas_match Allows agents and discovery to do Augeas match lookups
+ fstat Retrieve file stat data for a given file
+ resource Information about Puppet managed resources
+ sysctl Retrieve values for a given sysctl
+{% endhighlight %}
+
+For information on the input these plugins take and output they provide
+use the *mco plugin doc fstat* command.
+
+Currently, each data function can only accept one input while matches
+are restricted to a single output field per invocation.
+
+## Chaining RPC Requests
+
+The *rpc* application can chain commands one after the other. The
+example below uses the *package* agent to find machines with a specific
+version of mcollective and then schedules Puppet runs on those machines:
+
+{% highlight console %}
+% mco rpc package status package=mcollective -j|jgrep "data.properties.ensure=2.0.0-6.el6" |mco rpc puppetd runonce
+{% endhighlight %}
+
+Mcollective results can also be filtered using the opensource gem,
+jgrep. Mcollective data output is fully compatible with jgrep.
+
+## Seeing the Raw Data
+
+By default the *rpc* application will try to show human-readable data.
+To see the actual raw data, add the *-v* flag to disable the display
+helpers:
+
+{% highlight console %}
+% mco rpc nrpe runcommand command=check_load -I dev1 -v
+.
+.
+dev1 : OK
+ {:exitcode=>0, :output=>"OK - load average: 0.00, 0.00, 0.00", :perfdata=> "load1=0.000;1.500;2.000;0; load5=0.000;1.500;2.000;0; load15=0.000;1.500;2.000;0;"}
+{% endhighlight %}
+
+
+This data can also be returned in JSON format:
+
+{% highlight console %}
+% mco rpc nrpe runcommand command=check_load -I dev1 -j
+[
+ {
+ "action": "runcommand",
+ "agent": "nrpe",
+ "data": {
+ "exitcode": 0,
+ "output": "OK - load average: 0.00, 0.00, 0.00",
+ "perfdata": "load1=0.000;1.500;2.000;0; load5=0.000;1.500;2.000;0; load15=0.000;1.500;2.000;0;"
+ },
+ "statuscode": 0,
+ "statusmsg": "OK",
+ "sender": "dev1"
+ }
+]
+{% endhighlight %}
+
+## Error Messaging
+
+When an application encounters an error, it returns an explanatory
+string:
+
+{% highlight console %}
+% mco rpc rpcutil foo
+rpc failed to run: Attempted to call action foo for rpcutil but it's not declared in the DDL (MCollective::DDLValidationError)
+{% endhighlight %}
+
+By default only an abbreviated error string is shown that provides some
+insight into the nature of the problem. For more details, add the *-v*
+flag to show a full stack trace:
+
+{% highlight console %}
+% mco rpc rpcutil foo -v
+rpc failed to run: Attempted to call action foo for rpcutil but it's not declared in the DDL (MCollective::DDLValidationError)
+ from /usr/lib/ruby/site_ruby/1.8/mcollective/ddl.rb:303:in `validate_rpc_request'
+ from /usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:218:in `method_missing'
+ from /home/rip/.mcollective.d/lib/mcollective/application/rpc.rb:133:in `send'
+ from /home/rip/.mcollective.d/lib/mcollective/application/rpc.rb:133:in `main'
+ from /usr/lib/ruby/site_ruby/1.8/mcollective/application.rb:283:in `run'
+ from /usr/lib/ruby/site_ruby/1.8/mcollective/applications.rb:23:in `run'
+ from /usr/bin/mco:20
+{% endhighlight %}
+
+## Custom Applications
+The *rpc* application should suit most needs. However, sometimes the
+data being returned calls for customization such as custom aggregation,
+summarising or complete custom display.
+
+In such cases, a custom application may be useful For example, the
+*package* application provides concluding summaries and provides some
+basic safe guards for its use. The agent also provides the commonly
+required data. Typical *package* output looks like this:
+
+{% highlight console %}
+% mco package status kernel
+Do you really want to operate on packages unfiltered? (y/n): y
+
+ * [ ============================================================> ] 25 / 25
+
+
+ dev5 version = kernel-2.6.32-220.7.1.el6
+ dev9 version = kernel-2.6.32-220.2.1.el6
+ .
+ .
+
+---- package agent summary ----
+ Nodes: 25 / 25
+ Versions: 9 * 2.6.32-220.2.1.el6, 9 * 2.6.32-220.4.1.el6, 7 * 2.6.32-220.el6
+ Elapsed Time: 3.95 s
+{% endhighlight %}
+
+
+Notice how this application recognises that you are acting on all
+possible machines, an action which might have a big impact on your YUM
+servers. Consequently, *package* prompts for confirmation and, at the
+end of processing, displays a brief summary of the network status.
+
+While the behaviors of custom applications are not always consistant
+with each other, in general they accept the standard discovery flags.
+For details of which flags are accepted in a given application, use the
+*mco help appname* command.
+
+To discover which custom applications are available, run *mco* or *mco
+help*.
--- /dev/null
+---
+layout: default
+title: Configuration Guide
+---
+
+[SSLSecurity]: /mcollective/reference/plugins/security_ssl.html
+[AESSecurity]: /mcollective/reference/plugins/security_aes.html
+[Registration]: /mcollective/reference/plugins/registration.html
+[Auditing]: /mcollective/simplerpc/auditing.html
+[Authorization]: /mcollective/simplerpc/authorization.html
+[Subcollectives]: /mcollective/reference/basic/subcollectives.html
+[server_config]: /mcollective/configure/server.html
+
+> **Note:** There is a new [Server Configuration Reference][server_config] page with a more complete overview of MCollective's server daemon settings. A similar client configuration page is forthcoming.
+
+This guide tells you about the major configuration options in the daemon and client config files. There are options not mentioned
+here typically ones specific to a certain plugin.
+
+## Configuration Files
+There are 2 configuration files, one for the client and one for the server, these default to */etc/mcollective/server.cfg* and */etc/mcollective/client.cfg*.
+
+Configuration is a simple *key = val* style configuration file.
+
+## Common Options
+
+|Key|Sample|Description|
+|---|------|-----------|
+|collectives|mcollective,subcollective|A list of [Subcollectives] to join - 1.1.3 and newer only|
+|main_collective|mcollective|The main collective to target - 1.1.3 and newer only|
+|logfile|/var/log/mcollective.log|Where to log|
+|loglevel|debug|Can be info, warn, debug, fatal, error|
+|identity|dev1.your.com|Identifier for this node, does not need to be unique, defaults to hostname if unset and must match _/\w\.\-/_ if set|
+|keeplogs|5|The amount of logs to keep|
+|max_log_size|2097152|Max size in bytes for log files before rotation happens|
+|libdir|/usr/libexec/mcollective:/site/mcollective|Where to look for plugins|
+|connector|activemq|Which _connector_ plugin to use for communication|
+|securityprovider|Psk|Which security model to use, see [SSL Security Plugin][SSLSecurity] and [AES Security Plugin][AESSecurity] for details on others|
+|rpchelptemplate|/etc/mcollective/rpc-help.erb|The path to the erb template used for generating help|
+|helptemplatedir|/etc/mcollective|The path that contains all the erb templates for generating help|
+|logger_type|file|Valid logger types, currently file, syslog or console|
+|ssl_cipher|aes-256-cbc|This sets the cipher in use by the SSL code, see _man enc_ for a list supported by OpenSSL|
+|direct_addressing|n|Enable or disable directed requests|
+|direct_addressing_threshold|10|When direct requests are enabled, send direct messages for less than or equal to this many hosts|
+|ttl|60|Sets the default TTL for requests - 1.3.2 and newer only|
+|logfacility|When using the syslog logger sets the facility, defaults to user|
+|default_discovery_method|The default method to use for discovery - _mc_ by default|
+|default_discovery_options|Options to pass to the discovery plugin, empty by default|
+
+## Server Configuration
+The server configuration file should be root only readable
+
+|Key|Sample|Description|
+|---|------|-----------|
+|daemonize|1|Runs the server in the background|
+|factsource|Facter|Which fact plugin to use|
+|registration|Agentlist|[Registration] plugin to use|
+|registerinterval|120|How many seconds to sleep between registration messages, setting this to zero disables registration|
+|registration_collective|development|Which sub-collective to send registration messages to|
+|classesfile|/var/lib/puppet/classes.txt|Where to find a list of classes configured by your configuration management system|
+|rpcaudit|1|Enables [SimpleRPC Auditing][Auditing]|
+|rpcauditprovider|Logfile|Enables auditing using _MCollective::Audit::Logfile_|
+|rpcauthorization|1|Enables [SimpleRPC Authorization][Authorization] globally|
+|rpcauthprovider|action_policy|Use the _MCollective::Util::ActionPolicy_ plugin to manage authorization|
+|rpclimitmethod|The method used for --limit-results. Can be either _first_ or _random_|
+|fact_cache_time|300|How long to cache fact results for before refreshing from source|
+
+The last example sets a option for the _discovery_ plugin, you can also set this in _/etc/mcollective/plugin.d/discovery.cfg_, in that case
+you'd just set _timeout=10_ in the file.
+
+## Client Configuration
+The client configuration file should be readable by everyone, it's not advised to put PSK's or client SSL certs in a world readable file, see below how to do that per user.
+
+|Key|Sample|Description|
+|---|------|-----------|
+|color|0|Disables the use of color in RPC results|
+
+## Plugin Configuration
+You can add free form config options for plugins, they take the general form like:
+
+{% highlight ini %}
+ plugin.pluginname.option = value
+{% endhighlight %}
+
+Each plugin's documentation should tell you what options are availble.
+
+Common plugin options are:
+
+|Key|Sample|Description|
+|---|------|-----------|
+|plugin.yaml|/etc/mcollective/facts.yaml:/other/facts.yaml|Where the yaml fact source finds facts from, multiples get merged|
+|plugin.psk|123456789|The pre-shared key to use for the Psk security provider|
+|plugin.psk.callertype|group|What to base the callerid on for the PSK plugin, uid, gid, user, group or identity|
+
+## Client Setup
+It's recommended that you do not set host, user, password and Psk in the client configuration file since these files are generally world readable unlike the server one that should be root readable only. I just set mine to *unset* so it's clear to someone who looks at the config file that it's not going to work without the settings below.
+
+You can also put client configuration in _~/.mcollective_ as an alternative to the method below, but you will need a full client.cfg then in that location.
+
+You can set various Environment variables per user to supply these values:
+
+{% highlight bash %}
+export STOMP_USER=user
+export STOMP_PASSWORD=password
+export MCOLLECTIVE_PSK=123456789
+{% endhighlight %}
+
+You an set options that will always apply using the _MCOLLECTIVE_EXTRA_OPTS_ as below:
+
+{% highlight bash %}
+export MCOLLECTIVE_EXTRA_OPTS="--dt 5 --timeout 3 --config /home/you/mcollective.cfg"
+{% endhighlight %}
+
+The client library will use these and so you can give each user who use the admin utilities their own username and rights.
--- /dev/null
+---
+layout: default
+title: Controlling the Daemon
+---
+
+The main daemon that runs on nodes keeps internal stats and supports reloading of agents and changing
+logging level without restarting.
+
+If you want to reload all the agents without restarting the daemon you can just send it signal *USR1*
+and it will reload its agents.
+
+You can send *USR2* to cycle the log level through DEBUG to FATAL and back again, just keep sending
+the signal and look at the logs.
+
+Reloading agents work in most cases though we recommend a full daemon restart in production use
+due to the nature of the ruby class loading system. If you are changing agent contents and relying
+on the reload behavior you might end up with agents not being in a consistent state.
+
+## Obtaining daemon statistics
+
+The daemon keeps a number of statistics about its operation, you can view these using the _inventory_
+application:
+
+{% highlight console %}
+% mco inventory example.com
+ Server Statistics:
+ Version: 2.2.0
+ Start Time: Mon Sep 24 17:37:28 +0100 2012
+ Config File: /etc/mcollective/server.cfg
+ Collectives: mcollective, fr_collective
+ Main Collective: mcollective
+ Process ID: 24473
+ Total Messages: 52339
+ Messages Passed Filters: 44118
+ Messages Filtered: 8221
+ Expired Messages: 0
+ Replies Sent: 29850
+ Total Processor Time: 527.06 seconds
+ System Time: 349.32 seconds
+
+.
+.
+.
+{% endhighlight %}
+
+The statistics mean:
+
+|Statistic |Meaning |
+|------------|-------------------------------------------|
+|Start Time |Local time on the node when the daemon started|
+|Collectives |All known collectives this agent responds on|
+|Main Collective |The primary collective|
+|Process ID |The process ID of the mcollectived|
+|Total Messages |Total messages received from the middleware|
+|Messages Passed Filters|Amount of messages that was determined to be applicable to this node based on filters|
+|Messages Filtered |Messages that did not apply to this node|
+|Expired Messages |Received messages that had expired their TTL values|
+|Replies Sent |Not all received messages result in replies, this counts the actual replies sent|
+|Total Processor Time |Processor time including user and system time consumed since start|
+|System Time |System Processor time only|
+
+You can get the raw values using the *rpcutil* agent using the *daemon_stats* action.
+
+{% highlight console %}
+% mco rpc rpcutil daemon_stats
+Discovering hosts using the mongo method .... 26
+
+ * [ ============================================================> ] 26 / 26
+
+.
+.
+.
+
+example.com
+ Agents: ["stomputil",
+ "nrpe",
+ "package",
+ "rpcutil",
+ "rndc",
+ "urltest",
+ "iptables",
+ "puppetd",
+ "discovery",
+ "service",
+ "eximng",
+ "filemgr",
+ "process"]
+ Config File: /etc/mcollective/server.cfg
+ Failed Filter: 168432
+ Passed Filter: 91231
+ PID: 1418
+ Replies: 91127
+ Start Time: 1347545937
+ Threads: ["#<Thread:0x7f44350964f8 sleep>",
+ "#<Thread:0x7f4434f7f538 sleep>",
+ "#<Thread:0x7f44390ce368 sleep>",
+ "#<Thread:0x7f44350981b8 run>"]
+ Times: {:cutime=>1111.13, :utime=>3539.21, :cstime=>1243.64, :stime=>5045.21}
+ Total Messages: 259842
+ TTL Expired: 179
+ Failed Security: 0
+ Security Validated: 259842
+ Version: 2.2.0
+
+
+Summary of Agents:
+
+ package = 26
+ process = 26
+ discovery = 26
+ service = 26
+ puppetd = 26
+ filemgr = 26
+ nrpe = 26
+ rpcutil = 26
+ stomputil = 26
+ iptables = 11
+ urltest = 7
+ libvirt = 4
+ eximng = 4
+ registration = 3
+ rndc = 3
+ angelianotify = 2
+
+Summary of Version:
+
+ 2.2.0 = 26
+
+Finished processing 26 / 26 hosts in 289.20 ms
+{% endhighlight %}
+
+## Obtaining running configuration settings
+
+All configuration settings of any mcollective daemon can be retrieved using the *get_config_item*
+action of the *rpcutil* agent:
+
+{% highlight console %}
+% mco rpc rpcutil get_config_item item=collectives -I example.com
+Discovering hosts using the mongo method .... 1
+
+ * [ ============================================================> ] 1 / 1
+
+
+example.com:
+ Property: collectives
+ Value: ["mcollective", "fr_collective"]
+
+
+Summary of Value:
+
+ mcollective = 1
+ fr_collective = 1
+
+
+Finished processing 1 / 1 hosts in 59.05 ms
+{% endhighlight %}
--- /dev/null
+---
+layout: default
+title: Getting Started
+---
+[Screencasts]: /mcollective/screencasts.html
+[ActiveMQ]: http://activemq.apache.org/
+[RabbitMQ]: http://www.rabbitmq.com/
+[EC2Demo]: /mcollective/ec2demo.html
+[Stomp]: http://stomp.codehaus.org/Ruby+Client
+[DepRPMs]: http://www.marionette-collective.org/activemq/
+[DebianBug]: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=562954
+[SecurityWithActiveMQ]: /mcollective/reference/integration/activemq_security.html
+[ActiveMQClustering]: http://www.devco.net/archives/2009/11/10/activemq_clustering.php
+[ActiveMQSamples]: http://github.com/puppetlabs/marionette-collective/tree/master/ext/activemq/examples/
+[ConfigurationReference]: /mcollective/reference/basic/configuration.html
+[Terminology]: /mcollective/terminology.html
+[SimpleRPCIntroduction]: /mcollective/simplerpc/
+[ControllingTheDaemon]: /mcollective/reference/basic/daemon.html
+[SSLSecurityPlugin]: /mcollective/reference/plugins/security_ssl.html
+[AESSecurityPlugin]: /mcollective/reference/plugins/security_aes.html
+[ConnectorActiveMQ]: /mcollective/reference/plugins/connector_activemq.html
+[ConnectorRabbitMQ]: /mcollective/reference/plugins/connector_rabbitmq.html
+[MessageFlowCast]: /mcollective/screencasts.html#message_flow
+[Plugins]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki
+[RedHatGuide]: gettingstarted_redhat.html
+[DebianGuide]: gettingstarted_debian.html
+[server_config]: /mcollective/configure/server.html
+
+*NOTE:* This is an older, deprecated version of the Getting Started documentation. Red Hat and CentOS users can refer to [our Redhat guide][RedHatGuide]. Debian users can refer to [our Debian guide][DebianGuide]. Users on less common platforms should adapt one of those two instead of continuing to read this page.
+
+Below find a rough guide to get you going, this assumes the client and server is on the same node, but servers don't need the client code installed.
+
+Look at the [Screencasts] page, there are [some screencasts dealing with basic architecture, terminology and so forth][MessageFlowCast] that you might find helpful before getting started.
+
+## Requirements
+We try to keep the requirements on external Gems to a minimum, you only need:
+
+ * A Stomp server, tested against [ActiveMQ] and [RabbitMQ]
+ * Ruby
+ * Rubygems
+ * [Ruby Stomp Client][Stomp]
+
+RPMs for these are available [here][DepRPMs].
+
+## ActiveMQ
+ActiveMQ is currently the most used middleware for MCollective, it would be our recommended choice and one
+that the community has most experience supporting. There is a specific connector for RabbitMQ if you wish
+to go that route though - see [ConnectorRabbitMQ] for details. This guide will focus on ActiveMQ.
+
+Full details on setting up and configuring ActiveMQ is out of scope for this, but you can follow these simple
+setup instructions for initial testing (make sure JDK is installed, see below for Debian specific issue regarding JDK):
+
+### Download and Install
+ 1. Download the ActiveMQ "binary" package (for Unix) from [ActiveMQ]
+ 1. Extract the contents of the archive:
+ 1. cd into the activemq directory
+ 1. Execute the activemq binary
+
+{% highlight console %}
+ % tar xvzf activemq-x.x.x.tar.gz
+ % cd activemq-x.x.x
+ % bin/activemq
+{% endhighlight %}
+
+Below should help you get stomp and a user going. For their excellent full docs please see [ActiveMQ].
+There are also tested configurations in [the ext directory][ActiveMQSamples]
+
+A spec file can be found in the *ext* directory on GitHub that can be used to build RPMs for RedHat/CentOS/Fedora
+you need *tanukiwrapper* which you can find from *jpackage*, it runs fine under OpenJDK that comes with recent
+versions of these Operating Systems. I've uploaded some RPMs and SRPMs [here][DepRPMs].
+
+For Debian systems you'd be better off using OpenJDK than Sun JDK, there's a known issue [#562954][DebianBug].
+
+### Configuring Stomp
+First you should configure ActiveMQ to listen on the Stomp protocol
+
+And then you should add a user or two, to keep it simple we'll just add one user, the template file will hopefully make it obvious where this goes, it should be in the _broker_ block:
+
+*Note: This config is for ActiveMQ 5.4*
+
+{% highlight xml %}
+<beans
+ xmlns="http://www.springframework.org/schema/beans"
+ xmlns:amq="http://activemq.apache.org/schema/core"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
+ http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd
+ http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
+
+ <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" useJmx="true">
+ <managementContext>
+ <managementContext createConnector="false"/>
+ </managementContext>
+
+ <plugins>
+ <statisticsBrokerPlugin/>
+ <simpleAuthenticationPlugin>
+ <users>
+ <authenticationUser username="mcollective" password="marionette" groups="mcollective,everyone"/>
+ <authenticationUser username="admin" password="secret" groups="mcollective,admin,everyone"/>
+ </users>
+ </simpleAuthenticationPlugin>
+ <authorizationPlugin>
+ <map>
+ <authorizationMap>
+ <authorizationEntries>
+ <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
+ <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <authorizationEntry queue="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
+ <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
+ </authorizationEntries>
+ </authorizationMap>
+ </map>
+ </authorizationPlugin>
+ </plugins>
+
+ <systemUsage>
+ <systemUsage>
+ <memoryUsage>
+ <memoryUsage limit="20 mb"/>
+ </memoryUsage>
+ <storeUsage>
+ <storeUsage limit="1 gb" name="foo"/>
+ </storeUsage>
+ <tempUsage>
+ <tempUsage limit="100 mb"/>
+ </tempUsage>
+ </systemUsage>
+ </systemUsage>
+
+ <transportConnectors>
+ <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
+ <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
+ </transportConnectors>
+ </broker>
+</beans>
+{% endhighlight %}
+
+This creates a user *mcollective* with the password *marionette* and give it access to read/write/admin */topic/mcollective.`*`*.
+
+Save the above code as activemq.xml and run activemq as - if installing from a package probably _/etc/activemq/activemq.xml_:
+
+Else your package would have a RC script:
+
+{% highlight console %}
+# /etc/init.d/activemq start
+{% endhighlight %}
+
+For further info about ActiveMQ settings you might need see [SecurityWithActiveMQ] and [ActiveMQClustering].
+
+There are also a few known to work and tested [configs in git][ActiveMQSamples].
+
+## mcollective
+
+### Download and Extract
+Grab a copy of the mcollective ideally you'd use a package for your distribution else there's a tarfile that
+you can use, you can extract it wherever you want, the RPMs or deps will put files in Operating System compatible
+locations. If you use the tarball you'll need to double check all the paths in the config files below.
+
+### Configure
+You'll need to tweak some configs in */etc/mcollective/client.cfg*, a full reference of config settings can be
+found [here][ConfigurationReference]:
+
+Mostly what you'll need to change is the *identity*, *plugin.activemq.1.`*`* and the *plugin.psk*:
+
+{% highlight ini %}
+ # main config
+ libdir = /usr/libexec/mcollective
+ logfile = /dev/null
+ loglevel = debug
+ identity = fqdn
+
+ # connector plugin config
+ connector = activemq
+ plugin.activemq.pool.size = 1
+ plugin.activemq.pool.1.host = stomp.your.net
+ plugin.activemq.pool.1.port = 61613
+ plugin.activemq.pool.1.user = unset
+ plugin.activemq.pool.1.password = unset
+
+ # security plugin config
+ securityprovider = psk
+ plugin.psk = abcdefghj
+{% endhighlight %}
+
+You should also create _/etc/mcollective/server.cfg_ here's a sample, a full reference of config settings can be found on the [Server Configuration Reference][server_config]:
+
+{% highlight ini %}
+ # main config
+ libdir = /usr/libexec/mcollective
+ logfile = /var/log/mcollective.log
+ daemonize = 1
+ keeplogs = 1
+ max_log_size = 10240
+ loglevel = debug
+ identity = fqdn
+ registerinterval = 300
+
+ # connector plugin config
+ connector = activemq
+ plugin.activemq.pool.size = 1
+ plugin.activemq.pool.1.host = stomp.your.net
+ plugin.activemq.pool.1.port = 61613
+ plugin.activemq.pool.1.user = mcollective
+ plugin.activemq.pool.1.password = password
+
+ # facts
+ factsource = yaml
+ plugin.yaml = /etc/mcollective/facts.yaml
+
+ # security plugin config
+ securityprovider = psk
+ plugin.psk = abcdefghj
+{% endhighlight %}
+
+Replace the *plugin.activemq.pool.1.host* with your server running ActiveMQ and replace the *plugin.psk* with a Pre-Shared Key of your own.
+
+The ActiveMQ connector supports other options like failover pools, see [ConnectorActiveMQ] for full details.
+
+### Create Facts
+By default - and for this setup - we'll use a simple YAML file for a fact source, later on you can use Puppet Labs Facter or something else.
+
+Create */etc/mcollective/facts.yaml* along these lines:
+
+{% highlight yaml %}
+ ---
+ location: devel
+ country: uk
+{% endhighlight %}
+
+### Start the Server
+If you installed from a package start it with the RC script, else look in the source you'll find a LSB compatible RC script to start it.
+
+### Test from a client
+If all is setup you can test with the client code:
+
+{% highlight console %}
+% mco ping
+your.domain.com time=74.41 ms
+
+---- ping statistics ----
+1 replies max: 74.41 min: 74.41 avg: 74.41
+{% endhighlight %}
+
+This sent a simple 'hello' packet out to the network and if you started up several of the *mcollectived.rb* processes on several machines you
+would have seen several replies, be sure to give each a unique *identity* in the config.
+
+At this point you can start exploring the discovery features for example:
+
+{% highlight console %}
+% mco find --with-fact country=uk
+your.domain.com
+{% endhighlight %}
+
+This searches all systems currently active for ones with a fact *country=uk*, it got the data from the yaml file you made earlier.
+
+If you use configuration management tools like puppet and the nodes are setup with classes with *classes.txt* in */var/lib/puppet* then you
+can search for nodes with a specific class on them - the locations will configurable soon:
+
+{% highlight console %}
+% mco find --with-class common::linux
+your.domain.com
+{% endhighlight %}
+
+Chef does not yet support such a list natively but we have some details on the wiki to achieve the same with Chef.
+
+The filter commands are important they will be the main tool you use to target only parts of your infrastructure with calls to agents.
+
+See the *--help* option to the various *mco `*`* commands for available options. You can now look at some of the available plugins and
+play around, you might need to run the server process as root if you want to play with services etc.
+
+### Plugins
+We provide limited default plugins, you can look on our sister project [MCollective Plugins][Plugins] where you will
+find various plugins to manage packages, services etc.
+
+### Further Reading
+From here you should look at the rest of the wiki pages some key pages are:
+
+ * [Screencasts] - Get a hands-on look at what is possible
+ * [Terminology]
+ * [Introduction to Simple RPC][SimpleRPCIntroduction] - a simple to use framework for writing clients and agents
+ * [ControllingTheDaemon] - Controlling a running daemon
+ * [AESSecurityPlugin] - Using AES+RSA for secure message encryption and authentication of clients
+ * [SSLSecurityPlugin] - Using SSL for secure message signing and authentication of clients
--- /dev/null
+---
+layout: default
+title: Getting Started
+---
+[Screencasts]: /mcollective/screencasts.html
+[ActiveMQ]: http://activemq.apache.org/
+[ActiveMQ Getting Started]: http://activemq.apache.org/getting-started.html
+[EC2Demo]: /mcollective/ec2demo.html
+[Stomp]: http://stomp.codehaus.org/Ruby+Client
+[DepRPMs]: http://www.marionette-collective.org/activemq/
+[DebianBug]: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=562954
+[SecurityWithActiveMQ]: /mcollective/reference/integration/activemq_security.html
+[ActiveMQClustering]: /mcollective/reference/integration/activemq_clusters.html
+[ActiveMQSamples]: http://github.com/puppetlabs/marionette-collective/tree/master/ext/activemq/examples/
+[ActiveMQSingleBrokerSample]: http://github.com/puppetlabs/marionette-collective/raw/master/ext/activemq/examples/single-broker/activemq.xml
+[ConfigurationReference]: /mcollective/reference/basic/configuration.html
+[Terminology]: /mcollective/terminology.html
+[SimpleRPCIntroduction]: /mcollective/simplerpc/
+[ControllingTheDaemon]: /mcollective/reference/basic/daemon.html
+[SSLSecurityPlugin]: /mcollective/reference/plugins/security_ssl.html
+[AESSecurityPlugin]: /mcollective/reference/plugins/security_aes.html
+[ConnectorActiveMQ]: /mcollective/reference/plugins/connector_activemq.html
+[ConnectorRabbitMQ]: /mcollective/reference/plugins/connector_rabbitmq.html
+[MessageFlowCast]: /mcollective/screencasts.html#message_flow
+[Plugins]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki
+[MCDownloads]: http://www.puppetlabs.com/downloads/mcollective/
+[RubyGems]: http://packages.debian.org/search?suite=default§ion=all&arch=any&searchon=names&keywords=rubygems
+[server_config]: /mcollective/configure/server.html
+
+Getting started using Debian based distribution like Debian squeeze and Ubuntu is easy as DEBs are available for most the required components. This guide walks you through the process.
+
+## Requirements
+We try to keep the requirements on external Gems to a minimum, you only need:
+
+ * A Stomp server, tested against [ActiveMQ]
+ * Ruby
+ * RubyGems
+ * [Ruby Stomp Client][Stomp]
+
+## Packages
+
+We strongly recommend you set up a local Apt repository that will host all the packages on your LAN, you can get the prerequisite packages here:
+
+ * [ActiveMQ]
+ * Java - OpenJDK that is included with your distribution
+ * Ruby - included with your distribution
+ * [RubyGems]
+ * Stomp Ruby Gem
+ * [MCollective][MCDownloads] - mcollective-2.2.x-1_all.deb, mcollective-common-2.2.x-1_all.deb, mcollective-client-2.2.x-1_all.deb
+
+The rest of this guide will assume you set up a Apt repository. Puppet Labs hosts a Apt repository with all these dependencies at _apt.puppetlabs.com_.
+
+## ActiveMQ
+
+ActiveMQ is currently the most used and tested middleware for use with MCollective.
+
+You need at least one ActiveMQ server on your network, all the nodes you wish to manage will connect to the central ActiveMQ server.
+Later on you can [cluster the ActiveMQ servers for availability and scale][ActiveMQClustering].
+
+### Install
+
+On the server that you chose to configure as the ActiveMQ server:
+
+{% highlight console %}
+% apt-get install openjdk-6-jre
+{% endhighlight %}
+
+ActiveMQ installation instructions can be found [here][ActiveMQ Getting Started].
+
+### Configuring
+
+[The ActiveMQ config reference][activemq_config] describes all of the ActiveMQ settings that MCollective cares about. For best use, skim the sections you care about while comparing it to an example activemq.xml file.
+
+[activemq_config]: /mcollective/deploy/middleware/activemq.html
+
+We recommend that new users:
+
+* Start with the [single-broker example config][ActiveMQSingleBrokerSample].
+* Change the [user account passwords](/mcollective/deploy/middleware/activemq.html#authentication-users-and-groups).
+* [Set up TLS](/mcollective/deploy/middleware/activemq.html#tls-credentials) and [use a TLS Stomp transport connector](/mcollective/deploy/middleware/activemq.html#transport-connectors).
+
+Other example config files are also available from [GitHub][ActiveMQSamples].
+
+### Starting
+
+Start the ActiveMQ service:
+
+{% highlight console %}
+ # /etc/init.d/activemq start
+{% endhighlight %}
+
+You should see it running in the process list:
+
+{% highlight console %}
+ # ps auxw|grep java
+ activemq 3012 0.1 14.5 1155112 152180 ? Sl Dec28 2:02 java -Dactivemq.home=/usr/share/activemq -Dactivemq.base=/usr/share/activemq -Dcom.sun.management.jmxremote -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Xmx512m -Djava.library.path=/usr/lib:/usr/lib64 -classpath /usr/share/java/tanukiwrapper.jar:/usr/share/activemq/bin/run.jar -Dwrapper.key=eg4_VvENzCmvtAKg -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=3000 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp org.apache.activemq.console.Main start
+{% endhighlight %}
+
+You should also see it listening on port 61613 or 61614 in your network stack, depending on whether you turned on TLS.
+
+You should open port 61613 or 61614 for all your nodes to connect to.
+
+## Marionette Collective
+
+There are a few packages supplied and you will have potentially two type of server:
+
+ * Nodes that you wish to manage using mcollective need the mcollective and mcollective-common packages
+ * Nodes that you wish to use to initiate requests from also known as clients need mcollective-client and mcollective-common packages
+
+A machine can be both at once, in which case you need to install all 3 packages. We'll work on the assumption here that you wish to both manage your machine and use it as a client by installing all 3 packages on our initial node.
+
+### Installation
+
+{% highlight console %}
+ # apt-get install mcollective mcollective-client mcollective-common
+ # gem install stomp
+{% endhighlight %}
+
+
+## Configuring
+You'll need to tweak some configs in */etc/mcollective/client.cfg*, a full reference of config settings can be
+found [here][ConfigurationReference]:
+
+We're assuming you called the machine running ActiveMQ *stomp.example.net*; please change as appropriate. Also note that the port should be 61614 if you turned on TLS.
+
+{% highlight ini %}
+ # main config
+ libdir = /usr/libexec/mcollective
+ logfile = /dev/null
+ loglevel = error
+
+ # connector plugin config
+ connector = activemq
+ plugin.activemq.pool.size = 1
+ plugin.activemq.pool.1.host = stomp.example.net
+ plugin.activemq.pool.1.port = 61613
+ plugin.activemq.pool.1.user = mcollective
+ plugin.activemq.pool.1.password = marionette
+
+ # security plugin config
+ securityprovider = psk
+ plugin.psk = abcdefghj
+{% endhighlight %}
+
+You should also create _/etc/mcollective/server.cfg_ here's a sample, a full reference of config settings can be found on the [Server Configuration Reference][server_config]:
+
+{% highlight ini %}
+ # main config
+ libdir = /usr/libexec/mcollective
+ logfile = /var/log/mcollective.log
+ daemonize = 1
+ loglevel = info
+
+ # connector plugin config
+ connector = activemq
+ plugin.activemq.pool.size = 1
+ plugin.activemq.pool.1.host = stomp.example.net
+ plugin.activemq.pool.1.port = 61613
+ plugin.activemq.pool.1.user = mcollective
+ plugin.activemq.pool.1.password = marionette
+
+
+ # facts
+ factsource = yaml
+ plugin.yaml = /etc/mcollective/facts.yaml
+
+ # security plugin config
+ securityprovider = psk
+ plugin.psk = abcdefghj
+{% endhighlight %}
+
+Replace the *plugin.psk* in both these files with a Pre-Shared Key of your own.
+
+### Create Facts
+By default - and for this setup - we'll use a simple YAML file for a fact source, later on you can use Puppet Labs Facter or something else.
+
+Create */etc/mcollective/facts.yaml* along these lines:
+
+{% highlight yaml %}
+ ---
+ location: devel
+ country: uk
+{% endhighlight %}
+
+### Start the Server
+
+The packages include standard init script, just start the server:
+
+{% highlight console %}
+ # /etc/init.d/mcollective restart
+{% endhighlight %}
+
+You should see in the log file somethig like:
+
+{% highlight console %}
+ # tail /var/log/mcollective.log
+ I, [2010-12-29T11:15:32.321744 #11479] INFO -- : mcollectived:33 The Marionette Collective 1.1.0 started logging at info level
+{% endhighlight %}
+
+### Test connectivity
+
+If all is fine and you see this log message you can test with the client code:
+
+{% highlight console %}
+% mco ping
+your.domain.com time=74.41 ms
+
+---- ping statistics ----
+1 replies max: 74.41 min: 74.41 avg: 74.41
+{% endhighlight %}
+
+This sends out a simple 'hello' packet to all the machines, as we only installed one you should have just one reply.
+
+If you install the _mcollective_ and _mcollective-common_ packages along wit the facts and server.cfg you should see more nodes show up here.
+
+You can explore other aspects of your machines:
+
+{% highlight console %}
+% mco find --with-fact country=uk
+your.domain.com
+{% endhighlight %}
+
+This searches all systems currently active for ones with a fact *country=uk*, it got the data from the yaml file you made earlier.
+
+If you use confiuration management tools like puppet and the nodes are setup with classes with *classes.txt* in */var/lib/puppet* then you
+can search for nodes with a specific class on them - the locations will configurable soon:
+
+{% highlight console %}
+% mco find --with-class common::linux
+your.domain.com
+{% endhighlight %}
+
+The filter commands are important they will be the main tool you use to target only parts of your infrastructure with calls to agents.
+
+See the *--help* option to the various *mco `*`* commands for available options. You can now look at some of the available plugins and
+play around, you might need to run the server process as root if you want to play with services etc.
+
+### Plugins
+We provide limited default plugins, you can look on our sister project [MCollective Plugins][Plugins] where you will
+find various plugins to manage packages, services etc.
+
+### Further Reading
+From here you should look at the rest of the wiki pages some key pages are:
+
+ * [Screencasts] - Get a hands-on look at what is possible
+ * [Terminology]
+ * [Introduction to Simple RPC][SimpleRPCIntroduction] - a simple to use framework for writing clients and agents
+ * [ControllingTheDaemon] - Controlling a running daemon
+ * [AESSecurityPlugin] - Using AES+RSA for secure message encryption and authentication of clients
+ * [SSLSecurityPlugin] - Using SSL for secure message signing and authentication of clients
--- /dev/null
+---
+layout: default
+title: Getting Started
+---
+[Screencasts]: /mcollective/screencasts.html
+[ActiveMQ]: http://activemq.apache.org/
+[EC2Demo]: /mcollective/ec2demo.html
+[Stomp]: http://stomp.codehaus.org/Ruby+Client
+[DepRPMs]: http://www.marionette-collective.org/activemq/
+[DebianBug]: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=562954
+[SecurityWithActiveMQ]: /mcollective/reference/integration/activemq_security.html
+[ActiveMQClustering]: /mcollective/reference/integration/activemq_clusters.html
+[ActiveMQSamples]: http://github.com/puppetlabs/marionette-collective/tree/master/ext/activemq/examples/
+[ActiveMQSingleBrokerSample]: http://github.com/puppetlabs/marionette-collective/raw/master/ext/activemq/examples/single-broker/activemq.xml
+[ConfigurationReference]: /mcollective/reference/basic/configuration.html
+[Terminology]: /mcollective/terminology.html
+[SimpleRPCIntroduction]: /mcollective/simplerpc/
+[ControllingTheDaemon]: /mcollective/reference/basic/daemon.html
+[SSLSecurityPlugin]: /mcollective/reference/plugins/security_ssl.html
+[AESSecurityPlugin]: /mcollective/reference/plugins/security_aes.html
+[ConnectorActiveMQ]: /mcollective/reference/plugins/connector_activemq.html
+[ConnectorRabbitMQ]: /mcollective/reference/plugins/connector_rabbitmq.html
+[MessageFlowCast]: /mcollective/screencasts.html#message_flow
+[Plugins]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki
+[MCDownloads]: http://www.puppetlabs.com/downloads/mcollective/
+[EPEL]: http://fedoraproject.org/wiki/EPEL
+[server_config]: /mcollective/configure/server.html
+
+Getting started using Red Hat based distribution like Red Hat Enterprise Linux and CentOS is easy as RPMs are available for all the required components. This guide walks you through the process.
+
+## Requirements
+We try to keep the requirements on external Gems to a minimum, you only need:
+
+ * A Stomp server, tested against [ActiveMQ]
+ * Ruby
+ * Rubygems
+ * [Ruby Stomp Client][Stomp]
+
+## Packages
+
+We strongly recommend you set up a local Yum repository that will host all the packages on your LAN, you can get the prerequisite packages here:
+
+ * [ActiveMQ][MCDownloads] - activemq-5.4.0-2.el5.noarch.rpm, activemq-info-provider-5.4.0-2.el5.noarch.rpm, tanukiwrapper-3.2.3-1jpp.`*`.rpm
+ * Java - OpenJDK that is included with your distribution
+ * Ruby - included with your distribution
+ * RubyGems - [EPEL]
+ * Stomp Ruby Gem - [EPEL]
+ * [MCollective][MCDownloads] - mcollective-2.2.x-1.el5.noarch.rpm, mcollective-common-2.2.x-1.el5.noarch.rpm, mcollective-client-2.2.x-1.el5.noarch.rpm
+
+The rest of this guide will assume you set up a Yum repository. Puppet Labs hosts a Yum repository with all these dependencies at _yum.puppetlabs.com_.
+
+## ActiveMQ
+ActiveMQ is currently the most used and tested middleware for use with MCollective.
+
+You need at least one ActiveMQ server on your network, all the nodes you wish to manage will connect to the central ActiveMQ server.
+Later on your can [cluster the ActiveMQ servers for availability and scale][ActiveMQClustering].
+
+### Install
+
+On the server that you chose to configure as the ActiveMQ server:
+
+{% highlight console %}
+% yum install java-1.6.0-openjdk activemq
+{% endhighlight %}
+
+### Configuring
+
+[The ActiveMQ config reference][activemq_config] describes all of the ActiveMQ settings that MCollective cares about. For best use, skim the sections you care about while comparing it to an example activemq.xml file.
+
+[activemq_config]: /mcollective/deploy/middleware/activemq.html
+
+We recommend that new users:
+
+* Start with the [single-broker example config][ActiveMQSingleBrokerSample].
+* Change the [user account passwords](/mcollective/deploy/middleware/activemq.html#authentication-users-and-groups).
+* [Set up TLS](/mcollective/deploy/middleware/activemq.html#tls-credentials) and [use a TLS Stomp transport connector](/mcollective/deploy/middleware/activemq.html#transport-connectors).
+
+Other example config files are also available from [GitHub][ActiveMQSamples].
+
+
+### Starting
+
+Start the ActiveMQ service:
+
+{% highlight console %}
+ # /etc/init.d/activemq start
+{% endhighlight %}
+
+You should see it running in the process list:
+
+{% highlight console %}
+ # ps auxw|grep java
+ activemq 3012 0.1 14.5 1155112 152180 ? Sl Dec28 2:02 java -Dactivemq.home=/usr/share/activemq -Dactivemq.base=/usr/share/activemq -Dcom.sun.management.jmxremote -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Xmx512m -Djava.library.path=/usr/lib:/usr/lib64 -classpath /usr/share/java/tanukiwrapper.jar:/usr/share/activemq/bin/run.jar -Dwrapper.key=eg4_VvENzCmvtAKg -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=3000 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp org.apache.activemq.console.Main start
+{% endhighlight %}
+
+You should also see it listening on port 61613 in your network stack
+
+You should open port 61613 for all your nodes to connect to.
+
+## Marionette Collective
+
+There are a few packages supplied and you will have potentially two type of server:
+
+ * Nodes that you wish to manage using mcollective need the mcollective and mcollective-common packages
+ * Nodes that you wish to use to initiate requests from also known as clients need mcollective-client and mcollective-common packages
+
+A machine can be both at once, in which case you need to install all 3 packages. We'll work on the assumption here that you wish to both manage your machine and use it as a client by installing all 3 packages on our initial node.
+
+### Installation
+
+{% highlight console %}
+ # yum install mcollective mcollective-client mcollective-common rubygem-stomp
+{% endhighlight %}
+
+
+## Configuring
+You'll need to tweak some configs in */etc/mcollective/client.cfg*, a full reference of config settings can be
+found [here][ConfigurationReference]:
+
+We're assuming you called the machine running ActiveMQ *stomp.example.net* please change as appropriate
+
+{% highlight ini %}
+ # main config
+ libdir = /usr/libexec/mcollective
+ logfile = /dev/null
+ loglevel = error
+
+ # connector plugin config
+ connector = activemq
+ plugin.activemq.pool.size = 1
+ plugin.activemq.pool.1.host = stomp.example.net
+ plugin.activemq.pool.1.port = 61613
+ plugin.activemq.pool.1.user = mcollective
+ plugin.activemq.pool.1.password = marionette
+
+ # security plugin config
+ securityprovider = psk
+ plugin.psk = abcdefghj
+{% endhighlight %}
+
+You should also create _/etc/mcollective/server.cfg_ here's a sample, a full reference of config settings can be found on the [Server Configuration Reference][server_config]:
+
+{% highlight ini %}
+ # main config
+ libdir = /usr/libexec/mcollective
+ logfile = /var/log/mcollective.log
+ daemonize = 1
+ loglevel = info
+
+ # connector plugin config
+ connector = activemq
+ plugin.activemq.pool.size = 1
+ plugin.activemq.pool.1.host = stomp.example.net
+ plugin.activemq.pool.1.port = 61613
+ plugin.activemq.pool.1.user = mcollective
+ plugin.activemq.pool.1.password = marionette
+
+ # facts
+ factsource = yaml
+ plugin.yaml = /etc/mcollective/facts.yaml
+
+ # security plugin config
+ securityprovider = psk
+ plugin.psk = abcdefghj
+{% endhighlight %}
+
+Replace the *plugin.psk* in both these files with a Pre-Shared Key of your own.
+
+### Create Facts
+By default - and for this setup - we'll use a simple YAML file for a fact source, later on you can use Puppet Labs Facter or something else.
+
+Create */etc/mcollective/facts.yaml* along these lines:
+
+{% highlight yaml %}
+ ---
+ location: devel
+ country: uk
+{% endhighlight %}
+
+### Start the Server
+
+The packages include standard init script, just start the server:
+
+{% highlight console %}
+ # /etc/init.d/mcollective restart
+{% endhighlight %}
+
+You should see in the log file somethig like:
+
+{% highlight console %}
+ # tail /var/log/mcollective.log
+ I, [2010-12-29T11:15:32.321744 #11479] INFO -- : mcollectived:33 The Marionette Collective 1.1.0 started logging at info level
+{% endhighlight %}
+
+### Test connectivity
+
+If all is fine and you see this log message you can test with the client code:
+
+{% highlight console %}
+% mco ping
+your.domain.com time=74.41 ms
+
+---- ping statistics ----
+1 replies max: 74.41 min: 74.41 avg: 74.41
+{% endhighlight %}
+
+This sends out a simple 'hello' packet to all the machines, as we only installed one you should have just one reply.
+
+If you install the _mcollective_ and _mcollective-common_ packages along wit the facts and server.cfg you should see more nodes show up here.
+
+You can explore other aspects of your machines:
+
+{% highlight console %}
+% mco find --with-fact country=uk
+your.domain.com
+{% endhighlight %}
+
+This searches all systems currently active for ones with a fact *country=uk*, it got the data from the yaml file you made earlier.
+
+If you use confiuration management tools like puppet and the nodes are setup with classes with *classes.txt* in */var/lib/puppet* then you
+can search for nodes with a specific class on them - the locations will configurable soon:
+
+{% highlight console %}
+% mco find --with-class common::linux
+your.domain.com
+{% endhighlight %}
+
+The filter commands are important they will be the main tool you use to target only parts of your infrastructure with calls to agents.
+
+See the *--help* option to the various *mco `*`* commands for available options. You can now look at some of the available plugins and
+play around, you might need to run the server process as root if you want to play with services etc.
+
+### Plugins
+We provide limited default plugins, you can look on our sister project [MCollective Plugins][Plugins] where you will
+find various plugins to manage packages, services etc.
+
+### Further Reading
+From here you should look at the rest of the wiki pages some key pages are:
+
+ * [Screencasts] - Get a hands-on look at what is possible
+ * [Terminology]
+ * [Introduction to Simple RPC][SimpleRPCIntroduction] - a simple to use framework for writing clients and agents
+ * [ControllingTheDaemon] - Controlling a running daemon
+ * [AESSecurityPlugin] - Using AES+RSA for secure message encryption and authentication of clients
+ * [SSLSecurityPlugin] - Using SSL for secure message signing and authentication of clients
--- /dev/null
+---
+layout: default
+title: Message Flow
+toc: false
+---
+[MessageFormat]: /mcollective/reference/basic/messageformat.html
+[ActiveMQClusters]: /mcollective/reference/integration/activemq_clusters.html
+[SecurityWithActiveMQ]: /mcollective/reference/integration/activemq_security.html
+[ScreenCast]: /mcollective/screencasts.html#message_flow
+
+The diagram below shows basic message flow on a MCollective system. There is also a [screencast][ScreenCast] that shows this process, recommend you watch that.
+
+The key thing to take away from this diagram is the broadcast paradigm that is in use, one message only leaves the client and gets broadcast to all nodes. We'll walk you through each point below.
+
+
+
+|Step|Description|
+|----|-----------|
+|A|A single messages gets sent from the workstation of the administrator to the middleware. The message has a filter attached saying only machines with the fact _cluster=c_ should perform an action.|
+|B|The middleware network broadcasts the message to all nodes. The middleware network can be a [cluster of multiple servers in multiple locations, networks and data centers][ActiveMQClusters].|
+|C|Every node gets the message and validates the filter|
+|D|Only machines in _cluster=c_ act on the message and sends a reply, depending on your middleware only the workstation will get the reply.|
+
+For further information see:
+
+ * [Messages and their contents][MessageFormat]
+ * [Clustering ActiveMQ brokers][ActiveMQClusters]
+ * [Security, authentication and authorization][SecurityWithActiveMQ]
--- /dev/null
+---
+layout: default
+title: Message Format
+---
+[SecurityPlugins]: http://github.com/puppetlabs/marionette-collective/tree/master/plugins/mcollective/security/
+[SimpleRPCIntroduction]: /mcollective/simplerpc/
+[MessageFlow]: messageflow.html
+[ScreenCast]: /mcollective/screencasts.html#message_flow
+
+The messages that gets put on the middleware attempts to contain everything that mcollective needs to function, avoiding where possible special features in the Middle Ware. This will hopefully make it easier to create Connector plugins for other middleware.
+
+At present the task of encoding and decoding messages lies with the _MCollective::Security::`*`_ classes, see the provided [security plugins][SecurityPlugins] as a examples.
+
+Abstracting the encoding away from the security plugins is a goal for future refactorings, till then each security plugin will need to at least conform to the following structure.
+
+In general this is all hidden from the developers, especially if you use [Simple RPC][SimpleRPCIntroduction]. If you want to implement your own security or serialization you will need to know exactly how all of this sticks together.
+
+There is also a [screencast][ScreenCast] that shows this process and message format, recommend you watch that.
+
+## Message Flow
+For details of the flow of messages and how requests / replies travel around the network see the [MessageFlow] page.
+
+## History
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/04/23|Add _agent_ and _collective_ to the request hashes|7113|
+
+### Requests sent to agents
+A sample request that gets sent to the connector can be seen here, each component is described below:
+
+{% highlight ruby %}
+{:filter =>
+ {"cf_class" => ["common::linux"],
+ "fact" => [{:fact=>"country", :value=>"uk"}],
+ "agent" => ["package"]},
+ :senderid => "devel.your.com",
+ :msgtarget => "/topic/mcollective.discovery.command",
+ :agent: => 'discovery',
+ :collective' => 'mcollective',
+ :body => body,
+ :hash => "2d437f2904980ac32d4ebb7ca1fd740b",
+ :msgtime => 1258406924,
+ :ttl => 60,
+ :requestid => "0b54253cb5d04eb8b26ea75bbf468cbc"}
+{% endhighlight %}
+
+Once this request is created the security plugin will serialize it and sent it to the connector, in the case of the PSK security plugin this is done using Marshal.
+
+#### :filter
+The filter will be evaluated by each node, if it passes the node will dispatch the message to an agent.
+
+You can see all these types of filter in action in the _MCollection::Optionparser_ class.
+
+Each filter is an array and you can have multiple filters per type which will be applied with an _AND_
+Valid filter types are:
+
+##### CF Class
+
+This will look in a list of classes/recipes/cookbooks/roles applied by your
+configuration management system and match based on that
+
+{% highlight ruby %}
+filter["cf_class"] = ["common::linux"]
+{% endhighlight %}
+
+##### MCollective Agent
+
+This will look through the list of known agents and match against that.
+
+{% highlight ruby %}
+filter["agent"] = ["package"]
+{% endhighlight %}
+
+##### Facts
+
+Since facts are key => value pairs this is a bit more complex than normal as you need to build a nested Hash.
+
+{% highlight ruby %}
+filter["fact"] = [{:fact => "country", :value => "uk"}]
+{% endhighlight %}
+
+Regular expression matches are:
+
+{% highlight ruby %}
+filter["fact"] = [{:fact => "country", :value => "/uk/"}]
+{% endhighlight %}
+
+As of version 1.1.0 this has been extended to support more comparison operators:
+
+{% highlight ruby %}
+filter["fact"] = [{:fact => "country", :value => "uk", :operator => "=="}]
+{% endhighlight %}
+
+Valid operators are: ==, =~, <=, =>, >=, =<, > < and !=
+
+As regular expressions are now done using their own operator backwards compatability
+is lost and in mixed version environment 1.1.x clients doing regex matches on facts
+will be treated as equality on 1.0.x and older clients.
+
+##### Identity
+
+The identity is the configured identity in the server config file, many hosts can have the same identity it's just another level of filter doesn't really mean much like a hostname that would need to be unique.
+
+{% highlight ruby %}
+filter["identity"] = ["foo.bar.com"]
+{% endhighlight %}
+
+#### :senderid
+
+The value of _identity_ in the configuration file.
+
+#### :msgtarget
+
+The Middleware topic or channel the message is being sent to. This is not being used since later 1.1.x and since 1.3.1 it is not included in messages anymore.
+
+#### :body
+
+The contents of the body will vary by what ever the security provider choose to impliment, the PSK security provider simply Marshal encodes the body into a serialized format ready for transmission.
+
+This ensures that variable types etc remain in tact end to end. Other security providers might use JSON etc, the decoding of this is also handled by the security provider so its totally up to the provider to decide.
+
+In the case of [Simple RPC][SimpleRPCIntroduction] the entire RPC request and replies will be in the body of the messages, it's effectively a layer on top of the basic message flow.
+
+#### :hash
+
+This is an example of something specific to the security provider, this is used only by the PSK provider so it's optional and specific to the PSK provider
+
+#### :msgtime
+
+The unix timestamp that the message was sent at.
+
+#### :ttl
+
+Each request has a TTL, messages older than this gets discarded. Added in version 1.3.1
+
+#### :requestid
+
+This is a unique id for each message that gets sent, replies will have the same id attached to them for validation.
+
+### Replies from Agents
+Replies are very similar to requests, I'll show a reply below and only highlight the differences between requests.
+
+{% highlight ruby %}
+{:senderid => "devel.your.com",
+ :senderagent => "package",
+ :msgtarget => "/topic/mcollective.package.reply",
+ :body => body,
+ :hash => "2d437f2904980ac32d4ebb7ca1fd740b",
+ :msgtime => 1258406924,
+ :requestid => "0b54253cb5d04eb8b26ea75bbf468cbc"}
+{% endhighlight %}
+
+Once this reply is created the security plugin will serialize it and sent it to the connector, in the case of the PSK security plugin this is done using serialization tools like Marshal or YAML depending on Security Plugin.
+
+#### :senderagent
+This is the agent name that sent the reply
+
+#### :requestid
+The id that was contained in the request we are replying to. Agents do not generally tend to generate messages - they only reply - so this should always be provided.
--- /dev/null
+---
+layout: default
+title: Subcollectives
+---
+
+[ActiveMQClustering]: /mcollective/reference/integration/activemq_clusters.html
+[activemq_authorization]: /mcollective/deploy/middleware/activemq.html#authorization-group-permissions
+[activemq_detailed]: /mcollective/deploy/middleware/activemq.html#detailed-restrictions
+[activemq_subcollectives]: /mcollective/deploy/middleware/activemq.html#detailed-restrictions-with-multiple-subcollectives
+[activemq_filtering]: /mcollective/deploy/middleware/activemq.html#destination-filtering
+[activemq_authentication]: /mcollective/deploy/middleware/activemq.html#authentication-users-and-groups
+
+## Overview
+
+By default all servers are part of a single broadcast domain, if you have an
+agent on all machines in your network and you send a message directed to
+machines with that agent they will all get it regardless of filters.
+
+This works well for the common use case but can present problems in the
+following scenarios:
+
+ * You have a very big and busy network. With thousands of machines responding
+ to requests every 10 seconds very large numbers of messages will be created
+ and you might want to partition the traffic.
+ * You have multiple data centers on different continents, the latency and
+ volume of traffic will be too big. You'd want to duplicate monitoring and
+ management automations in each datacenter but as it's all one broadcast
+ domain you still see large amount of traffic on the slow link.
+ * You can't just run multiple seperate installs because you still wish to
+ retain the central management feature of MCollective.
+ * Securing a flat network can be problematic. SimpleRPC has a security
+ framework that is aware of users and network topology but the core network
+ doesnt.
+
+We've introduced the concept of sub collectives that lets you define broadcast
+domains and configure a mcollective server to belong to one or many of these domains.
+
+## Partitioning Approaches
+
+Determining how to partition your nework can be a very complex subject and
+requires an understanding of your message flow, where requestors sit and also
+the topology of your middleware clusters.
+
+Most middleware solutions will only send traffic where they know there exist an
+interest in this traffic. Therefore if you had an agent on only 10 of 1000
+machines only those 10 machines will receive the associated traffic. This is an
+important distinction to keep in mind.
+
+
+
+We'll be working with a small 52 node collective that you can see above, the
+collective has machines in many data centers spread over 4 countries. There are
+3 ActiveMQ servers connected in a mesh.
+
+Along with each ActiveMQ node is also a Puppet Master, Nagios instance and other
+shared infrastructure components.
+
+An ideal setup for this network would be:
+
+ * MCollective NRPE and Puppetd Agent on each of 52 servers
+ * Puppet Commander on each of the 3 ActiveMQ locations
+ * Nagios in each of the locations monitoring the machines in its region
+ * Regional traffic will be isolated and contained to the region as much as
+ possible
+ * Systems Administrators and Registration data retain the ability to target the
+ whole collective
+
+The problem with a single flat collective is that each of the 3 Puppet
+Commanders will get a copy of all the traffic, even traffic they did not request
+they will simply ignore the wrong traffic. The links between Europe and US will
+see a large number of messages traveling over them. In a small 52 node traffic
+this is managable but if each of the 4 locations had thousands of nodes the
+situation will rapidly become untenable.
+
+It seems natural then to create a number of broadcast domains - subcollectives:
+
+ * A global collective that each machines belongs to
+ * UK, DE, ZA and US collectives that contains just machines in those regions
+ * An EU collective that has UK, DE and ZA machines
+
+Visually this arrangement might look like the diagram below:
+
+
+
+Notice how subcollectives can span broker boundaries - our EU collective has nodes
+that would connect to both the UK and DE brokers.
+
+We can now configure our Nagios and Puppet Commanders to communicate only to the
+sub collectives and the traffic for these collectives will be contained
+regionally.
+
+The graph below shows the impact of doing this, this is the US ActiveMQ instance
+showing traffic before partitioning and after. You can see even on a small
+network this can have a big impact.
+
+
+
+## Configuring MCollective
+
+Configuring the partitioned collective above is fairly simple. We'll look at
+one of the DE nodes for reference:
+
+{% highlight ini %}
+collectives = mcollective,de_collective,eu_collective
+main_collective = mcollective
+{% endhighlight %}
+
+The _collectives_ directive tells the node all the collectives it should belong
+to and the _main`_`collective_ instructs Registration where to direct messages
+to.
+
+## Testing
+
+Testing that it works is pretty simple, first we need a _client.cfg_ that
+configures your client to talk to all the sub collectives:
+
+{% highlight ini %}
+collectives = mcollective,uk_collective,us_collective,de_collective,eu_collective,us_collective,za_collective
+main_collective = mcollective
+{% endhighlight %}
+
+You can now test with _mco ping_:
+
+{% highlight console %}
+$ mco ping -T us_collective
+host1.us.my.net time=200.67 ms
+host2.us.my.net time=241.30 ms
+host3.us.my.net time=245.24 ms
+host4.us.my.net time=275.42 ms
+host5.us.my.net time=279.90 ms
+host6.us.my.net time=283.61 ms
+host7.us.my.net time=370.56 ms
+
+
+---- ping statistics ----
+7 replies max: 370.56 min: 200.67 avg: 270.96
+{% endhighlight %}
+
+By specifying other collectives in the -T argument you should see the sub
+collectives and if you do not specify anything you should see all machines.
+
+Clients don't need to know about all collectives, only the ones they intend
+to communicate with.
+
+You can discover the list of known collectives and how many nodes are in each
+using the _inventory_ application:
+
+{% highlight console %}
+$ mco inventory --list-collectives
+
+ * [ ==================================== ] 52 / 52
+
+ Collective Nodes
+ ========== =====
+ za_collective 2
+ us_collective 7
+ uk_collective 19
+ de_collective 24
+ eu_collective 45
+ mcollective 52
+
+ Total nodes: 52
+
+{% endhighlight %}
+
+## Partitioning for Security
+
+Another possible advantage from subcollectives is security. While the SimpleRPC
+framework has a security model that is aware of the topology the core network
+layer does not. Even if you only give someone access to run SimpleRPC requests
+against some machines they can still use _mco ping_ to discover other nodes on
+your network.
+
+By creating a subcollective of just their nodes and restricting them on the
+middleware level to just that collective you can effectively and completely
+create a secure isolated zone that overlays your exiting network.
+
+These restrictions have to be configured on the middleware server, outside of MCollective itself. The method will vary based on the middleware you use; the suggestions below are for ActiveMQ, the main recommended middleware.
+
+### Identifying Subcollectives on ActiveMQ
+
+The ActiveMQ connector plugin identifies subcollectives with the **first segment** of every destination (topic or queue) name.
+
+So for direct node addressing, for example, the default `mcollective` collective would use the `mcollective.nodes` queue, and `uk_collective` would use the `uk_collective.nodes` queue. For the package agent, they would use the `mcollective.package.agent` and `uk_collective.package.agent` topics, respectively.
+
+This makes it easy to use ActiveMQ destination wildcards to control access to a given collective.
+
+### Per-Subcollective Authorization
+
+To control subcollective access, identify the set of topics and queues that collective will use, then use ActiveMQ's authorization rules to secure them.
+
+* [See the "Authorization" section of the ActiveMQ config reference][activemq_authorization] for details.
+* The ["Detailed Restrictions"][activemq_detailed] example shows all of the topics and queues used by the default collective; you can copy/paste/modify these for a small number of collectives.
+* The ["Detailed Restrictions with Multiple Subcollectives"][activemq_subcollectives] example uses a snippet of ERB template to manage any number of collectives.
+
+You must then [configure multiple ActiveMQ user accounts][activemq_authentication] for your site's admins. Give each user membership in the groups they'll need to manage their collectives.
+
+### Destination Filters in a Network of Brokers
+
+In a [network of brokers][ActiveMQClustering], you can also prevent propagation across the network of sub collective
+traffic. See the ["Destination Filtering"][activemq_filtering] section of the ActiveMQ config reference for details.
--- /dev/null
+---
+layout: default
+title: EC2 Demo Creation
+toc: false
+---
+[Bundling]: http://support.rightscale.com/12-Guides/01-RightScale_Dashboard_User_Guide/02-Clouds/01-EC2/08-EC2_Image_Locator/Register_an_AMI#Step_2.3a_Bundle_the_Instance
+[Console]: https://console.aws.amazon.com/ec2
+
+Things to improve in next build:
+
+ * set _plugin.urltest.syslocation_ to the availability zone the AMI is running on to improve mc-urltest output
+
+## RightScale AMI
+Start up ami _ami-efe4cf9b_ or a newer RightScale EC2 image
+
+## Packages Needed
+Install the following RPMs:
+
+{% highlight console %}
+facter
+tanukiwrapper
+activemq
+mcollective
+mcollective-client
+mcollective-common
+rubygem-stomp
+rubygems
+ruby-shadow
+puppet
+net-snmp-libs
+lm_sensors
+net-snmp
+perl-Socket6
+nrpe
+perl-Crypt-DES
+perl-Digest-SHA1
+nagios-plugins
+perl-Digest-HMAC
+perl-Net-SNMP
+xinetd
+dialog
+rubygem-rdialog
+{% endhighlight %}
+
+Gram and install _passmakr-1.0.0.gem_ from _http://passmakr.googlecode.com/_
+
+## File modifications
+Most of the files needed are in SVN in the _ext/ec2demo_ directory.
+
+{% highlight console %}
+.
+|-- etc
+| |-- activemq
+| | `-- activemq.xml.templ
+| |-- mcollective
+| | |-- client.cfg.templ
+| | `-- server.cfg.templ
+| |-- nagios
+| | |-- command-plugins.cfg
+| | |-- nrpe.cfg
+| | `-- nrpe.d
+| | |-- check_disks.cfg
+| | |-- check_load.cfg
+| | |-- check_swap.cfg
+| | |-- check_totalprocs.cfg
+| | `-- check_zombieprocs.cfg
+| |-- rc.d
+| | `-- rc.local
+| `-- xinetd.d
+| `-- nrpe
+|-- opt
+| `-- rightscale
+| `-- etc
+| `-- motd-complete
+|-- root
+| `-- README.txt
+`-- usr
+ |-- lib
+ | `-- ruby
+ | `-- site_ruby
+ | `-- 1.8
+ | `-- facter
+ | `-- rightscale.rb
+ `-- local
+ |-- bin
+ | `-- start-mcollective-demo.rb
+ `-- etc
+ `-- mcollective-node.motd
+{% endhighlight %}
+
+## Bundling Changes
+Bundling up is based on [RightScale docs][bundling].
+
+You need to copy your key, cert and have your credentials all into _/mnt_:
+
+{% highlight console %}
+% cp /dev/null /root/.bash_history
+% rm -rf /var/tmp/mcollective/
+
+% ec2-bundle-vol -d /mnt -k pk-xx.pem -c cert-xx.pem -u 481328239245 -r i386
+% ec2-upload-bundle -b mcollective-041-demo -m image.manifest.xml -a xx -s xxx
+{% endhighlight %}
+
+Now register the AMI in the [AWS console][Console] then make public after testing
--- /dev/null
+---
+layout: default
+title: Release Tasks
+toc: false
+---
+
+Notes on what to do when we release
+
+ * update Changelog
+ * update ReleaseNotes
+ * update rakefile for version
+ * update website news section
+ * tag release
+ * update release notes with release date
+ * build and release rpms
+ * build and release debs using ami-0db89079
+ * send mail
+ * Announce on freshmeat
+
+## Building RPMs
+Boot up an instance of the EC2 demo AMIs
+
+{% highlight console %}
+# gem install rake
+# git clone git://github.com/puppetlabs/marionette-collective.git
+# cd marionette-collective
+# git checkout 0.x.x
+# rake rpm
+{% endhighlight %}
+
+Copy the RPMs and test it.
+
+## Building debs
+Boot up an instance of _ami-0db89079_ and do more or less the following:
+
+**Note: There's some bug, you might need to run _make deb_ twice to make it work**
+
+{% highlight console %}
+# apt-get update
+# apt-get install rake irb rdoc build-essential subversion devscripts dpatch cdbs rubygems git-core
+# git clone git://github.com/puppetlabs/marionette-collective.git
+# cd marionette-collective
+# git checkout 0.x.x
+# rake deb
+{% endhighlight %}
+
+Copy do test installs on the machine make sure it looks fine and ship them off the AMI, shut it. Make sure to copy the source deb as well.
--- /dev/null
+---
+layout: default
+title: Users Documentation and Reference
+toc: false
+---
+
+Index to basic users documentation.
+
+## Basic Operations and Configuration
+
+ 1. [Getting Started](basic/gettingstarted.html)
+ 1. [Configuration Guide](basic/configuration.html)
+ 1. [Message Flow](basic/messageflow.html)
+ 1. [Network Partitioning using Subcollectives](basic/subcollectives.html)
+ 1. [Message Format](basic/messageformat.html)
+
+### User Interface
+
+ 1. [Using MCollective Command Line Applications](basic/basic_cli_usage.html)
+ 1. [Node Reports](ui/nodereports.html)
+ 1. [CLI Filters](ui/filters.html)
+
+### Integration
+
+ 1. [Using with Puppet](integration/puppet.html)
+ 1. [Using with Chef](integration/chef.html)
+ 1. [ActiveMQ Security](integration/activemq_security.html)
+ 1. [ActiveMQ SSL](integration/activemq_ssl.html)
+ 1. [ActiveMQ Clusters](integration/activemq_clusters.html)
+ 1. [ActiveMQ Connector](plugins/connector_activemq.html)
+ 1. [RabbitMQ Connector](plugins/connector_rabbitmq.html)
+
+### Plugin Types
+
+ 1. [Data Plugins](plugins/data.html)
+ 1. [Discovery Plugins](plugins/discovery.html)
+ 1. [Result Aggregation Plugins](plugins/aggregate.html)
+ 1. [The single application plugin system](plugins/application.html)
+ 1. [Registration](plugins/registration.html)
+ 1. [Fact Source Plugins](plugins/facts.html)
+ 1. [Validator Plugins](plugins/validator.html)
+
+### Plugins
+
+ 1. [The _rpcutil_ helper Agent](plugins/rpcutil.html)
+ 1. [AES+RSA based Security Plugin](plugins/security_aes.html)
+ 1. [OpenSSL based Security Plugin](plugins/security_ssl.html)
+ 1. [Stomp Connector](plugins/connector_stomp.html)
+ 1. [ActiveMQ Connector](plugins/connector_activemq.html)
+ 1. [RabbitMQ Connector](plugins/connector_rabbitmq.html)
+ 1. [User Contributed Plugins](http://projects.puppetlabs.com/projects/mcollective-plugins/wiki)
+
+### Development
+
+ 1. [Release Tasks](development/releasetasks.html)
+ 1. [EC2 Demo Creation](development/ec2_demo.html)
--- /dev/null
+---
+layout: default
+title: ActiveMQ Clustering
+toc: false
+---
+[MessageFlow]: /mcollective/reference/basic/messageflow.html
+[NetworksOfBrokers]: http://activemq.apache.org/networks-of-brokers.html
+[SampleConfig]: http://github.com/puppetlabs/marionette-collective/tree/master/ext/activemq/
+[fuse_cluster]: http://fusesource.com/docs/broker/5.5/clustering/index.html
+[activemq_network]: /mcollective/deploy/middleware/activemq.html#settings-for-networks-of-brokers
+
+Relying on existing middleware tools and not re-inventing the transport wheel ourselves means we can take advantage of a lot of the built in features they provide. One such feature is clustering in ActiveMQ that allows for highly scalable and flexible network layouts.
+
+We'll cover here how to build a multi data center network of ActiveMQ servers with a NOC, the NOC computers would not need to send any packets direct to the nodes they manage and thanks to our meta data based approach to addressing machines they do not even need to know IPs or hostnames.
+
+There is an example of a 3 node cluster in the [ext/activemq directory of the MCollective source][SampleConfig].
+
+## Network Design
+
+### Network Layout
+
+
+
+The diagram above shows our sample network, I am using the same techniques to put an ActiveMQ in each of 4 countries and then having local nodes communicate to in-country ActiveMQ nodes.
+
+* These are the terminals of your NOC staff, they run the client code, these could also be isolated web servers for running admin tools etc.
+* Each location has its own instance of ActiveMQ and nodes would only need to communicate to their local ActiveMQ. This greatly enhances security in a setup like this.
+* The ActiveMQ instances speak to each other using a protocol called OpenWire, you can run this over IPSec or you could use the native support for SSL.
+* These are the servers being managed, they run the server code. No direct communications needs to be in place between NOC and managed servers.
+
+Refer to the [MessageFlow][] document to see how messages would traverse the middleware in a setup like this.
+
+### General Observations
+The main design goal here is to promote network isolation, the staff in your NOC are often high churn people, you'll get replacement staff relatively often and it's a struggle to secure what information they carry and how and when you can trust them.
+
+Our model of using middleware and off-loading authentication and authorization onto the middleware layer means you only need to give NOC people appropriate access to the middleware and not to each individual machine.
+
+Our usage of meta data to address machines rather than hostnames or ip address means you do not need to divulge this information to NOC staff, from their point of view they access machines like this:
+
+* All machines in _datacenter=a_
+* AND all machines with puppet class _/webserver/_
+* AND all machines with Facter fact _customer=acmeinc_
+
+In the end they can target the machines they need to target for maintenance commands as above without the need for any info about hostnames, ips, or even what/where data center A is.
+
+This model works particularly well in a Cloud environment where hostnames are dynamic and not under your control, in a cloud like Amazon S3 machines are better identified by what AMI they have booted and in what availability zones they are rather than what their hostnames are.
+
+## ActiveMQ Configuration
+
+ActiveMQ supports many types of cluster; we think their Network of Brokers model works best for MCollective.
+
+You will need to configure your ActiveMQ servers with everything from the ["Settings for Networks of Brokers" section of the ActiveMQ config reference][activemq_network]. Note the comments about the bi-directional connections: In the example network described above, you could either configure a pair of connectors on each datacenter broker to connect them to the NOC, or configure several pairs of connectors on the NOC broker to connect it to every datacenter. Do whichever makes sense for your own convenience and security needs.
+
+There is also a set of example config files in the [ext/activemq directory of the MCollective source][SampleConfig]; refer to these while reading the config reference.
+
+See [the ActiveMQ docs][NetworksOfBrokers] or [the Fuse docs][fuse_cluster] for more detailed info about networks of brokers.
+
--- /dev/null
+---
+layout: default
+title: ActiveMQ Security
+toc: false
+---
+
+[Security]: http://activemq.apache.org/security.html
+[Wildcard]: http://activemq.apache.org/wildcards.html
+[activemq_config]: /mcollective/deploy/middleware/activemq.html
+[mcollective_username]: /mcollective/reference/plugins/connector_activemq.html#configuring-mcollective
+[mcollective_tls]: ./activemq_ssl.html#configuring-mcollective-servers
+
+As part of rolling out MCollective you need to think about security. The various examples in the quick start guide and on this blog has allowed all agents to talk to all nodes all agents. The problem with this approach is that should you have untrusted users on a node they can install the client applications and read the username/password from the server config file and thus control your entire architecture.
+
+The default format for message topics is compatible with [ActiveMQ wildcard patterns][Wildcard] and so we can now do fine grained controls over who can speak to what.
+
+General information about [ActiveMQ Security can be found on their wiki][Security].
+
+## Configuring Security in activemq.xml
+
+[The ActiveMQ config reference][activemq_config] contains all relevant info for configuring security is activemq.xml. The most relevant sections are:
+
+* [Topic and Queue Names](/mcollective/deploy/middleware/activemq.html#topic-and-queue-names) --- Info about the destinations that MCollective uses.
+* [Transport Connectors](/mcollective/deploy/middleware/activemq.html#transport-connectors) --- URL structure for insecure and TLS transports.
+* [TLS Credentials](/mcollective/deploy/middleware/activemq.html#tls-credentials) --- For use with TLS transports.
+* [Authentication](/mcollective/deploy/middleware/activemq.html#authentication-users-and-groups) --- Establishing user accounts and groups.
+* [Authorization](/mcollective/deploy/middleware/activemq.html#authorization-group-permissions) --- Limiting access to destinations based on group membership.
+* [Destination Filtering](/mcollective/deploy/middleware/activemq.html#destination-filtering) --- Preventing certain messages from crossing between datacenters.
+
+
+
+## Configuring Security in MCollective
+
+MCollective clients and servers need security credentials that line up with ActiveMQ's expectations. Specifically:
+
+* [An ActiveMQ username and password][mcollective_username]
+* [TLS credentials, if necessary][mcollective_tls]
+
--- /dev/null
+---
+layout: default
+title: ActiveMQ TLS
+toc: false
+---
+
+[keystores]: /mcollective/deploy/middleware/activemq_keystores.html
+[sslcontext]: /mcollective/deploy/middleware/activemq.html#tls-credentials
+[transport]: /mcollective/deploy/middleware/activemq.html#transport-connectors
+
+[activemq_connector]: /mcollective/reference/plugins/connector_activemq.html
+[ssl_security]: /mcollective/reference/plugins/security_ssl.html
+[aes_security]: /mcollective/reference/plugins/security_aes.html
+
+You can configure MCollective and ActiveMQ to do end-to-end encryption over TLS. This allows you to use MCollective's fast and efficient [SSL security plugin][ssl_security] instead of the slow and hard to configure [AES security plugin][aes_security].
+
+There are two main approaches:
+
+* [CA-verified TLS](#ca-verified-tls) encrypts traffic, and also lets you restrict middleware connections --- only nodes with valid certificates from the site's CA can connect.
+* [Anonymous TLS](#anonymous-tls) encrypts messages to prevent casual traffic sniffing, and will prevent the passwords MCollective uses to connect to ActiveMQ from being stolen. However, it doesn't check whether nodes are allowed to connect, so you have to trust your username and password security.
+
+This feature requires:
+
+* MCollective 2.0.0 or newer
+* ActiveMQ 5.5.0 or newer
+* Stomp gem 1.2.2 or newer
+* The [activemq connector][activemq_connector] plugin (included with MCollective 2.0.0 and newer)
+
+## CA-Verified TLS
+
+**(Recommended For Most Users)**
+
+### Summary
+
+This approach configures MCollective and ActiveMQ to only accept connections when the peer has a certificate signed by the site's CA.
+
+**Benefits:**
+
+This approach gives extra security, and your MCollective servers will generally already have the credentials they need since you can re-use Puppet certificates.
+
+**Drawbacks:**
+
+You will need to go out of your way to issue keys and certificates to your admin users, which is an extra step when onboarding a new admin.
+
+
+### Step 1: Configure ActiveMQ to Use TLS
+
+Do the following steps to get ActiveMQ configured:
+
+* Follow [the ActiveMQ keystores guide][keystores] to create Java keystores for ActiveMQ.
+* [Configure activemq.xml's `<sslContext>` element to point at the keystores.][sslcontext]
+* [Configure the proper URIs in the broker's transport connectors.][transport]
+* Restart ActiveMQ.
+
+At this point, MCollective servers and clients should be unable to connect to ActiveMQ, since they do not yet have credentials configured.
+
+### Step 2: Configure MCollective Servers
+
+For the MCollective daemon you can use your existing Puppet certificates by editing the _server.cfg_ file:
+
+{% highlight ini %}
+connector = activemq
+# Optional:
+# plugin.activemq.base64 = yes
+plugin.activemq.pool.size = 1
+plugin.activemq.pool.1.host = stomp.example.com
+plugin.activemq.pool.1.port = 61614
+plugin.activemq.pool.1.user = mcollective
+plugin.activemq.pool.1.password = secret
+plugin.activemq.pool.1.ssl = true
+plugin.activemq.pool.1.ssl.ca = /var/lib/puppet/ssl/ca/ca_crt.pem
+plugin.activemq.pool.1.ssl.key = /var/lib/puppet/ssl/private_keys/<NAME>.pem
+plugin.activemq.pool.1.ssl.cert = /var/lib/puppet/ssl/certs/<NAME>.pem
+{% endhighlight %}
+
+* Set the correct paths to each node's private key and certificate; they will be named after the node's Puppet certname and located in the ssldir.
+ * You can locate a node's private key by running `sudo puppet agent --configprint hostprivkey`, the certificate with `sudo puppet agent --configprint hostcert`, and the CA certificate with `sudo puppet agent --configprint localcacert`.
+* Set the correct username and password.
+
+### Step 3: Configure MCollective Clients
+
+Each client will now need a TLS certificate issued by the Puppet CA in order to be able to
+connect to the ActiveMQ:
+
+{% highlight bash %}
+# puppet cert generate ripienaar
+notice: ripienaar has a waiting certificate request
+notice: Signed certificate request for ripienaar
+notice: Removing file Puppet::SSL::CertificateRequest ripienaar at '/var/lib/puppet/ssl/ca/requests/ripienaar.pem'
+notice: Removing file Puppet::SSL::CertificateRequest ripienaar at '/var/lib/puppet/ssl/certificate_requests/ripienaar.pem'
+{% endhighlight %}
+
+Copy the certificates to your user:
+
+{% highlight bash %}
+# mkdir /home/rip/.mcollective.d
+# cp /var/lib/puppet/ssl/ca/ca_crt.pem /home/rip/.mcollective.d/
+# cp /var/lib/puppet/ssl/private_keys/ripienaar.pem /home/rip/.mcollective.d/ripienaar-private.pem
+# cp /var/lib/puppet/ssl/public_keys/ripienaar.pem /home/rip/.mcollective.d/ripienaar.pem
+# cp /var/lib/puppet/ssl/certs/ripienaar.pem /home/rip/.mcollective.d/ripienaar-cert.pem
+# chown -R rip:rip /home/rip/.mcollective.d
+{% endhighlight %}
+
+You can now configure the mcollective client config in _/home/rip/.mcollective_ to use these:
+
+{% highlight ini %}
+connector = activemq
+# Optional:
+# plugin.activemq.base64 = yes
+plugin.activemq.pool.size = 1
+plugin.activemq.pool.1.host = stomp.example.com
+plugin.activemq.pool.1.port = 61614
+plugin.activemq.pool.1.user = ripienaar
+plugin.activemq.pool.1.password = secret
+plugin.activemq.pool.1.ssl = true
+plugin.activemq.pool.1.ssl.ca = /home/rip/.mcollective.d/ca_crt.pem
+plugin.activemq.pool.1.ssl.key = /home/rip/.mcollective.d/ripienaar-private.pem
+plugin.activemq.pool.1.ssl.cert = /home/rip/.mcollective.d/ripienaar-cert.pem
+{% endhighlight %}
+
+If you are using the SSL security plugin, you can use these same files by setting `/home/rip/.mcollective.d/ripienaar.pem` as the public key.
+
+### Finish: Verify Encryption
+
+If you want to be sure of this, you should now verify with tcpdump or wireshark that the connection and traffic
+really is all encrypted.
+
+### Troubleshooting Common Errors
+
+You will get some obvious errors from this code if any files are missing, but the errors from SSL validation will be pretty
+hard to understand.
+
+There are two main scenarios where ActiveMQ will reject an MCollective conneciton:
+
+#### MCollective Uses Wrong CA to Verify ActiveMQ Cert
+
+When the client connects using a CA set in _plugin.activemq.pool.1.ssl.ca_ that does not match the one
+in the ActiveMQ _truststore.jks_:
+
+{% highlight console %}
+failed: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
+{% endhighlight %}
+
+And in the ActiveMQ log:
+
+{% highlight console %}
+Transport failed: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca
+{% endhighlight %}
+
+#### MCollective Certs Aren't Signed by the Right CA
+
+When your client has the correct CA but his certificates are not signed by that CA:
+
+{% highlight console %}
+failed: SSL_connect returned=1 errno=0 state=SSLv3 read finished A: sslv3 alert certificate unknown
+{% endhighlight %}
+
+And in the ActiveMQ log:
+
+{% highlight console %}
+sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
+{% endhighlight %}
+
+
+
+
+
+## Anonymous TLS
+
+**(Less Recommended)**
+
+### Summary
+
+This approach configures MCollective and ActiveMQ to encrypt traffic via TLS, but accept connections from anyone.
+
+**Benefits:**
+
+This approach requires less configuration, especially when adding new admin users.
+
+**Drawbacks:**
+
+This approach has some relative security drawbacks. Depending on your site's security needs, these may not concern you --- since MCollective's application-level security plugins will prevent an attacker from issuing agent commands and owning your servers, attacks like those below would only result in denial of service plus some leakage of inventory data via lower-level discovery protocols.
+
+* Although attackers can't sniff MCollective's ActiveMQ passwords, there's nothing to prevent them from logging in if they steal a password through some other means. (With CA-verified TLS, on the other hand, they would also require a private key and certificate, which provides some additional security depth.)
+* An attacker able to run a man-in-the-middle attack at your site could fool your MCollective servers into trusting a malicious ActiveMQ server.
+
+
+### Step 1: Configure ActiveMQ to Use Anonymous TLS
+
+* Follow [the ActiveMQ keystores guide][keystores] to create a Java keystore for ActiveMQ. _You can skip the truststore._
+* [Configure activemq.xml's `<sslContext>` element to point at the keystore.][sslcontext] _You can skip the `trustStore` and `trustStorePassword` attributes._
+* [Configure the proper URIs in the broker's transport connectors][transport], but _leave off the `?needClientAuth=true` portion._
+* Restart ActiveMQ.
+
+
+### Step 2: Configure MCollective Servers and Clients
+
+Set the following settings in the `server.cfg` and `client.cfg` (or `~/.mcollective`) files:
+
+{% highlight ini %}
+connector = activemq
+# Optional:
+# plugin.activemq.base64 = yes
+plugin.activemq.pool.size = 1
+plugin.activemq.pool.1.host = stomp.example.com
+plugin.activemq.pool.1.port = 61614
+plugin.activemq.pool.1.user = mcollective
+plugin.activemq.pool.1.password = secret
+plugin.activemq.pool.1.ssl = true
+plugin.activemq.pool.1.ssl.fallback = true
+{% endhighlight %}
+
+The `plugin.activemq.pool.1.ssl.fallback` setting tells the plugin that it is allowed to:
+
+* Connect to an unverified server
+* Connect without presenting its own SSL credentials
+
+...if it is missing any of the `.ca` `.cert` or `.key` settings or cannot find the files they reference. If the settings _are_ present (and point to correct files), MCollective will try to do a verified connection.
+
+
+### Finish: Verify Encryption
+
+If you want to be sure of this, you should now verify with tcpdump or wireshark that the connection and traffic
+really is all encrypted.
--- /dev/null
+---
+layout: default
+title: Using With Chef
+toc: false
+---
+[FactsOpsCodeOhai]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsOhai
+
+If you're a Chef user you are supported in both facts and classes filters.
+
+## Facts
+There is a [community plugin to enable Ohai][FactsOpsCodeOhai] as a fact source.
+
+Using this plugin Ohai facts will be converted from:
+
+{% highlight javascript %}
+ "languages": {
+ "java": {
+ "runtime": {
+ "name": "OpenJDK Runtime Environment",
+ "build": "1.6.0-b09"
+ },
+ "version": "1.6.0"
+ },
+{% endhighlight %}
+
+to:
+
+{% highlight ruby %}
+ "languages.java.version"=>"1.6.0",
+ "languages.java.runtime.name"=>"OpenJDK Runtime Environment",
+ "languages.java.runtime.build"=>"1.6.0-b09",
+{% endhighlight %}
+
+So you can use the flattened versions of the information provided by Ohai in filters, reports etc.
+
+{% highlight console %}
+% mco find --with-fact languages.java.version=1.6.0
+{% endhighlight %}
+
+## Class Filters
+Chef does not provide a list of roles and recipes that has been applied to a node, to use with MCollective you need to create such a list.
+
+It's very easy with Chef to do this in a simple cookbook. Put the following code in a cookbook and arrange for it to run *last* on your node.
+
+This will create a list of all roles and recipes in _/var/tmp/chefnode.txt_ on each node for us to use:
+
+{% highlight ruby %}
+ruby_block "store node data locally" do
+ block do
+ state = File.open("/var/tmp/chefnode.txt", "w")
+
+ node.run_state[:seen_recipes].keys.each do |recipe|
+ state.puts("recipe.#{recipe}")
+ end
+
+ node.run_list.roles.each do |role|
+ state.puts("role.#{role}")
+ end
+
+ state.close
+ end
+end
+{% endhighlight %}
+
+You should configure MCollective to use this file by putting the following in your _server.cfg_
+
+{% highlight ini %}
+classesfile = /var/tmp/chefnode.txt
+{% endhighlight %}
+
+You can now use your roles and recipe lists in filters:
+
+{% highlight console %}
+% mco find --with-class role.webserver --with-class /apache/
+{% endhighlight %}
--- /dev/null
+---
+layout: default
+title: Using with Puppet
+toc: false
+---
+[FactsRLFacter]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/FactsFacterYAML
+[PluginSync]: http://docs.reductivelabs.com/guides/plugins_in_modules.html
+[AgentPuppetd]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AgentPuppetd
+[AgentPuppetca]: http://github.com/puppetlabs/mcollective-plugins/tree/master/agent/puppetca/
+[AgentPuppetRal]: http://github.com/puppetlabs/mcollective-plugins/tree/master/agent/puppetral/
+[PuppetCommander]: http://github.com/puppetlabs/mcollective-plugins/tree/master/agent/puppetd/commander/
+[RapidRuns]: http://www.devco.net/archives/2010/08/05/rapid_puppet_runs_with_mcollective.php
+[EC2Bootstrap]: http://www.devco.net/archives/2010/07/14/bootstrapping_puppet_on_ec2_with_mcollective.php
+[PuppetRalBlog]: http://www.devco.net/archives/2010/07/07/puppet_resources_on_demand.php
+[SchedulingPuppet]: http://www.devco.net/archives/2010/03/17/scheduling_puppet_with_mcollective.php
+[ManagingPuppetd]: http://www.devco.net/archives/2009/11/30/managing_puppetd_with_mcollective.php
+[CloudBootstrap]: http://vuksan.com/blog/2010/07/28/bootstraping-your-cloud-environment-with-puppet-and-mcollective/
+[ServiceAgent]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AgentService
+[PackageAgent]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AgentPackage
+[Facter2YAML]: http://www.semicomplete.com/blog/geekery/puppet-facts-into-mcollective.html
+
+If you're a Puppet user you are supported in both facts and classes filters.
+
+There are a number of community plugins related to Puppet:
+
+ * Manage the Puppetd, request runs, enable and disable - [AgentPuppetd]
+ * Manage the Puppet CA, sign, list and revoke certificates - [AgentPuppetca]
+ * Use the Puppet Ral to create resources on demand, a distributed *ralsh* - [AgentPuppetRal]
+ * Schedule your puppetd's controlling concurrency and resource usage - [PuppetCommander]
+ * The [ServiceAgent] and [PackageAgent] use the Puppet RAL to function on many operating systems
+
+There are also several blog posts related to Puppet and MCollective:
+
+ * Running puppet on a number of nodes as quick as possible - [RapidRuns]
+ * Bootstrapping a Puppet + EC2 environment with Puppet - [EC2Bootstrap]
+ * Using the Puppet RAL with MCollective - [PuppetRalBlog]
+ * General scheduling of Puppet Runs - [SchedulingPuppet]
+ * Managing Puppetd - [ManagingPuppetd]
+ * Bootstrapping your cloud environment - [CloudBootstrap]
+
+## Facts
+There is a [community plugin to enable Facter][FactsRLFacter] as a fact source.
+
+So you can use the facts provided by Facter in filters, reports etc.
+
+{% highlight console %}
+$ mco find --with-fact lsbdistrelease=5.4
+{% endhighlight %}
+
+This includes facts pushed out with [Plugin Sync][PluginSync].
+
+A less resource intensive approach has can be found [here][Facter2YAML], it converts the Puppet scope into a YAML file that the YAML fact source then loads. This is both less resource intensive and much faster.
+
+## Class Filters
+Puppet provides a list of classes applied to a node by default in */var/puppet/state/classes.txt* or */var/lib/puppet/state/classes.txt* (depending on which Puppet version you are using. The latter is true for 0.25.5 onwards) , we'll use this data with *--with-class* filters.
+
+You should configure MCollective to use this file by putting the following in your *server.cfg*
+
+
+{% highlight ini %}
+classesfile = /var/lib/puppet/classes.txt
+{% endhighlight %}
+
+or if using Puppet 0.23.0 and onwards
+
+{% highlight ini %}
+classesfile = /var/lib/puppet/state/classes.txt
+{% endhighlight %}
+
+You can now use your classes lists in filters:
+
+{% highlight console %}
+$ mco find --with-class /apache/
+{% endhighlight %}
--- /dev/null
+---
+layout: default
+title: Aggregate Plugins
+---
+[DDL]: /mcollective/reference/plugins/ddl.html
+[Examples]: https://github.com/puppetlabs/marionette-collective/tree/master/plugins/mcollective/aggregate
+
+## Overview
+MCollective Agents return data and we try to provide as much usable user
+interface for free. To aid in this we require agents to have [DDL] files that
+describe the data that the agent returns.
+
+DDL files are used to configure the client but also to assist with user
+interface generation. They are used to ask questions that an action needs but
+also to render the results when the replies come in. For example we turn
+*:freecpu* into "Free CPU" when displaying the data based on the DDL.
+
+Previously if data that agents returned required any summarization this had to
+be done using a custom application. Here is an example from *mco nrpe*:
+
+{% highlight console %}
+% mco nrpe check_load
+.
+.
+Finished processing 25 / 25 hosts in 556.48 ms
+
+ OK: 25
+ WARNING: 0
+ CRITICAL: 0
+ UNKNOWN: 0
+{% endhighlight %}
+
+Here to get the summary of results displayed in a way that has contextual
+relevance to the nrpe plugin a custom application had to be written and anyone
+who interacts with the agent using other RPC clients would not get the benefit
+of this summary.
+
+By using aggregate plugins and updating the DDL we can now provide such a
+summary in all result sets and display it using the *mco rpc* application and
+any calls to *printrpc*.
+
+{% highlight console %}
+% mco rpc nrpe runcommand command=check_load
+Discovering hosts using the mongo method .... 25
+
+ * [============================================================> ] 25 / 25
+
+Summary of Exit Code:
+
+ OK : 25
+ WARNING : 0
+ UNKNOWN : 0
+ CRITICAL : 0
+
+
+Finished processing 25 / 25 hosts in 390.70 ms
+{% endhighlight %}
+
+Here you get a similar summary as before, all that had to be done was a simple
+aggregate plugin be written and distributed with your clients.
+
+The results are shown as above using *printrpcstats* but you can also get access to
+the raw data so you can decide to render it in some other way - perhaps using a
+graph on a web interface.
+
+We provide a number of aggregate plugins with MCollective and anyone can write
+more.
+
+For examples that already use functions see the *rpcutil* agent - its
+*collective_info*, *get_fact*, *daemon_stats* and *get_config_item* actions all
+have summaries applied.
+
+*NOTE:* This feature is available since version 2.1.0
+
+## Using existing plugins
+
+### Updating the DDL
+At present MCollective supplies 3 plugins *average()*, *summary()* and *sum()*
+you can use these in any agent, here is an example from the *rpcutil* agent DDL
+file:
+
+{% highlight ruby %}
+action "get_config_item", :description => "Get the active value of a specific config property" do
+ output :value,
+ :description => "The value that is in use",
+ :display_as => "Value"
+
+ summarize do
+ aggregate summary(:value)
+ end
+end
+{% endhighlight %}
+
+We've removed a few lines from this example DDL block leaving only the relevant
+lines. You can see the agent outputs data called *:value* and we reference that
+output in the summary function *summary(:value)*, the result would look like
+this:
+
+### Viewing summaries on the CLI
+{% highlight console %}
+% mco rpc rpcutil get_config_item item=collectives
+.
+.
+dev8
+ Property: collectives
+ Value: ["mcollective", "uk_collective"]
+
+Summary of Value:
+
+ mcollective = 25
+ uk_collective = 15
+ fr_collective = 9
+ us_collective = 1
+
+Finished processing 25 / 25 hosts in 349.70 ms
+{% endhighlight %}
+
+You can see that the value in this case contains arrays, the *summary()*
+function produce the table in the output showing the data distribution.
+
+### Producing summaries in your own clients
+You can enable the same display in your own code, here is ruby code that has the
+same affect as the CLI call above:
+
+{% highlight ruby %}
+require 'mcollective'
+
+include MCollective::RPC
+
+c = rpcclient("rpcutil")
+
+printrpc c.get_config_item(:item => "collectives")
+
+printrpcstats :summarize => true
+{% endhighlight %}
+
+Without passing in the *:summarize => true* you would not see the summaries
+
+### Getting access to the raw summary results
+If you wanted to do something else entirely like produce a graph on a web page
+of the summaries you can get access to the raw data, here's some ruby code to
+show all computed summaries:
+
+{% highlight ruby %}
+require 'mcollective'
+
+include MCollective::RPC
+
+c = rpcclient("rpcutil")
+c.progress = false
+
+c.get_config_item(:item => "collectives")
+
+c.stats.aggregate_summary.each do |summary|
+ puts "Summary of type: %s" % summary.result_type
+ puts "Display format: '%s'" % summary.aggregate_format
+ puts
+ pp summary.result
+end
+{% endhighlight %}
+
+As you can see you will get an array of summaries this is because each DDL can
+use many aggregate calls, this would be an array of all the computed summaries:
+
+{% highlight console %}
+Summary of type: collection
+Display format: '%13s = %s'
+
+{:type=>:collection,
+ :value=>
+ {"mcollective"=>25,
+ "fr_collective"=>9,
+ "us_collective"=>1,
+ "uk_collective"=>15},
+ :output=>:value}
+{% endhighlight %}
+
+There are 2 types of result *:collection* and *:numeric*, in the case of numeric
+results the :value would just be a number.
+
+The *aggregate_format* is either a user supplied format or a dynamically
+computed format to display the summary results on the console. In this case
+each pair of the hash should be displayed using the format to produce a nice
+right justified list of keys and values.
+
+## Writing your own function
+We'll cover writing your own function by looking at the Nagios one from earlier
+in this example. You can look at [the functions supplied with
+MCollective][Examples] for more examples using other types than the one below.
+
+First lets look at the DDL for the existing *nrpe* Agent:
+
+{% highlight ruby %}
+action "runcommand", :description => "Run a NRPE command" do
+ input :command,
+ :prompt => "Command",
+ :description => "NRPE command to run",
+ :type => :string,
+ :validation => '\A[a-zA-Z0-9_-]+\z',
+ :optional => false,
+ :maxlength => 50
+
+ output :output,
+ :description => "Output from the Nagios plugin",
+ :display_as => "Output",
+ :default => ""
+
+ output :exitcode,
+ :description => "Exit Code from the Nagios plugin",
+ :display_as => "Exit Code",
+ :default => 3
+
+ output :perfdata,
+ :description => "Performance Data from the Nagios plugin",
+ :display_as => "Performance Data",
+ :default => ""
+end
+{% endhighlight %}
+
+You can see it will return an *:exitcode* item and from the default value you
+can gather this is going to be a number. Nagios defines 4 possibly exit codes
+for a Nagios plugin and we need to convert this *:exitcode* into a string like
+WARNING, CRITICAL, UNKNOWN or OK.
+
+Usually when writing any kind of summarizer for an array of results your code
+might contain 3 phases.
+
+Given a series of Nagios results like this:
+
+{% highlight ruby %}
+[
+ {:exitcode => 0, :output => "OK", :perfdata => ""},
+ {:exitcode => 2, :output => "failure", :perfdata => ""}
+]
+{% endhighlight %}
+
+You would write a nagios_states() function that does roughly this:
+
+{% highlight ruby %}
+def nagios_states(results)
+ # set initial values
+ result = {}
+ status_map = ["OK", "WARNING", "CRITICAL", "UNKNOWN"]
+ status_map.each {|s| result[s] = 0}
+
+ # loop over all the data, increment the count for OK etc
+ results.each do |result|
+ status = status_map[result[:exitcode]]
+ result[status] += 1
+ end
+
+ # return the result hash, {"OK" => 1, "CRITICAL" => 1, "WARN" => 0, "UNKNOWN" => 0}
+ return result
+end
+{% endhighlight %}
+
+You could optimise the code but you can see there are 3 major stages in the life
+of this code.
+
+ * Set initial values for the return data
+ * Loop the data building up the state
+ * Return the data.
+
+Given this, here is our Nagios exitcode summary function, it is roughly the same
+code with a bit more boiler plate to plugin into mcollective, but the same code
+can be seen:
+
+{% highlight ruby %}
+module MCollective
+ class Aggregate
+ class Nagios_states<Base
+ # Before function is run processing
+ def startup_hook
+ # :collection or :numeric
+ @result[:type] = :collection
+
+ # set default aggregate_format if it is undefined
+ @aggregate_format = "%10s : %s" unless @aggregate_format
+
+ @result[:value] = {}
+
+ @status_map = ["OK", "WARNING", "CRITICAL", "UNKNOWN"]
+ @status_map.each {|s| @result[:value][s] = 0}
+
+ end
+
+ # Determines the average of a set of numerical values
+ def process_result(value, reply)
+ if value
+ status = @status_map[value]
+ @result[:value][status] += 1
+ else
+ @result["UNKNOWN"] += 1
+ end
+ end
+
+ # Post processing hook that returns the summary result
+ def summarize
+ result_class(@result[:type]).new(@result, @aggregate_format, @action)
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+This shows that an aggregate function has the same 3 basic parts. First we set
+the initial state using the *startup_hook*. We then process each result as it
+comes in from the network using *process_result*. Finally we turn that into a
+the result objects that you saw earlier in the ruby client examples using the
+*summarize* method.
+
+### *startup_hook*
+Each function needs a startup hook, without one you'll get exceptions. The
+startup hook lets you set up the initial state.
+
+The first thing to do is set the type of result this will be. Currently we
+support 2 types of result either a plain number indicated using *:numeric* or a
+complex *:collection* type that can be a hash with keys and values.
+
+Functions can take display formats in the DDL, in this example we set
+*@aggregate_format* to a *printf* default that would display a table of results
+but we still let the user supply his own format.
+
+We then just initialize the result hash to and build a map from the English
+representation of the Nagios status codes.
+
+### *process_result*
+Every reply that comes in from the network gets passed into your
+*process_result* method. The first argument will be just the single value the
+DDL indicates you are interested in but you'll also get the whole rely so you
+can get access to other reply values and such.
+
+This gets called each time, we just look at the value and increment each Nagios
+status or treat it as an unknown - in case the result data is missformed.
+
+### *summarize*
+The summarize method lets you take the state you built up and convert that into
+an answer. The summarize method is optional what you see here is the default
+action if you do not supply one.
+
+The *result_class* method accepts either *:collection* or *:numeric* as
+arguments and it is basically a factory for the correct result structure.
+
+### Deploy and update the DDL
+You should deploy this function into your *libdir/aggregate* directory called
+*nagios_states.rb* on the client machines - no harm deploying it everywhere
+though.
+
+Update the DDL so it looks like:
+
+{% highlight ruby %}
+action "runcommand", :description => "Run a NRPE command" do
+ .
+ .
+ .
+
+ if respond_to?(:summarize)
+ summarize do
+ aggregate nagios_states(:exitcode)
+ end
+ end
+end
+{% endhighlight %}
+
+Add the last few lines - we check that we're running in a version of MCollective
+that supports this feature and then we call our function with the *:exitcode*
+results.
+
+
--- /dev/null
+---
+layout: default
+title: Single Executable Application Plugin
+---
+[Clients]: ../../simplerpc/clients.html
+[SimpleRPCAgents]: ../../simplerpc/agents.html
+
+
+## Overview
+The Marionette Collective 1.1.1 and newer supports a single executable - called
+mco - and have a plugin type called application that lets you create
+applications for this single executable.
+
+In the past we tended to write small standalone scripts to interact with
+MCollective, this had a number of issues:
+
+ * Large number of executables in _/usr/sbin_
+ * Applications behave inconsistently with regard to error handling and reporting
+ * Discovering new applications is difficult since they are all over the filesystem
+ * Installation and packaging of plugins is complex
+
+We've attempted to address these concerns by creating a single point of access
+for all applications - the _mco_ script - with unified help, error reporting and
+option parsing.
+
+Below you can see the single executable system in use:
+
+{% highlight console %}
+The Marionette Collective version 2.0.0
+
+usage: /usr/bin/mco: command <options>
+
+Known commands:
+
+ cap controller exim
+ facts filemgr find
+ help inventory iptables
+ nettest nrpe package
+ pgrep ping plugin
+ puppetd rpc service
+ virt
+
+Type 'mco help' for a detailed list of commands and 'mco help command'
+to get detailed help for a command
+
+{% endhighlight %}
+
+{% highlight console %}
+$ mco help
+The Marionette Collection version 2.0.0
+
+ facts Reports on usage for a specific fact
+ filemgr Generic File Manager Client
+ find Find hosts matching criteria
+ help Application list and RPC agent help
+ inventory Shows an inventory for a given node
+ ping Ping all nodes
+ rpc Generic RPC agent client application
+{% endhighlight %}
+
+{% highlight console %}
+$ mco rpc package status package=zsh
+Determining the amount of hosts matching filter for 2 seconds .... 51
+
+ * [ ============================================================> ] 51 / 51
+
+
+ test.com:
+ Properties:
+ {:provider=>:yum,
+ :release=>"3.el5",
+ :arch=>"x86_64",
+ :version=>"4.2.6",
+ :epoch=>"0",
+ :name=>"zsh",
+ :ensure=>"4.2.6-3.el5"}
+{% endhighlight %}
+
+These applications are equivalent to the old mc-rpc and similar applications but without the problem of lots of files in _/usr/sbin_.
+
+## Basic Application
+Applications goes in _libdir/mcollective/application/echo.rb_, the one below is a simple application that speaks to a hypothetical _echo_ action of a _helloworld_ agent. This agent has been demonstrated in : writing [agents][SimpleRPCAgents].
+
+{% highlight ruby %}
+class MCollective::Application::Echo<MCollective::Application
+ description "Reports on usage for a specific fact"
+
+ option :message,
+ :description => "Message to send",
+ :arguments => ["-m", "--message MESSAGE"],
+ :type => String,
+ :required => true
+
+ def main
+ mc = rpcclient("helloworld")
+
+ printrpc mc.echo(:msg => configuration[:message], :options => options)
+
+ printrpcstats
+ end
+end
+{% endhighlight %}
+
+Here's the application we wrote in action:
+
+{% highlight console %}
+$ mco echo
+The message option is mandatory
+
+Please run with --help for detailed help
+{% endhighlight %}
+
+{% highlight console %}
+$ mco echo -m test
+
+ * [ ============================================================> ] 1 / 1
+
+
+example.com
+ Message: test
+ Time: Mon Jan 31 21:27:03 +0000 2011
+
+
+Finished processing 1 / 1 hosts in 68.53 ms
+{% endhighlight %}
+
+Most of the techniques documented in SimpleRPC [Clients] can be reused here, we've just simplified a lot of the common used patterns like CLI arguments and incorporated it all in a single framework.
+
+## Reference
+
+### Usage Messages
+
+To add custom usage messages to your application we can add lines like this:
+
+{% highlight ruby %}
+class MCollective::Application::Echo<MCollective::Application
+ description "Reports on usage for a specific fact"
+
+ usage "mco echo [options] --message message"
+end
+{% endhighlight %}
+
+You can add several of these messages by just adding multiple such lines.
+
+### Application Options
+
+A standard options hash is available simply as options you can manipulate it and pass it into the RPC Client like normal. See the SimpleRPC [Clients] reference for more on this.
+
+### CLI Argument Parsing
+
+There are several options available to assist in parsing CLI arguments. The most basic option is:
+
+{% highlight ruby %}
+class MCollective::Application::Echo<MCollective::Application
+ option :message,
+ :description => "Message to send",
+ :arguments => ["-m", "--message MESSAGE"]
+end
+{% endhighlight %}
+
+In this case if the user used either _-m message_ or _--message message_ on the CLI the desired message would be in _configuration`[`:message`]`_
+
+#### Required Arguments
+You can require that a certain parameter is always passed:
+
+{% highlight ruby %}
+option :message,
+ :description => "Message to send",
+ :arguments => ["-m", "--message MESSAGE"],
+ :required => true
+{% endhighlight %}
+
+#### Argument data types
+CLI arguments can be forced to a specific type, we also have some additional special types that the default ruby option parser cant handle on its own.
+
+You can force data to be of type String, Fixnum etc:
+
+{% highlight ruby %}
+option :count,
+ :description => "Count",
+ :arguments => ["--count MESSAGE"],
+ :type => Fixnum
+{% endhighlight %}
+
+You can force an argument to be boolean:
+
+{% highlight ruby %}
+option :detail,
+ :description => "Detailed view",
+ :arguments => ["--detail"],
+ :type => :bool
+{% endhighlight %}
+
+If you have an argument that can be called many times you can force that to build an array:
+
+{% highlight ruby %}
+option :args,
+ :description => "Arguments",
+ :arguments => ["--argument ARG"],
+ :type => :array
+{% endhighlight %}
+
+Here if you supplied multiple arguments _configuration`[`:args`]`_ will be an array with all the options supplied.
+
+#### Argument validation
+You can validate input passed on the CLI:
+
+{% highlight ruby %}
+option :count,
+ :description => "Count",
+ :arguments => ["--count MESSAGE"],
+ :type => Fixnum,
+ :validate => Proc.new {|val| val < 10 ? true : "The message count has to be below 10" }
+{% endhighlight %}
+
+Should the supplied value be 10 or more a error message will be displayed.
+
+#### Disabling standard sections of arguments
+By default every Application get all the RPC options enabling filtering, discovery etc. In some cases this is undesirable so we let users disable those.
+
+{% highlight ruby %}
+class MCollective::Application::Echo<MCollective::Application
+ exclude_argument_sections "common", "filter", "rpc"
+end
+{% endhighlight %}
+
+This application will only have --help, --verbose and --config as options, all the other options will be removed.
+
+#### Post argument parsing hook
+Right after all arguments are parsed you can have a hook in your program called, this hook could perhaps parse the remaining data on _ARGV_ after option parsing is complete.
+
+{% highlight ruby %}
+class MCollective::Application::Echo<MCollective::Application
+ description "Reports on usage for a specific fact"
+
+ def post_option_parser(configuration)
+ unless ARGV.empty?
+ configuration[:message] = ARGV.shift
+ else
+ STDERR.puts "Please specify a message on the command line"
+ exit 1
+ end
+ end
+
+ def main
+ # use configuration[:message] here to access the message
+ end
+end
+{% endhighlight %}
+
+#### Validating configuration
+After the options are parsed and the post hook is called you can validate the contents of the configuration:
+
+{% highlight ruby %}
+class MCollective::Application::Echo<MCollective::Application
+ description "Reports on usage for a specific fact"
+
+ # parse the first argument as a message
+ def post_option_parser(configuration)
+ configuration[:message] = ARGV.shift unless ARGV.empty?
+ end
+
+ # stop the application if we didnt receive a message
+ def validate_configuration(configuration)
+ raise "Need to supply a message on the command line" unless configuration.include?(:message)
+ end
+
+ def main
+ # use configuration[:message] here to access the message
+ end
+end
+{% endhighlight %}
+
+### Exiting your application
+You can use the normal _exit_ Ruby method at any time to exit your application and you can supply any
+exit code as normal.
+
+The supplied applications have a standard exit code convention, if you want your applications to exhibit
+the same behavior use the _halt_ helper. The exit codes are below:
+
+|Code|Description |
+|----|-----------------------------------------------------|
+|0 |Nodes were discovered and all passed |
+|0 |No discovery was done but responses were received |
+|1 |No nodes were discovered |
+|2 |Nodes were discovered but some responses failed |
+|3 |Nodes were discovered but no responses were received |
+|4 |No discovery were done and no responses were received|
+
+{% highlight ruby %}
+class MCollective::Application::Echo<MCollective::Application
+ description "Reports on usage for a specific fact"
+
+ def main
+ mc = rpcclient("echo")
+
+ printrpc mc.echo(:msg => "Hello World", :options => options)
+
+ printrpcstats
+
+ halt mc.stats
+ end
+end
+{% endhighlight %}
+
+As you can see you pass the _halt_ helper an instance of the RPC Client statistics and it will then
+use that to do the right exit code.
+
--- /dev/null
+---
+layout: default
+title: ActiveMQ Connector
+toc: false
+---
+
+[STOMP]: http://stomp.codehaus.org/
+[wildcard]: http://activemq.apache.org/wildcards.html
+[subcollectives]: /reference/basic/subcollectives.html
+[activemq_config]: /mcollective/deploy/middleware/activemq.html
+
+
+The ActiveMQ connector uses the [STOMP][] rubygem to connect to ActiveMQ servers. It is specifically optimized for ActiveMQ
+and uses features in ActiveMQ 5.5.0 and later.
+
+This plugin requires version _1.2.2_ or newer of the Stomp gem.
+
+## Differences between ActiveMQ connector and Stomp Connector
+
+### Topic and Queue Names
+
+The new connector uses different destination names from the old stomp connector.
+
+MCollective uses the following destination names. This list uses standard [ActiveMQ destination wildcards][wildcard]. "COLLECTIVE" is the name of the collective being used; by default, this is `mcollective`, but if you are using [subcollectives][], each one is implemented as an equal peer of the default collective.
+
+Topics:
+
+- `ActiveMQ.Advisory.>` (built-in topics that all ActiveMQ producers and consumers need all permissions on)
+- `COLLECTIVE.*.agent` (for each agent plugin, where the `*` is the name of the plugin)
+
+Queues:
+
+- `COLLECTIVE.nodes` (used for direct addressing; this is a single destination that uses JMS selectors, rather than a group of destinations)
+- `COLLECTIVE.reply.>` (where the continued portion is a request ID)
+
+Note especially that:
+
+* We can now do direct addressing to specific nodes.
+* Replies now go directly to the instigating client instead of being brodcast on a topic.
+
+This has big impact on overall CPU usage by clients on busy networks, and also optimizes the traffic flow on
+networks with many brokers.
+
+
+## Configuring ActiveMQ
+
+See [the ActiveMQ config reference][activemq_config] for details on configuring ActiveMQ for this connector. As recommended at the top of the reference, you should skim the sections you care about and edit an example config file while reading.
+
+
+## Configuring MCollective
+
+MCollective clients and servers use the same connector settings, although the value of settings involving credentials will vary.
+
+### Failover Pools
+
+A sample configuration can be seen below. Note this plugin does not support the old style config of the Stomp connector.
+
+{% highlight ini %}
+connector = activemq
+plugin.activemq.pool.size = 2
+plugin.activemq.pool.1.host = stomp1
+plugin.activemq.pool.1.port = 61613
+plugin.activemq.pool.1.user = me
+plugin.activemq.pool.1.password = secret
+
+plugin.activemq.pool.2.host = stomp2
+plugin.activemq.pool.2.port = 61613
+plugin.activemq.pool.2.user = me
+plugin.activemq.pool.2.password = secret
+{% endhighlight %}
+
+This gives it 2 servers to attempt to connect to, if the first one fails it will use the second. Usernames and passwords can be set
+with the environment variables `STOMP_USER`, `STOMP_PASSWORD`.
+
+If you do not specify a port it will default to _61613_
+
+You can also specify the following options for the Stomp gem, these are the defaults in the Stomp gem: <!-- last checked: v. 1.1.6 of the gem -->
+
+{% highlight ini %}
+plugin.activemq.initial_reconnect_delay = 0.01
+plugin.activemq.max_reconnect_delay = 30.0
+plugin.activemq.use_exponential_back_off = true
+plugin.activemq.back_off_multiplier = 2
+plugin.activemq.max_reconnect_attempts = 0
+plugin.activemq.randomize = false
+plugin.activemq.timeout = -1
+{% endhighlight %}
+
+### Message Priority
+
+ActiveMQ messages support priorities, you can pass in the needed priority header by setting:
+
+{% highlight ini %}
+plugin.activemq.priority = 4
+{% endhighlight %}
--- /dev/null
+---
+layout: default
+title: RabbitMQ Connector
+toc: false
+---
+[STOMP]: http://stomp.codehaus.org/
+[RabbitStomp]: http://www.rabbitmq.com/stomp.html
+[RabbitCLI]: http://www.rabbitmq.com/management-cli.html
+
+The RabbitMQ connector uses the [STOMP] rubygem to connect to RabbitMQ servers.
+
+This code will only work with version _1.2.2_ or newer of the Stomp gem.
+
+## Differences between RabbitMQ connector and Stomp Connector
+
+The RabbitMQ connector requires MCollective 2.0.0 or newer.
+
+While this plugin still uses the Stomp protocol to connect to RabbitMQ it does use a nubmer of
+RabbitMQ specific optimizations to work well and as such is a Stomp connector specific to the
+RabbitMQ broker.
+
+## Configuring RabbitMQ
+
+Basic installation of the RabbitMQ broker is out of scope for this document apart from the
+basic broker you need to enable the [Stomp plugi][RabbitStomp] and the [CLI Management Tool][RabbitCLI].
+
+With that in place you need to create a few exchanges, topics and queues for each of your
+sub collectives.
+
+First we create a virtual host, user and some permissions on the vhost:
+
+{% highlight console %}
+rabbitmqadmin declare vhost=/mcollective
+rabbitmqadmin declare user=mcollective password=changeme
+rabbitmqadmin declare permission vhost=/mcollective user=mcollective configure=.* write=.* read=.*
+{% endhighlight %}
+
+And then we need to create two exchanges that the mcollective plugin needs:
+
+{% highlight console %}
+rabbitmqadmin declare exchange vhost=/mcollective name=mcollective_broadcast type=topic
+rabbitmqadmin declare exchange vhost=/mcollective name=mcollective_directed type=direct
+{% endhighlight %}
+
+## Configuring MCollective
+
+### Common Options
+
+### Failover Pools
+A sample configuration can be seen below.
+
+{% highlight ini %}
+direct_addressing = 1
+
+connector = rabbitmq
+plugin.rabbitmq.vhost = /mcollective
+plugin.rabbitmq.pool.size = 2
+plugin.rabbitmq.pool.1.host = rabbit1
+plugin.rabbitmq.pool.1.port = 61613
+plugin.rabbitmq.pool.1.user = mcollective
+plugin.rabbitmq.pool.1.password = changeme
+
+plugin.rabbitmq.pool.2.host = rabbit2
+plugin.rabbitmq.pool.2.port = 61613
+plugin.rabbitmq.pool.2.user = mcollective
+plugin.rabbitmq.pool.2.password = changeme
+{% endhighlight %}
+
+This gives it 2 servers to attempt to connect to, if the first one fails it will use the second. Usernames and passwords can be set
+with the environment variables STOMP_USER, STOMP_PASSWORD.
+
+If you do not specify a port it will default to _61613_
+
+You can also specify the following options for the Stomp gem, these are the defaults in the Stomp 1.2.2 gem:
+
+{% highlight ini %}
+plugin.rabbitmq.initial_reconnect_delay = 0.01
+plugin.rabbitmq.max_reconnect_delay = 30.0
+plugin.rabbitmq.use_exponential_back_off = true
+plugin.rabbitmq.back_off_multiplier = 2
+plugin.rabbitmq.max_reconnect_attempts = 0
+plugin.rabbitmq.randomize = false
+plugin.rabbitmq.timeout = -1
+plugin.rabbitmq.vhost = /
+{% endhighlight %}
--- /dev/null
+---
+layout: default
+title: STOMP Connector
+toc: false
+---
+[STOMP]: http://stomp.codehaus.org/
+[ConnectorActiveMQ]: /mcollective/reference/plugins/connector_activemq.html
+[ConnectorRabbitMQ]: /mcollective/reference/plugins/connector_rabbitmq.html
+
+*NOTE:* This connector is being deprecated and will not be supported in versions newer than 2.2.x. Please move to one of the [ConnectorActiveMQ] or [ConnectorRabbitMQ].
+
+The stomp connector uses the [STOMP] rubygem to connect to compatible servers. This is known to work with ActiveMQ and Stompserver. Anecdotal evidence suggests it works with RabbitMQ's Stomp plugin.
+
+This code will only work with version _1.1_ and _1.1.6_ or newer of the Stomp gem, the in between versions have threading issues.
+
+As this connector tries to be as generic as possible it is hard to support all the advanced features of MCollective using it. We do not recommend you use the directed mode
+using this plugin, instead look towards specific ones written for ActiveMQ or your chosen middleware.
+
+## Middleware Layout
+
+For broadcast messages this connector will create _topics_ with names like _/topic/<collective>.<agent>.command_ and replies will go to
+_/topic/<collective>.<agent>.reply_
+
+For directed messages it will create queues with names like _/queue/<collective>.mcollective.<md5 hash of identity>_.
+
+You should configure appropriate ACLs on your middleware to allow this scheme
+
+## Configuring
+
+### Common Options
+The most basic configuration method is supported in all versions of the gem:
+
+{% highlight ini %}
+connector = stomp
+plugin.stomp.base64 = false
+plugin.stomp.host = stomp.my.net
+plugin.stomp.port = 61613
+plugin.stomp.user = me
+plugin.stomp.password = secret
+{% endhighlight %}
+
+You can override all of these settings using environment variables STOMP_SERVER, STOMP_PORT, STOMP_USER, STOMP_PASSWORD. It is recommended that your _client.cfg_ do not have usernames and passwords in it, users should set their own in the environment.
+
+If you are seeing issues with the Stomp gem logging protocol errors and resetting your connections, especially if you are using Ruby on Rails then set the _plugin.stomp.base64_ to true, this adds an additional layer of encoding on packets to make sure they don't interfere with UTF8 encoding used in Rails.
+
+### Failover Pools
+Newer versions of the Stomp gem supports failover between multiple Stomp servers, you need at least _1.1.6_ to use this.
+
+If you are using version _1.1.9_ and newer of the Stomp Gem and this method of configuration you will also receive more detailed
+logging about connections, failures and other significant events.
+
+{% highlight ini %}
+connector = stomp
+plugin.stomp.pool.size = 2
+plugin.stomp.pool.host1 = stomp1
+plugin.stomp.pool.port1 = 61613
+plugin.stomp.pool.user1 = me
+plugin.stomp.pool.password1 = secret
+
+plugin.stomp.pool.host2 = stomp2
+plugin.stomp.pool.port2 = 61613
+plugin.stomp.pool.user2 = me
+plugin.stomp.pool.password2 = secret
+{% endhighlight %}
+
+This gives it 2 servers to attempt to connect to, if the first one fails it will use the second. As before usernames and passwords can be set with STOMP_USER, STOMP_PASSWORD.
+
+If you do not specify a port it will default to _6163_
+
+When using pools you can also specify the following options, these are the defaults in the Stomp 1.1.6 gem:
+
+{% highlight ini %}
+plugin.stomp.pool.initial_reconnect_delay = 0.01
+plugin.stomp.pool.max_reconnect_delay = 30.0
+plugin.stomp.pool.use_exponential_back_off = true
+plugin.stomp.pool.back_off_multiplier = 2
+plugin.stomp.pool.max_reconnect_attempts = 0
+plugin.stomp.pool.randomize = false
+plugin.stomp.pool.timeout = -1
+plugin.stomp.pool.connect_timeout = 30
+{% endhighlight %}
+
+### Message Priority
+
+As of version 5.4 of ActiveMQ messages support priorities, you can pass in the needed
+priority header by setting:
+
+{% highlight ini %}
+plugin.stomp.priority = 4
+{% endhighlight %}
--- /dev/null
+---
+layout: default
+title: Data Plugins
+---
+[DDL]: /mcollective/reference/plugins/ddl.html
+[DiscoveryPlugins]: /mcollective/reference/plugins/discovery.html
+
+## Overview
+Up to MCollective 2.0 the discovery system could only discover against
+installed agents, configuration management classes or facts and the node
+identities. We're extending this to support discovery against many
+sources through a simple plugin system.
+
+*NOTE:* This feature is available since version 2.1.0
+
+The basic idea is that you could do discovery statements like the ones
+below:
+
+{% highlight console %}
+% mco find -S "fstat('/etc/rsyslog.conf').md5=/4edff591f6e38/"
+{% endhighlight %}
+
+{% highlight console %}
+% mco find -S "sysctl('net.ipv4.conf.all.forwarding').value=1"
+{% endhighlight %}
+
+{% highlight console %}
+% mco find -S "sysctl('net.ipv4.conf.all.forwarding').value=1 and % location=dc1"
+{% endhighlight %}
+
+You could also use these data sources in your own agents or other
+plugins:
+
+{% highlight ruby %}
+action "query" do
+ reply[:value] = Data.sysctl(request[:sysctl_name]).value
+end
+{% endhighlight %}
+
+*NOTE:* As opposed to the [DiscoveryPlugins] which are used by the client
+to communicate to the nodes using direct addressing, data plugins on the other
+hand refer to data that the nodes can provide, and hence this uses the normal
+broadcast discovery paradigm.
+
+These new data sources are plugins so you can provide via the plugin
+system and they require DDL documents. The DDL will be used on both the
+client and the server to provide strict validation and configuration.
+
+The DDL for these plugins will affect the client libraries in the
+following ways:
+
+ * You will get errors if you try to discover using unknown functions
+ * Your input argument values will be validated against the DDL
+ * You will only be able to use output properties that are known in the DDL
+ * If a plugin DDL says it needs 5 seconds to run your discovery and maximum run times will increase by 5 seconds automatically
+
+On the servers the DDL will:
+
+ * be used to validate known plugins
+ * be used to validate input arguments
+ * be used to validate requests for known output values
+
+## Viewing or retrieving results from a data plugin
+
+You can view the output from a data plugin using the *rpcutil* agent:
+
+{% highlight console %}
+% mco rpc rpcutil get_data source=fstat query=/etc/hosts
+.
+.
+your.node.net
+ atime: 2012-06-14 21:41:54
+ atime_age: 54128
+ atime_seconds: 1339706514
+ ctime: 2012-01-18 20:28:34
+ ctime_age: 12842128
+ ctime_seconds: 1326918514
+ gid: 0
+ md5: 54fb6627dbaa37721048e4549db3224d
+ mode: 100644
+ mtime: 2010-01-12 13:28:22
+ mtime_age: 76457740
+ mtime_seconds: 1263302902
+ name: /etc/hosts
+ output: present
+ present: 1
+ size: 158
+ type: file
+ uid: 0
+{% endhighlight %}
+
+The same action can be used to retrieve data programatically.
+
+## Writing a data plugin
+
+### The Ruby logic for the plugin
+The data plugins should not change the system in anyway, you should take
+care to create plugins that only reads the state of the system. If you
+want to affect the status of the system you should write Agents.
+
+These plugins are kept simple as they will be typed on the command line
+so the following restrictions are present:
+
+ * They can only take 1 input argument
+ * They can only return simple String, Numeric or Booleans no Hashes or complex data types
+ * They should be fast as these will impact discovery times and agent run times.
+
+Writing data plugins is easy and mimic the basics of writing agents,
+below we have a simple *sysctl* plugin that was used in the examples
+earlier:
+
+{% highlight ruby linenos %}
+module MCollective
+ module Data
+ class Sysctl_data<Base
+ activate_when { File.executable?("/sbin/sysctl") && Facter["kernel"] == "Linux" }
+
+ query do |sysctl|
+ shell = Shell.new("/sbin/sysctl %s" % sysctl)
+ shell.runcommand
+
+ if shell.status.exitstatus == 0
+ value = shell.stdout.chomp.split(/\s*=\s*/)[1]
+
+ if value
+ value = Integer(value) if value =~ /^\d+$/
+ value = Float(value) if value =~ /^\d+\.\d+$/
+ end
+
+ result[:value] = value
+ end
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+The class names have to be *Something_data* and they must inherit from
+*Base* as in the example here. The file would be saved in the *libdir*
+as *data/sysctl_data.rb* and *data/sysctl_data.ddl*.
+
+This plugin will only be activated if the file */sbin/sysctl* exist, is
+executable and if the system is a Linux server. This allow us to install
+it on a Windows machine where it will just be disabled and those
+machines will never be discovered using this function.
+
+We then create a block that would be the main body of the query. We use
+the *MCollective::Shell* class to run sysctl, parse the output and save
+it into the *result* hash.
+
+The result hash is the only way to return values from these plugins. You
+can only save simple strings, numbers or booleans in the result.
+
+### The DDL for the plugin
+As mentioned every data plugin requires a DDL. These DDL files mimic
+those of the [SimpleRPC Agents][DDL].
+
+Below you'll find a DDL for the above sysctl data plugin:
+
+{% highlight ruby linenos %}
+metadata :name => "Sysctl values",
+ :description => "Retrieve values for a given sysctl",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://marionette-collective.org/",
+ :timeout => 1
+
+dataquery :description => "Sysctl values" do
+ input :query,
+ :prompt => "Variable Name",
+ :description => "Valid Variable Name",
+ :type => :string,
+ :validation => /^[\w\-\.]+$/,
+ :maxlength => 120
+
+ output :value,
+ :description => "Kernel Parameter Value",
+ :display_as => "Value"
+end
+{% endhighlight %}
+
+The *timeout* must be set correctly, if your data source is slow you
+need to reflect that in the timeout here. The timeout will be used on
+the clients to decide how long to wait for discovery responses from the
+network so getting this wrong will result in nodes not being discovered.
+
+Each data plugin can only have one *dataquery* block with exactly 1
+*input* block but could have multiple *output* blocks.
+
+It's important to get the validation correct, here we only accept the
+characters we know are legal in sysctl variables on Linux. We will
+specifically never allow backticks to be used in arguments to avoid
+accidental shell exploits.
+
+Note the correlation between output names and the use in discovery and
+agents here we create an output called *value* this means we would use
+it in discovery as:
+
+{% highlight console %}
+% mco find -S "sysctl('net.ipv4.conf.all.forwarding').value=1"
+{% endhighlight %}
+
+And we would output the result from our plugin code as:
+
+{% highlight ruby linenos %}
+result[:value] = value
+{% endhighlight %}
+
+And in any agent where we might use the data source:
+
+{% highlight ruby linenos %}
+something = Data.sysctl('net.ipv4.conf.all.forwarding').value
+{% endhighlight %}
+
+These have to match everywhere, you cannot reference undeclared data and
+you cannot use input that does not validate against the DDL declared
+validations.
+
+Refer to the full [DDL] documentation for details on all possible values
+of the *metadata*, *input* and *output* blocks.
+
+## Auto generated documentation
+As with agents the DDL can be used to generate documentation, if you
+wanted to know what the input and output values are for a specific
+plugin you can use *mco plugin doc* to see generated documentation.
+
+{% highlight console %}
+% mco plugin doc sysctl
+Sysctl values
+=============
+
+Retrieve values for a given sysctl
+
+ Author: R.I.Pienaar <rip@devco.net>
+ Version: 1.0
+ License: ASL 2.0
+ Timeout: 1
+ Home Page: http://marionette-collective.org/
+
+QUERY FUNCTION INPUT:
+
+ Description: Valid Variable Name
+ Prompt: Variable Name
+ Type: string
+ Validation: (?-mix:^[\w\-\.]+$)
+ Length: 120
+
+QUERY FUNCTION OUTPUT:
+
+ value:
+ Description: Kernel Parameter Value
+ Display As: Value
+
+{% endhighlight %}
+
+## Available plugins for a node You can use the *mco inventory*
+application to see remotely what plugins a node has available:
+
+{% highlight console %}
+% mco inventory your.node
+Inventory for your.node:
+
+ .
+ .
+ .
+
+ Data Plugins:
+ fstat sysctl
+
+{% endhighlight %}
--- /dev/null
+---
+layout: default
+title: Data Definition Language
+---
+[WritingAgents]: /mcollective/reference/basic/basic_agent_and_client.html
+[SimpleRPCClients]: /mcollective/simplerpc/clients.html
+[ResultsandExceptions]: /mcollective/simplerpc/clients.html#results_and_exceptions
+[SimpleRPCAuditing]: /mcollective/simplerpc/auditing.html
+[SimpleRPCAuthorization]: /mcollective/simplerpc/authorization.html
+[WritingAgentsScreenCast]: http://mcollective.blip.tv/file/3808928/
+[DDLScreenCast]: http://mcollective.blip.tv/file/3799653
+[RPCUtil]: /mcollective/reference/plugins/rpcutil.html
+[ValidatorPlugins]: /mcollective/reference/plugins/validator.html
+
+As with other remote procedure invocation systems MCollective has a DDL that defines what remote methods are available, what inputs they take and what outputs they generate.
+
+In addition to the usual procedure definitions we also keep meta data about author, versions, license and other key data points.
+
+The DDL is used in various scenarios:
+
+ * The user can access it in the form of a human readable help page
+ * User interfaces can access it in a way that facilitate auto generation of user interfaces
+ * The RPC client auto configures and use appropriate timeouts in waiting for responses
+ * Before sending a call over the network inputs get validated so we do not send unexpected data to remote nodes.
+ * Module repositories can use the meta data to display a standard view of available modules to assist a user in picking the right ones.
+ * The server will validate incoming requests prior to sending it to agents
+
+We've created [a screencast showing the capabilities of the DDL][DDLScreenCast] that might help give you a better overview.
+
+**NOTE:** As of version 2.1.1 the DDL is required on all servers before an agent will be activated
+
+## Examples
+We'll start with a few examples as I think it's pretty simple what they do, and later on show what other permutations are allowed for defining inputs and outputs.
+
+A helper agent called [_rpcutil_][RPCUtil] is included that helps you gather stats, inventory etc about the running daemon. This helper has a full DDL included, see the plugins dir for this agent.
+
+The typical service agent is a good example, it has various actions that all more or less take the same input. All but status would have almost identical language.
+
+### Meta Data
+First we need to define the meta data for the agent itself:
+
+{% highlight ruby linenos %}
+metadata :name => "SimpleRPC Service Agent",
+ :description => "Agent to manage services using the Puppet service provider",
+ :author => "R.I.Pienaar",
+ :license => "GPLv2",
+ :version => "1.1",
+ :url => "http://projects.puppetlabs.com/projects/mcollective-plugins/wiki",
+ :timeout => 60
+{% endhighlight %}
+
+It's fairly obvious what these all do, *:timeout* is how long the MCollective daemon will let the threads run.
+
+## Required versions
+As of MCollective 2.1.2 you can indicate which is the lowest version of MCollective needed to use a plugin. Plugins that do not meet the requirement can not be used.
+
+{% highlight ruby linenos %}
+requires :mcollective => "2.0.0"
+{% endhighlight %}
+
+You should add this right after the metadata section in the DDL
+
+## Actions, Input and Output
+Defining inputs and outputs is the hardest part, below first the *status* action:
+
+{% highlight ruby linenos %}
+action "status", :description => "Gets the status of a service" do
+ display :always # supported in 0.4.7 and newer only
+
+ input :service,
+ :prompt => "Service Name",
+ :description => "The service to get the status for",
+ :type => :string,
+ :validation => '^[a-zA-Z\-_\d]+$',
+ :optional => false,
+ :default => "mcollective",
+ :maxlength => 30
+
+ output :status,
+ :description => "The status of service",
+ :display_as => "Service Status",
+ :default => "unknown status"
+end
+{% endhighlight %}
+
+As you see we can define all the major components of input and output parameters. *:type* can be one of various values and each will have different parameters, more on that later.
+
+As of version 2.1.1 the outputs can define a default value. For agents the reply structures are pre-populated with all the defined outputs, if no default is supplied a default of nil will be set.
+
+As of version 2.3.1 the inputs can also define default values, this is only processed and applied for non optional inputs.
+
+By default mcollective only show data from actions that failed, the *display* line above tells it to always show the results. Possible values are *:ok*, *:failed* (the default behavior) and *:always*.
+
+Finally the service agent has 3 almost identical actions - *start*, *stop* and *restart* - below we use a simple loop to define them all in one go.
+
+{% highlight ruby linenos %}
+["start", "stop", "restart"].each do |act|
+ action act, :description => "#{act.capitalize} a service" do
+ input :service,
+ :prompt => "Service Name",
+ :description => "The service to #{act}",
+ :type => :string,
+ :validation => '^[a-zA-Z\-_\d]+$',
+ :optional => false,
+ :maxlength => 30
+
+ output :status,
+ :description => "The status of service after #{act}",
+ :display_as => "Service Status",
+ :default => "unknown status"
+ end
+end
+{% endhighlight %}
+
+All of this code just goes into a file, no special class or module bits needed, just save it as *service.ddl* in the same location as the *service.rb*.
+
+Importantly you do not need to have the *service.rb* on a machine to use the DDL, this means on machines that are just used for running client programs you can just drop the *.ddl* files into the agents directory.
+
+You can view a human readable version of this using *mco plugin doc <agent>* command:
+
+{% highlight console %}
+% mco plugin doc service
+SimpleRPC Service Agent
+=======================
+
+Agent to manage services using the Puppet service provider
+
+ Author: R.I.Pienaar
+ Version: 1.1
+ License: GPLv2
+ Timeout: 60
+ Home Page: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki
+
+
+
+ACTIONS:
+========
+ restart, start, status, stop
+
+ restart action:
+ ---------------
+ Restart a service
+
+ INPUT:
+ service:
+ Description: The service to restart
+ Prompt: Service Name
+ Type: string
+ Validation: ^[a-zA-Z\-_\d]+$
+ Length: 30
+
+
+ OUTPUT:
+ status:
+ Description: The status of service after restart
+ Display As: Service Status
+{% endhighlight %}
+
+### Optional Inputs
+The input block has a mandatory *:optional* field, when true it would be ok if a client attempts to call the agent without this input supplied. If it is supplied though it will be validated.
+
+### Types of Input
+As you see above the input block has *:type* option, types can be *:string*, *:list*, *:boolean*, *:integer*, *:float* or *:number*
+
+#### :string type
+The string type validates initially that the input is infact a String, then it validates the length of the input and finally matches the supplied Regular Expression.
+
+Both *:validation* and *:maxlength* are required arguments for the string type of input.
+
+If you want to allow unlimited length text you can make *:maxlength => 0* but use this with care.
+
+As of version 2.2.0 a new plugin type called [Validator Plugins][ValidatorPlugins] exist that allow you to supply your own validations for *:string* types.
+
+#### :list type
+List types provide a list of valid options and only those will be allowed, see an example below:
+
+{% highlight ruby linenos %}
+input :action,
+ :prompt => "Service Action",
+ :description => "The action to perform",
+ :type => :list,
+ :optional => false,
+ :list => ["stop", "start", "restart"]
+{% endhighlight %}
+
+In user interfaces this might be displayed as a drop down list selector or another kind of menu.
+
+#### :boolean type
+
+The value input should be either _true_ or _false_ actual boolean values. This feature was introduced in version _0.4.9_.
+
+#### :integer type
+
+The value input should be an integer number like _1_ or _100_ but not _1.1_. This feature was introduced in version _1.3.2_
+
+#### :float type
+
+The value input should be a floating point number like _1.0_ but not _1_. This feature was introduced in version _1.3.2_
+
+#### :number type
+
+The value input should be an integer or a floating point number. This feature was introduced in version _1.3.2_
+
+#### :any type
+
+The value input can be any type, this allows you to send rich objects like arrays of hashes around, it effectively disables validation of the type of input.
+
+The :any type is deprecated and will be removed after version 2.2.x.
+
+### Accessing the DDL
+While programming client applications or web apps you can gain access to the DDL for any agent in several ways:
+
+{% highlight ruby linenos %}
+require 'mcollective'
+
+config = MCollective::Config.instance
+config.loadconfig(options[:config])
+
+ddl = MCollective::DDL.new("service")
+puts ddl.help("#{config.configdir}/rpc-help.erb")
+{% endhighlight %}
+
+This will produce the text help output from the above example, you can supply any ERB template to format the output however you want.
+
+You can also access the data structures directly:
+
+{% highlight ruby linenos %}
+ddl = MCollective::DDL.new("service")
+puts "Meta Data:"
+pp ddl.meta
+
+puts
+puts "Status Action:"
+pp ddl.action_interface("status")
+{% endhighlight %}
+
+{% highlight ruby linenos %}
+Meta Data:
+{:license=>"GPLv2",
+ :author=>"R.I.Pienaar",
+ :name=>"SimpleRPC Service Agent",
+ :timeout=>60,
+ :version=>"1.1",
+ :url=>"http://projects.puppetlabs.com/projects/mcollective-plugins/wiki",
+ :description=>"Agent to manage services using the Puppet service provider"}
+
+Status Action:
+{:action=>"status",
+ :input=>
+ {:service=>
+ {:validation=>"^[a-zA-Z\\-_\\d]+$",
+ :maxlength=>30,
+ :prompt=>"Service Name",
+ :type=>:string,
+ :optional=>false,
+ :description=>"The service to get the status for"}},
+ :output=>
+ {"status"=>
+ {:display_as=>"Service Status", :description=>"The status of service"}},
+ :description=>"Gets the status of a service"}
+
+{% endhighlight %}
+
+The ddl object is also available on any *rpcclient*:
+
+{% highlight ruby linenos %}
+service = rpcclient("service")
+pp service.ddl.meta
+{% endhighlight %}
+
+In the case of accessing it through the service as in this example, if there was no DDL file on the machine for the service agent you'd get a *nil* back from the ddl accessor.
+
+### Input Validation
+As mentioned earlier the client does automatic input validation using the DDL, if validation fails you will get an *MCollective::DDLValidationError* exception thrown with an appropriate message.
--- /dev/null
+---
+layout: default
+title: Discovery Plugins
+---
+[DDL]: /mcollective/reference/plugins/ddl.html
+
+## Overview
+Up to MCollective 2.0.0 the discovery system could only discover against
+the network by doing broadcasts over the middleware.
+
+The _direct addressing_ capability introduced in 2.0.0 enables users to
+communicate with a node without doing broadcast if they know the
+configured identity of that node.
+
+In version 2.1.0 we are introducing a new kind of plugin that works on
+the client system to do discovery against any data source that can
+return a list of identities such as flatfiles or databases.
+
+## Configuring and using discovery plugins
+Your mcollective client has a setting called *default_discovery_method*
+that defaults to *mc*, if you change this in your _client.cfg_ to
+another known plugin you can use that instead.
+
+To get a list of known discovery plugins use the _mco plugin_
+application:
+
+{% highlight console %}
+% mco plugin doc
+Please specify a plugin. Available plugins are:
+
+Discovery Methods:
+ flatfile Flatfile based discovery for node identities
+ mc MCollective Broadcast based discovery
+ mongo MongoDB based discovery for databases built using registration
+{% endhighlight %}
+
+Each plugin can have a different set of capabilities, for example a
+flatfile with only hostnames cannot do class or fact based filters and
+you will receive an error if you tried to do so. You can see the
+capabilities of each plugin using the _mco plugin_ application:
+
+{% highlight console %}
+$ mco plugin doc flatfile
+flatfile
+========
+
+Flatfile based discovery for node identities
+
+ Author: R.I.Pienaar <rip@devco.net>
+ Version: 0.1
+ License: ASL 2.0
+ Timeout: 0
+ Home Page: http://marionette-collective.org/
+
+DISCOVERY METHOD CAPABILITIES:
+ Filter based on mcollective identity
+{% endhighlight %}
+
+Here you can see the only capability that this plugin has is to filter
+against identities.
+
+These plugins require DDL files to be written and distributed when
+installing each plugin.
+
+When using the mcollective CLI you can choose which plugin to use per
+request, some plugins require arguments like the file to discover
+against:
+
+{% highlight console %}
+$ mco rpc rpcutil ping --dm flatfile --do /some/text/file
+{% endhighlight %}
+
+In the case of the flatfile plugin there is a convenient shortcut
+available on all client applications that has the same effect as above:
+
+{% highlight console %}
+$ mco rpc rpcutil ping --nodes /some/text/file
+{% endhighlight %}
+
+Any request that uses the compound filters using *-S* will be forced to
+use the network broadcast discovery method.
+
+## Writing a discovery plugin
+Writing your own discovery plugin is very simple, you need to provide
+one method that returns an array of node names.
+
+The plugins only need to be present on the client machines but no harm
+in installing them on all machines. They need to be installed into the
+*discovery* directory in the usual plugin directory. You can use the
+*mco plugin package* command to create RPM or DEB packages for these
+plugins.
+
+{% highlight ruby linenos %}
+module MCollective
+ class Discovery
+ class Flatfile
+ def self.discover(filter, timeout, limit=0, client=nil)
+ unless client.options[:discovery_options].empty?
+ file = client.options[:discovery_options].first
+ else
+ raise "The flatfile discovery method needs a path to a text file"
+ end
+
+ raise "Cannot read the file %s specified as discovery source" % file unless File.readable?(file)
+
+ discovered = []
+
+ hosts = File.readlines(file).map{|l| l.chomp}
+
+ unless filter["identity"].empty?
+ filter["identity"].each do |identity|
+ identity = Regexp.new(identity.gsub("\/", "")) if identity.match("^/")
+
+ if identity.is_a?(Regexp)
+ discovered = hosts.grep(identity)
+ elsif hosts.include?(identity)
+ discovered << identity
+ end
+ end
+ else
+ discovered = hosts
+ end
+
+ discovered
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+This is the *flatfile* plugin that is included in the distribution. You
+can see it using the *client.options\[:discovery_options\]* array to get
+access to the file supplied using the *--do* command line argument,
+reading that file and doing either string or Regular Expression matching
+against it finally returning the list of nodes.
+
+As mentioned each plugin needs a DDL, the DDL for this plugin is very
+simple:
+
+{% highlight ruby linenos %}
+metadata :name => "flatfile",
+ :description => "Flatfile based discovery for node identities",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :license => "ASL 2.0",
+ :version => "0.1",
+ :url => "http://marionette-collective.org/",
+ :timeout => 0
+
+discovery do
+ capabilities :identity
+end
+{% endhighlight %}
+
+Here we expose just the one capability, valid capabilities would be
+*:classes*, *:facts*, *:identity*, *:agents* and *:compound*. In
+practise you cannot create a plugin that supports the *:compound*
+capability as mcollective will force the use of the *mc* plugin if you
+use those.
--- /dev/null
+---
+layout: default
+title: Writing Fact Plugins
+toc: false
+---
+[SimpleRPCAuthorization]: /mcollective/simplerpc/authorization.html
+[Registration]: registration.html
+
+Fact plugins are used during discovery whenever you run the agent with queries like *-W country=de*.
+
+The default setup uses a YAML file typically stored in */etc/mcollective/facts.yaml* to read facts. There are however many fact systems like Reductive Labs Facter and Opscode Ohai or you can come up with your own. The facts plugin type lets you write code to access these tools.
+
+Facts at the moment should be simple *variable = value* style flat Hashes, if you have a hierarchical fact system like Ohai you can flatten them into *var.subvar = value* style variables.
+
+## Details
+Implementing a facts plugin is made simple by inheriting from *MCollective::Facts::Base*, in that case you just need to provide 1 method, the YAML plugin code can be seen below:
+
+For releases in the 1.0.x release cycle and older, use this plugin format:
+
+{% highlight ruby linenos %}
+module MCollective
+ module Facts
+ require 'yaml'
+
+ # A factsource that reads a hash of facts from a YAML file
+ class Yaml<Base
+ def self.get_facts
+ config = MCollective::Config.instance
+
+ facts = {}
+
+ YAML.load_file(config.pluginconf["yaml"]).each_pair do |k, v|
+ facts[k] = v.to_s
+ end
+
+ facts
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+For releases 1.1.x and onward use this format:
+
+{% highlight ruby linenos %}
+module MCollective
+ module Facts
+ require 'yaml'
+
+ class Yaml_facts<Base
+ def load_facts_from_source
+ config = MCollective::Config.instance
+
+ facts = {}
+
+ YAML.load_file(config.pluginconf["yaml"]).each_pair do |k, v|
+ facts[k] = v.to_s
+ end
+
+ facts
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+If using the newer format in newer releases of mcollective you do not need to worry about caching or
+thread safety as the base class does this for you. You can force reloading of facts by creating a
+*force_reload?* method that should return *true* or *false*. Returning *true* will force the cache
+to be rebuilt.
+
+You can see that all you have to do is provide *self.get_facts* which should return a Hash as described above.
+
+There's a sample using Puppet Labs Facter on the plugins project if you wish to see an example that queries an external fact source.
+
+Once you've written your plugin you can save it in the plugins directory and configure mcollective to use it:
+
+{% highlight ini %}
+factsource = yaml
+{% endhighlight %}
+
+This will result in *MCollective::Facts::Yaml* or *MCollective::Facts::Yaml_facts* being used as source for your facts.
--- /dev/null
+---
+layout: default
+title: Registration
+toc: false
+---
+[RegistrationMonitor]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AgentRegistrationMonitor
+
+MCollective supports the ability for each node to register with a central inventory. The core functionality
+of Mcollective doesn't require registration internally; it's simply provided as a framework to enable you to
+build inventory systems or Web UIs.
+
+## Details
+
+Registration plugins are easy to write. You can configure your nodes to use your own or the provided one.
+
+The one we provide simply sends a list of agents to the inventory. It's available in the plugins directory
+under *registration/agentlist.rb* and can be seen in its entirety below:
+
+{% highlight ruby %}
+module MCollective
+ module Registration
+ # A registration plugin that simply sends in the list of agents we have
+ class Agentlist<Base
+ def body
+ MCollective::Agents.agentlist
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+You can see it's very simple, you just need to subclass *MCollective::Registration::Base* to ensure they get
+loaded into the plugin system and provide a _body_ method whose return value will be sent to the registration agent(s).
+
+The registration plugin can decide if a message should be sent or not. If your plugin responds with a _nil_ value the
+message will not be sent. This can be useful if you wish to only send registration data when some condition has changed.
+On large collectives, registration messages can be quite high-volume. It's worthwhile to sample the size of your
+registration messages and multiply by the number of nodes to determine an appropriate frequency to send them.
+
+To configure it to be used you just need the following in your config:
+
+{% highlight ini %}
+registerinterval = 300
+registration = Agentlist
+
+# only valid since 1.3.0 and newer
+registration_collective = development
+{% endhighlight %}
+
+This will cause the plugin to be called every 300 seconds to the development collective, if you do not configure
+a target collective explicitely it will target the main collective for the given node.
+
+We do not provide the receiving end of this in the core mcollective. You will need to write an agent called
+*registration* and do something useful with the data you receive from all the nodes. You can find
+[a simple monitoring system][RegistrationMonitor] built using this method as an example. The receiving agent
+can simply be installed as an extra mcollectived plugin on a node which participates in the collective.
+
+You need to note a few things about the receiving agents:
+
+ * They need to be fast. You'll receive a lot of registration messages so if your agent talks to a database that
+ is slow you'll run into problems
+ * They should not return anything other than *nil*. The MCollective server will interpret *nil* from an agent as
+ an indication that you do not want to send back any reply. Replying to registration requests is almost always undesired.
+
+There's nothing preventing you from running more than one type of receiving agent in your collective, you can have one
+on your monitor server as above and another with completely different code on a web server feeding a local
+cache for your web interfaces. As long as both agents are called *registration* you'll be fine. However this
+does mean you can't run more than one registration receiver on the same server.
--- /dev/null
+---
+layout: default
+title: The rpcutil Agent
+toc: false
+---
+
+We include an agent with a few utilities and helpers there to assist in retrieving information about the running mcollectived.
+
+We aim to add the ability to initiate reloads and so forth in this agent too in future, this
+will require further internal refactoring though.
+
+## _inventory_ Action
+
+Retrieves an inventory of the facts, classes and agents and plugins on the system, takes no arguments
+and returns a hash like this:
+
+{% highlight ruby %}
+{:agents => ["rpcutil", "discovery"],
+ :facts => {"mcollective"=>1},
+ :classes => ["common::linux", "motd"],
+ :data_plugins=>["sysctl_data", "fstat_data"],
+ :collectives=>["mcollective"],
+ :main_collective=>"mcollective",
+ :version=>"2.0.0"}
+{% endhighlight %}
+
+## _daemon`_`stats_ Action
+
+Retrieves statistics about the running daemon, how many messages it's handled, passed, dropped etc.
+
+See the DDL for the agent for a full reference
+
+{% highlight ruby %}
+{:configfile=>"/etc/mcollective/server.cfg",
+ :validated=>46,
+ :threads=> ["#<Thread:0xb7dcf480 sleep>",
+ "#<Thread:0xb7fba704 sleep>",
+ "#<Thread:0xb7dcfb88 run>"],
+ :starttime=>1284305683,
+ :agents=>["rpcutil", "discovery"],
+ :unvalidated=>0,
+ :pid=>15499,
+ :times=>{:cutime=>0.0, :utime=>0.15, :cstime=>0.0, :stime=>0.02},
+ :passed=>46,
+ :total=>46,
+ :filtered=>0,
+ :replies=>45}
+{% endhighlight %}
+
+Replies will always be less than received since the current message has not been sent yet when the stats are gathered.
+
+## _get`_`fact_ Action
+
+Retrieves a single fact from the server
+
+{% highlight ruby %}
+{:fact => "mcollective",
+ :value => 1}
+{% endhighlight %}
+
+## _agent`_`inventory_ Action
+
+Returns a list of all agents with their meta data like version, author, license etc
+
+{% highlight ruby %}
+{:agents=> [
+ {:agent=>"discovery",
+ :license=>"Apache License, Version 2",
+ :author=>"R.I.Pienaar <rip@devco.net>"},
+
+ {:agent=>"rpcutil",
+ :license=>"Apache License, Version 2.0",
+ :name=>"Utilities and Helpers for SimpleRPC Agents",
+ :url=>"http://marionette-collective.org/",
+ :description=> "General helpful actions that expose stats and internals to SimpleRPC clients",
+ :version=>"1.0",
+ :author=>"R.I.Pienaar <rip@devco.net>",
+ :timeout=>3}
+ ]
+}
+{% endhighlight %}
+
+## _get`_`config`_`item_ Action
+
+Retrieves the active value for any configuration item on a server
+
+{% highlight ruby %}
+{:item => "loglevel",
+ :value => "debug"}
+{% endhighlight %}
+
+## _ping_ Action
+
+A simple lightweight ping action that just returns each nodes local time
+
+{% highlight ruby %}
+{:pong => 1295471526}
+{% endhighlight %}
+
+## _collective`_`info_ Action
+
+Retrieves the main and sub collectives configured
+
+For a server configured with:
+
+{% highlight ruby %}
+collectives = mcollectivedev,subdev1
+main_collective = mcollectivedev
+{% endhighlight %}
+
+The following structure gets returned:
+
+{% highlight ruby %}
+{:collectives=>["mcollectivedev", "subdev1"],
+ :main_collective=>"mcollectivedev"}
+{% endhighlight %}
--- /dev/null
+---
+layout: default
+title: OpenSSL based Security Plugin
+---
+[SimpleRPCAuthorization]: /mcollective/simplerpc/authorization.html
+[Registration]: registration.html
+[SSLSecurity]: security_ssl.html
+[SecurityOverview]: ../../security.html
+
+## Overview
+This plugin impliments a AES encryption and RSA public / private key based security system
+for The Marionette Collective.
+
+Please review the [Security Overview][SecurityOverview] for a general discussion about security in Marionette Collective.
+
+The design goals of this plugin are:
+
+ * Each actor - clients and servers - can have their own set of public and private keys
+ * All actors are uniquely and cryptographically identified
+ * Requests are encrypted using the clients private key and anyone that has
+ the public key can see the request. Thus an atacker may see the requests
+ given access to a public key of the requester.
+ * Request TTLs and Message Times are cryptographically secured and tampered
+ messages are not accepted by default. This is a first line defence in message
+ replaying and tampering.
+ * Replies are encrypted using the calling clients public key. Thus no-one but
+ the caller can view the contents of replies.
+ * Servers can all have their own SSL keys, or share one, or reuse keys created
+ by other PKI using software like Puppet
+ * Requests from servers - like registration data - can be secured even to external
+ eavesdropping depending on the level of configuration you are prepared to do
+ * Given a network where you can ensure third parties are not able to access the
+ middleware public key distribution can happen automatically
+
+During the design of this system we considered the replies back to clients to be most
+important piece of information on the network. This is secured in a way that only
+the client can decrypt the replies he gets. An attacker will need to gain access
+to every private key of every client to see all the reply data.
+
+Serialization uses Marshal or YAML, which means data types in and out of mcollective
+will be preserved from client to server and reverse.
+
+## Compared to the SSL plugin
+
+The earlier [SSLSecurity] only provided message signing and identification of clients, this
+new plugin builds on this adding payload encryption, identification of servers and optional
+automatic key distribution.
+
+The [SSLSecurity] plugin puts less drain on resources, if you do not specifically need encryption
+you should consider using that one instead.
+
+## Deployment Scenarios
+
+There are various modes of deployment, the more secure you wish to be the more work you have
+to do in terms of key exchange and initial setup.
+
+This plugin is designed to allow you to strike a balance between security and setup cost
+the sections below discuss the possible deployment scenarios to help you choose an approach.
+
+In all of the below setups the following components are needed:
+
+ * Each user making requests - the client - needs a public and private key pair
+ * Each server receiving requests need the public key of each client
+
+In cases where you wish to use [Registration] or initiate requests from the server for any
+reason the following are needed:
+
+ * Each server needs a public and private key pair
+ * Each other server that wish to receive these requests need the public key of the sending server
+
+In this scenario each server will act as a client making RPC requests to the collective network
+for any agent called _registration_. So in this scenario the server acts as a client and therefore
+need a key-pair to identify it.
+
+### Manual key distribution with each server and client having unique keys
+
+In this setup each client and each server needs a unique set of keys. You need to
+distribute these keys manually and securely - perhaps using Puppet.
+
+The setup cost is great, to enable registration the nodes receiving registration data
+need to have the public key of every other node stored locally before registration
+data can be received.
+
+If you do not use Puppet or some other PKI system that provide access to keys you need
+create keypairs for each node and client.
+
+This is the most secure setup protecting all replies and registration data. Rogue people
+on the network who do not compromise a running host cannot make requests on the network.
+
+Attackers who compromise a server can only make registration requests assuming you deployed
+a strictly configured [Authorization][SimpleRPCAuthorization] system they cannot use those
+machines as starting points to inject requests for the rest of your network.
+
+By gaining access to your Middleware an attacker will not be able to observe the contents of
+requests, replies or registration messages. Attackers need to compromise servers and gain
+access to private keys before they can start observing parts of the exchange.
+
+|Feature / Capability|Supported|
+|--------------------|---------|
+|Clients are uniquely identified using cryptographic means|yes|
+|Anyone with the client public key can observe request contents|yes|
+|Attackers can gain access to the client public key by just listening on the network|no|
+|Replies back to the client are readable only by client that initiated the request|yes|
+|Attackers can create new certificates and start using them to make requests as clients|no|
+|Servers are uniquely identified using cryptographic means|yes|
+|Anyone with the server public key can observe registration contents|yes|
+|Attackers can gain access to the server public keys by just listening on the network|no|
+|Registration data can be protected from rogue agents posing as registration agents|yes|
+|Attackers can create new nodes and inject registration data for those new nodes|no|
+
+To configure this scenario use the following options and manually copy public keys to the
+_plugin.aes.client`_`cert`_`dir_ directory:
+
+|Settings|Value|Descritpion|
+|--------|-----|-----------|
+|plugin.aes.send_pubkey|0|Do not send public keys|
+|plugin.aes.learn_pubkeys|0|Do not learn public keys|
+
+### Automatic public key distribution with each server and client having unique keys
+
+Here we enable the _plugin.aes.learn`_`pubkeys_ feature on all servers. Your public keys
+will now be distributed automatically on demand but you loose some security in that anyone
+with access to your network or middleware can observe the contents of replies and registration
+data
+
+You still need to create keys for every node - or use Puppets. You still need to create keys
+for every user.
+
+In order to protect against attackers creating new certificates and making requests on your network
+deploy a [Authorization][SimpleRPCAuthorization] plugin that denies unknown clients.
+
+|Feature / Capability|Supported|
+|--------------------|---------|
+|Clients are uniquely identified using cryptographic means|yes|
+|Anyone with the client public key can observe request contents|yes|
+|Attackers can gain access to the client public key by just listening on the network|*yes*|
+|Replies back to the client are readable only by client that initiated the request|yes|
+|Attackers can create new certificates and start using them to make requests as clients|*yes*|
+|Servers are uniquely identified using cryptographic means|yes|
+|Anyone with the server public key can observe registration contents|yes|
+|Attackers can gain access to the server public keys by just listening on the network|*yes*|
+|Registration data can be protected from rogue agents posing as registration agents|yes|
+|Attackers can create new nodes and inject registration data for those new nodes|*yes*|
+
+To configure this scenario use the following options and ensure the _mcollectived_ can write
+to the _plugin.aes.client`_`cert`_`dir_ directory:
+
+|Settings|Value|Descritpion|
+|--------|-----|-----------|
+|plugin.aes.send_pubkey|1|Send public keys|
+|plugin.aes.learn_pubkeys|1|Learn public keys|
+
+### Manual public key distribution with servers sharing a key pair and clients having unique keys
+
+This is comparable to the older SSL plugin where all servers shared the same public / private
+pair. Here anyone who is part of the network can decrypt the traffic related to registration
+but replies to clients are still securely encrypted and visable only to them.
+
+You will not need to create unique keys for every server, you can simply copy the same one out
+everywhere. You still need to create keys for every user.
+
+If you do not use registration, this is a very secure setup that requires a small configuration
+overhead.
+
+|Feature / Capability|Supported|
+|--------------------|---------|
+|Clients are uniquely identified using cryptographic means|yes|
+|Anyone with the client public key can observe request contents|yes|
+|Attackers can gain access to the client public key by just listening on the network|no|
+|Replies back to the client are readable only by client that initiated the request|yes|
+|Attackers can create new certificates and start using them to make requests as clients|no|
+|Servers are uniquely identified using cryptographic means|*no*|
+|Anyone with the server public key can observe registration contents|yes|
+|Attackers can gain access to the server public keys by just listening on the network|no|
+|Registration data can be protected from rogue agents posing as registration agents|*no*|
+
+To configure this scenario use the following options and ensure the _mcollectived_ can write
+to the _plugin.aes.client`_`cert`_`dir_ directory:
+
+|Settings|Value|Descritpion|
+|--------|-----|-----------|
+|plugin.aes.send_pubkey|0|Do not send public keys|
+|plugin.aes.learn_pubkeys|0|Do not learn public keys|
+
+### Automatic public key distribution with servers sharing a key and client having unique keys
+
+This is comparable to the older SSL plugin where all servers shared the same public / private
+pair. Here anyone who is part of the network can decrypt the traffic related to registration
+but replies to clients are still securely encrypted and visable only to them.
+
+Here we enable the _plugin.aes.learn`_`pubkeys_ feature on all servers. Your public keys
+will now be distributed automatically on demand but you loose some security in that anyone
+with access to your network or middleware can observe the contents of replies and registration
+data
+
+You will not need to create unique keys for every server, you can simply copy the same one out
+everywhere. You still need to create keys for every user.
+
+In order to protect against attackers creating new certificates and making requests on your network
+deploy a [Authorization][SimpleRPCAuthorization] plugin that denies unknown clients.
+
+|Feature / Capability|Supported|
+|--------------------|---------|
+|Clients are uniquely identified using cryptographic means|yes|
+|Anyone with the client public key can observe request contents|yes|
+|Attackers can gain access to the client public key by just listening on the network|*yes*|
+|Replies back to the client are readable only by client that initiated the request|yes|
+|Attackers can create new certificates and start using them to make requests as clients|*yes*|
+|Servers are uniquely identified using cryptographic means|*no*|
+|Anyone with the server public key can observe registration contents|yes|
+|Attackers can gain access to the server public keys by just listening on the network|*yes*|
+|Registration data can be protected from rogue agents posing as registration agents|*no*|
+
+To configure this scenario use the following options and ensure the _mcollectived_ can write
+to the _plugin.aes.client`_`cert`_`dir_ directory:
+
+|Settings|Value|Descritpion|
+|--------|-----|-----------|
+|plugin.aes.send_pubkey|1|Send public keys|
+|plugin.aes.learn_pubkeys|1|Learn public keys|
+
+## Creating keys
+
+Keys are created using OpenSSL. The filenames of public keys are significant you should name
+them so that they are unique for your network and they should match on the client and servers.
+
+{% highlight console %}
+ % openssl genrsa -out server-private.pem 1024
+ % openssl rsa -in server-private.pem -out server-public.pem -outform PEM -pubout
+{% endhighlight %}
+
+Client and Server keys are made using the same basic method.
+
+## Reusing Puppet keys
+
+Puppet managed nodes will all have keys created by Puppet already, you can reuse these if your
+_mcollectived_ runs as root.
+
+Generally Puppet stores these in _/var/lib/puppet/ssl/private`_`keys/fqdn.pem_ and _/var/lib/puppet/ssl/public`_`keys/fqdn.pem_
+simply configure these paths for your _server`_`private_ and _server`_`public_ options.
+
+Clients will still need their own keys made and distributed.
+
+## Future Roadmap
+
+ * Depending on performance of the initial system we might validate request certificates are
+ signed by a known CA this will provide an additional level of security preventing attackers
+ from creating their own keys and using those on the network without also compromising the CA.
+ * Private keys will be secured with a password
+
+
+## Configuration Options
+
+### Common Options
+
+|Setting|Example / Default|Description
+|-------|-----------------|-----------|
+|securityprovider|aes_security|Enables this security provider|
+|plugin.aes.serializer|yaml or marshal|Serialization to use|
+|plugin.aes.send_pubkey|0 or 1|Send the public key with every request|
+|plugin.aes.learn_pubkeys|0 or 1|Receive public keys from the network and cache them locally|
+
+### Client Options
+
+|Setting|Example / Default|Description
+|-------|-----------------|-----------|
+|plugin.aes.client_private|/home/user/.mcollective.d/user-private.pem|The private key path for the user. File must be /\w\.\-/|
+|plugin.aes.client_public|/home/user/.mcollective.d/user.pem|The public key path for the user. File must be /\w\.\-/|
+
+### Server Options
+
+|Setting|Example / Default|Description
+|-------|-----------------|-----------|
+|plugin.aes.client_cert_dir|/etc/mcollective/ssl/clients|Where to store and load client public keys|
+|plugin.aes.server_private|/etc/mcollective/ssl/server-private.pem|Server private key. File must be /\w\.\-/|
+|plugin.aes.server_public|/etc/mcollective/ssl/server-public.pem|Server public key. File must be /\w\.\-/|
+|plugin.aes.enforce_ttl|1|Enforce TTL and Message time security, warn only when disabled. 1.3.2 and newer only|
+
--- /dev/null
+---
+layout: default
+title: OpenSSL based Security Plugin
+---
+[SimpleRPCAuthorization]: /mcollective/simplerpc/authorization.html
+[Registration]: registration.html
+[AESPlugin]: security_aes.html
+[SecurityOverview]: ../../security.html
+
+## Overview
+Implements a public/private key based message validation system using SSL
+public and private keys.
+
+Please review the [Security Overview][SecurityOverview] for a general discussion about security in Marionette Collective.
+
+The design goal of the plugin are two fold:
+
+ * give different security credentials to clients and servers to avoid a compromised server from sending new client requests.
+ * create a token that uniquely identify the client - based on the filename of the public key. This creates a strong identity token for [SimpleRPCAuthorization].
+ * As of 1.3.2 it cryptographically protect the TTL and Message Time properties of requests. Aiding in securing against replay atacks.
+
+Serialization uses Marshal or YAML, which means data types in and out of mcollective
+will be preserved from client to server and reverse.
+
+Validation is as default and is provided by *MCollective::Security::Base*
+
+Initial code was contributed by Vladimir Vuksan and modified by R.I.Pienaar
+
+An [alternative plugin][AESPlugin] exist that encrypts data but is more work to setup and maintain.
+
+## Setup
+
+### Nodes
+To setup you need to create a SSL key pair that is shared by all nodes.
+
+The certificate names must match /\w\.\-/.
+
+{% highlight console %}
+ % openssl genrsa -out server-private.pem 1024
+ % openssl rsa -in server-private.pem -out server-public.pem -outform PEM -pubout
+{% endhighlight %}
+
+Distribute the private and public file to */etc/mcollective/ssl* on all the nodes.
+Distribute the public file to */etc/mcollective/ssl* everywhere the client code runs.
+
+server.cfg:
+
+{% highlight ini %}
+ securityprovider = ssl
+ plugin.ssl_server_private = /etc/mcollective/ssl/server-private.pem
+ plugin.ssl_server_public = /etc/mcollective/ssl/server-public.pem
+ plugin.ssl_client_cert_dir = /etc/mcollective/ssl/clients/
+{% endhighlight %}
+
+TTL and Message Times are protected by default 2.0.0, this means older clients will not be able to
+communicate with servers running this version of the security plugin. You can make it warn but not
+deny older clients:
+
+{% highlight ini %}
+ plugin.ssl.enforce_ttl = 0
+{% endhighlight %}
+
+### Users and Clients
+Now you should create a key pair for every one of your clients, here we create one
+for user john - you could also if you are less concerned with client id create one
+pair and share it with all clients:
+
+The certificate names must match /\w\.\-/.
+
+{% highlight console %}
+ % openssl genrsa -out john-private.pem 1024
+ % openssl rsa -in john-private.pem -out john-public.pem -outform PEM -pubout
+{% endhighlight %}
+
+Each user has a unique userid, this is based on the name of the public key.
+In this example case the userid would be *'john-public'*.
+
+Store these somewhere like:
+
+{% highlight console %}
+ /home/john/.mc/john-private.pem
+ /home/john/.mc/john-public.pem
+{% endhighlight %}
+
+Every users public key needs to be distributed to all the nodes, save the john one
+in a file called:
+
+{% highlight console %}
+ /etc/mcollective/ssl/clients/john-public.pem
+{% endhighlight %}
+
+If you wish to use [Registration] or auditing that sends connections over MC to a
+central host you will need also put the *server-public.pem* in the clients directory.
+
+You should be aware if you do add the node public key to the clients dir you will in
+effect be weakening your overall security. You should consider doing this only if
+you also set up an Authorization method that limits the requests the nodes can make.
+
+client.cfg:
+
+{% highlight ini %}
+ securityprovider = ssl
+ plugin.ssl_server_public = /etc/mcollective/ssl/server-public.pem
+ plugin.ssl_client_private = /home/john/.mc/john-private.pem
+ plugin.ssl_client_public = /home/john/.mc/john-public.pem
+{% endhighlight %}
+
+If you have many clients per machine and dont want to configure the main config file
+with the public/private keys you can set the following environment variables:
+
+{% highlight console %}
+ export MCOLLECTIVE_SSL_PRIVATE=/home/john/.mc/john-private.pem
+ export MCOLLECTIVE_SSL_PUBLIC=/home/john/.mc/john-public.pem
+{% endhighlight %}
+
+### Serialization Method
+
+You can choose either YAML or Marshal, the default is Marshal. The view with optional Marshal encoding is to have a serializer supported by other languages other than Ruby to enable future integration with those.
+
+To use YAML set this in both *client.cfg* and *server.cfg*:
+
+{% highlight ini %}
+plugin.ssl_serializer = yaml
+{% endhighlight %}
--- /dev/null
+---
+layout: default
+title: Validator Plugins
+---
+[DDL]: /mcollective/reference/plugins/ddl.html
+
+## Overview
+MCollective provides extensive input data validation to prevent attacks and
+injections into your agents preventing attack vectors like Shell Injection
+Attacks.
+
+Traditionally we shipped a number of pre-made validator plugins that could be
+used in agents and DDL files but you were not capable fo adding your own easily.
+
+As of version 2.2.0 you can write new Validator plugins that allow you to extend
+the DDL and Agent validation methods.
+
+## Writing A New Validator
+We'll write a new validator plugin that can validate a string matches valid Exim
+message IDs like *1Svk5S-0001AW-I5*.
+
+Validator plugins and their DDL files goes in the libdir in the *validator*
+directory on both the servers and the clients.
+
+### The Ruby Plugin
+The basic validator plugin that will validate any data against this regular
+expression can be seen here:
+
+{% highlight ruby %}
+module MCollective
+ module Validator
+ class Exim_msgidValidator
+ def self.validate(msgid)
+ Validator.typecheck(msgid, :string)
+
+ raise "Not a valid Exim Message ID" unless msgid.match(/(?:[+-]\d{4} )?(?:\[\d+\] )?(\w{6}\-\w{6}\-\w{2})/)
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+All you need to do is provide a *self.validate* method that takes 1 argument and
+do whatever validation you want to do against the input data.
+
+Here we first confirm it is a string and then we do the regular expression match
+against that. Any Exception that gets raised will result in validation failing.
+
+### The DDL
+As with other plugins these plugins need a DDL file, all they support is the
+metadata section.
+
+{% highlight ruby %}
+metadata :name => "Exim Message ID",
+ :description => "Validates that a string is a Exim Message ID",
+ :author => "R.I.Pienaar <rip@devco.net>",
+ :license => "ASL 2.0",
+ :version => "1.0",
+ :url => "http://devco.net/",
+ :timeout => 1
+{% endhighlight %}
+
+## Using the Validator in a DDL
+You can use the validator in any DDL file, here is a snippet matching an input
+using the new *exim_msgid* validator:
+
+{% highlight ruby %}
+action "retrymsg", :description => "Retries a specific message" do
+ display :ok
+
+ input :msgid,
+ :prompt => "Message ID",
+ :description => "Valid message id currently in the mail queue",
+ :type => :string,
+ :validation => :exim_msgid,
+ :optional => false,
+ :maxlength => 16
+
+ output :status,
+ :description => "Status Message",
+ :display_as => "Status"
+end
+{% endhighlight %}
+
+Note here we are using our new validator to validate the *msgid* input.
+
+## Using the Validator in an Agent
+Agents can also have validation, traditionally this included the normal things
+like regular expressions but now here you can also use the validator plugins:
+
+{% highlight ruby %}
+action "retrymsg" do
+ validate :msgid, :exim_msgid
+
+ # call out to exim to retry the message
+end
+{% endhighlight %}
+
+Here we've extended the basic *validate* helper of the RPC Agent with our own
+plugin and used it to validate a specific input.
+
+## Listing available Validators
+You can obtain a list of validators using the *plugin* application:
+
+{% highlight ruby %}
+% mco plugin doc
+
+Please specify a plugin. Available plugins are:
+
+.
+.
+.
+
+Validator Plugins:
+ array Validates that a value is included in a list
+ exim_msgid Validates that a string is a Exim Message ID
+ ipv4address Validates that a value is an ipv4 address
+ ipv6address Validates that a value is an ipv6 address
+ length Validates that the length of a string is less or equal to a specified value
+ regex Validates that a string matches a supplied regular expression
+ shellsafe Validates that a string is shellsafe
+ typecheck Validates that a value is of a certain type
+
+{% endhighlight %}
+
+Note our new *exim_msgid* plugin appears in this list.
+
--- /dev/null
+---
+layout: default
+title: Discovery Filters
+---
+
+[FactPlugin]: /mcollective/reference/plugins/facts.html
+
+Using filters to control discovery and addressing is a key concept in mcollective.
+You can use facts, classes, agents and server identities in filters and combine
+to narrow down what hosts you will affect.
+
+To determine if your client support filters use the _--help_ option:
+
+
+{% highlight console %}
+$ mco rpc --help
+.
+.
+.
+Host Filters
+ -W, --with FILTER Combined classes and facts filter
+ -F, --wf, --with-fact fact=val Match hosts with a certain fact
+ -C, --wc, --with-class CLASS Match hosts with a certain config management class
+ -A, --wa, --with-agent AGENT Match hosts with a certain agent
+ -I, --wi, --with-identity IDENT Match hosts with a certain configured identity
+{% endhighlight %}
+
+If you see a section as above then the client supports filters, this is the default
+for all clients using SimpleRPC.
+
+All filters support Regular Expressions and some support comparisons like greater than
+or less than.
+
+Filters are applied in a combined manner, if you supply 5 filters they must all match
+your nodes.
+
+## Fact Filters
+
+Filtering on facts require that you've correctly set up a [FactPlugin]. The examples below
+show common fact filters.
+
+Install the ZSH package on machines with the fact _country=de_:
+
+{% highlight console %}
+% mco rpc package install zsh -F country=de
+{% endhighlight %}
+
+Install the ZSH package on machines where the _country_ fact starts with the letter _d_:
+
+{% highlight console %}
+% mco rpc package install zsh -F country=/^d/
+{% endhighlight %}
+
+{% highlight console %}
+% mco rpc package install zsh -F country=~^d
+{% endhighlight %}
+
+Install the ZSH package on machines with more than 2 CPUs, other available operators
+include _==, <=, >=, <, >, !=_. For facts where the comparison and the
+actual fact is numeric it will do a numerical comparison else it wil do alphabetical:
+
+{% highlight console %}
+% mco rpc package install zsh -F "physicalprocessorcount>=2"
+{% endhighlight %}
+
+## Agent, Identity and Class filters
+
+These filters all work on the same basic pattern, they just support equality or regular
+expressions:
+
+Install ZSH on machines with hostnames starting with _web_:
+
+{% highlight console %}
+% mco rpc package install zsh -I /^web/
+{% endhighlight %}
+
+Install ZSH on machines with hostnames _web1.example.com_:
+
+{% highlight console %}
+% mco rpc package install zsh -I web1.example.com
+{% endhighlight %}
+
+## Combining Fact and Class filters
+
+As a short-hand you can combine Fact and Class filters into a single filter:
+
+Install ZSH on machines in Germany that has classes matching _/apache/_:
+
+{% highlight console %}
+% mco rpc package install zsh -W "/apache/ country=de"
+{% endhighlight %}
+
--- /dev/null
+---
+layout: default
+title: Node Reports
+---
+[Subcollectives]: ../basic/subcollectives.html
+
+As we have all facts, classes and agents for nodes we can do some custom reporting on all of these.
+
+The _mco inventory_ tool is a generic node and network reporting tool, it has basic scripting abilities.
+
+**Note: This is an emerging feature, the scripting language is likely to change**
+
+## Node View
+To obtain a full inventory for a given node you can run _mco inventory_ like this:
+
+{% highlight console %}
+ % mco inventory your.node.com
+ Inventory for your.node.com:
+
+
+ Server Statistics:
+ Start Time: Mon Sep 13 18:24:46 +0100 2010
+ Config File: /etc/mcollective/server.cfg
+ Process ID: 5197
+ Total Messages: 62
+ Messages Passed Filters: 62
+ Messages Filtered: 0
+ Replies Sent: 61
+ Total Processor Time: 0.18 seconds
+ System Time: 0.01 seconds
+
+ Agents:
+ discovery echo nrpe
+ package process puppetd
+ rpctest service
+
+ Configuration Management Classes:
+ aliases apache
+ <snip>
+
+ Facts:
+ architecture => i386
+ country => de
+ culturemotd => 1
+ customer => rip
+ diskdrives => xvda
+ <snip>
+{% endhighlight %}
+
+This gives you a good idea of all the details available for a node.
+
+## Collective List
+
+We have a concept of [Subcollectives] and you can use the inventory application to get a quick
+report on all known collectives:
+
+{% highlight console %}
+$ mco inventory --list-collectives
+
+ * [ ===================================== ] 52 / 52
+
+ Collective Nodes
+ ========== =====
+ za_collective 2
+ us_collective 7
+ uk_collective 19
+ de_collective 24
+ eu_collective 45
+ mcollective 52
+
+ Total nodes: 52
+
+{% endhighlight %}
+
+## Collective Map
+
+You can also create a DOT format graph of your collective:
+
+{% highlight console %}
+$ mco inventory --collective-graph collective.dot
+
+Retrieving collective info....
+Graph of 52 nodes has been written to collective.dot
+{% endhighlight %}
+
+The graph will be a simple dot graph that can be viewed with Graphviz, Gephi or
+other compatible software.
+
+## Custom Reports
+
+You can create little scriptlets and pass them into *mco inventory* with the *--script* option.
+
+You have the following data available to your reports:
+
+| Variable | Description |
+|----------|-------------|
+|time|The time the report was started, normal Ruby Time object|
+|identity|The sender id|
+|facts|A hash of facts|
+|agents|An array of agents|
+|classes|An array of CF Classes|
+
+### printf style reports
+
+Lets say you now need a report of all your IBM hardware listing hostname, serial number and product name you can write a scriptlet like this:
+
+{% highlight ruby linenos %}
+inventory do
+ format "%s:\t\t%s\t\t%s"
+
+ fields { [ identity, facts["serialnumber"], facts["productname"] ] }
+end
+{% endhighlight %}
+
+And if saved as _inventory.mc_ run it like this:
+
+{% highlight console %}
+ % mco inventory -W "productname=/IBM|BladeCenter/" --script inventory.mc
+ xx12: 99xxx21 BladeCenter HS22 -[7870B3G]-
+ xx9: 99xxx46 BladeCenter HS22 -[7870B3G]-
+ xx10: 99xxx29 BladeCenter HS22 -[7870B3G]-
+ yy1: KDxxxFR IBM System x3655 -[79855AY]-
+ xx5: 99xxx85 IBM eServer BladeCenter HS21 -[8853GLG]-
+ <snip>
+{% endhighlight %}
+
+We'll add more capabilities in the future, for now you can access *facts* as a hash, *agents* and *classes* as arrays as well as *identity* as a string.
+
+
+### Perl format style reports
+To use this you need to install the *formatr* gem, once that's installed you can create a report scriptlet like below:
+
+{% highlight ruby linenos %}
+formatted_inventory do
+ page_length 20
+
+ page_heading <<TOP
+
+ Node Report @<<<<<<<<<<<<<<<<<<<<<<<<<
+ time
+
+Hostname: Customer: Distribution:
+-------------------------------------------------------------------------
+TOP
+
+ page_body <<BODY
+
+@<<<<<<<<<<<<<<<< @<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+identity, facts["customer"], facts["lsbdistdescription"]
+ @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
+ facts["processor0"]
+BODY
+end
+{% endhighlight %}
+
+Here we create a paged report - 20 nodes per page - with a heading section and a 2 line report per node with identity, customer, distribution and processor.
+
+The output looks like this:
+
+{% highlight console %}
+ % mco inventory -W "/dev_server/" --script inventory.mc
+
+ Node Report Sun Aug 01 10:30:57 +0100
+
+ Hostname: Customer: Distribution:
+ -------------------------------------------------------------------------
+
+ dev1.one.net rip CentOS release 5.5 (Final)
+ AMD Athlon(tm) 64 X2 Dual Core Processor
+
+ dev1.two.net xxxxxxx CentOS release 5.5 (Final)
+ AMD Athlon(tm) 64 X2 Dual Core Processor
+{% endhighlight %}
+
+Writing these reports are pretty ugly I freely admit, however it avoids designing our own reporting engine and it's pretty good for kicking out simple reports. You can see the *perlform* man page for details of the reporting layouts, ours is pretty close to that thanks to Formatr
--- /dev/null
+---
+layout: default
+title: Release Notes
+toc: false
+---
+
+This is a list of release notes for various releases, you should review these
+before upgrading as any potential problems and backward incompatible changes
+will be highlighted here.
+
+<a name="2_3_1"> </a>
+
+## 2.3.1 - 2013/02/14
+
+This is the second release in the new development series of MCollective. This
+release features enhancements and bug fixes.
+
+This release is for early adopters, production users should consider the 2.2.x
+series.
+
+### New Features and Improvements
+
+ * Initial work towards online help, improved logging and internationalization
+ * The output from *--help* has been made clearer
+ * The output of a failed reply in the default *printrpc* method has been improved
+
+### Bug Fixes
+
+ * The vendored JSON gem was updated to version 1.5.5 due to CVE-2013-0269
+ * The RPC client inadvertently lost the ability to set discovery_timeout, this has been restored
+ * Plugins with underscores in their name were not packagable on Debian, we now change underscores to dashes
+ * The STOMP connector will not be maintained further and has been removed
+ * A config file reading race condition were fixed, we no longer attempt to use config details before parsing the config file thus always using defaults.
+ * Dependencies on packaged plugins have been made more specific to ensure updates work correctly
+ * When an argument to the rpc application fails to parse the command will fail instead of continue with unexpected side effects
+ * Processing of *--no-response* was broken in 2.3.0, this has been fixed
+
+### Removed Functionality
+
+ * The STOMP adapter has been deprecated and removed
+
+### Online Help and Internationalization
+
+Starting in this release a number of errors and messages will start showing error codes along
+with the error text and we have a method for obtaining detailed information about each coded
+message.
+
+An example log line can be seen here:
+
+{% highlight console %}
+puppetd.rb:26 PLMC34: setting meta data in agents have been deprecated, DDL files are now being used for this information. Please update the 'puppetd.rb' agent
+{% endhighlight %}
+
+And an example CLI error string:
+
+{% highlight console %}
+% mco rpc rpcutil get_fact
+
+The rpc application failed to run: PLMC30: Action 'get_fact' needs a 'fact' argument
+
+Use the 'mco doc PLMC30' command for details about this error, use -v for full error backtrace details
+{% endhighlight %}
+
+You can now use the *mco doc PLMC30* command to get additional information about this error
+and any other error code you might see.
+
+Only a small number of errors and log lines have been updated for the new system and
+we will soon publish web versions of these help documents too which should help when
+searching for resolution to common errors.
+
+### Backwards Compatibility and Upgrading
+
+The STOMP connector has been removed, if you are using it please move to the RabbitMQ
+or ActiveMQ one before upgrading. Especially if you use Debian which would avoid the
+package upgrading from failing
+
+### Changes since 2.3.0
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|*2013/02/14*|*Release 2.3.1*|19265|
+|2013/02/14|Initial work towards internationalization and online help|18663|
+|2013/02/14|Update vendored JSON gem for CVE-2013-0269|19265|
+|2013/02/13|Restore the ability to set a discovery timeout on a RPC client|19238|
+|2013/02/12|Replace underscores in plugin names with dashes to keep Debian happy|19200|
+|2013/02/12|Fix package building on certain Debian systems|19141|
+|2013/02/12|Remove the stomp connector|19146|
+|2013/02/07|Read the client config before trying to use any configuration options|19105|
+|2013/01/22|When an argument fails to parse in the rpc application fail rather than continue with unintended consequences|18773|
+|2013/01/22|The fix the *--no-response* argument to the rpc application that broke due to 18438|18513|
+|2013/01/22|Set *=* dependencies on the various packages that form a plugin rather than *>=*|18758|
+|2013/01/21|Improve presentation of the --help output for applications|18447|
+|2013/01/21|When a request failed via *reply.fail*, only show the message and not the half built data|18434|
+
+<a name="2_2_3"> </a>
+
+## 2.2.3 - 2013/02/14
+
+This is a maintenance release to the current production version of MCollective.
+This release is a bug fix only release.
+
+### Bug Fixes
+
+ * The vendored JSON gem was updated to version 1.5.5 due to CVE-2013-0269
+ * The RPC client inadvertently lost the ability to set discovery_timeout, this has been restored
+ * Plugins with underscores in their name were not packagable on Debian, we now change underscores to dashes
+ * The STOMP adapter will not be maintained past this release series, we now issue deprecation warnigns
+ * A config file reading race condition were fixed, we no longer attempt to use config details before parsing the config file thus always using defaults.
+ * Dependencies on packaged plugins have been made more specific to ensure updates work correctly
+
+### Backwards Compatibility and Upgrading
+
+This release should be 100% backwards compatible with 2.2.0, 2.2.1 and 2.2.2, when upgrading
+from earlier releases please review the Release notes for 2.0.0.
+
+If you packaged any plugins with a underscore in their name, future packages will have a dash
+instead, this might cause upgrade problems.
+
+We are deprecating the STOMP connector, if you are using this connector please consider moving to the
+ActiveMQ or RabbitMQ specific ones.
+
+### Changes since 2.2.2
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|*2013/02/14*|*Release 2.2.3*|19265|
+|2013/02/14|Update vendored JSON gem for CVE-2013-0269|19265|
+|2013/02/13|Restore the ability to set a discovery timeout on a RPC client|19238|
+|2013/02/12|Replace underscores in plugin names with dashes to keep Debian happy|19200|
+|2013/02/12|Fix package building on certain Debian systems|19141|
+|2013/02/12|Deprecate the stomp connector|19146|
+|2013/02/07|Read the client config before trying to use any configuration options|19105|
+|2013/01/22|Set *=* dependencies on the various packages that form a plugin rather than *>=*|18758|
+
+<a name="2_0_1"> </a>
+
+## 2.0.1 - 2013/02/14
+
+This is a maintenance release against our unsupported past production release, it brings no
+visible changes or bug fixes we only updated the vendored JSON gem to version 1.5.5 due to
+CVE-2013-0269
+
+<a name="2_2_2"> </a>
+
+## 2.2.2 - 2013/01/17
+
+This is a maintenance release to the current production version of MCollective.
+This release is a bug fix only release.
+
+### Bug Fixes
+
+ * Add the package iteration number as dependency for common packages
+ * The :any validator has been restored
+ * Packaging non-agent plugins failed when providing custom paths
+ * Packaging on RHEL5 systems failed due to an undefined buildroot
+ * When available packages will be built using rpmbuild-md5
+ * Help for data plugins with no input queries are now rendered correctly
+ * The rpcutil#get_data action now supports data plugins without input queries
+ * The RPM packages will now require Ruby > 1.8 to improve packaging for 1.9.x
+
+### Backwards Compatibility and Upgrading
+
+This release should be 100% backwards compatible with 2.2.0 and 2.2.1, when upgrading
+from earlier releases please review the Release notes for 2.0.0.
+
+### Changes since 2.2.1
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|*2013/02/17*|*Release 2.2.2*|18258|
+|2013/01/03|Add the package iteration number as a dependency for the common packages|18273|
+|2012/12/24|Restore the :any validator|18265|
+|2012/12/19|Do not fail when packaging non-agent packages using custom paths|17281|
+|2012/12/19|Require Ruby > 1.8 in the RPM specs for Ruby 1.9|17149|
+|2012/11/08|Define a specific buildroot to support RHEL5 systems correctly|17516|
+|2012/11/08|Use the correct rpmbuild commands on systems with rpmbuild-md5|17515|
+|2012/10/22|Correctly show help for data plugins without any input queries|17137|
+|2012/10/22|Allow the rpcutil#get_data action to work with data queries that takes no input|17138|
+
+<a name="2_3_0"> </a>
+
+## 2.3.0 - 2012/01/10
+
+This is the first release in the new development series of MCollective. This
+release features small enhancements and bug fixes.
+
+This release is for early adopters, production users should consider the 2.2.x
+series.
+
+### Enhancements and behaviour changes
+
+ * Data queries can be written without any input queries
+ * Required inputs can now supply default values in their DDLs
+ * Support for Ruby 1.9 was improved in the packages
+ * The generated plugin documentation has been updated to show defaults and optional items
+ * Errors in agents will now log backtraces on the servers to assist with debugging
+ * libdirs will now be expanded to absolute paths and using relative ones will raise an error
+ * Various error and logging improvements
+ * Various improvements to the plugin packager
+
+### Bug fixes
+
+ * Packaging non-agent plugins with custom paths caused an unexpected failure
+ * The plugin packager works correctly on RHEL5 now after previously using an incorrect buildroot
+ * Correctly handle custom formats passed to the aggregation plugins from the DDL
+ * Failure in one aggregate plugin does not impact other aggregate functions
+ * The chosen timeout for agents when using direct addressing could be wrong in some cases
+ * Data plugins can now return BigNum data like those found in timestamps
+ * Aggregate functions support non string data
+ * Boolean flags in applications can now support --noop and --no-noop style flags
+ * Data results were not raising the correct exception, this was not causing problems in practice but caused the mcollective-test gem to fail
+
+### Input defaults in the DDL
+
+You can now provide input defaults for required inputs in the DDL meaning if not
+supplied they will default to the supplied format.
+
+{% highlight ruby %}
+action "get_fact", :description => "Retrieve a single fact from the fact store" do
+ input :fact,
+ :prompt => "The name of the fact",
+ :description => "The fact to retrieve",
+ :type => :string,
+ :validation => '^[\w\-\.]+$',
+ :optional => false,
+ :maxlength => 40,
+ :default => "operatingsystems"
+end
+{% endhighlight %}
+
+The DDL file above defines a input *fact* that is required and sets a default value to
+*operatingsystem*.
+
+Previously the following command would have failed stating the input is required, now it
+will default to the supplied value and continue without error:
+
+{% highlight ruby %}
+$ mco rpc rpcutil get_fact
+{% endhighlight %}
+
+The defaults processing is done on the client side and not on the servers meaning at no
+point does a non compliant request get published by the clients and older MCollective servers
+will process these requests correctly.
+
+### Backwards Compatibility and Upgrading
+
+This release can cohabit with older versions with the only potential upgrade problem being
+the changes to how the libdir variable is handled.
+
+In the past a libdir could be:
+
+{% highlight ini %}
+libdir = /usr/libexec/mcollective:.mcollective.d
+{% endhighlight %}
+
+This would have the effect of looking for *.mcollective.d* in the current directory.
+
+This represented a security risk and would fail on the server side when daemonizing.
+We now force all libdir paths to be fully qualified and raises an error at start should
+you have relative paths.
+
+### Changes since 2.2.1
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2012/01/10|Raise the correct exception when trying to access unknown data items in a Data results|18466|
+|2013/01/10|Fix failing documentation generation for data plugins|18437|
+|2013/01/09|Correctly support negative boolean flags declared as --[no]-foo|18438|
+|2013/01/03|Add the package iteration number as a dependency for the common packages|18273|
+|2012/12/21|The libdirs supplied in the config file now has to be absolute paths to avoid issues when daemonising|16018|
+|2012/12/20|Logs the error and backtrace when an action fails|16414|
+|2012/12/20|Display the values of :optional and :default in DDL generated help|16616|
+|2012/12/20|Allow the query string for the get_data action in rpcutil to be 200 chars|18200|
+|2012/12/19|Do not fail when packaging non-agent packages using custom paths|17281|
+|2012/12/19|Require Ruby > 1.8 in the RPM specs for Ruby 1.9|17149|
+|2012/12/18|Allow required inputs to specify default data in DDLs|17615|
+|2012/11/12|When disconnecting set the connection to nil|17384|
+|2012/11/08|Define a specific buildroot to support RHEL5 systems correctly|17516|
+|2012/11/08|Use the correct rpmbuild commands on systems with rpmbuild-md5|17515|
+|2012/10/22|Correctly show help for data plugins without any input queries|17137|
+|2012/10/22|Allow the rpcutil#get_data action to work with data queries that takes no input|17138|
+|2012/10/03|Improve text output when providing custom formats for aggregations|16735|
+|2012/10/03|Correctly process supplied formats when displaying aggregate results|16415|
+|2012/10/03|Prevent one failing aggregate function from impacting others|16411|
+|2012/10/03|When validation fails indicate which input key has the problem|16617|
+|2012/09/26|Data queries can be written without any input queries meaning they take no input|16424|
+|2012/09/26|Use correct timeout for agent requests when using direct addressing|16569|
+|2012/09/26|Allow BigNum data to be used in data plugin replies|16503|
+|2012/09/26|Support non string data in the summary aggregate function|16410|
+
+
+<a name="2_2_1"> </a>
+
+## 2.2.1 - 2012/10/17
+
+This is a maintenance release to the current production version of MCollective.
+This release is a bug fix only release.
+
+### Bug Fixes
+
+ * Various display and stability improvements with aggregate plugins
+ * Improve error messages
+ * Data queries that does not take an input still had to provide a bogus query input, now not needed
+ * When using direct addressing and identity filter the client timeout was incorrect
+ * BigNum type data can now be used in data plugin replies
+
+### Backwards Compatibility and Upgrading
+
+This release should be 100% backwards compatible with 2.2.0, when upgrading from earlier releases
+pleas reivew the Release notes for 2.0.0.
+
+### Changes since 2.1.0
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|*2012/10/17*|*Release 2.2.1*|16965|
+|2012/10/03|Improve text output when providing custom formats for aggregations|16735|
+|2012/10/03|Correctly process supplied formats when displaying aggregate results|16415|
+|2012/10/03|Prevent one failing aggregate function from impacting others|16411|
+|2012/10/03|When validation fails indicate which input key has the problem|16617|
+|2012/09/26|Data queries can be written without any input queries meaning they take no input|16424|
+|2012/09/26|Use correct timeout for agent requests when using direct addressing|16569|
+|2012/09/26|Allow BigNum data to be used in data plugin replies|16503|
+|2012/09/26|Support non string data in the summary aggregate function|16410|
+|2012/09/14|Package discovery plugins that was left out for debian|16413|
+
+<a name="2_2_0"> </a>
+
+## 2.2.0 - 2012/09/13
+
+This is the next production release of MCollective. It brings to an end active
+support for versions 2.1.1 and older.
+
+### Major Enhancements
+
+ * A new plugin type called data plugins were added making network discovery extendible by users
+ * Discovery is now pluggable allowing network based, database based, file based or any other data source to be used as a source of truth
+ * Automatic result summarization methods can be declared in the DDL and users can write their own
+ * A RabbitMQ specific Direct Addressing capable connector was added
+ * Agent DDLs must be present on the servers, input validation is done against the DDL and prior to running user code
+ * DDL files can define default values for returned data - all declared data fields are pre-populated by agents
+ * DDL files can store general usage information that gets rendered via the help application
+ * DDL files can declare the minimum version mcollective they need to be functional and loading plugins on older mcollective versions will fail
+ * New validation logic in DDL files and Agents can now be delivered using plugins
+ * A thread safe caching system was added that users can use in their Agents to store information between invocations
+ * Code generators to assist writing agents
+ * Support deterministic random node selection
+ * Display mode can be overriden on the CLI using the new *--display* option
+ * The plugin packager will now keep source debs and rpms and has had major improvements done
+ * A new application called *completion* was added to assist in writing shell completion systems. ZSH and Bash examples are in *ext/*
+ * Various improvements to documentation was made especially around using the CLI tools and discovery available plugins
+
+### Bug Fixes
+
+ * The vendored systemu gem has been updated to remove a rude error message
+ * Improved error reporting in many areas
+ * Boolean and numeric data is correctly parsed on the RPC application command line
+ * Improved parsing of compound filters
+ * Batched requests will now all have the same request id thus improving consistency of auditing information
+
+### Deprecations
+
+ * Remove the traditional Client#discovered_req method
+ * The metadata section in the agent is being removed as the DDL is now present everywhere
+
+### Data Plugins
+
+A new plugin type called _data plugins_ have been added, these plugins are
+usable in discovery requests and in any agent code.
+
+You can use these plugins to expose any node side data to your client discovery
+command line, an example can be seen below, this will discover all nodes where
+_/etc/syslog.conf_ has a md5 sum matching the regular expression _/19ff4997e/_:
+
+{% highlight console %}
+$ mco rpc rpcutil ping -S "fstat('/etc/rsyslog.conf').md5 = /19ff4997e/"
+{% endhighlight %}
+
+For full information see the plugins documentation on our website. The _fstat_
+plugin seen above is included at the moment, more will be added in due course
+but as always users can also write their own suitable to their needs.
+
+### Custom Discovery Sources
+
+A new type of plugin that can be used as alternative data sources for discovery
+data has been added. The traditional network broadcast mode is supported and
+remains the default but a new flat file one was added.
+
+Custom discovery sources can be made the default for a client using the
+*default_discovery_method* configuration option but can be selected on the
+command line using _--disc-method_.
+
+All applications now have a _--nodes_ option that takes as an argument a flat
+file full of mcollective identity names, one per line.
+
+Users can write their own discovery plugins and distribute it using the normal
+plugin packager. A complex example can be seen in the community plugin site
+for the MongoDB registration plugin.
+
+In the event that the _-S_ filter is used the network discovery mode will be
+forced so that data source plugins in discovery queries will always work as
+expected.
+
+This feature requires Direct Addressing.
+
+### DDL files on the servers
+
+The DDL files now have to be on the servers and the clients. On the servers the
+results will be pre-populated with default data for all defined output values of
+a specific action and you can now supply defaults.
+
+Additionally input will be validated on each node prior to running the agent
+code providing consistent input validation on client and server. This should
+remove the need to add *validate* statements to agents.
+
+An example for a Nagios plugin can be seen below, here we default to *UNKNOWN*
+so that even if the action fails to run we will still see valid data being
+returned thats appropriate for the specific use case.
+
+{% highlight ruby %}
+action "runcommand", :description => "Run a NRPE command" do
+ output :exitcode,
+ :description => "Exit Code from the Nagios plugin",
+ :display_as => "Exit Code",
+ :default => 3
+end
+{% endhighlight %}
+
+### Summarization Plugins
+
+Often custom applications are written just to summarize data like the *facts*
+application or *nrpe* ones.
+
+We have added a new plugin type that allows you to define summarization logic
+and included a few of our own. These summaries are declared in the DDL, here is
+a section from the new DDL for the *get_fact* action:
+
+{% highlight ruby %}
+action "get_fact", :description => "Retrieve a single fact from the fact store" do
+ output :value,
+ :description => "The value of the fact",
+ :display_as => "Value"
+
+ summarize do
+ aggregate summary(:value)
+ end
+end
+{% endhighlight %}
+
+Here we are using the *summarize* block to say that we wish to summarize the
+output *:value*. The *summary(:value)* is the call to a custom plugin and you
+can provide your own.
+
+Now when interacting with this action you will see summaries produced
+automatically:
+
+{% highlight ruby %}
+% mco rpc rpcutil get_fact fact=operatingsystemrelease
+.
+.
+dev2
+ Fact: operatingsystemrelease
+ Value: 6.2
+
+
+Summary of Value:
+
+ 6.2 = 19
+ 6.3 = 7
+
+Finished processing 26 / 26 hosts in 294.97 ms
+{% endhighlight %}
+
+The last section of the rpc output shows the summarization in action.
+
+The NRPE plugin on GitHub shows an example of a Nagios specific aggregation
+function and the plugin packager supports distributing these plugins.
+
+### Validation Plugins
+
+Users can now write their own plugins to perform input validation, these
+validations are usable in DDL files and agents.
+
+Below is a snippet from a DDL file using a custom *exim_msgid* validation
+plugin:
+
+{% highlight ruby %}
+ input :msgid,
+ :prompt => "Message ID",
+ :description => "Valid message id currently in the mail queue",
+ :type => :string,
+ :validation => :exim_msgid,
+ :optional => false,
+ :maxlength => 16
+{% endhighlight %}
+
+And a snippet using the same plugin inside your agent:
+
+{% highlight ruby %}
+action "retrymsg" do
+ validate :msgid, :exim_msgid
+
+ # call out to exim to retry the message
+end
+{% endhighlight %}
+
+The error messages shown when validation fails are more user friendly than
+before, in this example the new error would be *Not a valid Exim Message ID*
+where in the past it would have been *value should match ^(?:[+-]\d{4})?(?:\[\d+\] )?(\w{6}\-\w{6}\-\w{2})/*
+
+### Code generation
+
+Code for agents and data sources can now be generated to assist development, you
+can use the _plugin_ command to create a basic skeleton agent or data source
+including the DDL files.
+
+{% highlight console %}
+$ mco plugin generate agent myagent actions=do_something,do_something_else
+{% endhighlight %}
+
+Defaults used in the metadata templates can be set in the config file:
+
+{% highlight ini %}
+plugin.metadata.url=http://devco.net
+plugin.metadata.author=R.I.Pienaar <rip@devco.net>
+plugin.metadata.license=ASL2.0
+plugin.metadata.version=0.0.1
+{% endhighlight %}
+
+All generator produced output will have these settings set, the other fields are
+constructed using a pattern convenient for using in your editor as a template.
+
+### Backwards Compatibility and Upgrading
+
+As of this version every agent on every node and client must have a DDL file. If
+the DDL file is not present or not valid the agent will not activate. Further
+input validation is done according to the content of the DDL prior to running
+any actions. You should therefore prepare for this upgrade by writing and
+deploying DDL files for all your agents.
+
+Version 2.0.0 and 2.2.0 can co-exist on the same network. If a new client uses
+any of the new features added such as data plugins the older clients will simply
+refuse to run the request but requests using features shared between versions
+will continue to work.
+
+When you first start this version of mcollectived you will see warnings logged
+similar to the one below:
+
+{% highlight ruby %}
+puppetd.rb:26: setting meta data in agents have been deprecated, DDL files are now being used for this information.
+{% endhighlight %}
+
+This is only a warning and not a critical problem. The next major release will
+remove support for metadata in agents.
+
+Upgrading from versions prior to 2.0.0 was not tested, please refer to the
+release notes for 2.0.0.
+
+<a name="2_1_1"> </a>
+
+## 2.1.1 - 2012/07/12
+
+This release features major new features, enhancements and bug fixes.
+
+This release is for early adopters, production users should consider the 2.0.x
+series.
+
+### Major Enhancements
+
+ * A new discovery source was added capable of querying agent properties
+ * When doing limited discovery you can now supply a random seed for deterministic random selection
+ * A *get_data* action has been added to the *rpcutil* agent to retrieve the result of a data plugin
+ * RPC Agents must have DDLs on the MCollective Servers, agents will not load without them
+ * Output values can now have defaults assigned in the DDL, the server will set those defaults before running an action
+ * A new plugin type used to summarize sets of replies has been added. Summarization is declared in the DDL for an Agent
+
+### Bug Fixes
+
+ * Correctly parse numeric and boolean input arguments in the RPC application
+
+### Deprecations
+
+ * The old *Client#discovered_req* is removed along with the *controller* application that used it
+ * Parsing compound filters were improved wrt complex regular expressions
+ * Metadata sections in agents are not needed anymore and deprecation notices are logged when they are found
+
+### Summarization Plugins
+
+Often custom applications are written just to summarize data like the *facts*
+application or *nrpe* ones.
+
+We have added a new plugin type that allows you to define summarization logic
+and included a few of our own. These summaries are declared in the DDL, here is
+a section from the new DDL for the *get_fact* action:
+
+{% highlight ruby %}
+action "get_fact", :description => "Retrieve a single fact from the fact store" do
+ output :value,
+ :description => "The value of the fact",
+ :display_as => "Value"
+
+ summarize do
+ aggregate summary(:value)
+ end
+end
+{% endhighlight %}
+
+Here we are using the *summarize* block to say that we wish to summarize the
+output *:value*. The *summary(:value)* is the call to a custom plugin and you
+can provide your own.
+
+Now when interacting with this action you will see summaries produced
+automatically:
+
+{% highlight ruby %}
+% mco rpc rpcutil get_fact fact=operatingsystemrelease
+.
+.
+dev2
+ Fact: operatingsystemrelease
+ Value: 6.2
+
+
+Summary of Value:
+
+ 6.2 = 19
+ 6.3 = 7
+
+Finished processing 26 / 26 hosts in 294.97 ms
+{% endhighlight %}
+
+The last section of the rpc output shows the summarization in action.
+
+The NRPE plugin on GitHub shows an example of a Nagios specific aggregation
+function and the plugin packager supports distributing these plugins.
+
+### DDL files on the servers
+
+The DDL files now have to be on the servers and the clients. On the servers the
+results will be pre-populated with default data for all defined output values of
+a specific action and you can now supply defaults.
+
+An example for a Nagios plugin can be seen below, here we default to *UNKNOWN*
+so that even if the action fails to run we will still see valid data being
+returned thats appropriate for the specific use case.
+
+{% highlight ruby %}
+action "runcommand", :description => "Run a NRPE command" do
+ output :exitcode,
+ :description => "Exit Code from the Nagios plugin",
+ :display_as => "Exit Code",
+ :default => 3
+end
+{% endhighlight %}
+
+As the servers now have the DDL the *metadata* section at the top of agents are
+not needed anymore and deprecations will be logged when the mcollectived starts
+up warning you of this.
+
+### Backwards Compatibility and Upgrading
+
+As this release now requires DDL files to exist before an agent can be loaded in
+the server you might have to adjust your deployment strategy and possibly write
+some DDLs for your custom agents. The DDL files have to be on both client and
+servers.
+
+The servers will now pre-populate the replies with all output defined in the DDL
+and supply defaults if no default is provided in the DDL it will default to nil.
+This might potentially change the behavior of custom applications that are
+designed around the approach of checking if a field is included in the results
+or not.
+
+When you first start this version of mcollectived you will see warnings logged
+similar to the one below:
+
+{% highlight ruby %}
+puppetd.rb:26: setting meta data in agents have been deprecated, DDL files are now being used for this information.
+{% endhighlight %}
+
+This is only a warning and not a critical problem. Once 2.2.0 is out we will be
+updating all the agents to remove metadata sections in favour of those in the DDL.
+You should also remove metadata from your own agents.
+
+### Changes since 2.1.0
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2012/07/11|Add a --display option to RPC clients that overrides the DDL display mode|15273|
+|2012/07/10|Do not add a metadata to agents created with the generator as they are now deprecated|15445|
+|2012/07/03|Correctly parse numeric and boolean data on the CLI in the rpc application|15344|
+|2012/07/03|Fix a bug related to parsing regular expressions in compound statements|15323|
+|2012/07/02|Update vim snippets in ext for new DDL features|15273|
+|2012/06/29|Create a common package for agent packages containing the DDL for servers and clients|15268|
+|2012/06/28|Improve parsing of compound filters where the first argument is a class|15271|
+|2012/06/28|Add the ability to declare automatic result summarization in the DDL files for agents|15031|
+|2012/06/26|Surpress subscribing to reply queues when no reply is expected|15226|
+|2012/06/25|Batched RPC requests will now all have the same requestid|15195|
+|2012/06/25|Record the request id on M::Client and in the RPC client stats|15194|
+|2012/06/24|Use UUIDs for the request id rather than our own weak implementation|15191|
+|2012/06/18|The DDL can now define defaults for outputs and the RPC replies are pre-populated|15087|
+|2012/06/18|Remove unused agent help code|15084|
+|2012/06/18|Remove unused code from the *discovery* agent related to inventory and facts|15083|
+|2012/06/18|Nodes will now refuse to load RPC agents without DDL files|15082|
+|2012/06/18|The Plugin Name and Type is now available to DDL objects|15076|
+|2012/06/15|Add a get_data action to the rpcutil agent that can retrieve data from data plugins|15057|
+|2012/06/14|Allow the random selection of nodes to be deterministic|14960|
+|2012/06/12|Remove the Client#discovered_req method and add warnings to the documentation about its use|14777|
+|2012/06/11|Add a discovery source capable of doing introspection on running agents|14945|
+|2012/06/11|Only do identity filter optimisations for the *mc* discovery source|14942|
+
+<a name="2_1_0"> </a>
+
+## 2.1.0 - 2012/06/08
+
+This is the first release in the new development series of MCollective. This
+releas features major new features and enhancements.
+
+This release is for early adopters, production users should consider the 2.0.x
+series.
+
+### Major Enhancements
+
+ * Discovery requests can now run custom data plugins on nodes to facilitate discovery against any node-side data
+ * Discovery sources are now pluggable, one supporting flat files are included in this release
+ * All applications now have a --nodes option to read a text file of identities to operate on
+ * A new _completion_ application was added to assist with shell completion systems, zsh and bash tab completion plugins are in ext
+ * Users can now use a generator to create skeleton agents and data sources
+
+### Changes in behavior
+
+ * The _mco controller_ application is being deprecated for the next major release and has now been removed from the development series
+ * The _mco find_ application is now a discovery client so it's output mode has changed slightly but the functionality stays the same
+
+### Bug Fixes
+
+ * Numerous small improvement to user facing errors and status outputs have been made
+ * Sub collectives combined with direct addressing has been fixed
+ * Various packaging issues were resolved
+ * The ActiveMQ and Stomp connectors will now by default handle dual homed IPv6 and IPv4 hosts better in cases where the IPv6 target isn't reachable
+
+### Data Plugins
+
+A new plugin type called _data plugins_ have been added, these plugins are
+usable in discovery requests and in any agent code.
+
+You can use these plugins to expose any node side data to your client discovery
+command line, an example can be seen below, this will discover all nodes where
+_/etc/syslog.conf_ has a md5 sum matching the regular expression
+_/19ff4997e/_:
+
+{% highlight console %}
+$ mco rpc rpcutil ping -S "fstat('/etc/rsyslog.conf').md5 = /19ff4997e/"
+{% endhighlight %}
+
+For full information see the plugins documentation on our website. The _fstat_
+plugin seen above is included at the moment, more will be added in due course
+but as always users can also write their own suitable to their needs.
+
+### Custom Discovery Sources
+
+Since the introduction of direct addressing mode in 2.0.0 you've been able to
+pragmatically specify arbitrary host lists as discovery data but this was never
+exposed to the user interface.
+
+We now introduce plugins that can be used as alternative data sources and
+include the traditional network broadcast mode and a flat file one. The hope
+is that more will be added in future perhaps integrating with systems like
+PuppetDB. There is also one that uses the MongoDB Registration plugin to build
+a local node cache.
+
+Custom discovery sources can be made the default for a client using the
+*default_discovery_method* configuration option but can be selected on the
+command line using _--disc-method_.
+
+All applications now have a _--nodes_ option that takes as an argument a flat
+file full of mcollective identity names, one per line.
+
+Users can write their own discovery plugins and distribute it using the normal
+plugin packager.
+
+In the event that the _-S_ filter is used the network discovery mode will be
+forced so that data source plugins in discovery queries will always work as
+expected.
+
+### Code generation
+
+Code for agents and data sources can now be generated to assist development,
+you can use the _plugin_ command to create a basic skeleton agent or data source
+including the DDL files.
+
+{% highlight console %}
+$ mco plugin generate agent myagent actions=do_something,do_something_else
+{% endhighlight %}
+
+Defaults used in the metadata templates can be set in the config file:
+
+{% highlight ini %}
+plugin.metadata.url=http://devco.net
+plugin.metadata.author=R.I.Pienaar <rip@devco.net>
+plugin.metadata.license=ASL2.0
+plugin.metadata.version=0.0.1
+{% endhighlight %}
+
+All generator produced output will have these settings set, the other fields
+are constructed using a pattern convenient for using in your editor as a
+template.
+
+### Backwards Compatibility and Upgrading
+
+This release can co-exist with 2.0.0 but using the new discovery data plugins in a
+mixed environment will result in the old nodes not being discovered and they will
+log exceptions in their logs. This was done by choice and ensures the safest
+possible upgrade path.
+
+When the 2.0.0 collective is running with directed mode enabled a client using the
+new discovery plugins will be able to communicate wth the older nodes without
+problem.
+
+### Changes since 2.0.0
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2012/06/07|Force discovery state to be reset when changing collectives in the RPC client|14874|
+|2012/06/07|Create code generators for agents and data plugins|14717|
+|2012/06/07|Fix the _No response from_ report to be correctly formatted|14868|
+|2012/06/07|Sub collectives and direct addressing mode now works correctly|14668|
+|2012/06/07|The discovery method is now pluggable, included is one supporting flat files|14255|
+|2012/05/28|Add an application to assist shell completion systems with bash and zsh completion plugins|14196|
+|2012/05/22|Improve error messages from the packager when a DDL file cannot be found|14595|
+|2012/05/17|Add a dependency on stomp to the rubygem|14300|
+|2012/05/17|Adjust the ActiveMQ and Stomp connect_timeout to allow IPv4 fall back to happen in dual homed hosts|14496|
+|2012/05/16|Add a plugable data source usable in discovery and other plugins|14254|
+|2012/05/04|Improve version dependencies and upgrade experience of debian packages|14277|
+|2012/05/03|Add the ability for the DDL to load DDL files from any plugin type|14293|
+|2012/05/03|Rename the MCollective::RPC::DDL to MCollective::DDL to match its larger role in all plugins|14254|
+
+<a name="2_0_0"> </a>
+
+## 2.0.0 - 2012/04/30
+
+This is the next production release of MCollective. It brings to an
+end active support for versions 1.3.3 and older.
+
+This release brings to general availability all the features added in the
+1.3.x development series.
+
+### Major Enhancements
+
+ * Complete messaging protocol rewrite to enable direct style connectivity that would allow programs to bypass normal discovery instead using their own data sources
+ * An additional more robust messaging paradigm supporting a more assured addressing and delivery scheme
+ * Batched mode allowing users to address machines in small groups thus avoiding thundering herd and enabling more granular changes
+ * A more complete language for expressing discovery that includes and/or/not style queries across the infrastructure
+ * Improved Stomp connection security using normal industry standard Certificate Authority validated TLS
+ * New connector that uses ActiveMQ specific features for better performance and scalability
+ * Security of the SSL and AES security plugins have been improved for tamper protection by middle men
+ * A message validity period has been introduced to lower the window of message replay attacks
+ * Better error handling and better logging for Stomp connections
+ * JSON output from the 'rpc' application
+ * Ability to pipe RPC requests into each other creating a chain of related RPC calls
+ * Better validations, better error handling and better documentation creation from the DDL
+ * Performance improvements in the CLI, more consistently formatted output of received data
+ * A Ruby GEM of the client is now made available on rubygems.org
+ * The rc script for Debian based systems have been improved to prevent duplicate daemons from running
+ * Built in packager for plugins into native OS packages - RedHat and Debian supported
+ * MS Windows Support
+
+### Point to Point comms
+
+Previously MCollective could only broadcast messages and was tied to a discovery model.
+
+The messaging layer now supports per node destinations that allows you to address a node, even if its down,
+doesn't yet exist or if you cannot come up with a filter that would match a group of arbitrarily selected
+nodes.
+
+When this mode is in use the user configure which machine to communicate with using either text, arrays or
+JSON data. It will then communicate directly to those nodes via the middleware and if any of them are down
+you will get the usual no responses report after DDL configured timeout, this is a smooth transparent to the
+end user mix in communication modes.
+
+It is ideal for building deployers, web apps and so forth where you know exactly which nodes should be there
+and you'd like to influence the MCollective network addressing, perhaps from a CMDB you built yourself.
+
+This is the start towards an assured style of delivery, you can consider it the TCP to MCollective's UDP.
+Both modes of communication will be supported in the future and both will have access to all the same agents
+and clients.
+
+This is feature is enabled using the *direct_addressing* configuration option. At present only the new
+ActiveMQ connector supports this at scale. The ActiveMQ connector is now the recommended standard connector
+combined with Apache ActiveMQ. More brokers could be supported in future.
+
+### Pluggable / Optional Discovery
+
+If the user did _mco rpc rpcutil ping -I box.example.com -I another.example.com_ mcollective will now just
+assume you know what she wants, it won't do a discover to confirm those machines exist or not, it will just go
+and communicate with them. This is a big end user visible speed improvement. If however you did a filter
+like *-I /example.com/* mcollective cannot know which machines you want to reach and so a traditional
+broadcast discovery is done first.
+
+When the direct addressing mode is enabled various behind the scenes optimizations are being done:
+
+ * If a discovery is done and it finds you only want to address 10 or fewer nodes it will use direct mode for that
+ request. This avoids a second needless broadcast. This is less efficient to the middleware but does not send
+ needless messages to uninterested nodes that would then just ignore them.
+ * The _rpc_ application supports piping output from one to the next. Example of this below.
+
+{% highlight console %}
+$ mco rpc package update package=foo -W customer=acme -j|mco rpc service restart service=bar
+{% endhighlight %}
+
+This will update a package on machines matching *customer=foo* and then restart the service *bar* on those
+machines.
+
+The first request is doing traditional discovery based on the fact while the 2nd request is not doing
+discovery at all, it uses the JSON output enabled by -j as discovery data and then restart the service on only
+those machines.
+
+These abilities are exposed in the SimpleRPC client API and you can write your own schemes, query your own
+databases etc
+
+### Batching
+
+Often the speed of MCollective is a problem, you want to install a package on thousands of machines but your
+APT or YUM server isn't up to the task.
+
+You can now do batching of requests:
+
+{% highlight console %}
+$ mco package update myapp --batch 10 --batch-sleep 60
+{% endhighlight %}
+
+This performs the update as usual but only affecting machines in groups of 10 and sleeps for a minute between.
+
+You can also access this functionality via the API please see the docs for usage. Any existing script or
+application should support this functionality without any code changes.
+
+The results, error reporting, statistics reporting and so forth all stays consistent with non batched
+behavior.
+
+At any time you can interrupt the process and only the current group of machines will have been affected.
+
+The batching requires a direct addressing capable collective as it is built using the new direct to node
+communications and pluggable discovery features
+
+### ActiveMQ specific connector
+
+A new connector plugin has been added that is specific to ActiveMQ and is compatible with the new direct
+addressing communication system.
+
+You will need to change your ActiveMQ configuration to support this plugin, see the documentation for this
+plugin and the examples in _ext/activemq_ have also been updated for the new plugin.
+
+Anyone who use ActiveMQ is strongly recommended to use this plugin as it uses a few ActiveMQ specific
+optimizations that can have a big performance enhancing effect on your collective.
+
+### Packaging Agent plugins
+
+Distributing agents has been a problem as they are just files that have limited meta data and attached.
+
+We now support packaging agents into rpm or deb packages, your agent must have a DDL file for this to work:
+
+{% highlight console %}
+$ mco plugin package . --vendor "My Company"
+Successfully built RPM 'mcollective-exim_ng-client-0.1-1.noarch.rpm'
+Successfully built RPM 'mcollective-exim_ng-common-0.1-1.noarch.rpm'
+Successfully built RPM 'mcollective-exim_ng-agent-0.1-1.noarch.rpm'
+{% endhighlight %}
+
+The packages will have meta data like Author, Version and so forth as per your DDL file.
+
+Users can provide their own packaging implementations for other package managers or custom layouts using the
+MCollective plugin system.
+
+### Full verified CA
+
+When using the new ActiveMQ specific connector combined with Stomp version 1.2.2 or newer you can get full CA
+verified connection handling ensuring that only clients using signed certificates can connect to ActiveMQ.
+
+The documentation for the ActiveMQ SSL setup now includes instructions on setting up ActiveMQ and your clients
+using the built in Puppet CA but any CA could be used to manage these certificates.
+
+This feature will work best when ActiveMQ 5.6.0 is released in a few weeks since there will then be a NIO+SSL
+Stomp connector. The current SNAPSHOT release of ActiveMQ has this feature as well as the most recent Service
+Pack release of the Fuse Message Broker.
+
+### MS Windows Support
+
+The MS Windows platform is now supported as both a client and a server. The _ext/windows_ directory has some
+helpers and read me documentation that has been confirmed to work but we have not yet completed packaging
+ourselves so this is still a manual process.
+
+Combined with Puppet 2.7.12 or newer the Package and Service agents can be used to manage Windows resources
+using the same commands as those on Linux via mcollective.
+
+### New Discovery Language
+
+Previously dicovery was very limited, filters were simply run one after the other and you could not do
+anything complex like a mix of OR and AND boolean logic.
+
+A new compact discovery language was introduced perfect for use on the command line, an example below:
+
+{% highlight console %}
+$ mco find -S "((fqdn=/example.com/ or fqdn=/another.com/) or customer=acme) and apache and physicalprocessorcount>2"
+{% endhighlight %}
+
+The EBNF for this language can be seen below, it's available on the command line and the API
+
+ compound = ["("] expression [")"] {["("] expression [")"]}
+ expression = [!|not]statement ["and"|"or"] [!|not] statement
+ char = A-Z | a-z | < | > | => | =< | _ | - |* | / { A-Z | a-z | < | > | => | =< | _ | - | * | / | }
+ int = 0|1|2|3|4|5|6|7|8|9{|0|1|2|3|4|5|6|7|8|9|0}
+
+### Backwards Compatibility and Upgrading
+
+This release is not compatible with older versions. Client scripts and agents written for older versions will
+continue to work but a network hosting both 2.0.0 clients and older one will effectively be split into 2
+networks. While planning your upgrade you should plan to have machines running the client for both versions
+to retain full control during upgrade. The upgrade is best done in an scheduled window where all machines are
+updated together.
+
+While upgrading you must ensure that the plugins that come with the release are updated at the same time as
+the release. Older security and connector plugins will not function with this release. This also means if
+you wrote your own connector or security plugin you will need to port these prior to upgrading.
+
+Past this it should be a simple matter of updating using your operating systems package manager.
+
+We recommend you switch to the new ActiveMQ based connector plugin away from the previous generic Stomp one as
+this is the primary supported method of deployment and the generic Stomp one will be deprecated in future.
+Additionally the Stomp connector does not support the new direct messaging communications mode.
+
+In order to upgrade to the new ActiveMQ connector you will need to change your broker setup including ACLs,
+transport connectors, message policies and inter broker connections. Sample configuration files for single
+and multi broker setups can be found in the Git repository or the tar file in _ext/activemq_
+
+<a name="1_3_3"> </a>
+
+## 1.3.3 - 2012/04/05
+
+This is a release in the development series of MCollective. It feature major new features and bug fixes.
+
+This release is for early adopters, production users should consider the 1.2.x series.
+
+### Major Enhancements
+
+ * The MS Windows platform is now supported, packaging is still outstanding
+ * Agents can now be packaged to native OS packages using the new _mco plugin_ command
+ * _mco help rpc_ now show the help for the rpc application, _mco plugin doc puppetd_ shows the help for the puppetd agent
+ * Full CA verified Stomp is supported and documented between ActiveMQ and MCollective using Stomp > 1.2.2
+ * Application exit codes have been standardized using a new _halt_ helper function
+ * A new validator that allows users to check if a supplied value is one of a fixed list
+ * The syslog facility can now be set in the configuration file
+ * The client libraries are now available as a Ruby Gem
+ * Batch mode can now be enabled and disabled at will in an application
+ * The client config files now default to console based logging at warn level
+
+### Bug Fixes
+
+ * nil or empty results are correctly displayed by printrpc
+ * Some exceptions under Ruby 1.9.3 when using run() related to nil exit code has been fixed
+ * Various exceptions have been silence in inventory application, stomp plugin, rpc application and others
+ * Previous SSL_read errors when using the Stomp+TLS configuration is now avoided on Ruby 1.8
+
+### Packaging Agent plugins
+
+Distributing agents has been a problem as they are just files that have limited meta data and attached.
+
+We now support packaging agents into rpm or deb packages, your agent must have a DDL file
+for this to work:
+
+{% highlight console %}
+$ mco plugin package . --vendor "My Company"
+Successfully built RPM 'mcollective-exim_ng-client-0.1-1.noarch.rpm'
+Successfully built RPM 'mcollective-exim_ng-common-0.1-1.noarch.rpm'
+Successfully built RPM 'mcollective-exim_ng-agent-0.1-1.noarch.rpm'
+{% endhighlight %}
+
+The packages will have meta data like Author, Version and so forth as per your DDL file.
+
+We support building all the main plugin types in this manner but need to restructure the plugins
+repository to support this layout.
+
+To use this you need to install the fpm gem, you must install 0.4.3 and not a newer version, we are
+currently working on removing the fpm dependency as it's proven to be too unreliable to use.
+
+Users can provide their own packaging implementations for other package managers or custom layouts
+using the MCollective plugin system.
+
+### Full verified CA
+
+When using the new ActiveMQ specific connector combined with Stomp version 1.2.2 or newer you can
+get full CA verified connection handling ensuring that only clients using signed certificates
+can connect to ActiveMQ.
+
+The documentation for the ActiveMQ SSL setup now includes instructions on setting up ActiveMQ and your
+clients using the built in Puppet CA but any CA could be used to manage these certificates.
+
+This feature will work best when ActiveMQ 5.6.0 is released in a few weeks since there will then be a NIO+SSL
+Stomp connector. The current SNAPSHOT release of ActiveMQ has this feature as well as the most recent Service
+Pack release of the Fuse Message Broker.
+
+### MS Windows Support
+
+The MS Windows platform is now supported as both a client and a server. The _ext/windows_ directory
+has some helpers and read me documentation that has been confirmed to work but we have not yet
+completed packaging ourselves so this is still a manual process.
+
+Combined with Puppet 2.7.12 or newer the Package and Service agents can be used to manage Windows
+resources using the same commands as those on Linux via mcollective.
+
+### Backwards compatibility
+
+This release is backwards compatible with version 1.3.2, if you are coming from an older version please
+review earlier release notes.
+
+If you have been using the ActiveMQ specific plugin and its SSL settings you will now need to enable
+fallback mode as it will now only connect to ActiveMQ machines that present the correct CA certificate
+and will refuse to use anonymous certificates
+
+{% highlight ini %}
+plugin.activemq.pool.1.ssl.fallback = 1
+{% endhighlight %}
+
+### Changes since 1.3.2
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2012/04/04|Use the MCollective::SSL utility class for crypto functions in the SSL security plugin|13615|
+|2012/04/02|Support reading public keys from SSL Certificates as well as keys|13534|
+|2012/04/02|Move the help template to the common package for both Debian and RedHat|13434|
+|2012/03/30|Support Stomp 1.2.2 CA verified connection to ActiveMQ|10596|
+|2012/03/27|_mco help rpc_ now shows the help for the rpc application|13350|
+|2012/03/22|Add a mco command that creates native OS packaging for plugins|12597|
+|2012/03/21|Default to console based logging at warning level for clients|13285|
+|2012/03/20|Work around SSL_read errors when using SSL or AES plugins and Stomp+SSL in Ruby < 1.9.3|13207|
+|2012/03/16|Improve logging for SSL connections when using Stomp Gem newer than 1.2.0|13165|
+|2012/03/14|Simplify handling of signals like TERM and INT and remove pid file on exit|13105|
+|2012/03/13|Create a conventional place to store implemented_by scripts|13064|
+|2012/03/09|Handle exceptions added to the Stomp 1.1 compliant versions of the Stomp gem|13020|
+|2012/03/09|Specifically enable reliable communications while using the pool style syntax|13040|
+|2012/03/06|Initial support for the Windows Platform|12555|
+|2012/03/05|Application plugins can now disable any of 3 sections of the standard CLI argument parsers|12859|
+|2012/03/05|Fix base 64 encoding and decoding of message payloads that would previous raise unexpected exceptions|12950|
+|2012/03/02|Treat :hosts and :nodes as equivalents when supplying discovery data, be more strict about flags discover will accept|12852|
+|2012/03/02|Allow exit() to be used everywhere in application plugins, not just in the main method|12927|
+|2012/03/02|Allow batch mode to be enabled and disabled on demand during the life of a client|12854|
+|2012/02/29|Show the progress bar before sending any requests to give users feedback as soon as possible rather than after first result only|12865|
+|2012/02/23|Do not log exceptions in the RPC application when a non existing action is called with request parameters|12719|
+|2012/02/17|Log miscellaneous Stomp errors at error level rather than debug|12705|
+|2012/02/17|Improve subscription tracking by using the subID feature of the Stomp gem and handle duplicate exceptions|12703|
+|2012/02/15|Improve error handling in the inventory application for non responsive nodes|12638|
+|2012/02/14|Comply to Red Hat guideline by not setting mcollective to start by default after RPM install|9453|
+|2012/02/14|Allow building the client libraries as a gem|9383|
+|2012/02/13|On Red Hat like systems read /etc/sysconfig/mcollective in the init script to allow modification of the environment|7441|
+|2012/02/13|Make the handling of symlinks to the mco script more robust to handle directories with mc- in their name|6275|
+|2012/02/01|systemu and therefore MC::Shell can sometimes return nil exit code, the run() method now handles this better by returning -1 exit status|12082|
+|2012/01/27|Improve handling of discovery data on STDIN to avoid failures when run without a TTY and without supplying discovery data|12084|
+|2012/01/25|Allow the syslog facility to be configured|12109|
+|2012/01/13|Add a RPC agent validator to ensure input is one of list of known good values|11935|
+|2012/01/09|The printrpc helper did not correctly display empty strings in received output|11012|
+|2012/01/09|Add a halt method to the Application framework and standardize exit codes|11280|
+|2011/11/21|Remove unintended dependency on _pp_ in the ActiveMQ plugin|10992|
+|2011/11/17|Allow reply to destinations to be supplied on the command line or API|9847|
+
+
+<a name="1_3_2"> </a>
+
+## 1.3.2 - 2011/11/17
+
+This is a release in the development series of MCollective. It feature major new features.
+
+This release is for early adopters, production users should consider the 1.2.x series.
+
+### Enhancements
+
+ * Handling of syntax errors in Application plugins have been improved
+ * The limit method can now be set per RPC Client instance
+ * Optionally show response distribution in the _ping_ application with the _--graph_ option
+ * Expose a statistic about expired messages via the _rpcutil_ agent and show them in the inventory application.
+ * Remove all the _mc-_ scripts that has been ported to applications
+ * AES and TTL security plugins prevent tampering with the TTL and Message Times
+ * The RPC client can now raise an exception rather than exit on failure - ideal for use in web apps
+ * Discovery during requests that has a specific limit count set have been sped up
+ * Specific types for :number, :float and :integer has been aded to the DDL and the RPC application has special handling for them
+ * Caller ID, Certificate Names and Identity Names can now only be word characters, full stop and dash
+ * Security plugins are now quicker to ignore miss directed messages
+ * The client now unsubscribes from topics it does not need anymore
+ * SimpleRPC now supports performing actions in batches with a sleep between each batch
+ * A direct request capable ActiveMQ specific plugin has been included
+ * Message TTLs can be set globally in the config or in the API
+
+### ActiveMQ specific connector
+
+A new connector plugin has been added that is specific to ActiveMQ and is compatible
+with the new direct addressing communication system.
+
+You will need to change your ActiveMQ configuration to support this plugin, see the
+documentation for this plugin and the examples in _ext/activemq_ have also been
+updated for the new plugin.
+
+Anyone who use ActiveMQ is strongly recommended to use this plugin as it uses a
+few ActiveMQ specific optimizations that can have a big performance enhancing effect
+on your collective.
+
+### Batching
+
+Often the speed of MCollective is a problem, you want to install a package on thousands
+of machines but your APT or YUM server isn't up to the task.
+
+You can now do batching of requests:
+
+{% highlight console %}
+$ mco package update myapp --batch 10 --batch-sleep 60
+{% endhighlight %}
+
+This performs the update as usual but only affecting machines in groups of 10 and
+sleeps for a minute between.
+
+You can also access this functionality via the API please see the docs for usage.
+Any existing script or application should support this functionality without any
+code changes.
+
+The results, error reporting, statistics reporting and so forth all stays consistant
+with non batched behavior.
+
+The batching requires a direct addressing capable collective.
+
+### Backwards Compatibility
+
+As this release does a few more tweaks to the security system it might not work with older
+versions of MCollective.
+
+Hopefully this will be the last release in this dev cycle to break backwards compatibility
+as we're nearing the next major release.
+
+#### Identities, Certificates and Caller ID names
+
+These items have been tightened up to only match _\w\.-_. Plugins like the registration
+ones might assume it is safe to just write files based on names contained in these fields
+so rather than expect everyone to write secure code the framework now just enforce
+a safe approach to these.
+
+This means if you have cases that would violate this rule you would need to change that
+configuration prior to upgrading to 1.3.2
+
+#### AES and SSL plugins are more secure
+
+If you use the AES or SSL plugins you will need to plan your rollout carefully, these plugins
+are not capable of communicating with older versions of MCollective.
+
+#### Changes since 1.3.1
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/11/16|Imrpove error reporting for code errors in application plugins|10883|
+|2011/11/15|The limit method is now configurable on each RPC client as well as the config file|7772|
+|2011/11/15|Add a --graph option to the ping application that shows response distribution|10864|
+|2011/11/14|An ActiveMQ specific connector was added that supports direct connections|7899|
+|2011/11/11|SimpleRPC clients now support native batching with --batch|5939|
+|2011/11/11|The client now unsubscribes from topics when it's idle minimising the risk of receiving missdirected messages|10670|
+|2011/11/09|Security plugins now ignore miss directed messages early thus using fewer resources|10671|
+|2011/10/28|Support ruby-1.9.2-p290 and ruby-1.9.3-rc1|10352|
+|2011/10/27|callerid, certificate names, and identity names can now only have \w . and - in them|10327|
+|2011/10/25|When discovery information is provided always accept it without requiring reset first|10265|
+|2011/10/24|Add :number, :integer and :float to the DDL and rpc application|9902|
+|2011/10/22|Speed up discovery when limit targets are set|10133|
+|2011/10/22|Do not attempt to validate TTL and Message Times on replies in the SSL plugin|10226|
+|2011/10/03|Allow the RPC client to raise an exception rather than exit on failure|9360|
+|2011/10/03|Allow the TTL of requests to be set in the config file and the SimpleRPC API|9399|
+|2011/09/26|Cryptographically secure the TTL and Message Time of requests when using AES and SSL plugins|9400|
+|2011/09/20|Update default shipped configurations to provide a better out of the box experience|9452|
+|2011/09/20|Remove deprecated mc- scripts|9402|
+|2011/09/20|Keep track of messages that has expired and expose the stat in rpcutil and inventory application|9456|
+
+<a name="1_3_1"> </a>
+
+## 1.3.1 - 2011/09/16
+
+This is a release in the development series of MCollective. It feature major new features
+and bug fixes.
+
+This release is for early adopters, production users should consider the 1.2.x series.
+
+### Enhancements
+
+ * Messaging has been completely reworked internally to be more generic and easier to integrate
+ with other middleware
+ * When using Stomp 1.1.9 detailed connection logs are kept showing connections, reconnections
+ and communication errors
+ * A new point to point - but still via the middleware - communications ability has been introduced
+ * When point to point comms is enabled, favour this mode when small number of nodes are being addressed
+ * Add -j to any SimpleRPC client. Clients using _printrpc_ will automatically support a new JSON output format
+ * A new rich discovery language was added using the -S flag
+ * SimpleRPC validators can now also validate boolean data
+ * The default location of _classes.txt_ has changed to be in line with Puppet defaults.
+ * A default TTL of 60 seconds are set on all messages. This is a start towards replay protection and is needed
+ for the new point to point comms style
+ * Discovery is now optional. If you supply an identity filter discovery will be bypassed. Additionally discovery
+ can be supplied in arrays, text or JSON formats. This requires the new point to point comms model.
+
+### Bug Fixes
+
+ * Missing DDL files on the servers are now logged at debug level to minimise noise in the logs
+ * The RC scripts set RUBYLIB, remove this and rely on the operating system to be set up correctly
+ * Invalid fact filters supplied on the CLI now raises an error rather than create empty filters
+
+### New Discovery Language
+
+Previously dicovery was very limited, filters were simply run one after the other and you could not do
+anything complex like a mix of OR and AND boolean logic.
+
+A new compact discovery language was introduced perfect for use on the command line, an example below:
+
+{% highlight console %}
+$ mco find -S "((fqdn=/example.com/ or fqdn=/another.com/) or customer=acme) and apache and physicalprocessorcount>2"
+{% endhighlight %}
+
+The EBNF for this language can be seen below, it's available on the command line and the API
+
+ compound = ["("] expression [")"] {["("] expression [")"]}
+ expression = [!|not]statement ["and"|"or"] [!|not] statement
+ char = A-Z | a-z | < | > | => | =< | _ | - |* | / { A-Z | a-z | < | > | => | =< | _ | - | * | / | }
+ int = 0|1|2|3|4|5|6|7|8|9{|0|1|2|3|4|5|6|7|8|9|0}
+
+### Point to Point comms
+
+Previously MCollective could only broadcast messages and was tied to a discovery model. This is in line
+with the initial goals of the project, having solved that we want to mix in a more traditional messaging
+style.
+
+The messaging layer now supports per node destinations that allows you to address a node, even if its down,
+doesn't yet exist or if you cannot come up with a filter that would match a group of arbitrarily selected
+nodes.
+
+When this mode is in use you tell it using either text, arrays or JSON data which machines to communicate with
+it will then talk directly to those nodes via the middleware and if any of them are down you will get the
+usual no responses report after DDL configured timeout, this is a smooth transparent to the end user mix
+in communication modes.
+
+It is ideal for building deployers, web apps and so forth where you know exactly which nodes should be there
+and you'd like to influence the MCollective network addressing, perhaps from a CMDB you built yourself.
+
+This is the start towards an assured style of delivery, you can consider it the TCP to MCollective's UDP.
+Both modes of communication will be supported in the future and both will have access to all the same agents
+clients etc.
+
+This is feature is still maturing, you enable it using the _direct\_\addressing_ configuration option. At
+present the STOMP connector supports it but it is not optimized for networks larger than 20 to 30 hosts. A
+new connector is being developed that uses ActiveMQ features to achieve this efficiently.
+
+### Pluggable / Optional Discovery
+
+If you did _mco rpc rpcutil ping -I box.example.com -I another.example.com_ mcollective will now just assume
+you know what you want, it won't do a discover to confirm those machines exist or not, it will just go and
+talk with them. This is a big end user visible speed improvement. If however you did a filter like _-I /example.com/_
+it cannot know which machines you want to reach and so a traditional broadcast discovery is done first.
+
+When the direct addressing mode is enabled various behind the scenes optimizations are being done:
+
+ * If a discovery is done and it finds you only want to address 10 or fewer nodes it will use direct mode for that
+ request. This avoids a second needless broadcast. This is less efficient to the middleware but does not send
+ needless messages to uninterested nodes that would then just ignore them.
+ * The _rpc_ application supports piping output from one to the next. Example of this below.
+
+{% highlight console %}
+$ mco rpc package update package=foo -W customer=acme -j|mco rpc service restart service=bar
+{% endhighlight %}
+
+This will update a package on machines matching _customer=foo_ and then restart the service _bar_ on those machines.
+
+The first request is doing traditional discovery based on the fact while the 2nd request is not doing discovery
+at all, it uses the JSON output enabled by -j as discovery data and then restart the service on only those machines.
+
+These abilities are exposed in the SimpleRPC client API and you can write your own schemes, query your own databases etc
+
+### Backwards Compatibility
+
+This is a big release and the entire messaging system has been redesigned, rewritten and has had features added.
+As such there might be problems running mixed 1.2.x and 1.3.1 networks, we'd ask users to test this in lab situations
+and provide us feedback to improve the eventual transition from 1.2.x to 1.4.x. We did though aim to maintain backward
+compatibility and the intention is to fix any bugs reported where a default configured 1.3.x cannot co-habit with a
+previous 1.2.x build.
+
+Enabling the new direct addressing mode is a big configuration change both in your collective and the middleware as such
+soon as you enable it there will be compatibility issues until all your nodes are up to the same level. Specifically old
+nodes will just ignore your direct requests.
+
+The default location for _classes.txt_ has changed to _/var/lib/puppet/state/classes.txt_ you need to ensure
+this file exists or configure either MCollective or Puppet accordingly else your classes filters will break
+
+Messages are now valid for only 60 seconds, nodes will _ignore_ messages older than 60 seconds. This means
+your clocks have to be in sync on your entire collective. We use UTC time for the TTL check so your machines
+can be in different time zones. At present the 60 second threshold is hard coded, it will become configurble on a
+per message basis in future.
+
+#### Changes since 1.3.0
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/09/9|Use direct messaging where possible for identity filters and make the rpc application direct aware|8466|
+|2011/08/29|Enforce a 60 second TTL on all messages by default|8325|
+|2011/08/29|Change the default classes.txt file to be in line with Puppet defaults|9133|
+|2011/08/06|Add reload-agents and reload-loglevel commands to the redhat RC script|7730|
+|2011/08/06|Avoid reloading the authorization class over and over from disk on each request|8703|
+|2011/08/06|Add a boolean validator to SimpleRPC agents|8799|
+|2011/08/06|Justify text results better when using printrpc|8807|
+|2011/07/22|Add --version to the mco utility|7822|
+|2011/07/22|Add missing meta data to the discovery agent|8497|
+|2011/07/18|Raise an error if invalid format fact filters are supplied|8419|
+|2011/07/14|Add a rich discovery query language|8181|
+|2011/07/08|Do not set RUBYLIB in the RC scripts, the OS should do the right thing|8063|
+|2011/07/07|Add a -j argument to all SimpleRPC clients that causes printrpc to produce JSON data|8280|
+|2011/06/30|Add the ability to do point to point comms for requests affecting small numbers of hosts|7988|
+|2011/06/21|Add support for Stomp Gem version 1.1.9 callback based logging|7960|
+|2011/06/21|On the server side log missing DDL files at debug and not warning level|7961|
+|2011/06/16|Add the ability for nodes to subscribe to per-node queues, off by default|7225|
+|2011/06/12|Remove assumptions about middleware structure from the core and move it to the connector plugins|7619|
+
+<a name="1_2_1"> </a>
+
+## 1.2.1 - 2011/06/30
+
+This is a maintenance release in the production series of MCollective and is a recommended
+upgrade for all users of 1.2.0.
+
+### Bug Fixes
+
+ * Improve error handling in the inventory application
+ * Fix compatablity problems with RedHat 4 init scripts
+ * Allow . in Fact names
+ * Allow applications to use the exit method
+ * Correct parsing of the MCOLLECTIVE_EXTRA_OPTS environment variable
+
+### Backwards compatibility
+
+This release should be 100% backward compatable with version 1.2.0
+
+#### Changes since 1.2.0
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/06/02|Correct parsing of MCOLLECTIVE_EXTRA_OPTS in cases where no config related settings were set|7755|
+|2011/05/23|Allow applications to use the exit method as would normally be expected|7626|
+|2011/05/16|Allow _._ in fact names|7532|
+|2011/05/16|Fix compatibility issues with RH4 init system|7448|
+|2011/05/15|Handle failures from remote nodes better in the inventory app|7524|
+|2011/05/06|Revert unintended changes to the Debian rc script|7420|
+|2011/05/06|Remove the _test_ agent that was accidentally checked in|7425|
+
+<a name="1_3_0"> </a>
+
+## 1.3.0 - 2011/06/08
+
+This is a release in the development series of mcollective. It features major
+new features, some bug fixes and internal structure refactoring.
+
+This release is for early adopters, production users should consider the 1.2.x series.
+
+### Enhancements
+
+ * Agents can now programatically declare if they should work on a node
+ * Applications can now use the exit method as normal and clean disconnects will be done
+ * The target collective for registration messages is configurable. In the past it defaulted to main_collective
+
+### Bug Fixes
+
+ * Error reporting in applications, agents and mcolletive core has been improved
+ * The RC script works better on Red Hat 4 based systems
+
+### Other Changes
+
+ * The connector layer is being improved to make it easier to use other middleware.
+ This release starts this process but it's far from complete.
+ * The sshkey plugin was removed from core and moved to the plugins project
+
+### Backwards Compatibility
+
+If you were using the sshkey plugin you need to ensure your CM system is copying it out prior to this
+upgrade as the packages will not contain it anymore.
+
+If you have your own connectors other than the STOMP one we supply you should wait to upgrade till 1.3.1
+at which point you will need to make extensive changes to your plugins internals. If your CM is copying
+out the connector you have to ensure that when this version of MCollective start that the new plugin is
+in place.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/06/07|Exceptions raised during option parsing were not handled and resulted in stack traces|7796|
+|2011/06/06|Remove the sshkey, it's being moved to the plugin repository|7794|
+|2011/06/02|Correct parsing of MCOLLECTIVE_EXTRA_OPTS in cases where no config related settings were set|7755|
+|2011/05/31|Disconnect from the middleware when an application calls exit|7712|
+|2011/05/29|Validations failure in RPC agents will now raise the correct exceptions as documented|7711|
+|2011/05/25|Make the target collective for registration messages configurable|7650|
+|2011/05/24|Rename the connector plugins send method to publish to avoid issues ruby Object#send|7623|
+|2011/05/23|Log a warning when the CF file parsing fails rather than raise a whole ruby exception|7627|
+|2011/05/23|Allow applications to use the exit method as would normally be expected|7626|
+|2011/05/22|Refactor subscribe and unsubscribe so that middleware structure is entirely contained in the connectors|7620|
+|2011/05/21|Add the ability for agents to programatically declare if they should work on a node|7583|
+|2011/05/20|Improve error reporting in the single application framework|7574|
+|2011/05/16|Allow _._ in fact names|7532|
+|2011/05/16|Fix compatibility issues with RH4 init system|7448|
+|2011/05/15|Handle failures from remote nodes better in the inventory app|7524|
+|2011/05/06|Revert unintended changes to the Debian rc script|7420|
+|2011/05/06|Remove the _test_ agent that was accidentally checked in|7425|
+
+<a name="1_2_0"> </a>
+
+## 1.2.0 - 2011/05/04
+
+This is the next production release of MCollective. It brings to an
+end active support for versions 1.1.4 and older.
+
+This release brings to general availability all the features added in the
+1.1.x development series.
+
+### Enhancements
+
+ * The concept of sub-collectives were introduced that help you partition
+ your MCollective traffic for network isolation, traffic management and security
+ * The single executable framework has been introduced replacing the old
+ _mc-\*_ commands
+ * A new AES+RSA security plugin was added that provides strong encryption,
+ client authentication and message security
+ * New fact matching operators <=, >=, <, >, !=, == and =~.
+ * Actions can be written in external scripts and therefore other languages
+ than Ruby, wrappers exist for PHP, Perl and Python
+ * Plugins can now be configured using the _plugins.d_ directory
+ * A convenient and robust exec wrapper has been written to assist in calling
+ external scripts
+ * The _MCOLLECTIVE\_EXTRA\_OPTS_ environment variable has been added that will
+ add options to all client scripts
+ * Network timeout handling has been improved to better take account of latency
+ * Registration plugins can elect to skip sending of registration data by
+ returning _nil_, previously nil data would be published
+ * Multiple libdirs are supported
+ * The logging framework is pluggable and easier to use
+ * Fact plugins can now force fact cache invalidation. The YAML plugin will
+ force a cache clear as soon as the source YAML file updates
+ * The _ping_ application now supports filters
+ * Network payload can now be Base64 encoded avoiding issues with Unicode characters
+ in older Stomp gems
+ * All fact plugins are now cached and only updated every 300 seconds
+ * The progress bar now resizes based on terminal dimensions
+ * DDL files with missing output blocks will not invalidate the whole DDL
+ * Display of DDL assisted complex data has been improved to be more readable
+ * Stomp messages can have a priority header added for use with recent versions
+ of ActiveMQ
+ * Almost 300 unit tests have been written, lots of old code and any new code being
+ written is subject to continuos testing on Ruby 1.8.5, 1.8.6 and 1.9.2
+ * Improved the Red Hat RC script to be more compliant with distribution policies
+ and to reuse the builtin functions
+
+### Deprecations and removed functionality
+
+ * The old _mc-\*_ commands are being removed in favor for the new _mco_ command.
+ The old style is still available and your existing scripts will keep working but
+ porting to the new single executable system is very easy and encouraged.
+ * _MCOLLECTIVE_TIMEOUT_ and _MCOLLECTIVE_DTIMEOUT_ were removed in favor of _MCOLLECTIVE\_EXTRA\_OPTS_
+ * _mc-controller_ could exit all mcollectived instances, this feature was not ported
+ to the new _mco controller_ application
+
+### Bug Fixes
+
+ * mcollectived and all of the standard supplied client scripts now disconnects
+ cleanly from the middleware avoiding exceptions in the ActiveMQ logs
+ * Communications with the middleware has been made robust by adding a timeout
+ while sending
+ * Machines that do not pass security validation are now handled as having not
+ responded at all
+ * When a fire and forget request was sent, replies were still sent, they are
+ now suppressed
+
+### Backwards compatibility
+
+This release can communicate with machines running older versions of mcollective
+there are though a few steps to take to ensure a smooth upgrade.
+
+#### Backward compatible sub-collective setup
+
+{% highlight ini %}
+topicprefix = /topic/mcollective
+{% endhighlight %}
+
+This has to change to:
+
+{% highlight ini %}
+topicprefix = /topic/
+main_collective = mcollective
+collectives = mcollective
+{% endhighlight %}
+
+#### Security Plugins
+
+The interface for the _encodereply_ method on the security plugins have changed
+if you are using any of the community plugins or wrote your own you should update
+them with the latest at the time you upgrade to 1.2.0
+
+#### Fact Plugins
+
+The interface to the fact plugins have been greatly simplified, this means you need to
+update to new plugins at the time you upgrade to 1.2.0
+
+You can place these new plugins into the plugindir before upgrading. The old mcollective
+will not use these plugins and the new one will not touch the old ones. This will allow
+for a clean rollback.
+
+Once the new version is deployed you will immediately have caching on all fact types
+at 300 seconds you can tune this using the fact_cache_time setting in the configuration file.
+
+#### New fact selectors
+
+The new fact selectors are only available on newer versions of mcollective. If a client
+attempts to use them and an older version of the server is on the network those older
+servers will treat all fact lookups as ==
+
+#### Changes since 1.1.4
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/05/03|Improve Red Hat RC script by using distro builtin functions|7340|
+|2011/05/01|Support setting a priority on Stomp messages|7246|
+|2011/04/30|Handle broken and incomplete DDLs better and improve the format of DDL output|7191|
+|2011/04/23|Encode the target agent and collective in requests|7223|
+|2011/04/20|Make the SSL Cipher used a config option|7191|
+|2011/04/20|Add a clear method to the PluginManager that deletes all plugins, improve test isolation|7176|
+|2011/04/19|Abstract the creation of request and reply hashes to simplify connector plugin development|5701|
+|2011/04/15|Improve the shellsafe validator and add a Util method to do shell escaping|7066|
+|2011/04/14|Update Rakefile to have a mail_patches task|6874|
+|2011/04/13|Update vendored systemu library for Ruby 1.9.2 compatibility |7067|
+|2011/04/12|Fix failing tests on Ruby 1.9.2|7067|
+|2011/04/11|Update the DDL documentation to reflect the _mco help_ command|7042|
+|2011/04/11|Document the use filters on the CLI|5917|
+|2011/04/11|Improve handling of unknown facts in Util#has_fact? to avoid exceptions about nil#clone|6956|
+|2011/04/11|Correctly set timeout on the discovery agent to 5 seconds as default|7045|
+|2011/04/11|Let rpcutil#agent_inventory supply _unknown_ for missing values in agent meta data|7044|
+
+<a name="1_1_4"> </a>
+
+## 1.1.4 - 2011/04/07
+
+This is a release in the development series of mcollective. It features major
+new features and some bug fixes.
+
+This release is for early adopters, production users should consider the 1.0.x series.
+
+### Actions in other languages
+
+We have implemented the ability to write actions in languages other than Ruby.
+This is done via simple JSON API documented in [in our docs](simplerpc/agents.html#actions-in-external-scripts)
+
+The _ext_ directory on [GitHub](https://github.com/puppetlabs/marionette-collective/tree/master/ext/action_helpers)
+hosts wrappers for PHP, Perl and Python that makes using this interface easier.
+
+{% highlight ruby %}
+action "test" do
+ implemented_by "/some/external/script"
+end
+{% endhighlight %}
+
+Special thanks to the community members who contributed the wrappers.
+
+### Enhancements
+
+ * Actions can now be written in any language
+ * Plugin configuration can be kept in _/etc/mcollective/plugin.d_
+ * _mco inventory_ now shows collective and sub-collective membership
+ * mc-controller has been deprecated for _mco controller_
+ * Agents are now ran using new instances of the classes rather than reuse the exiting
+ one to avoid concurrency related problems
+
+### Bug Fixes
+
+ * When mcollectived exits it now cleanly disconnects from the Middleware
+ * The _rpcutil_ agent is less strict about valid Fact names
+ * The default configuration files have been updated for sub-collectives
+
+### Backwards Compatibility
+
+This release will be backward compatible with version _1.1.3_ for compatibility
+with earlier releases see the notes for _1.1.3_ and the sub collective related
+configuration changes.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/03/28|Correct loading of vendored JSON gem|6877|
+|2011/03/28|Show collective and sub collective info in the inventory application|6872|
+|2011/03/23|Disconnect from the middleware when mcollectived disconnects|6821|
+|2011/03/21|Update rpcutil ddl file to be less strict about valid fact names|6764|
+|2011/03/22|Support reading configuration from configfir/plugin.d for plugins|6623|
+|2011/03/21|Update default configuration files for sub-collectives|6741|
+|2011/03/16|Add the ability to implement actions using external scripts|6705|
+|2011/03/15|Port mc-controller to the Application framework and deprecate the exit command|6637|
+|2011/03/13|Only cache registration and discovery agents, handle the rest as new instances|6692|
+|2011/03/08|PluginManager can now create new instances on demand for a plugin type|6622|
+
+<a name="1_1_3"> </a>
+
+## 1.1.3 - 2011/03/07
+
+This is a release in the development series of mcollective. It features major
+new features and some bug fixes.
+
+This release is for early adopters, production users should consider the 1.0.x series.
+
+### Enhancements
+
+ * Add the ability to partition collectives into sub-collectives for security and
+ network traffic management
+ * Add a exec wrapper for agents that provides unique environments and cwds in a
+ thread safe manner as well as avoid zombie processes
+ * Automatically pass Application options to rpcclient when options are not
+ specifically provided
+ * Rename _/usr/sbin/mc_ to _/usr/bin/mco_
+
+### Bug Fixes
+
+ * Missing _libdirs_ will not cause crashes anymore
+ * Parse _MCOLLECTIVE\_EXTRA\_OPTS_ correctly with multiple options
+ * _file`_`logger_ failures are handled better
+ * Improve middleware communication in unreliable settings by adding timeouts
+ around middleware operations
+
+### Backwards Compatibility
+
+The configuration format has changed slightly to accomodate the concept of
+collective names and sub-collectives.
+
+In older releases the configuration was:
+
+{% highlight ini %}
+topicprefix = /topic/mcollective
+{% endhighlight %}
+
+This has to change to:
+
+{% highlight ini %}
+topicprefix = /topic/
+main_collective = mcollective
+collectives = mcollective
+{% endhighlight %}
+
+When setup as above a old and new version will be compatible but as soon as you
+start configuring the new sub-collective feature you will loose compatiblity
+between versions.
+
+Various defaults apply, if you configure it with these exactly topic and
+collective names you can leave off the _main`_`collective_ and _collectives_
+directives as the above settings would be their defaults
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/03/04|Rename /usr/sbin/mc to /usr/bin/mco|6578|
+|2011/03/01|Wrap rpcclient in applications ensuring that options is always set|6308|
+|2011/02/28|Make communicating with the middleware more robust by including send calls in timeouts|6505|
+|2011/02/28|Create a wrapper to safely run shell commands avoiding zombies|6392|
+|2011/02/19|Introduce Sub-collectives for network partitioning|5967|
+|2011/02/19|Improve error handling when parsing arguments in the rpc application|6388|
+|2011/02/19|Fix error logging when file_logger creation fails|6387|
+|2011/02/17|Correctly parse MCOLLECTIVE\_EXTRA\_OPTS in the new unified binary framework|6354|
+|2011/02/15|Allow the signing key and Debian distribution to be customized|6321|
+|2011/02/14|Remove inadvertently included package.ddl|6313|
+|2011/02/14|Handle missing libdirs without crashing|6306|
+
+<a name="1_0_1"> </a>
+
+## 1.0.1 - 2011/02/16
+
+### Release Focus and Notes
+
+This is a minor bug fix release.
+
+### Bugs Fixed
+
+ * The YAML fact plugin failed to remove deleted facts from memory
+ * The _-_ character is now allowed in Fact names for the rpcutil agent
+ * Machines that fali security validations were not reported correctly,
+ they are now treated as having not responded at all
+ * Timeouts on RPC requests were too aggressive and did not allow for slow networks
+
+### Backwards Compatibility
+
+This release will be backward compatible with older releases.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/02/02|Include full Apache 2 license text|6113|
+|2011/01/29|The YAML fact plugin kept deleted facts in memory|6056|
+|2012/01/04|Use the LSB based init script on SUSE|5762|
+|2010/12/30|Allow - in fact names|5727|
+|2010/12/29|Treat machines that fail security validation like ones that did not respond|5700|
+|2010/12/25|Allow for network and fact source latency when calculating client timeout|5676|
+|2010/12/25|Increase the rpcutil timeout to allow for slow facts|5679|
+
+## 1.1.2 - 2011/02/14
+
+This is a release in the development series of mcollective. It features minor
+bug fixes and features.
+
+This release is for early adopters, production users should consider the 1.0.x series.
+
+### Bug Fixes
+
+ * The main fix in this release is a packaging bug in Debian systems that prevented
+ both client and server from being installed on the same machine.
+ * Backwards compatibility fix for fact filters that are empty strings
+
+### Enhancement
+
+ * Registration plugins can now return nil which will skip that specific registration
+ message. This will enable plugins to decide based on some node state if a message
+ should be sent or not.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/02/13|Surpress replies to SimpleRPC clients who did not request results|6305|
+|2011/02/11|Fix Debian packaging error due to the same file in multiple packages|6276|
+|2011/02/11|The application framework will now disconnect from the middleware for consistency|6292|
+|2011/02/11|Returning _nil_ from a registration plugin will skip registration|6289|
+|2011/02/11|Set loglevel to warn by default if not specified in the config file|6287|
+|2011/02/10|Fix backward compatibility with empty fact strings|6278|
+
+## 1.1.1 - 2011/02/02
+
+This is a release in the development series of mcollective. It features major new
+features and numerous bug fixes. Please pay careful attention to the upgrading
+section as there is some changes that are not backward compatible.
+
+This release is for early adopters, production users should consider the 1.0.x series.
+
+### AES+RSA Security Plugin
+
+A new security plugin that encrypts the payloads, uniquely identify senders and secure
+replies from inspection by other people on the collective has been written. The plugin
+can re-use Puppet certificates and supports distributing of public keys if you wish.
+
+This plugin and its deployment is very complex and it has a visible performance impact
+but we felt it was a often requested feature and so decided to implement it.
+
+Full documentation for this plugin can be found [in our docs](reference/plugins/security_aes.html), please read them very
+carefully should you choose to deploy this plugin.
+
+### Single Executable Framework
+
+In the past a lot of the CLI tools have behaved inconsistently as the mc scripts were
+mostly just written to serve immediate needs, we are starting a process of improving
+these scripts and making them more robust.
+
+The first step is to create a new framework for CLI commands, we call these Single Executable
+Applications. A new executable called _mc_ is being distributed with this release:
+
+{% highlight console %}
+$ mc
+The Marionette Collective version 1.1.1
+
+/usr/sbin/mc: command (options)
+
+Known commands: rpc filemgr inventory facts ping find help
+{% endhighlight %}
+
+{% highlight console %}
+$ mc help
+The Marionette Collection version 1.1.1
+
+ facts Reports on usage for a specific fact
+ filemgr Generic File Manager Client
+ find Find hosts matching criteria
+ help Application list and RPC agent help
+ inventory Shows an inventory for a given node
+ ping Ping all nodes
+ rpc Generic RPC agent client application
+{% endhighlight %}
+
+{% highlight console %}
+$ mc rpc package status package=zsh
+Determining the amount of hosts matching filter for 2 seconds .... 51
+
+ * [ ============================================================> ] 51 / 51
+
+
+ test.com:
+ Properties:
+ {:provider=>:yum,
+ :release=>"3.el5",
+ :arch=>"x86_64",
+ :version=>"4.2.6",
+ :epoch=>"0",
+ :name=>"zsh",
+ :ensure=>"4.2.6-3.el5"}
+{% endhighlight %}
+
+You can see these commands behave just like their older counter parts but is more integrated
+and discovering available commands is much easier.
+
+Agent help that was in the past available through _mc-rpc --ah agentname_ is now available through
+_mc help agentname_ and error reporting is short single line reports by default but by adding
+_-v_ to the command line you can get full Ruby backtraces.
+
+We've maintained backward compatibility by creating wrappers for all the old mc scripts but these
+might be deprecated in future.
+
+These application live in the normal plugin directories and should make it much easier to distribute
+plugins in future.
+
+We will port the scripts for plugins to this framework and encourage you to do the same when writing
+new CLI tools. An example of a ported CLI can be seen in the _filemgr_ agent.
+
+Find the documentation for these plugins [here](reference/plugins/application.html).
+
+### Miscellaneous Improvements
+
+The logging system has been ra-efactored to not use a Signleton, logging messages are now simply:
+
+{% highlight ruby %}
+MCollective::Log.notice("hello world")
+{% endhighlight %}
+
+A backwards compatible wrapper exist to prevent existing code from breaking.
+
+In some cases - like when using MCollective from within Rails - the STOMP
+gem would fail to decode the payloads. We've worked with the authors and
+a new release was made that makes this more robust but we've also enabled
+Base64 encoding on the Stomp connector for those who can't upgrade the Gem
+and who are running into this problem.
+
+### Bug Fixes
+
+
+ * Machines that do not pass security checks are handled as having not responded
+ so that these are listed in the usual stat for non responsive hosts
+ * The - character is now allowed in Fact names by the DDL for rpcutil
+ * Version 1.1.0 introduced a bug with reloading agents from disks using USR1 and mc-controller
+
+### Enhancements
+
+ * New AES+RSA based security plugin was added
+ * Create a new single executable framework and port several mc scripts
+ * Security plugins have access to the callerid they are responding to
+ * The logging methods have been improved by removing the use of Singletons
+ * The STOMP connector can now Base64 encode all sent data to deal with en/decoding issues by the gem
+ * The rpcutil agent has a new _ping_ action
+ * the _mc ping_ client now supports standard filters
+ * DDL documentation has been updated to show you can disable type validations in the DDL
+ * Fact plugins can now force fact cache invalidation, the YAML plugin will immediately load new facts when mtime on the file change
+ * Improve _1.0.0_ compatibility for _foo=/bar/_ style fact matches at the expense of _1.1.0_ compatibility
+
+### Upgrading
+
+Upgrading should be mostly painless as most things are backward compatible.
+
+We discovered that we broke backward compatibility with _1.0.0_ and _0.4.x_ Fact filters. A filter in the form
+_foo=/bar/_ would be treated as an equality filter and not a regular expression.
+
+This releases fixes this compatibility with older versions at the expense of compatibility with _1.1.0_. If you
+are upgrading from _1.1.0_ keep this in mind and plan accordingly, once you've upgraded a client its requests that
+contain these filters will not be correctly parsed on servers running _1.1.0_.
+
+The security plugins have changed slightly, if you wrote your own security plugin the interface to _encodereply_
+has changed slightly. All the bundled security plugins have been updated already and older ones will just
+keep working.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2011/02/02|Load the DDL from disk once per printrpc call and not for every result|5958|
+|2011/02/02|Include full Apache 2 license text|6113|
+|2011/01/31|Create a new single executable application framework|5897|
+|2011/01/30|Fix backward compatibility with old foo=/bar/ style fact searches|5985|
+|2011/01/30|Documentation update to reflect correct default identity behavior|6073|
+|2011/01/29|Let the YAML file force fact reloads when the files update|6057|
+|2011/01/29|Add the ability for fact plugins to force fact invalidation|6057|
+|2011/01/29|Document an approach to disable type validation in the DDL|6066|
+|2011/01/19|Add basic filters to the mc-ping command|5933|
+|2011/01/19|Add a ping action to the rpcutil agent|5937|
+|2011/01/17|Allow MC::RPC#printrpc to print single results|5918|
+|2011/01/16|Provide SimpleRPC style results when accessing the MC::Client results directly|5912|
+|2011/01/11|Add an option to Base64 encode the STOMP payload|5815|
+|2011/01/11|Fix a bug with forcing all facts to be strings|5832|
+|2011/01/08|When using reload_agents or USR1 signal no agents would be reloaded|5808|
+|2011/01/04|Use the LSB based init script on SUSE|5762|
+|2011/01/04|Remove the use of a Singleton in the logging class|5749|
+|2011/01/02|Add AES+RSA security plugin|5696|
+|2010/12/31|Security plugins now have access to the callerid of the message they are replying to|5745|
+|2010/12/30|Allow - in fact names|5727|
+|2010/12/29|Treat machines that fail security validation like ones that did not respond|5700|
+
+## 1.1.0 - 2010/12/29
+
+This is the first in a new development series, as such there will be rapid changes
+and new features. We cannot guarantee the changes will be backward compatible but
+we will as before try to keep these releases solid and production quality.
+
+Production users who do not wish to have rapid change should use release 1.0.0.
+
+This release focus mainly on getting all the community contributed code into a release
+and addressing some issues I had but wasn't comfortable fixing them late in the
+previous development series.
+
+Please read these notes carefully we are **removing** some old functionality and changing
+some internals, you need to carefully review the text below.
+
+### Bug Fixes
+
+ * The progress bar will now try hard to detect screen size and adjust itself,
+ failing back to a dumb mode if it can't work it out.
+ * rpcutil timeout was too short when considering slow facts and network latency
+
+### Improvements
+
+ * libdir can now be multiple directories specified with : separation - Thanks to Richard Clamp
+ * Logging is now pluggable, 3 logger types are supported - file, syslog and console. Thanks to
+ Nicolas Szalay for the initial Syslog code
+ * A new experimental ssh agent based security system. Thanks to Jordan Sissel
+ * New fact matching operators <=, >=, <, >, !=, == and =~. Thanks to Mike Pountney
+ * SimpleRPC fact_filter method can now take any valid fact string as input in addition to the old format
+ * A message gets logged at startup showing mcollective version and logging level
+ * The fact plugin format has been changed, simplified, made thread safe and global caching added.
+ This breaks backward compatibility with old fact sources
+ * Creating options hashes has been simplified by adding a helper that creates them for you
+ * Calculating the client timeout has been improved by including for latency and fact source slowness
+ * Audit log lines are now on one line and include a ISO 8601 format date
+
+### Removed Functionality
+
+ * The old MCOLLECTIVE_TIMEOUT and MCOLLECTIVE_DTIMEOUT were removed, a new MCOLLECTIVE_EXTRA_OPTS
+ was added which should allow much more flexibility. Supply any command line options in this var
+
+### Upgrading
+
+Upgrading should be easy the only backward incompatible change is the Facts format. If you only use
+the included YAML plugin the upgrade will just work if you use the packages. If you use either the
+facter or ohai plugins you will need to download new plugins from the community plugin page.
+
+If you wrote your own Facts plugin you will need to change it a bit:
+
+ * The old get_facts method should now be load_facts_from_source
+ * The class for facts have to be in the form MCollective::Facts::Foo_facts and the filename should match
+
+This is all, your facts can now be much simpler as threading and caching is handled in the base class.
+
+You can place these new plugins into the plugindir before upgrading. The old mcollective will not use
+these plugins and the new one will not touch the old ones. This will allow for a clean rollback.
+
+Once the new version is deployed you will immediately have caching on all fact types at 3000 seconds
+you can tune this using the fact_cache_time setting in the configuration file.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2010/12/28|Adjust the logfile audit format to include local time and all on one line|5694|
+|2010/12/26|Improve the SimpleRPC fact_filter helper to support new fact operators|5678|
+|2010/12/25|Increase the rpcutil timeout to allow for slow facts|5679|
+|2010/12/25|Allow for network and fact source latency when calculating client timeout|5676|
+|2010/12/25|Remove MCOLLECTIVE_TIMEOUT and MCOLLECTIVE_DTIMEOUT environment vars in favor of MCOLLECTIVE_EXTRA_OPTS|5675|
+|2010/12/25|Refactor the creation of the options hash so other tools don't need to know the internal formats|5672|
+|2010/12/21|The fact plugin format has been changed and simplified, the base now provides caching and thread safety|5083|
+|2010/12/20|Add parameters <=, >=, <, >, !=, == and =~ to fact selection|5084|
+|2010/12/14|Add experimental sshkey security plugin|5085|
+|2010/12/13|Log a startup message showing version and log level|5538|
+|2010/12/13|Add a console logger|5537|
+|2010/12/13|Logging is now pluggable and a syslog plugin was provided|5082|
+|2010/12/13|Allow libdir to be an array of directories for agents and ddl files|5253|
+|2010/12/13|The progress bar will now intelligently figure out the terminal dimensions|5524|
+
+## 1.0.0 - 2010/12/13
+
+### Release Focus and Notes
+
+This is a bug fix and minor improvement release.
+
+We will maintain the 1.0.x branch as a stable supported branch. The features
+currently in the branch will be frozen and we'll only do bug fixes.
+
+A new 1.1.x series of releases will be done where we will introduce new features.
+Once the 1.1.x code base reaches a mature point it will become the new stable
+release and so forth.
+
+### Bug Fixes
+
+ * Settings like retry times were ignored in the Stomp connector
+ * The default init script had incorrect LSB comments
+ * The rpcutil DDL has better validation and will now match all facts
+
+### New Features and Enhancements
+
+ * You can now send RPC requests to a subset of discovered nodes
+ * SimpleRPC custom_request can now be used to create fire and forget requests
+ * Clients can now cleanly disconnect from the middleware. Bundled clients have been
+ updated. This should cause fewer exceptions in ActiveMQ logs
+ * Rather than big exceptions many clients will now log errors only
+ * mc-facts has been reworked to be a SimpleRPC client, this speeds it up significantly
+ * Add get_config_item to rpcutil to retrieve a running config value from a server
+ * YAML facts are now forced to be all strings and is thread safe
+ * On RedHat based systems the requirement for the LSB packages has been removed
+
+The first feature is a major new feature in SimpleRPC. If you expose a service redundantly
+on your network using MCollective you wouldn't always want to send requests to all the
+nodes providing the service. You can now limit the requests to an arbitrary amount
+using the new --limit-nodes option which will also take a percentage. A shortcut -1 has
+been added that is the equivalent to --limit-nodes 1
+
+### Backwards Compatibility
+
+This release will be backward compatible with older releases.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2010/12/04|Remove the LSB requirements for RedHat systems|5451|
+|2010/11/23|Make the YAML fact source thread safe and force all facts to strings|5377|
+|2010/11/23|Add get_config_item to rpcutil to retrieve a running config value from a server|5376|
+|2010/11/20|Convert mc-facts into a SimpleRPC client|5371|
+|2010/11/18|Added GPG signing to Rake packaging tasks (SIGNED=1)|5355|
+|2010/11/17|Improve error messages from clients in the case of failure|5329|
+|2010/11/17|Add helpers to disconnect from the middleware and update all bundled clients|5328|
+|2010/11/16|Correct LSB provides and requires in default init script|5222|
+|2010/11/16|Input validation on rpcutil has been improved to match all valid facts|5320|
+|2010/11/16|Add the ability to limit the results to a subset of hosts|5306|
+|2010/11/15|Add fire and forget mode to SimpleRPC custom_request|5305|
+|2010/11/09|General connection settings to the Stomp connector was ignored|5245|
+
+## 0.4.10 - 2010/10/18
+
+### Release Focus and Notes
+
+This is a bug fix and minor improvement release.
+
+### Bug Fixes
+
+ * Multiple RPC proxy classes in the same script would not all share the same command line options
+ * Ruby 1.9.x compatibility has been improved
+ * A major bug in registration has been fixed, any exception in the registration logic would have
+ resulted in a high CPU consuming loop
+
+The last bug is a major issue it will result in the _mcollectived_ consuming lots of CPU, updating to
+this version of MCollective is strongly suggested. Should you run into this problem on a large scale
+you can use _mc-controller exit_ to exit all your _mcollectived_ processes at the same time.
+
+### New Features and Enhancements
+
+ * The PSK security plugin can now be configured to set the callerid to a few different values
+ useful for cases where you want to do group based RPC Authorization for example.
+ * Info logging has been minimised by demoting the 'not targeted at us' message to debug
+ * Document the 'exit' option to mc-controller
+
+### Backwards Compatibility
+
+This release will be backward compatible with older releases.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2010/10/18|Document exit command to mc-controller|152|
+|2010/10/13|Log messages that don't pass the filters at debug level|149|
+|2010/10/03|Preserve options in cases where RPC::Client instances exist in the same program|148|
+|2010/09/30|Add the ability to set different types of callerid in the PSK plugin|145|
+|2010/09/30|Improve Ruby 1.9.x compatibility|142|
+|2010/09/29|Improve error handling in registration to avoid high CPU usage loops|143|
+
+
+## 0.4.9 - 2010/09/21
+
+### Release Focus and Notes
+
+This is a bug fix and minor improvement release.
+
+### Bug Fixes
+
+ * Internal data structure related to Agent meta data has been fixed, no user impact from this
+ * When using per-user config files the _rpc-help.erb_ template could not be found
+ * The log files will now rotate by default keeping 5 x 2MB files
+ * The config were parsed multiple times in complex scripts, this has been eliminated
+ * MCollective::RPC loaded a bunch of unneeded stuff into Object, this has been cleaned up
+ * Various packaging related tweaks were done
+
+### New Features
+
+ * We ship a new agent called _rpcutil_ with the base system, you can use this agent to get inventory etc from your _mcollectived_. _mc-inventory_ has been rewritten to use this agent and should serve as a good reference for what you can get from the agent.
+ * The DDL now support :boolean style inputs, mc-rpc also turn true/false on the command line into booleans when needed
+
+### Backwards Compatibility
+
+This release will be backward compatible with older releases.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2010/09/20|Improve Debian packaging task|140|
+|2010/09/20|Add :boolean type support to the DDL|138|
+|2010/09/19|Refactor MCollective::RPC to add less unneeded stuff to Object|137|
+|2010/09/18|Prevent duplicate config loading with multiple clients active|136|
+|2010/09/18|Rotate the log file by default, keeping 5 x 2MB files|135|
+|2010/09/18|Write a overview document detailing security of the collective|131|
+|2010/09/18|Add MCollective.version, set it during packaging and include it in the rpcutil agent|134|
+|2010/09/13|mc-inventory now use SimpleRPC and the rpcutil agent and display server stats|133|
+|2010/09/13|Make the path to the rpc-help.erb configurable and set sane default|130|
+|2010/09/13|Make the configfile used available in the Config class and add to rpcutil|132|
+|2010/09/12|Rework internal statistics and add a rpcutil agent|129|
+|2010/09/12|Fix internal memory structures related to agent meta data|128|
+|2010/08/24|Update the OpenBSD port for changes in 0.4.8 tarball|125|
+|2010/08/23|Fix indention/block error in M:R:Stats|124|
+|2010/08/23|Fix permissions in the RPM for files in /etc|123|
+|2010/08/23|Fix language in two error messages|122|
+
+## 0.4.8 - 2010/08/20
+
+### Release Focus and Notes
+
+This is a bug fix and minor improvement release.
+
+### Bug Fixes
+
+ * The RPM packages now require redhat-lsb since our RC scripts need it
+ * The rake tasks do not attempt to build rpms on all platforms
+ * Some plugin missing related exceptions are now handled gracefully
+ * The Rakefile had a few warnings cleaned up
+
+### Notable New Features
+
+ * Users can now have a _~/.mcollective_ file which will be preferred over over _/etc/mcollective/client.cfg_ if it exists. You can still use _--config_ to override.
+
+ * The SSL Security plugin can now use "either YAML or Marshal for serialization":/reference/plugins/security_ssl.html#serialization_method, this means other programming languages can be used as clients. A sample Perl client is included in the ext directory. Marshal remains the default for backwards compatibility
+
+ * _mc-inventory_ can now be used to create "custom reports using a small reporting DSL":/reference/ui/nodereports.html, this enable you to build custom reports listing all your machines and gives you access to facts, agents and classes lists.
+
+ * The log level for the _mcollectived_ can be adjusted during run time using the _USR2_ unix process signal.
+
+### Backwards Compatibility
+
+This release will be backward compatible with older releases. If you choose to use YAML in the SSL plugin you need matching versions on the client.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2010/08/19|Fix missing help template in debian packages|90|
+|2010/08/18|Clean up some hardlink warnings in the Rakefile|117|
+|2010/08/18|Include the website in the main repo and add a simple Rake task|118|
+|2010/08/17|Handle exceptions for missing plugins better|115|
+|2010/08/17|Add support for ~/.mcollective as a config file|114|
+|2010/08/07|SSL security plugin can use either YAML or Marshal|94|
+|2010/08/06|Multiple YAML files can now be used as fact source|112|
+|2010/08/06|Allow log level to be adjusted at run time with USR2|113|
+|2010/07/31|Add basic report scripting support to mc-inventory|111|
+|2010/07/06|Removed 'rpm' from the default rake task|109|
+|2010/07/06|Add redhat-lsb to the server RPM dependencies|108|
+
+## 0.4.7 - 2010/06/29
+
+### Release Focus and Notes
+
+This is a bug fix and incremental improvement release focusing on small improvements in the DDL mostly.
+
+### Data Definition Language
+
+We've extended the use of the DDL in the RPC client. We've integrated the DDL into _printrpc_ helper. The output is dynamic showing field names in human readable format rather than hash dumps.
+
+We're also using color to improve the display of the results, the color display can be disabled with the new _color_ configuration option.
+
+A "screencast of the DDL integration":http://mcollective.blip.tv/file/3799653 and color usage has been recorded.
+
+### Bug Fixes
+
+A serious issue has been fixed with regard to complex agents. If you attempted to use multiple agents from the same script errors such as duplicate discovery results or simply not working.
+
+The default fact source has been changed to YAML, it was inadvertently set to Facter in the past.
+
+Some previously unhandled exceptions are now being handled correctly and passed onto the clients as failed requests rather than no responses at all.
+
+### Backwards Compatibility
+
+This release will be backward compatible with older releases. The change to YAML fact source by default might impact you if you did not previously specify a fact source in the configuration files.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+| 2010/06/27 | Change default factsource to YAML|106|
+| 2010/06/27 | Added VIM snippets to create DDLs and Agents|102|
+| 2010/06/26 | DDL based help now works better with Symbols in in/output|105|
+| 2010/06/23 | Whitespace at the end of config lines are now stripped|100|
+| 2010/06/22 | printrpc will now inject some colors into results|99|
+| 2010/06/22 | Recover from syntax and other errors in agents|98|
+| 2010/06/17 | The agent a MC::RPC::Client is working on is now available|97|
+| 2010/06/17 | Integrate the DDL with data display helpers like printrpc|92|
+| 2010/06/15 | Avoid duplicate topic subscribes in complex clients|95|
+| 2010/06/15 | Catch some unhandled exceptions in RPC Agents|96|
+| 2010/06/15 | Fix missing help template file from RPM|90|
+
+## 0.4.6 - 2010/06/14
+
+### Release Focus and Notes
+
+This release is a major feature release.
+
+We're focusing mainly on the Stomp connector and on the SimpleRPC agents in this release though a few smaller additions were made.
+
+### Stomp Connector
+
+We've historically been stuck just using RubyGem Stomp 1.1 due to multi threading bugs in the newer releases. All attempts to contact the authors failed. Recently though I had some luck and these issues are resolved in the RubyGem Stomp 1.1.6 release.
+
+This means we can take advantage of a lot of new features such as connection pooling for failover/ha and also SSL TLS between nodes and ActiveMQ server.
+
+See "Stomp Connector":/reference/plugins/connector_stomp.html for details.
+
+### RPC Agent Data Description Language
+
+I've been working since around February on building introspection, automatically generated documentation and the ability for user interfaces to be auto generated for agents, even ones you write your self.
+
+This feature is documented in "DDL":/simplerpc/ddl.html but a quick example of a DDL document might help make it clear:
+
+### CLI Utilities changes
+
+ * _mc-facts_ now take all the standard filters so you can make reports for just subsets of machines
+ * A new utility _mc-inventory_ has been added, it will show agents, facts and classes for a node
+ * _mc-rpc_ has a new option _--agent-help_ that will use the DDL and display auto generated documentation for an agent.
+ * _mc-facts_ output is sorted for better readability
+
+### Backwards Compatibility
+
+This release will be backward compatible with older releases, the new Stomp features though require the newer Ruby Gem.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+| 2010/06/12 | Qualify the Process class to avoid clashes in the discovery agent|88|
+| 2010/06/12 | Add mc-inventory which shows agents, classes and facts for a node|87|
+| 2010/06/11 | mc-facts now supports standard filters|86|
+| 2010/06/11 | Add connection pool retry options and SSL for connection|85|
+| 2010/06/11 | Add support for specifying multiple stomp hosts for failover|84|
+| 2010/06/10 | Tighten up handling of filters to avoid nil's getting into them|83|
+| 2010/06/09 | Sort the mc-facts output to be more readable|82|
+| 2010/06/08 | Fix deprecation warnings in newer Stomp gems|81|
+
+## 0.4.5 - 2010/06/03
+
+### Release Focus and Notes
+
+This release is a major feature release.
+
+The focus of this release is to finish up some of the more enterprise like features, we now have fine grained Authorization and Authentication and a new security model that uses SSL keys.
+
+### Security Plugin
+
+Vladimir Vuksan contributed the base code of a new "SSL based security plugin":/reference/plugins/security_ssl.html . This plugin builds on the old PSK plugin but gives each client a unique certificate pair. The nodes all share a certificate and only store client public keys. This means that should one node be compromised it cannot be used to control the rest of the network.
+
+### Authorization Plugin
+
+We've developed new hooks and plugins for SimpleRPC that enable you to write "fine grained authorization systems":/simplerpc/authorization.html . You can do authorization on every aspect of the request and you'll have access to the caller userid as provided by the security plugin. Combined with the above SSL plugin this can be used to build powerful and secure Authentication and Authorization systems.
+
+A sample plugin can be found "here":http://code.google.com/p/mcollective-plugins/wiki/ActionPolicy
+
+### Enhancements for Web Development
+
+Web apps doesn't tie in well with our discover, request, wait model. We've made it easier for web apps to maintain their own cached discovery data using the "Registration:/reference/plugins/registration.html system and then based on that do requests that would not require any discovery.
+
+### Fire and Forget requests
+
+It is often desirable to just send a request and not wait for any reply. We've made it easy to do requests like this] with the addition of a new request parameter on the SimpleRPC client class.
+
+Requests like this will not take any time to do discovery and you will not be able to get results back from the agents.
+
+### Reloading Agents
+
+To make it a bit easier to manage daemons and agents you can now send the _mcollectived_ a _USR1_ signal and it will re-read all it's agents from disk.
+
+### Backwards Compatibility
+
+This release when used with the old style PSK plugin should be perfectly backward compatible with your existing agents. To use some of the newer features like authorization will require config and/or agent changes.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+| 2010/06/01 | Improve the main discovery agent by adding facts and classes to its inventory action|79|
+| 2010/05/30 | Add various helpers to get reports as text instead of printing them|43|
+| 2010/05/30 | Add a custom_request method to call SimpleRPC agents with your own discovery|75|
+| 2010/05/30 | Refactor RPC::Client to be more generic and easier to maintain|75|
+| 2010/05/29 | Fix a small scoping issue in Security::Base|76|
+| 2010/05/25 | Add option --no-progress to disable progress bar for SimpleRPC|74|
+| 2010/05/23 | Add some missing dependencies to the RPMs|72|
+| 2010/05/22 | Add an option _:process_results_ to the client|71|
+| 2010/05/13 | Fix help output that still shows old branding|70|
+| 2010/04/27 | The supplied generic stompclient now accepts STOMP_PORT in the environment|68|
+| 2010/04/26 | Add a SimpleRPC Client helper to reset filters|64|
+| 2010/04/26 | Listen for signal USR1 and reload all agents from disk|65|
+| 2010/04/12 | Add SimpleRPC Authorization support|63|
+
+
+## 0.4.4 - 2010/04/03
+
+### Release Focus and Notes
+
+This release is mostly a bug fix release.
+
+We've cleaned up the logs a bit so you'll see fewer exceptions logged, also if you have Rails + Puppet on a node the logs will stay in the standard format. We are handling some exceptions better which has improved stability of the registration system.
+
+If you do some slow queries in your discovery agent you can bump the timeout up in the config using _plugin.discovery.timeout_.
+
+All the scripts now use _/usr/bin/env ruby_ rather than hardcoded paths to deal better with Ruby's in weird places.
+
+Several other small annoyances was fixed or improved.
+
+### mc-controller
+
+We've always had a tool that let you control a network of mcollective instances remotely, it lagged behind a bit with the core, we've fixed it up and documented it "here":/reference/basic/daemon.html . You can use it to reload agents from disk without restarting the daemon for example or get stats or shut down the entire mcollective network.
+
+### Backwards Compatibility
+
+No changes that impacts backward compatibility has been made.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+| 2010/03/27 | Make it easier to construct SimpleRPC requests to use with the standard client library|60|
+| 2010/03/27 | Manipulating the filters via the helper methods will force rediscovery|59|
+| 2010/03/23 | Prevent Activesupport when brought in by Facter from breaking our logs|57|
+| 2010/03/23 | Clean up logging for messages not targeted at us|56|
+| 2010/03/19 | Add exception handling to the registration base class|55|
+| 2010/03/03 | Use /usr/bin/env ruby instead of hardcoded paths|54|
+| 2010/02/17 | Improve mc-controller and document it|46|
+| 2010/02/08 | Remove some close coupling with Stomp to easy creating of other connectors|49|
+| 2010/02/01 | Made the discovery agent timeout configurable using plugin.discovery.timeout|48|
+| 2010/01/25 | mc-controller now correctly loads/reloads agents.|45|
+| 2010/01/25 | Building packages has been improved to ensure rdocs are always included|44|
+
+
+## 0.4.3 - 2010/01/24
+
+### Release Focus and Notes
+
+This release fixes a few bugs and introduce a major new SimpleRPC feature for auditing requests.
+
+### Auditing
+
+We've created an "auditing framework for SimpleRPC":/simplerpc/auditing.html, each request gets passed to an audit plugin for processing. We ship one that simply logs to a file on each node and there's a "community plugin":http://code.google.com/p/mcollective-plugins/wiki/AuditCentralRPCLog that logs everything on a central logging host.
+
+In future we might add auditing to the client libraries so requests will be logged where they are sent as well as auditing of replies being sent, this will be driven by requests from the community though.
+
+### New _fail!_ method for SimpleRPC
+
+Till now while writing agents you can use the _fail_ method to set statuses in the reply, this however did not also raise exceptions and terminate execution of the agent immediately.
+
+Often the existing behavior is required but it did lead to some awkward code when you did want to just exit the agent immediately as well as set a fail status. We've added a _fail!_ method that works just like _fail_ except it stops execution of your agent immediately.
+
+### Backwards Compatibility
+
+No changes that impacts backward compatibility has been made.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+| 2010/01/23 | Handle ctrl-c during discovery without showing exceptions to users|34|
+| 2010/01/21 | Force all facts in the YAML fact source to be strings|41|
+| 2010/01/19 | Add SimpleRPCAuditing audit logging to SimpleRPC clients and Agents| |
+| 2010/01/18 | The SRPM we provide will now build outside of the Rake environment|40|
+| 2010/01/18 | Add a _fail!_ method to RPC::Agent|37|
+| 2010/01/18 | mc-rpc can now be used without supplying arguments|38|
+| 2010/01/18 | Don't raise an error if no user/pass is given to the stomp connector, try unauthenticated mode|35|
+| 2010/01/17 | Improve error message when Regex validation failed on SimpleRPC input|36|
+
+
+## 0.4.2 - 2010/01/14
+
+### Release Focus and Notes
+
+This release fixes a few bugs, add some command line improvements and brings major changes to the Debian packaging.
+
+### Packaging
+
+Firstly we've had some amazing work done by Riccardo Setti to make us Debian packages that complies with Debian and Ubuntu policy, this release use these new packages. It has some unfortunate changes to file layout detailed below but overall I think it's a big win to get us in line with Distribution policies and standards.
+
+The only major change is that in the past we used _/usr/libexec/mcollective_ as the libdir, but Debian does not have this directory and it is not in the LFHS anymore so we now use _/usr/share/mcollective/plugins_ as the lib dir. You need to move your plugins there and update both client and server configs.
+
+The RedHat packages will move to this convention too in the next release since I think it's the better location and complies with LFHS.
+
+### Command Line Improvements
+
+We've streamlined the command line a bit, nothings changed we've just added some flags.
+
+The _--with-class_, _--with-fact_, _--with-agent_ and _--with-identity_ now all have a short form _-C_, _-F_, _-A_ and _-I_ respectively.
+
+We've added a new filter option _--with_ and a short form _-W_ that combines _--with-class_ and _--with-fact_ into one filter type, use case would be:
+
+{% highlight console %}
+ % mc-find-hosts -W "/centreon/ country=de roles::dev_server/"
+{% endhighlight %}
+
+This would find hosts with class regex matched _/centreon/_, class _roles::dev_server_ and fact matching _country=de_. Hopefully this saves on some typing.
+
+You can also now set the environment variables _MCOLLECTIVE_TIMEOUT_ and _MCOLLECTIVE_DTIMEOUT_ which saves you from typing _--timeout_ and _--discovery-timeout_ often, especially useful on very fast networks.
+
+### Other fixes and improvements
+
+ * We've added the COPYING file to all the packages
+ * We've made the init script more LSB compliant
+ * A bug related to discovery in SimpleRPC was fixed
+
+### Backwards Compatibility
+
+The only backwards issue is the Debian packages. They've been tested to upgrade cleanly but you need to change the config as above.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+| 2010/01/13 | New packaging for Debian provided by Riccardo Setti|29|
+| 2010/01/07 | Improved LSB compliance of the init script - thanks Riccardo Setti|32|
+| 2010/01/07 | Multiple calls to SimpleRPC client would reset discovered hosts|31|
+| 2010/01/04 | Timeouts can now be changed with MCOLLECTIVE_TIMEOUT and MCOLLECTIVE_DTIMEOUT environment vars|25|
+| 2010/01/04 | Specify class and fact filters easier with the new -W or --with option|27 |
+| 2010/01/04 | Added COPYING file to RPMs and tarball|28|
+| 2010/01/04 | Make shorter filter options -C, -I, -A and -F|26|
+
+## 0.4.1 - 2010/01/02
+
+### Release Focus and Notes
+
+This is a bug fix release to address some shortcomings and issues found in Simple RPC.
+
+The main issue is around handling of meta data in agents, the documented approach did not work, we've now solved this by adding a number of hooks into the processing of Simple RPC agents.
+
+We've also made logging and config retrieval a bit easier in agents - and documented this.
+
+You can now call the _mc-rpc_ command a bit easier:
+
+{% highlight console %}
+ % mc-rpc --agent helloworld --action echo --argument msg="hello world"
+ % mc-rpc helloworld echo msg="hello world"
+{% endhighlight %}
+
+The 2 calls are the same, you can pass as many arguments in _key=val_ pairs as needed at the end.
+
+### Backwards Compatibility
+
+No issues with backward compatibility, should be a simple upgrade.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+| 2010/01/02 | Added hooks to plug into the processing of requests, also enabled setting meta data and timeouts|14|
+| 2010/01/02 | Created readers for @config and @logger in the SimpleRPC agent|23|
+| 2009/12/30 | Don't send out any requests if no nodes were discovered|17|
+| 2009/12/30 | Added :discovered and :discovered_nodes to client stats|20|
+| 2009/12/30 | Add a empty_filter? helper to the RPC mixin|18|
+| 2009/12/30 | Fix formatting bug with progress bar|21|
+| 2009/12/29 | Simplify mc-rpc command line|16|
+| 2009/12/29 | Fix layout issue when printing hosts that did not respond|15|
+
+
+## 0.4.0 - 2009/12/29
+
+### Release Focus and Notes
+
+This release introduced a major new feature - Simple RPC - a framework for easily building clients and agents. More than that it's a set of conventions and standards that will help us build generic clients like web based ones capable of talking to all agents.
+
+We think this feature is ready for wide use, it's well documented and we've done extensive testing. We'll be porting some of our own code over to it once this release is out and we do anticipate there might be some _0.4.x_ releases to round off a few issues that might remain. We do not currently have any open tickets against Simple RPC.
+
+We've also added the ability to create more complex queries such as:
+
+{% highlight console %}
+--with-class /dev_server/ --with-class /rails/
+{% endhighlight %}
+
+This does an _AND_ operation on the puppet classes on the node and finds only nodes with both _/dev_server/_ *AND* _/rails/_ classes. This new functionality applies to all types of filter.
+
+We've made the _--with-class_ filters more generic in comments, documentation etc with an eye to be more usable in Chef and other Configuration Management environments.
+
+### Backwards Compatibility
+
+Unfortunately introducing the new filtering methods has some backward compatibility issues, if you had clients/agents with code like:
+
+{% highlight ruby %}
+ options[:filter]["agent"] = "some agent"
+{% endhighlight %}
+
+You should now change that to:
+
+{% highlight ruby %}
+ options[:filter]["agent"] << "some agent"
+{% endhighlight %}
+
+As each filter is an array now. If you do not change the code it will still work as before but you will not be able to use the compound filtering feature on filter types that you've forced to be a string and there might be some other undesired side effects. We've tried though to at least not break old code, they just can't use the new features.
+
+You were also able to test easily in the past if you're running unfiltered using
+something like:
+
+{% highlight ruby %}
+ if options[:filter] == {}
+{% endhighlight %}
+
+Now that's much harder and we've added a helper to make this easier:
+
+{% highlight ruby %}
+ if MCollective::Util.empty_filter?(options[:filter])
+{% endhighlight %}
+
+This new method is compatible with both the old and new filter method so you can start using it before you finish the first issue mentioned here.
+
+We've also made the class filter more generic, in the past you did class filters like this:
+
+{% highlight ruby %}
+ options[:filter]["puppet_class"] << /apache/
+{% endhighlight %}
+
+Now you have to adjust it to:
+
+{% highlight ruby %}
+ options[:filter]["cf_class"] << /apache/
+{% endhighlight %}
+
+Old code will keep working but you should change to this name for filters to be consistent with the rest of the code base.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+| 2009/12/28 | Add support for other configuration management systems like chef in the --with-class filters|13|
+| 2009/12/28 | Add a _Util.empty`_`filter?_ to test for an empty filter| |
+| 2009/12/27 | Create a new client framework SimpleRPCIntroduction|6|
+| 2009/12/27 | Add support for multiple filters of the same type|3|
+
+## 0.3.0 - 2009/12/17
+
+### Release Focus and Notes
+
+Primarily a bug fix release. Only new feature is to allow the user to create _MCollective::Util::`*`_ classes and put those in the plugins directory. This is useful for more complex agents and clients.
+
+### Backwards Compatibility
+
+This release should not break any older code, if it does it's a bug.
+
+### Changes
+
+|Date|Description|Ticket|
+|----|-----------|------|
+|2009/12/16|Improvements for newer versions of Ruby where TERM signal was not handled|7|
+|2009/12/07|MCollective::Util is now a module and plugins can drop in util classes in the plugin dir| |
+|2009/12/07|The Rakefile now works with rake provided on Debian 4 systems|2|
+|2009/12/07|Improvements in the RC script for Debian and older Ubuntu systems|5|
+
+## 0.2.0 - 2009/12/02
+
+### Release Focus and Notes
+
+First numbered release
+
+### Backwards Compatibility
+
+n/a
+
+### Changes
+
+n/a
--- /dev/null
+---
+layout: default
+title: Screen Casts
+toc: false
+---
+[blip]: http://mcollective.blip.tv/
+[slideshare]: http://www.slideshare.net/mcollective
+[ec2demo]: /mcollective/ec2demo.html
+[Terminology]: /mcollective/terminology.html
+[SimpleRPCIntroduction]: /mcollective/simplerpc/
+[DDL]: /mcollective/reference/plugins/ddl.html
+
+We believe screen casts give the best introduction to new concepts, so we've recorded
+several to complement the documentation.
+
+There's a [blip] channel that has all the videos, you can subscribe and follow there.
+There is also a [slideshare] site where presentations will go that we do at conferences and events.
+
+## Introductions and Guides
+<ol>
+<li><a href="#introduction">Introducing MCollective</a></li>
+<li><a href="#ec2_demo">EC2 Hosted Demo</a></li>
+<li><a href="#message_flow">Message Flow, Terminology and Components</a></li>
+<li><a href="#writing_agents">Writing Agents</a></li>
+<li><a href="#simplerpc_ddl">Detailed information about the DDL</a></li>
+</ol>
+
+## Tools built using MCollective
+<ol>
+<li><a href="#simplerpc_ddl_irb">SimpleRPC DDL IRB</a></li>
+<li><a href="#mcollective_deployer">Software Deployer used by developers</a></li>
+<li><a href="#exim">Managing clusters of Exim Servers</a></li>
+<li><a href="#server_provisioner">Bootstrapping Puppet on EC2</a></li>
+</ol>
+
+<a name="introduction"> </a>
+
+### Introduction
+[This video](http://youtu.be/0i7VpvC2vMM) introduces the basic concepts behind MCollective. It predates the
+SimpleRPC framework but is still valid today.
+
+<iframe width="640" height="360" src="http://www.youtube-nocookie.com/embed/0i7VpvC2vMM" frameborder="0" allowfullscreen></iframe>
+
+<a name="ec2_demo"> </a>
+
+### EC2 Based Demo
+Sometimes you just want to know if a tool is right for you by getting hands on experience
+we've made a EC2 hosted demo where you can fire up as many nodes in a cluster as you want and
+get some experience.
+
+View the [ec2demo] page for more about this.
+
+<iframe width="640" height="360" src="http://www.youtube-nocookie.com/embed/Hw0Z1xfg050" frameborder="0" allowfullscreen></iframe>
+
+<a name="message_flow"> </a>
+
+### Message Flow, Terminology and Components
+This video introduces the messaging concepts you need to know about when using MCollective.
+It shows how the components talk with each other and what software needs to be installed where
+on your network. Viewing this prior to starting your deployment is recommended.
+
+We also have a page detailing the [Terminology]
+
+<iframe width="640" height="480" src="http://www.youtube-nocookie.com/embed/fIHW41W8jas" frameborder="0" allowfullscreen></iframe>
+
+View more <a href="http://www.slideshare.net/">webinars</a> from <a href="http://www.slideshare.net/mcollective">Marionette Collective</a>.
+<a name="writing_agents"> </a>
+
+### How to write an Agent, DDL and Client
+Writing agents are easy, we have good documentation that can be used as a reference, [this
+video](http://youtu.be/2Xhq_UqnqRE) should show you how to tie it all together though.
+See the [SimpleRPC Introduction][SimpleRPCIntroduction] for reference wiki pages after viewing this video.
+
+<iframe width="640" height="480" src="http://www.youtube-nocookie.com/embed/2Xhq_UqnqRE" frameborder="0" allowfullscreen></iframe>
+
+<a name="simplerpc_ddl"> </a>
+
+### The SimpleRPC DDL
+The Data Definition Language helps your clients produce more user friendly output, it ensures
+input gets validated while showing online help, and it enables dynamic generation of user interfaces.
+After you have watched the video, you can refer to the [DDL] wiki page for more information.
+
+<iframe width="640" height="480" src="http://www.youtube-nocookie.com/embed/xikjjXvN6nA" frameborder="0" allowfullscreen></iframe>
+
+<a name="simplerpc_ddl_irb"> </a>
+
+### SimpleRPC DDL enhanced IRB
+Based on the SimpleRPC DDL, this custom IRB shell supports command completion.
+
+<iframe width="640" height="480" src="http://www.youtube-nocookie.com/embed/xikjjXvN6nA" frameborder="0" allowfullscreen></iframe>
+
+<a name="mcollective_deployer"> </a>
+
+### Software Deployer using MCollective
+If you ever do commissioned work based on MCollective, this deployer written using SimpleRPC may be of use.
+It can be used by developers to deploy applications live into production using a defined
+API and process.
+
+<object width="640" height="385"><param name="movie" value="http://www.youtube.com/v/Fqt2SgnQn3k&hl=en_US&fs=1?rel=0"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/Fqt2SgnQn3k&hl=en_US&fs=1?rel=0" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="640" height="385"></embed></object>
+
+<a name="exim"> </a>
+
+### Managing Exim Clusters
+A command line and dialog-based UI written to manage clusters of Exim Servers.
+
+The code for this is [open source](http://github.com/puppetlabs/mcollective-plugins/tree/master/agent/exim/).
+Unfortunately it predates SimpleRPC; we hope to port it soon.
+
+<object width="640" height="385"><param name="movie" value="http://www.youtube.com/v/kNvoQCpJ1V4&hl=en_US&fs=1?rel=0"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/kNvoQCpJ1V4&hl=en_US&fs=1?rel=0" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="640" height="385"></embed></object>
+
+<a name="server_provisioner"> </a>
+
+### Bootstrapping Puppet on EC2 with MCollective
+Modern cloud environments present a lot of challenges to automation. However, with MCollective and
+some existing open source agents and plugins you can completely automate the entire process
+of provisioning EC2 nodes using Puppet.
+
+More detail is available [on this blog post](http://www.devco.net/archives/2010/07/14/bootstrapping_puppet_on_ec2_with_mcollective.php)
+
+<iframe width="640" height="480" src="http://www.youtube-nocookie.com/embed/-iEgz9MD3qA" frameborder="0" allowfullscreen></iframe>
--- /dev/null
+---
+layout: default
+title: Security Overview
+toc: false
+---
+[broadcast paradigm]: /mcollective/reference/basic/messageflow.html
+[SimpleRPC]: /mcollective/simplerpc/
+[Authorization]: /mcollective/simplerpc/authorization.html
+[Auditing]: /mcollective/simplerpc/auditing.html
+[SSL security plugin]: /mcollective/reference/plugins/security_ssl.html
+[AES security plugin]: /mcollective/reference/plugins/security_aes.html
+[ActiveMQ Security]: /mcollective/reference/integration/activemq_security.html
+[ActiveMQ TLS]: http://activemq.apache.org/how-do-i-use-ssl.html
+[ActiveMQ SSL]: /mcollective/reference/integration/activemq_ssl.html
+[ActiveMQ STOMP]: http://activemq.apache.org/stomp.html
+[MCollective STOMP Connector]: /mcollective/reference/plugins/connector_stomp.html
+[ActionPolicy]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AuthorizationActionPolicy
+[CentralAudit]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AuditCentralRPC
+[Subcollectives]: reference/basic/subcollectives.html
+
+
+Due to the [broadcast paradigm] of mcollective security is a complex topic to
+discuss.
+
+This discussion will focus on strong SSL and AES+RSA based security plugins,
+these are not the default or only option but is currently the most secure.
+Both the [SSL security plugin] and [AES security plugin] provide strong caller
+identification, this is used by the [SimpleRPC] framework for [Authorization]
+and [Auditing].
+
+As every organisation has its own needs almost all aspects of the security
+system is pluggable. This is an overview of the current state of SSL based
+Authentication, Authorization and Auditing.
+
+<center><img src="/mcollective/images/mcollective-aaa.png"></center>
+
+The image above is a reference to use in the following pages, it shows a
+MCollective Setup and indicates the areas of discussion.
+
+The focus here is on ActiveMQ, some of the details and capabilities will
+differ between middleware systems.
+
+## Client Connections and Credentials
+
+Every STOMP connection has a username and password, this is used to gain basic
+access to the ActiveMQ system. We have a [ActiveMQ Security] sample setup
+documented.
+
+ActiveMQ can use LDAP and other security providers, details of this is out of
+scope here, you should use their documentation or the recently released book
+for details of that.
+
+### MCollective Protocol Security
+The main protocol used by MCollective keeps track of message creation time and
+a per message TTL. Messages that are older than the TTL are not accepted in
+future we will also do full replay protection.
+
+Both the AES+RSA and the SSL plugin secures these 2 properties cryptographically
+to make sure they are not tampered with.
+
+### The AES+RSA securith plugin
+When using the [AES security plugin] each user also gets a private and public
+key, like with SSH you need to ensure that the private keys remain private
+and not shared between users.
+
+This plugin can be configured to distribute public keys at the cost of some
+security, you can also manually distribute keys for the most secure setup.
+
+The public / private key pair is used to encrypt using AES and then to encrypt
+the key used during the AES phase using RSA. This provides encrypted payloads
+securing the reply strucutres.
+
+The client embeds a _caller_ structure in each request, if RSA decryption
+pass the rest of the MCollective agents, auditing etc can securely know who
+initiated a request.
+
+This caller is used later during Authorization and Auditing.
+
+The plugin secures the TTL and Message Time properties of the message making sure
+someone cannot intercept, tamper and replay these messages.
+
+This plugin comes with a significant setup, maintenance and performance overhead
+if all you need is to securely identify users use the SSL security plugin instead.
+
+### The SSL security plugin
+When using the [SSL security plugin] each user also gets a private and public
+certificate, like with SSH you need to ensure that the private keys remain
+private and not be shared between users. The public part needs to be
+distributed to all nodes.
+
+The private key is used to cryptographically sign each request being made by a
+client, later the public key will be used to validate the signature for
+authenticity.
+
+The client embeds a _caller_ structure in each request, if SSL signature
+validation pass the rest of the MCollective agents, auditing etc can securely
+know who initiated a request.
+
+This caller is used later during Authorization and Auditing.
+
+The plugin secures the TTL and Message Time properties of the message making sure
+someone cannot intercept, tamper and replay these messages.
+
+## Connection to Middleware
+
+By default the connections between Middleware and Nodes are not encrypted, just
+signed using the SSL keys above. [ActiveMQ supports TLS][ActiveMQ TLS] and the
+[stomp connector][ActiveMQ STOMP] supports this including full CA based
+certificate verification.
+
+The [MCollective STOMP Connector] also supports TLS, to configure MCollective
+to speak TLS to your nodes please follow our notes about [ActiveMQ SSL].
+
+Enabling TLS throughout will secure your connections from any kind of sniffing
+and Man in The Middle attacks.
+
+## Middleware Authorization and Authentication
+
+As mentioned above ActiveMQ has it's own users and every node and client
+authenticates using these.
+
+In addition to this you can on the middleware layer restrict access to topics,
+you can for example run a development and production collective on the same
+ActiveMQ infrastructure and allow your developers access to just the development
+collective using these controls. They are not very fine grained but should be a
+import step to configure for any real setup.
+
+We have a sample [ActiveMQ Security] setup documented that has this kind of
+control.
+
+By combining this topic level restrictions with [Subcollectives] you can create
+virtually isolated groups of nodes and give certain users access to only those
+subcollectives. Effectively partitioning out a subset of machines and giving
+secure access to just those.
+
+## Node connections and credentials
+
+As with the client the node needs a username and password to connect to the
+middleware and can also use TLS.
+
+It's not a problem if all the nodes share a username and password for the
+connection since generally nodes do not make new requests. You can enable
+registration features that will see your nodes make connections, you should
+restrict this as outlined in the previous section.
+
+When using the [SSL security plugin] all the nodes share a same SSL private
+and public key, all replies are signed using this key. It would not be
+impossible to add a per node certificate setup but I do not think this will
+add a significant level of security over what we have today.
+
+When using the [AES security plugin] nodes can have their own sets of keys
+and registration data can be secured. Replies are encrypted using the clients
+key and so only the client who made the request can read the replies.
+
+## SimpleRPC Authorization
+
+The RPC framework has a pluggable [Authorization] system, you can create very
+fine grain control over requests, for example using the [ActionPolicy] setup you
+can create a policy like this:
+
+{% highlight text %}
+policy default deny
+allow cert=rip * * *
+allow cert=john * customer=acme acme::devserver
+allow cert=john enable disable status customer=acme *
+{% endhighlight %}
+
+This applied to the service agent will allow different level of access to
+actions to different people. The caller id is based directly on the SSL Private
+Key in use and subject to validation on every node.
+
+As with other aspects of mcollective authorization is tied closely with meta
+data like facts and classes so you can use these to structure your authorization
+as can be seen above.
+
+You can provide your own authorization layers to fit your ogranizational needs,
+they can be specific to an agent or apply to the entire collective.
+
+## SimpleRPC Auditing
+
+The RPC layer can keep detailed [Auditing] records of every request received,
+the audit log shows the - SSL signature or RSA verified - caller, what agent, action
+and any arguments that was sent for every request.
+
+The audit layer is a plugin based system, we provide one that logs to a file on
+every node and there are [community plugins][CentralAudit] that keeps a centralized
+log both in log files and in MongoDB NoSQL database.
+
+Which to use depends on your usecase, obviously a centralized auditing system
+for thousands of nodes is very complex and will require a special plugin to be
+developed the community centralized audit log is ok for roughly 100 nodes or
+so.
--- /dev/null
+---
+layout: default
+title: Writing SimpleRPC Agents
+---
+[WritingAgents]: /mcollective/reference/basic/basic_agent_and_client.html
+[SimpleRPCClients]: /mcollective/simplerpc/clients.html
+[ResultsandExceptions]: /mcollective/simplerpc/clients.html#Results_and_Exceptions
+[SimpleRPCAuditing]: /mcollective/simplerpc/auditing.html
+[SimpleRPCAuthorization]: /mcollective/simplerpc/authorization.html
+[DDL]: /mcollective/reference/plugins/ddl.html
+[WritingAgentsScreenCast]: http://mcollective.blip.tv/file/3808928/
+[RPCUtil]: /mcollective/reference/plugins/rpcutil.html
+[ValidatorPlugins]: /mcollective/reference/plugins/validator.html
+
+Simple RPC works because it makes a lot of assumptions about how you write agents, we'll try to capture those assumptions here and show you how to apply them to our Helloworld agent.
+
+We've recorded a [tutorial that will give you a quick look at what is involved in writing agents][WritingAgentsScreenCast].
+
+## Conventions regarding Incoming Data
+
+As you've seen in [SimpleRPCClients] our clients will send requests like:
+
+{% highlight ruby %}
+mc.echo(:msg => "Welcome to MCollective Simple RPC")
+{% endhighlight %}
+
+A more complex example might be:
+
+{% highlight ruby %}
+exim.setsender(:msgid => "1NOTVx-00028U-7G", :sender => "foo@bar.com")
+{% endhighlight %}
+
+Effectively this creates a hash with the members _:msgid_ and _:sender_.
+
+Your data types should be preserved if your Security plugin supports that - the default one does - so you can pass in Arrays, Hashes, OpenStructs, Hashes of Hashes but you should always pass something in and it should be key/value pairs like a Hash expects.
+
+You cannot use the a data item called _:process_results_ as this has special meaning to the agent and client. This will indicate to the agent that the client is'nt going to be waiting to process results. You might choose not to send back a reply based on this.
+
+## Sample Agent
+Here's our sample *Helloworld* agent:
+
+{% highlight ruby linenos %}
+module MCollective
+ module Agent
+ class Helloworld<RPC::Agent
+ # Basic echo server
+ action "echo" do
+ validate :msg, String
+
+ reply[:msg] = request[:msg]
+ end
+ end
+ end
+end
+
+{% endhighlight %}
+
+Strictly speaking this Agent will work but isn't considered complete - there's no meta data and no help.
+
+A helper agent called [_rpcutil_][RPCUtil] is included that helps you gather stats, inventory etc about the running daemon. It's a full SimpleRPC agent including DDL, you can look at it for an example.
+
+### Agent Name
+The agent name is derived from the class name, the example code creates *MCollective::Agent::Helloworld* and the agent name would be *helloworld*.
+
+<a name="Meta_Data_and_Initialization"> </a>
+
+### Meta Data and Initialization
+Simple RPC agents still need meta data like in [WritingAgents], without it you'll just have some defaults assigned, code below adds the meta data to our agent:
+
+**NOTE**: As of version 2.1.1 the _metadata_ section is deprecated, all agents must have DDL files with this information in them.
+
+{% highlight ruby linenos %}
+module MCollective
+ module Agent
+ class Helloworld<RPC::Agent
+ metadata :name => "helloworld",
+ :description => "Echo service for MCollective",
+ :author => "R.I.Pienaar",
+ :license => "GPLv2",
+ :version => "1.1",
+ :url => "http://projects.puppetlabs.com/projects/mcollective-plugins/wiki",
+ :timeout => 60
+
+ # Basic echo server
+ action "echo" do
+ validate :msg, String
+
+ reply[:msg] = request[:msg]
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+The added code sets our creator info, license and version as well as a timeout. The timeout is how long MCollective will let your agent run for before killing them, this is a very important number and should be given careful consideration. If you set it too low your agents will be terminated before their work is done.
+
+The default timeout for SimpleRPC agents is *10*.
+
+### Writing Actions
+Actions are the individual tasks that your agent can do:
+
+{% highlight ruby linenos %}
+action "echo" do
+ validate :msg, String
+
+ reply[:msg] = request[:msg]
+end
+{% endhighlight %}
+
+Creates an action called "echo". They don't and can't take any arguments.
+
+## Agent Activation
+In the past you had to copy an agent only to machines that they should be running on as
+all agents were activated regardless of dependencies.
+
+To make deployment simpler agents support the ability to determine if they should run
+on a particular platform. By default SimpleRPC agents can be configured to activate
+or not:
+
+{% highlight ini %}
+plugin.helloworld.activate_agent = false
+{% endhighlight %}
+
+You can also place the following in _/etc/mcollective/plugins.d/helloworld.cfg_:
+
+{% highlight ini %}
+activate_agent = false
+{% endhighlight %}
+
+This is a simple way to enable or disable an agent on your machine, agents can also
+declare their own logic that will get called each time an agent gets loaded from disk.
+
+{% highlight ruby %}
+module MCollective
+ module Agent
+ class Helloworld<RPC::Agent
+
+ activate_when do
+ File.executable?("/usr/bin/puppet")
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+If this block returns false or raises an exception then the agent will not be active on
+this machine and it will not be discovered.
+
+When the agent gets loaded it will test if _/usr/bin/puppet_ exist and only if it does
+will this agent be enabled.
+
+## Help and the Data Description Language
+We have a separate file that goes together with an agent and is used to describe the agent in detail, a DDL file for the above echo agent can be seen below:
+
+**NOTE**: As of version 2.1.1 the DDL files are required to be on the the nodes before an agent will be activated
+
+{% highlight ruby linenos %}
+metadata :name => "echo",
+ :description => "Echo service for MCollective",
+ :author => "R.I.Pienaar",
+ :license => "GPLv2",
+ :version => "1.1",
+ :url => "http://projects.puppetlabs.com/projects/mcollective-plugins/wiki",
+ :timeout => 60
+
+action "echo", :description => "Echos back any message it receives" do
+ input :msg,
+ :prompt => "Service Name",
+ :description => "The service to get the status for",
+ :type => :string,
+ :validation => '^[a-zA-Z\-_\d]+$',
+ :optional => false,
+ :maxlength => 30
+
+ output :msg,
+ :description => "The message we received",
+ :display_as => "Message"
+end
+{% endhighlight %}
+
+As you can see the DDL file expand on the basic syntax adding a lot of markup, help and other important validation data. This information - when available - helps in making more robust clients and also potentially auto generating user interfaces.
+
+The DDL is a complex topic, read all about it in [DDL].
+
+## Validating Input
+If you've followed the conventions and put the incoming data in a Hash structure then you can use a few of the provided validators to make sure your data that you received is what you expected.
+
+If you didn't use Hashes for input the validators would not be usable to you. In future validation will happen automatically based on the [DDL] so I strongly suggest you follow the agent design pattern shown here using hashes.
+
+In the sample action above we validate the *:msg* input to be of type *String*, here are a few more examples:
+
+{% highlight ruby linenos %}
+ validate :msg, /[a-zA-Z]+/
+ validate :ipaddr, :ipv4address
+ validate :ipaddr, :ipv6address
+ validate :commmand, :shellsafe
+ validate :mode, ["all", "packages"]
+{% endhighlight %}
+
+The table below shows the validators we support currently
+
+|Type of Check|Description|Example|
+|-------------|-----------|-------|
+|Regular Expressions|Matches the input against the supplied regular expression|validate :msg, /\[a-zA-Z\]+/|
+|Type Checks|Verifies that input is of a given ruby data type|validate :msg, String|
+|IPv4 Checks|Validates an ip v4 address, note 5.5.5.5 is technically a valid address|validate :ipaddr, :ipv4address|
+|IPv6 Checks|Validates an ip v6 address|validate :ipaddr, :ipv6address|
+|system call safety checks|Makes sure the input is a string and has no ><backtick, semi colon, dollar, ambersand or pipe characters in it|validate :command, :shellsafe|
+|Boolean|Ensures a input value is either real boolean true or false|validate :enable, :bool|
+|List of valid options|Ensures the input data is one of a list of known good values|validate :mode, \["all", "packages"\]|
+
+All of these checks will raise an InvalidRPCData exception, you shouldn't catch this exception as the Simple RPC framework catches those and handles them appropriately.
+
+We'll make input validators plugins so you can provide your own types of validation easily.
+
+Additionally if can escape strings being passed to a shell, escaping is done in line with the _Shellwords#shellescape_ method that is in newer version of Ruby:
+
+{% highlight ruby linenos %}
+ safe = shellescape(request[:foo])
+{% endhighlight %}
+
+As of version 2.2.0 you can add your own types of validation using [Validator Plugins][ValidatorPlugins].
+
+## Agent Configuration
+
+You can save configuration for your agents in the main server config file:
+
+{% highlight ini %}
+ plugin.helloworld.setting = foo
+{% endhighlight %}
+
+In your code you can retrieve the config setting like this:
+
+{% highlight ini %}
+ setting = config.pluginconf["helloworld.setting"] || ""
+{% endhighlight %}
+
+This will set the setting to whatever is the config file of "" if unset.
+
+## Accessing the Input
+As you see from the echo example our input is easy to get to by just looking in *request*, this would be a Hash of exactly what was sent in by the client in the original request.
+
+The request object is in instance of *MCollective::RPC::Request*, you can also gain access to the following:
+
+|Property|Description|
+|--------|-----------|
+|time|The time the message was sent|
+|action|The action it is directed at|
+|data|The actual hash of data|
+|sender|The id of the sender|
+|agent|Which agent it was directed at|
+
+Since data is the actual Hash you can gain access to your input like:
+
+{% highlight ruby %}
+ request.data[:msg]
+{% endhighlight %}
+
+OR
+
+{% highlight ruby %}
+request[:msg]
+{% endhighlight %}
+
+Accessing it via the first will give you full access to all the normal Hash methods where the 2nd will only give you access to *include?*.
+
+## Running Shell Commands
+
+A helper function exist that makes it easier to run shell commands and gain
+access to their _STDOUT_ and _STDERR_.
+
+We recommend everyone use this method for calling to shell commands as it forces
+*LC_ALL* to *C* as well as wait on all the children and avoids zombies, you can
+set unique working directories and shell environments that would be impossible
+using simple _system_ that is provided with Ruby.
+
+The simplest case is just to run a command and send output back to the client:
+
+{% highlight ruby %}
+reply[:status] = run("echo 'hello world'", :stdout => :out, :stderr => :err)
+{% endhighlight %}
+
+Here you will have set _reply`[`:out`]`_, _reply`[`:err`]`_ and _reply`[`:status`]`_ based
+on the output from the command
+
+You can append the output of the command to any string:
+
+{% highlight ruby %}
+out = []
+err = ""
+status = run("echo 'hello world'", :stdout => out, :stderr => err)
+{% endhighlight %}
+
+Here the STDOUT of the command will be saved in the variable _out_ and not sent
+back to the caller. The only caveat is that the variables _out_ and _err_ should
+have the _<<_ method, so if you supplied an array each line of output will be a
+single member of the array. In the example _out_ would be an array of lines
+while _err_ would just be a big multi line string.
+
+By default any trailing new lines will be included in the output and error:
+
+{% highlight ruby %}
+reply[:status] = run("echo 'hello world'", :stdout => :out, :stderr => :err)
+reply[:stdout].chomp!
+reply[:stderr].chomp!
+{% endhighlight %}
+
+You can shorten this to:
+
+{% highlight ruby %}
+reply[:status] = run("echo 'hello world'", :stdout => :out, :stderr => :err, :chomp => true)
+{% endhighlight %}
+
+This will remove a trailing new line from the _reply`[`:out`]`_ and _reply`[`:err`]`_.
+
+If you wanted this command to run from the _/tmp_ directory:
+
+{% highlight ruby %}
+reply[:status] = run("echo 'hello world'", :stdout => :out, :stderr => :err, :cwd => "/tmp")
+{% endhighlight %}
+
+Or if you wanted to include a shell Environment variable:
+
+{% highlight ruby %}
+reply[:status] = run("echo 'hello world'", :stdout => :out, :stderr => :err, :environment => {"FOO" => "BAR"})
+{% endhighlight %}
+
+The status returned will be the exit code from the program you ran, if the program
+completely failed to run in the case where the file doesn't exist, resources were
+not available etc the exit code will be -1
+
+You have to set the cwd and environment through these options, do not simply
+call _chdir_ or adjust the _ENV_ hash in an agent as that will not be safe in
+the context of a multi threaded Ruby application.
+
+## Constructing Replies
+
+### Reply Data
+The reply data is in the *reply* variable and is an instance of *MCollective::RPC::Reply*.
+
+{% highlight ruby %}
+reply[:msg] = request[:msg]
+{% endhighlight %}
+
+### Reply Status
+As pointed out in the [ResultsandExceptions] page results all include status messages and the reply object has a helper to create those.
+
+{% highlight ruby %}
+def rmmsg_action
+ validate :msg, String
+ validate :msg, /[a-zA-Z]+-[a-zA-Z]+-[a-zA-Z]+-[a-zA-Z]+/
+ reply.fail "No such message #{request[:msg]}", 1 unless have_msg?(request[:msg])
+
+ # check all the validation passed before doing any work
+ return unless reply.statuscode == 0
+
+ # now remove the message from the queue
+end
+
+{% endhighlight %}
+
+The number in *reply.fail* corresponds to the codes in [ResultsandExceptions] it would default to *1* so you could just say:
+
+{% highlight ruby %}
+reply.fail "No such message #{request[:msg]}" unless have_msg?(request[:msg])
+{% endhighlight %}
+
+This is hypothetical action that is supposed to remove a message from some queue, if we do have a String as input that matches our message id's we then check that we do have such a message and if we don't we fail with a helpful message.
+
+Technically this will just set *statuscode* and *statusmsg* fields in the reply to appropriate values.
+
+It won't actually raise exceptions or exit your action though you should do that yourself as in the example here.
+
+There is also a *fail!* instead of just *fail* it does the same basic function but also raises exceptions. This lets you abort processing of the agent immediately without performing your own checks on *statuscode* as above later on.
+
+## Actions in external scripts
+Actions can be implemented using other programming languages as long as they support JSON.
+
+{% highlight ruby %}
+action "test" do
+ implemented_by "/some/external/script"
+end
+{% endhighlight %}
+
+The script _/some/external/script_ will be called with 2 arguments:
+
+ * The path to a file with the request in JSON format
+ * The path to a file where you should write your response as a JSON hash
+
+You can also access these 2 file paths in the *MCOLLECTIVE_REPLY_FILE* and *MCOLLECTIVE_REQUEST_FILE* environment variables
+
+Simply write your reply as a JSON hash into the reply file.
+
+The exit code of your script should correspond to the ones in [ResultsandExceptions]. Any text in STDERR will be
+logged on the server at *error* level and used in the text for the fail text.
+
+Any text to STDOUT will be logged on the server at level *info*.
+
+These scripts can be placed in a standard location:
+
+{% highlight ruby %}
+action "test" do
+ implemented_by "script.py"
+end
+{% endhighlight %}
+
+This will search each configured libdir for _libdir/agent/agent_name/script.py_. If you specified a full path it will not try to find the file in libdirs.
+
+## Sharing code between agents
+Sometimes you have code that is needed by multiple agents or shared between the agent and client. MCollective has
+name space called *MCollective::Util* for this kind of code and the packagers and so forth supports it.
+
+Create a class with your shared code given a name like *MCollective::Util::Yourco* and save this file in the libdir in *util/yourco.rb*
+
+A sample class can be seen here:
+
+{% highlight ruby %}
+module MCollective
+ module Util
+ class Yourco
+ def dosomething
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+You can now use it in your agent or clients by first loading it from the MCollective lib directories:
+
+{% highlight ruby %}
+MCollective::Util.loadclass("MCollective::Util::Yourco")
+
+helpers = MCollective::Util::Yourco.new
+helpers.dosomething
+{% endhighlight %}
+
+## Authorization
+You can write a fine grained Authorization system to control access to actions and agents, please see [SimpleRPCAuthorization] for full details.
+
+## Auditing
+The actions that agents perform can be Audited by code you provide, potentially creating a centralized audit log of all actions. See [SimpleRPCAuditing] for full details.
+
+## Logging
+You can write to the server log file using the normal logger class:
+
+{% highlight ruby %}
+Log.debug ("Hello from your agent")
+{% endhighlight %}
+
+You can log at levels *info*, *warn*, *debug*, *fatal* or *error*.
+
+## Data Caching
+As of version 2.2.0 there is a system wide Cache you can use to store data that might be costly to create on each request.
+
+The Cache is thread safe and can be used even with multiple concurrent requests for the same agent.
+
+Imagine your agent interacts with a customer database on the node that is slow to read data from but this data does not
+change often. Using the cache you can arrange for this be read only every 10 minutes:
+
+{% highlight ruby %}
+action "get_customer_data" do
+ # Create a new cache called 'customer' with a 600 second TTL,
+ # noop if it already exist
+ Cache.setup(:customer, 600)
+
+ begin
+ customer = Cache.read(:customer, request[:customerid])
+ rescue
+ customer = Cache.write(:customer, request[:customerid], get_customer(request[:customerid])
+ end
+
+ # do something with the customer data
+end
+{% endhighlight %}
+
+Here we setup a new cache table called *:customer* if it does not already exist, the cache has a 10 minute validity.
+We then try to read a cached customer record for *request\[:customerid\]* and if it's not been put in the cache
+before or if it expired I create a new customer record using a method called *get_customer* and then save it
+into the cache.
+
+If you have critical code in an agent that can only ever be run once you can use the Mutex from the same cache
+to synchronize the code:
+
+{% highlight ruby %}
+action "get_customer_data" do
+ # Create a new cache called 'customer' with a 600 second TTL,
+ # noop if it already exist
+ Cache.setup(:customer, 600)
+
+ Cache(:customer).synchronize do
+ # Update customer record
+ end
+end
+{% endhighlight %}
+
+Here we are using the same Cache that was previously setup and just gaining access to the Mutex protecting the
+cache data. The code inside the synchronize block will only be run once so you won't get competing updates to
+your customer data.
+
+If the lock is held too long by anyone the mcollectived will kill the threads in line with the Agent timeout.
+
+## Processing Hooks
+We provide a few hooks into the processing of a message, you've already used this earlier to <a href="#Meta_Data_and_Initialization">set meta data</a>.
+
+You'd use these hooks to add some functionality into the processing chain of agents, maybe you want to add extra logging for audit purposes of the raw incoming message and replies, these hooks will let you do that.
+
+|Hook Function Name|Description|
+|------------------|-----------|
+|startup_hook|Called at the end of the initialize method of the _RPC::Agent_ base class|
+|before_processing_hook(msg, connection)|Before processing of a message starts, pass in the raw message and the <em>MCollective::Connector</em> class|
+|after_processing_hook|Just before the message is dispatched to the client|
+
+### *startup_hook*
+Called at the end of the _RPC::Agent_ standard initialize method use this to adjust meta parameters, timeouts and any setup you need to do.
+
+This will not be called right when the daemon starts up, we use lazy loading and initialization so it will only be called the first time a request for this agent arrives.
+
+### *before_processing_hook*
+Called just after a message was received from the middleware before it gets passed to the handlers. *request* and *reply* will already be set, the msg passed is the message as received from the normal mcollective runner and the connection is the actual connector.
+
+You can in theory send off new messages over the connector maybe for auditing or something, probably limited use case in simple agents.
+
+### *after_processing_hook*
+Called at the end of processing just before the response gets sent to the middleware.
+
+This gets run outside of the main exception handling block of the agent so you should handle any exceptions you could raise yourself. The reason it is outside of the block is so you'll have access to even status codes set by the exception handlers. If you do raise an exception it will just be passed onto the runner and processing will fail.
--- /dev/null
+---
+layout: default
+title: SimpleRPC Auditing
+toc: false
+---
+[SimpleRPCIntroduction]: index.html
+[AuditCentralRPCLog]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AuditCentralRPC
+
+# {{page.title}}
+
+As part of the [SimpleRPC][SimpleRPCIntroduction] framework we've added an auditing system that you can use to log all requests received into a file or even send it over mcollective to a central auditing system. What actually happens with audit data is pluggable and you can provide your own plugins to do what you need.
+
+The clients will include the _uid_ of the process running the client library in the requests and the audit function will have access to that on the requests.
+
+## Configuration
+To enable logging you should set an option to enable it and also one to configure which plugin to use:
+
+{% highlight ini %}
+rpcaudit = 1
+rpcauditprovider = Logfile
+{% endhighlight %}
+
+This sets it up to use _MCollective::Audit::Logfile_ plugin for logging evens.
+
+The client will embed a caller id - the Unix UID of the program running it or SSL cert - in requests which you can find in the _request_ object.
+
+## Logfile plugin
+
+Auditing is implemented using plugins that you should install in the normal plugin directory under _mcollective/audit/_. We have a sample Logfile plugin that you can see below:
+
+{% highlight ruby %}
+module MCollective
+ module RPC
+ class Logfile<Audit
+ require 'pp'
+
+ def audit_request(request, connection)
+ logfile = Config.instance.pluginconf["rpcaudit.logfile"] || "/var/log/mcollective-audit.log"
+
+ now = Time.now
+ now_tz = tz = now.utc? ? "Z" : now.strftime("%z")
+ now_iso8601 = "%s.%06d%s" % [now.strftime("%Y-%m-%dT%H:%M:%S"), now.tv_usec, now_tz]
+
+ File.open(logfile, "w") do |f|
+ f.puts("#{now_iso8601}: reqid=#{request.uniqid}: reqtime=#{request.time} caller=#{request.caller}@#{request.sender} agent=#{request.agent} action=#{request.action} data=#{request.data.pretty_print_inspect}")
+ end
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+As you can see you only need to provide one method called _audit_request_, you will get the request in the form of an _MCollective::RPC::Request_ object as well as the connection to the middleware should you wish to send logs to a central host.
+
+The Logfile plugin takes a configuration option:
+
+{% highlight ini %}
+plugin.rpcaudit.logfile = /var/log/mcollective-audit.log
+{% endhighlight %}
+
+We do not do log rotation of this file so you should do that yourself if you enable this plugin.
+
+This log lines like:
+
+{% highlight ruby %}
+2010-12-28T17:09:03.889113+0000: reqid=319719cc475f57fda3f734136a31e19b: reqtime=1293556143 caller=cert=nagios@monitor1 agent=nrpe action=runcommand data={:process_results=>true, :command=>"check_mailq"}
+{% endhighlight %}
+
+Other plugins can be found on the community site like [a centralized logging plugin][AuditCentralRPCLog].
--- /dev/null
+---
+layout: default
+title: SimpleRPC Authorization
+toc: false
+---
+[SimpleRPCIntroduction]: index.html
+[SecurityWithActiveMQ]: /mcollective/reference/integration/activemq_security.html
+[SimpleRPCAuditing]: /mcollective/simplerpc/auditing.html
+[ActionPolicy]: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AuthorizationActionPolicy
+
+As part of the [SimpleRPC][SimpleRPCIntroduction] framework we've added an authorization system that you can use to exert fine grained control over who can call agents and actions.
+
+Combined with [Connection Security][SecurityWithActiveMQ], [Centralized Auditing][SimpleRPCAuditing] and Crypto signed messages this rounds out a series of extremely important features for large companies that in combination allow for very precise control over your MCollective Cluster.
+
+The clients will include the _uid_ of the process running the client library in the requests and the authorization function will have access to that on the requests.
+
+There is a sample full featured plugin called [ActionPolicy] that you can use or get some inspiration from.
+
+## Writing Authorization Plugins
+
+Writing an Authorization plugin is pretty simple, the below example will only allow RPC calls from Unix UID 500.
+
+{% highlight ruby linenos %}
+module MCollective::Util
+ class AuthorizeIt
+ def self.authorize(request)
+ if request.caller != "uid=500"
+ raise("Not authorized")
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+Any exception thrown by your class will just result in the message not being processed or audited.
+
+You'd install this in your libdir where you should already have a Util directory for these kinds of classes.
+
+To use your authorization plugin in an agent simply do something like this:
+
+{% highlight ruby linenos %}
+module MCollective::Agent
+ class Service<RPC::Agent
+ authorized_by :authorize_it
+
+ # ...
+ end
+end
+{% endhighlight %}
+
+The call extra _authorized`_`by :authorize`_`it_ line tells your agent to use the _MCollective::Util::AuthorizeIt_ class for authorization.
+
+## Enabling RPC authorization globally
+You can enable a specific plugin on all RPC agents in the server config file. If you do this and an agent also specify it's own authorization the agent will take priority.
+
+{% highlight ini %}
+rpcauthorization = yes
+rpcauthprovider = action_policy
+{% endhighlight %}
+
+Note setting _rpcauthorization = no_ here doesn't disable it everywhere, agents that specify authorization will still be used. This boolean enables the global auth policy not the per agent.
--- /dev/null
+---
+layout: default
+title: Writing SimpleRPC Clients
+---
+[SimpleRPCIntroduction]: index.html
+[WritingAgents]: agents.html
+[RPCUtil]: /mcollective/reference/plugins/rpcutil.html
+[WritingAgentsScreenCast]: http://mcollective.blip.tv/file/3808928/
+[RubyMixin]: http://juixe.com/techknow/index.php/2006/06/15/mixins-in-ruby/
+[OptionParser]: http://github.com/puppetlabs/marionette-collective/blob/master/lib/mcollective/optionparser.rb
+[AppPlugin]: ../reference/plugins/application.html
+
+As pointed out in the [SimpleRPCIntroduction] page you can use the _mco rpc_ CLI
+to call agents and it will do it's best to print results in a sane way. When
+this is not enough you can write your own clients.
+
+Simple RPC clients can do most of what a normal [client][WritingAgents] can do
+but it makes a lot of things much easier if you stick to the Simple RPC
+conventions.
+
+This guide shows how to write standalone scripts to interact with your
+collective. There is a single executable system. You can apply most of
+the techniques documented here to writing plugins for that application system.
+See the full reference for the plugin system [here][AppPlugin]. You should try
+to write your general agent CLIs using this plugin system rather than the stand
+alone scripts detailed below as that promote a unified interface that behave in a
+consistant manner.
+
+We've recorded a [tutorial that will give you a quick look at what is involved
+in writing agents and a very simple client][WritingAgentsScreenCast].
+
+We'll walk through building a ever more complex example of Hello World here.
+
+## The Basic Client
+The client is mostly a bunch of helper methods that you use as a [Ruby
+Mixin][RubyMixin] in your own code, it provides:
+
+ * Standard command line option parsing with help output
+ * Ability to add your own command line options
+ * Simple access to agents and actions
+ * Tools to help you print results
+ * Tools to print stats
+ * Tools to construct your own filters
+ * While retaining full power of _MCollective::Client_ if you need the additional feature sets
+ * And being as simple or as complex to match your level of code proficiency
+
+We'll write a client for the _Helloworld_ agent that you saw in the
+[SimpleRPCIntroduction].
+
+## Call an Agent and print the result
+A basic hello world client can be seen below:
+
+{% highlight ruby linenos %}
+#!/usr/bin/ruby
+
+require 'mcollective'
+
+include MCollective::RPC
+
+mc = rpcclient("helloworld")
+
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+
+printrpcstats
+
+mc.disconnect
+{% endhighlight %}
+
+Save this into _hello.rb_ and run it with _--help_, you should see the standard basic help including filters for discovery.
+
+If you've set up the Agent and run the client you should see something along these lines:
+
+{% highlight ruby %}
+$ hello.rb
+
+Finished processing 44 hosts in 375.57 ms
+{% endhighlight %}
+
+While it ran you would have seen a little progress bar and then just the summary line. The idea is that if you're talking to a 1000 machine there's no point in seeing a thousand _OK_, you only want to see what failed and this is exactly what happens here, you're only seeing errors.
+
+If you run it with _--verbose_ you'll see a line of text for every host and also a larger summary of results.
+
+I'll explain each major line in the code below then add some more features from there:
+
+{% highlight ruby %}
+include MCollective::RPC
+
+mc = rpcclient("helloworld")
+{% endhighlight %}
+
+The first line pulls in the various helper functions that we provide, this is the Mixin we mentioned earlier.
+
+We then create a new client to the agent "helloworld" that you access through the _mc_ variable.
+
+{% highlight ruby %}
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+
+printrpcstats
+{% endhighlight %}
+
+To call a specific action you simply have to do _mc.echo_ this calls the _echo_ action, we pass a _:msg_ parameter into it with the string we want echo'd back. The parameters will differ from action to action. It returns a simple array of the results that you can print any way you want, we'll show that later.
+
+_printrpc_ and _printrpcstats_ are functions used to print the results and stats respectively.
+
+{% highlight ruby %}
+mc.disconnect
+{% endhighlight %}
+
+This cleanly disconnects the client from the middleware, some middleware tools like ActiveMQ will log confusing exceptions if you do not do this. It's good form to always disconnect but isn't strictly required.
+
+## Adjusting the output
+
+### Verbosely displaying results
+As you see there's no indication that discovery is happening and as pointed out we do not display results that are ok, you can force verbosity as below on individual requests:
+
+{% highlight ruby %}
+mc = rpcclient("helloworld")
+
+mc.discover :verbose => true
+
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC"), :verbose => true
+{% endhighlight %}
+
+Here we've added a _:verbose_ flag and we've specifically called the discover method. Usually you don't need to call discover it will do it on demand. Doing it this way you'll always see the line:
+
+{% highlight console %}
+Determining the amount of hosts matching filter for 2 seconds .... 44
+{% endhighlight %}
+
+Passing verbose to _printrpc_ forces it to print all the results, failures or not.
+
+If you just wanted to force verbose on for all client interactions, do:
+
+{% highlight ruby %}
+mc = rpcclient("helloworld")
+mc.verbose = true
+
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+{% endhighlight %}
+
+In this case everything will be verbose, regardless of command line options.
+
+### Disabling the progress indicator
+You can disable the twirling progress indicator easily:
+
+{% highlight ruby %}
+mc = rpcclient("helloworld")
+mc.progress = false
+{% endhighlight %}
+
+Now whenever you call an action you will not see the progress indicator.
+
+### Saving the reports in variables without printing
+You can retrieve the stats from the clients and also get text of reports without printing them:
+
+{% highlight ruby %}
+stats = mc.echo(:msg => "Welcome to MCollective Simple RPC").stats
+
+report = stats.report
+{% endhighlight %}
+
+_report_ will now have the text that would have been displayed by 'printrpcstats' you can also use _no_response_report_ to get report text for just the list of hosts that didnt respond.
+
+If you didn't want to just print the results out to STDOUT you can also get them back as just text:
+
+{% highlight ruby %}
+report = rpcresults mc.echo(:msg => "Welcome to MCollective Simple RPC")
+{% endhighlight %}
+
+
+## Applying filters programatically
+You can pass filters on the command line using the normal _--with-`*`_ options but you can also do it programatically. Here's a new version of the client that only calls machines with the configuration management class _/dev_server/_ and the fact _country=uk_
+
+{% highlight ruby %}
+mc = rpcclient("helloworld")
+
+mc.class_filter /dev_server/
+mc.fact_filter "country", "uk"
+
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+{% endhighlight %}
+
+You can set other filters like _agent`_`filter_, _identity`_`filter_ and _compound`_`filter_.
+
+The fact_filter method supports a few other forms in adition to above:
+
+{% highlight ruby %}
+mc.fact_filter "country=uk"
+mc.fact_filter "physicalprocessorcount", "4", ">="
+{% endhighlight %}
+
+This will limit it to all machines in the UK with more than 3 processors.
+
+## Resetting filters to empty
+If while using the client you wish to reset the filters to an empty set of filters - containing only the agent name that you're busy addressing you can do it as follows:
+
+{% highlight ruby %}
+mc = rpcclient("helloworld")
+
+mc.class_filter /dev_server/
+
+mc.reset_filter
+{% endhighlight %}
+
+After this code snippet the filter will only have an agent filter of _helloworld_ set.
+
+## Processing Agents in Batches
+By default the client will communicate with all machines at the same time.
+This might not be desired as you might affect a DOS on related components.
+
+You can instruct the client to communicate with remote agents in batches
+and sleep between each batch.
+
+Any client application has this capability using the _--batch_ and _--batch-sleep-time_
+command line options.
+
+You can also enable this programatically either per client or per request:
+
+{% highlight ruby %}
+mc = rpcclient("helloworld")
+mc.batch_size = 10
+mc.batch_sleep_time = 5
+
+mc.echo(:msg => "hello world")
+{% endhighlight %}
+
+{% highlight ruby %}
+mc = rpcclient("helloworld")
+
+mc.echo(:msg => "hello world", :batch_size => 10, :batch_sleep_time => 5)
+{% endhighlight %}
+
+By default batching is disabled and sleep time is 1
+
+Setting the batch_size to 0 will disable batch mode in both examples above,
+effectively overriding what was supplied on the command line.
+
+## Forcing Rediscovery
+By default it will only do discovery once per script and then re-use the results, you can though force rediscovery if you had to adjust filters mid run for example.
+
+{% highlight ruby %}
+mc = rpcclient("helloworld")
+
+mc.class_filter /dev_server/
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+
+mc.reset
+
+mc.fact_filter "country", "uk"
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+{% endhighlight %}
+
+Here we make one _echo_ call - which would do a discovery - we then reset the client, adjust filters and call it again. The 2nd call would do a new discovery and have new client lists etc.
+
+## Supplying your own discovery information
+
+A new core messaging mode has been introduced that enables direct non filtered communicatin to specific nodes. This has enabled us to provide an discovery-optional
+mode but only if the collective is configured to support direct messaging.
+
+{%highlight ruby %}
+mc = rpcclient("helloworld")
+
+mc.discover(:nodes => ["host1", "host2", "host3"]
+
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+{% endhighlight %}
+
+This will immediately, without doing discovery, communicate just with these 3 hosts. It will do normal failure reporting as with normal discovery based
+requests but will just be much faster as the 2 second discovery overhead is avoided.
+
+The goal with this feature is for cases such as deployment tools where you have a known expectation of which machines to deploy to and you always want
+to know if that fails. In that use case a discovery based approach is not 100% suitable as you won't know about down machines. This way you can provide
+your own source of truth.
+
+When using the direct mode messages have a TTL associated with them that defaults to 60 seconds. Since 1.3.2 you can set the TTL globally in the configuration
+file but you can also set it on the client:
+
+{%highlight ruby %}
+mc = rpcclient("helloworld")
+mc.ttl = 3600
+
+mc.discover(:nodes => ["host1", "host2", "host3"]
+
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+{% endhighlight %}
+
+With the TTL set to 3600 if any of the hosts are down at the time of the request the request will wait on the middleware and should they come back up
+before 3600 has passed since request time they will then perform the requested action.
+
+## Only sending requests to a subset of discovered nodes
+By default all nodes that get discovered will get the request. This isn't always desirable maybe you want to deploy only to a random subset of hosts or maybe you have a service exposed over MCollective that you want to treat as a HA service and so only speak with one host that provides the functionality.
+
+You can limit the hosts to talk to either using a number or a percentage, the code below shows both:
+
+{%highlight ruby %}
+mc = rpcclient("helloworld")
+
+mc.limit_targets = "10%"
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+{% endhighlight %}
+
+This will pick 10% of the discovered hosts - or 1 if 10% is less than 1 - and only target those nodes with your request. You can also set it to an integer.
+
+There are two possible modes for choosing the targets. You can configure a global default method but also set it on your client:
+
+{%highlight ruby %}
+mc = rpcclient("helloworld")
+
+mc.limit_targets = "10%"
+mc.limit_method = :random
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+{% endhighlight %}
+
+The above code will force a _random_ selection, you can also set it to _:first_
+
+## Gaining access to the full MCollective::Client
+If you wanted to work with the Client directly as in [WritingAgents] after perhaps setting up some queries or gathering data first you can gain access to the client, you might also need access to the options array that was parsed out from the command line and any subsequent filters that you added.
+
+{% highlight ruby %}
+mc = rpcclient("helloworld")
+
+client = mc.client
+options = mc.options
+{% endhighlight %}
+
+The first call will set up the CLI option parsing, create clients etc, you can then just grab the client and options and go on as per [WritingAgents]. This is a much quicker way to write full power clients, by just by short-circuiting the options parsing etc.
+
+## Dealing with the results directly
+The biggest reason that you'd write custom clients is probably if you wanted to do custom processing of the results, there are 2 options to do it.
+
+<a name="Results_and_Exceptions"> </a>
+
+### Results and Exceptions
+Results have a set structure and depending on how you access the results you will either get Exceptions or result codes.
+
+|Status Code|Description|Exception Class|
+|-----------|-----------|---------------|
+|0|OK| |
+|1|OK, failed. All the data parsed ok, we have a action matching the request but the requested action could not be completed.|RPCAborted|
+|2|Unknown action|UnknownRPCAction|
+|3|Missing data|MissingRPCData|
+|4|Invalid data|InvalidRPCData|
+|5|Other error|UnknownRPCError|
+
+Just note these now, I'll reference them later down.
+
+### Simple RPC style results
+Simple RPC provides a trimmed down version of results from the basic Client library. You'd choose to use this if you just want to do simple things or maybe you're just learning Ruby. You'll get to process the results _after_ the call is either done or timed out completely.
+
+This is an important difference between the two approaches, in one you can parse the results as it comes in, in the other you will only get results after processing is done. This would be the main driving facter for choosing one over the other.
+
+Here's an example that will print out results in a custom way.
+
+{% highlight ruby %}
+mc.echo(:msg => "hello world").each do |resp|
+ printf("%-40s: %s\n", resp[:sender], resp[:data][:msg])
+end
+{% endhighlight %}
+
+This will produce a result something like this:
+
+{% highlight console %}
+dev1.you.net : hello world
+dev2.you.net : hello world
+dev3.you.net : hello world
+{% endhighlight %}
+
+The _each_ in the above code just loops through the array of results. Results are an array of Hashes, the data you got for above has the following structure:
+
+{% highlight console %}
+[{:statusmsg=>"OK",
+ :sender=>"dev1.your.net",
+ :data=>{:msg => "hello world"},
+ :statuscode=>0},
+{:statusmsg=>"OK",
+ :sender=>"dev2.your.net",
+ :data=>{:msg => "hello world"},
+ :statuscode=>0}]
+{% endhighlight %}
+
+The _:statuscode_ matches the table above so you can make decisions based on each result's status.
+
+### Gaining access to MCollective::Client#req results
+You can get access to each result in real time, in this case you will need to handle the exceptions in the table above and you'll get a different style of result set. The result set will be exactly as from the full blown client.
+
+In this mode there will be no progress indicator, you'll deal with results as and when they come in not after the fact as in the previous example.
+
+{% highlight ruby %}
+mc.echo(:msg => "hello world") do |resp|
+ begin
+ printf("%-40s: %s\n", resp[:senderid], resp[:body][:data])
+ rescue RPCError => e
+ puts "The RPC agent returned an error: #{e}"
+ end
+end
+{% endhighlight %}
+
+The output will be the same as above
+
+In this mode the results you get will be like this:
+
+{% highlight ruby %}
+{:msgtarget=>"/topic/mcollective.helloworld.reply",
+ :senderid=>"dev2.your.net",
+ :msgtime=>1261696663,
+ :hash=>"2d37daf690c4bcef5b5380b1e0c55f0c",
+ :body=>{:statusmsg=>"OK", :statuscode=>0, :data=>{:msg => "hello world"}},
+ :requestid=>"2884afb0b52cb38ea4d4a3146d18ef5f",
+ :senderagent=>"helloworld"}
+{% endhighlight %}
+
+Note how here we need to catch the exceptions, just handing _:statuscode_ will not be enough as the RPC client will raise exceptions - all descendant from _RPCError_ so you can easily catch just those.
+
+You can additionally gain access to a SimpleRPC style result in addition to the more complex native client results:
+
+{% highlight ruby %}
+mc.echo(:msg => "hello world") do |resp, simpleresp|
+ begin
+ printf("%-40s: %s\n", simpleresp[:sender], simpleresp[:data][:msg])
+ rescue RPCError => e
+ puts "The RPC agent returned an error: #{e}"
+ end
+end
+{% endhighlight %}
+
+You can still use printrpc to print these style of results and gain advantage of the DDL and so forth:
+
+{% highlight ruby %}
+mc.echo(:msg => "hello world") do |resp, simpleresp|
+ begin
+ printrpc simpleresp
+ rescue RPCError => e
+ puts "The RPC agent returned an error: #{e}"
+ end
+end
+{% endhighlight %}
+
+You will need to handle exceptions yourself but you have a simpler result set to deal with
+
+## Adding custom command line options
+You can look at the _mco rpc_ script for a big sample, here I am just adding a simple _--msg_ option to our script so you can customize the message that will be sent and received.
+
+{% highlight ruby linenos %}
+#!/usr/bin/ruby
+
+require 'mcollective'
+
+include MCollective::RPC
+
+options = rpcoptions do |parser, options|
+ parser.define_head "Generic Echo Client"
+ parser.banner = "Usage: hello [options] [filters] --msg MSG"
+
+ parser.on('-m', '--msg MSG', 'Message to pass') do |v|
+ options[:msg] = v
+ end
+end
+
+unless options.include?(:msg)
+ puts("You need to specify a message with --msg")
+ exit! 1
+end
+
+mc = rpcclient("helloworld", :options => options)
+
+mc.echo(:msg => options[:msg]).each do |resp|
+ printf("%-40s: %s\n", resp[:sender], resp[:data][:msg])
+end
+{% endhighlight %}
+
+This version of the code should be run like this:
+
+{% highlight console %}
+% test.rb --msg foo
+dev1.you.net : foo
+dev2.you.net : foo
+dev3.you.net : foo
+{% endhighlight %}
+
+Documentation for the Options Parser can be found [in it's code][OptionParser].
+
+And finally if you add options as above rather than try to parse it yourself you will get help integration for free:
+
+{% highlight console %}
+% hello.rb --help
+Usage: hello [options] [filters] --msg MSG
+Generic Echo Client
+ -m, --msg MSG Message to pass
+
+Common Options
+ -c, --config FILE Load configuratuion from file rather than default
+ --dt SECONDS Timeout for doing discovery
+ --discovery-timeout
+ -t, --timeout SECONDS Timeout for calling remote agents
+ -q, --quiet Do not be verbose
+ -v, --verbose Be verbose
+ -h, --help Display this screen
+
+Host Filters
+ --wf, --with-fact fact=val Match hosts with a certain fact
+ --wc, --with-class CLASS Match hosts with a certain configuration management class
+ --wa, --with-agent AGENT Match hosts with a certain agent
+ --wi, --with-identity IDENT Match hosts with a certain configured identity
+{% endhighlight %}
+
+## Disabling command line parsing and supplying your options programatically
+
+Sometimes, perhaps when embedding an MCollective client into another tool like Puppet, you do not want MCollective to do any command line parsing as there might be conflicting command line options etc.
+
+This can be achieved by supplying an options hash to the SimpleRPC client:
+
+{% highlight ruby %}
+include MCollective::RPC
+
+options = MCollective::Util.default_options
+
+client = rpcclient("test", {:options => options})
+{% endhighlight %}
+
+This will create a RPC client for the agent test without any options parsing at all.
+
+To set options like discovery timeout and so forth you will need use either the client utilities or manipulate the hash upfront, the client utility methods is the best. The code below sets the discovery timeout in a way that does not require you to know any internal structures or the content of the options hash.
+
+{% highlight ruby %}
+options = MCollective::Util.default_options
+
+client = rpcclient("test", {:options => options})
+client.discovery_timeout = 4
+{% endhighlight %}
+
+Using this method of creating custom options hashes mean we can make internal changes to MCollective without affecting your code in the future.
+
+## Sending SimpleRPC requests without discovery and blocking
+
+Usually this section will not apply to you. The client libraries support sending a request without waiting for a reply. This could be useful if you want to clean yum caches but don't really care if it actually happens everywhere.
+
+You will loose these abilities:
+
+ * Knowing if your request was received by any agents
+ * Any stats about processing times etc
+ * Any information about the success or failure of your request
+
+The above should make it clear already that this is a limited use case, it's essentially a broadcast request with no feedback loop.
+
+The code below will send a request to the _runonce_ action for an agent _puppetd_, once the request is dispatched I will have no idea if it got handled etc, my code will just continue onwards.
+
+{% highlight ruby %}
+p = rpcclient("puppetd")
+
+p.identity_filter "your.box.com"
+reqid = p.runonce(:forcerun => true, :process_results => false)
+{% endhighlight %}
+
+This will honor any attached filters set either programatically or through the command line, it will send the request but will
+just not handle any responses. All it will do is return the request id.
+
+## Doing your own discovery
+For web applications you'd probably use cached copied of Registration data to do discovery rather than wait for MC to do discovery between each request.
+
+To do this, you'll need to construct a filter and a list of expected hosts and then do a custom call:
+
+{% highlight ruby %}
+puppet = rpcclient("puppetd")
+
+printrpc puppet.custom_request("runonce", {:forcerun => true}, "your.box.com", {"identity" => "your.box.com"})
+{% endhighlight %}
+
+This will do a call with exactly the same stats, block and other semantics as a normal call like:
+
+{% highlight ruby %}
+printrpc puppet.runonce(:forcerun => true)
+{% endhighlight %}
+
+But instead of doing any discovery it will use the host list and filter you supplied in the call.
+
+## The _rpcutil_ Helper Agent
+
+A helper agent called [_rpcutil_][RPCUtil] is included that helps you gather stats, inventory etc about the running daemon.
--- /dev/null
+---
+layout: default
+title: SimpleRPC Introduction
+toc: false
+---
+[WritingAgents]: /mcollective/reference/basic/basic_agent_and_client.html
+[SimpleRPCAgents]: /mcollective/simplerpc/agents.html
+[SimpleRPCClients]: /mcollective/simplerpc/clients.html
+[SimpleRPCAuditing]: /mcollective/simplerpc/auditing.html
+[SimpleRPCAuthorization]: /mcollective/simplerpc/authorization.html
+[DDL]: /mcollective/reference/plugins/ddl.html
+[SimpleRPCMessageFormat]: /mcollective/simplerpc/messageformat.html
+[RPCUtil]: /mcollective/reference/plugins/rpcutil.html
+[WritingAgentsScreenCast]: http://mcollective.blip.tv/file/3808928/
+[RestGateway]: http://github.com/puppetlabs/marionette-collective/blob/master/ext/mc-rpc-restserver.rb
+
+MCollective is a framework for writing feature full agents and clients and provides a [rich system to do that][WritingAgents]. MCollective's native Client though is very low level, a bit like TCP/IP is to HTTP. Like TCP/IP the native client does not provide any Authentication, Authorization etc.
+
+MCollective Simple RPC is a framework on top of the standard client that abstracts away a lot of the complexity and provides a lot of convention and standards. It's a bit like using HTTP ontop of TCP/IP to create REST services.
+
+SimpleRPC is a framework that provides the following:
+
+ * Provide simple conventions for writing [agents][SimpleRPCAgents] and [clients][SimpleRPCClients], favoring convention over custom design
+ * Very easy to write agents including input validation and a sensible feedback mechanism in case of error
+ * Provide [audit logging][SimpleRPCAuditing] abilities of calls to agents
+ * Provide the ability to do [fine grain Authorization][SimpleRPCAuthorization] of calls to agents and actions.
+ * Has a [Data Definition Language][DDL] used to describe agents and assist in giving hints to auto generating user interfaces.
+ * The provided generic calling tool should be able to speak to most compliant agents
+ * Should you need to you can still write your own clients, this should be very easy too
+ * Return data should be easy to print, in most cases the framework should be able to print a sensible output with a single, provided, function. The [DDL] is used here to improve the standard one-size-fits-all methods.
+ * The full capabilities of the standard Client classes shouldddl still be exposed in case you want to write advanced agents and clients
+ * A documented [standard message format][SimpleRPCMessageFormat] built ontop of the core format.
+
+
+We've provided full tutorials on [Writing Simple RPC Agents][SimpleRPCAgents] and [Clients][SimpleRPCClients]. There is also a [screencast that will give you a quick look at what is involved in writing agents][WritingAgentsScreenCast].
+
+
+A bit of code probably says more than lots of English, so here's a simple hello world Agent, it just echo's back everything you send it in the _:msg_ argument:
+
+{% highlight ruby linenos %}
+module MCollective
+ module Agent
+ class Helloworld<RPC::Agent
+ # Basic echo server
+ def echo_action
+ validate :msg, String
+
+ reply.data = request[:msg]
+ end
+ end
+ end
+end
+{% endhighlight %}
+
+The nice thing about using a standard abstraction for clients is that you often won't even need to write a client for it, we ship a standard client that you can use to call the agent above:
+
+{% highlight console %}
+ % mco rpc helloworld echo msg="Welcome to MCollective Simple RPC"
+ Determining the amount of hosts matching filter for 2 seconds .... 1
+
+ devel.your.com : OK
+ "Welcome to MCollective Simple RPC"
+
+
+
+ ---- rpctest#echo call stats ----
+ Nodes: 1
+ Start Time: Wed Dec 23 20:49:14 +0000 2009
+ Discovery Time: 0.00ms
+ Agent Time: 54.35ms
+ Total Time: 54.35ms
+{% endhighlight %}
+
+Note: This example is not complete. Please see the [agents][SimpleRPCAgents] and [clients][SimpleRPCClients] pages for a walkthrough.
+
+You could also use *mco rpc* like this and achieve the same result:
+
+{% highlight console %}
+ % mco rpc helloworld echo msg="Welcome to MCollective Simple RPC"
+{% endhighlight %}
+
+For multiple options just add more *key=val* pairs at the end
+
+But you can still write your own clients, it's incredibly simple, full details of a client is out of scope for the introduction - see the [SimpleRPCClients] page instead for full details - but here is some sample code to do the same call as above including full discovery and help output:
+
+{% highlight ruby linenos %}
+#!/usr/bin/ruby
+
+require 'mcollective'
+
+include MCollective::RPC
+
+mc = rpcclient("helloworld")
+
+printrpc mc.echo(:msg => "Welcome to MCollective Simple RPC")
+
+printrpcstats
+{% endhighlight %}
+
+With a standard interface come a lot of possibilities, just like the standard one-size-fits-all CLI client above you can make web interfaces, there's a [simple MCollective <-> REST bridge in the ext directory][RestGateway].
+
+A helper agent called [_rpcutil_][RPCUtil] is included that helps you gather stats, inventory etc about the running daemon.
--- /dev/null
+---
+layout: default
+title: SimpleRPC Message Format
+---
+[MessageFormat]: ../reference/basic/messageformat.html
+[ErrorCodes]: clients.html#dealing-with-the-results-directly
+
+SimpleRPC has a specific message structure that builds on the core
+[MessageFormat]. As such SimpleRPC is simply a plugin developed
+ontop of The Marionette Collective rather than an integrated part.
+
+The core messages has a _:body_ structure where agents and clients
+can send any data between nodes and clients. All the SimpleRPC
+structures below goes in this body. Filters, targets etc all use the
+standard core [MessageFormat].
+
+# Requests
+
+A basic SimpleRPC message can be seen below:
+
+{% highlight ruby %}
+{:agent => "echo",
+ :action => "echo",
+ :caller => "cert=rip",
+ :data => {:message => "hello world"},
+ :process_results => true}
+{% endhighlight %}
+
+This structure will be sent as the _:body_ of the core message, you might create
+this request using the command below:
+
+{% highlight ruby %}
+ e = rpcclient("echo")
+ e.echo(:message => "hello world")
+{% endhighlight %}
+
+## :agent
+
+Records the agent that this message is targetted at.
+
+## :action
+
+The action being called. As the core protocol has no concept of actions per
+agent this provides the needed data to route the request to the right code in
+the SimpleRPC agent
+
+## :caller
+
+The callerid initiating the request. This is redundant and might be removed
+later since the core message format also includes this information - the core
+did not always include it. Removing it will only break backwards compatability
+with really old versions.
+
+## :data
+
+The data being sent to the SimpleRPC action as its _request_ structure,
+technically this can be any data but by SimpleRPC convention this would be a
+hash with keys being of the Symbol type as per the example above
+
+## :process_results
+
+Indicates the client preference to receive a result or not, the SimpleRPC agent
+should not send a response at all if this is true.
+
+# Replies
+
+As with requests the replies just build on the core [MessageFormat] and would be
+in the body of the standard message.
+
+A typical rely would look like:
+
+{% highlight ruby %}
+{:statuscode => 0
+ :statusmsg => "OK",
+ :data => {:message => "hello world"}}
+{% endhighlight %}
+
+## :statuscode and :statusmsg
+
+The statuscode and statusmsg are related and is used for error propagation
+through the collective.
+
+These are the [documented errors clients receive][ErrorCodes] and will result
+in exceptions raised on the client in some cases.
+
+The agent's _fail_ and _fail!_ methods will manipulate these structures.
+
+## :data
+
+This is a freeform variable for any data being returned by agents. Technically
+it can be anything but by SimpleRPC convention it's a hash with keys being of
+type Symbol.
--- /dev/null
+---
+layout: default
+title: Terminology
+toc: false
+---
+[Software_agent]: http://en.wikipedia.org/wiki/Software_agent
+[Plugin]: http://en.wikipedia.org/wiki/Plugin
+[Publish_subscribe]: http://en.wikipedia.org/wiki/Publish_subscribe
+[Apache ActiveMQ]: http://activemq.apache.org/
+[SimpleRPCAgents]: /mcollective/simplerpc/agents.html
+[SimpleRPCIntroduction]: /mcollective/simplerpc/
+[WritingFactsPlugins]: /mcollective/reference/plugins/facts.html
+[Subcollective]: /mcollective/reference/basic/subcollectives.html
+[Registration]: /mcollective/reference/plugins/registration.html
+[SimpleRPCAuthorization]: /mcollective/simplerpc/authorization.html
+
+This page documents the various terms used in relation to mcollective.
+
+## Server
+The mcollective daemon, an app server for hosting Agents and managing
+the connection to your Middleware.
+
+## Node
+The Computer or Operating System that the Server runs on.
+
+## Agent
+A block of Ruby code that performs a specific role, the main reason for
+mcollective's existence is to host agents. Agents can perform tasks like
+manipulate firewalls, services, packages etc. See [Wikipedia][Software_agent].
+
+Docs to write your own can be seen in [SimpleRPCAgents]
+
+## Plugins
+Ruby code that lives inside the server and takes on roles like security, connection
+handling, agents and so forth. See [Wikipedia][Plugin]
+
+## Middleware
+A [publish subscribe][Publish_subscribe] based system like [Apache ActiveMQ].
+
+## Connector
+A plugin of the type *MCollective::Connector* that handles the communication with your chosen Middleware.
+
+## Name Space
+Currently messages are sent to the middleware directed at topics named */topic/mcollective.package.command*
+and replies come back on */topic/mcollective.package.reply*.
+
+In this example the namespace is "mcollective" and all servers and clients who wish to form part of the same
+Collective must use the same name space.
+
+Middleware can generally carry several namespaces and therefore several Collectives.
+
+## Collective
+A combination of Servers, Nodes and Middleware all operating in the same Namespace.
+
+Multiple collectives can be built sharing the same Middleware but kept separate by using different Namespaces.
+
+## Subcollective
+A server can belong to many Namespaces. A [Subcollective] is a Namespace that only a subset of a full collectives nodes belong to.
+
+Subcolllectives are used to partition networks and to control broadcast domains in high traffic networks.
+
+## Simple RPC
+A Remote Procedure Call system built ontop of MCollective that makes it very simple to write feature
+full agents and clients. See [SimpleRPCIntroduction].
+
+## Action
+Agents expose tasks, we call these tasks actions. Each agent like a exim queue management agent might
+expose many tasks like *mailq*, *rm*, *retry* etc. These are al actions provided by an agent.
+
+## Facts
+Discreet bits of information about your nodes. Examples could be the domain name, country,
+role, operating system release etc.
+
+Facts are provided by plugins of the type *MCollective::Facts*, you can read about writing
+your own in [WritingFactsPlugins]
+
+## Registration
+Servers can send regular messages to an agent called *registration*. The code that sends the
+registration messages are plugins of the type *MCollective::Registration*. See [Registration].
+
+## Security
+A plugin of the type *MCollective::Security* that takes care of encryption, authentication
+and encoding of messages on which will then be passed on to the Connector for delivery to the Collective.
+
+## Client
+Software that produce commands for agents to process, typically this would be a computer with
+the client package installed and someone using the commands like *mc-package* to interact with Agents.
+
+Often clients will use the *MCollective::Client* library to communicate to the Collective
+
+## User
+Servers and Clients all authenticate to the Middleware, user would generally refer to the username
+used to authenticate against the middleware.
+
+## Audit
+In relation to SimpleRPC an audit action is a step requests go through where they can get
+logged to disk or other similar action
+
+## Authorization
+In relation to SimpleRPC authorization is a processes whereby requests get allowed or denied
+based on some identifying information of the requester. See [SimpleRPCAuthorization].
Code Review / packages / trusty / mcollective.git / commitdiff
