clusterfork: a cluster admin tool

sudo apt-get install libgetopt-mixed-perl libconfig-simple-perl \
libio-interface-perl mc diff yum apt mutt

ARCH=`uname -a |cut -f 12 -d ' '`
if [[ ${ARCH} =~ 64 ]]; then ARCH="x86_64"; else ARCH="x86"; fi

sudo yum install perl-Config-Simple.noarch perl-Getopt-Mixed.noarch \
perl-Config-Simple.noarch perl-IO-Interface.$ARCH  mc.$ARCH \
diffutils.$ARCH yum.noarch mutt.$ARCH

$ clusterfork

        It looks like this is the 1st time you've run clusterfork
        as this user on this system.  An example .clusterforkrc file
        will be written to your home dir. Once you edit it to your
        specifications, run a non-destructive command with it
        (ie 'ls -lSh') to make sure it's working and examine the output
        so that you understand the workflow and the output.

        Remember that in order for clusterfork to work, passwordless ssh keys
        must be operational from the node where you execute clusterfork to the
        client nodes.  If you're going to use sudo to execute clusterfork, the
        root user public ssh key must be shared out to the clients.

        Typical cluster use implies a shared /home file system which means that
        the shared keys should only have to be installed once in
        /home/$USER/.ssh/authorized_keys.

        Please edit the ~/.clusterforkrc template that's just been written so that
        the next time things go smoother.

# This is the config file for the 'clusterfork' application (aka cf) which executes
# commands on a range of machines defined as below.  Use 'clusterfork -h'
# to view the help file

# Comments start with a pound ('#') sign and //cannot share the same line//
# with other configuration data.
# Strings do not need to be quoted unles they contain commas (imply list entries)

[ADMIN]

  # RPMDB - file that lists the RPMs that cf has been used to install
  RPMDB = /home/hmangala/BDUC_RPM_LIST.DB

  # ALLNODESFILE holds a list of ALL the IP nodes that this will support.
  # this should actually be generated outside of cf and written out if required.
  ALLNODESFILE = /home/hmangala/ALLNODESFILE

  # emails to notify of newly installed packages (note the escaping of the '@'
  EMAIL_LIST = "hmangala\@uci.edu, jsaska\@uci.edu, lopez\@uci.edu"

  # how many IP addresses to list on a line - 5 is pretty good.
  IPLISTWIDTH = 5

  # write all the results in this dir; comment out or assign to "" if you
  # want to write the results in the current working dir.
  # use fully qualified path; not '~/cf'.
  RESULTS_DIR = "/home/hmangala/cf"

  # command to install apps - if this is found in the command, triggers a routine to
  # email admins with updated install info.
  INSTALLCMD = "yum install -y"

[SGE]
# obviously only applies if you're running an SunGridEngine instance
  CELL          = bduc_nacs
  JOB_DIR       = /sge62/bduc_nacs/spool/qmaster/jobs
  EXECD_PORT    = 537
  QMASTER_PORT  = 536
  ROOT          = /sge62

[APPS]
# these will probably not change much among distros, but YMMV
 yum   = /usr/bin/yum
 diff  = /usr/bin/diff
 mutt  = /usr/bin/mutt
 mc    = /usr/bin/mc


[IPRANGE]
# you //definitely// need to change these.
# use ';' as separators, not commas.  Spaces are ignored.
  ADC_2X = 10.255.78.[10:22 26 35:49]  ;  10.255.78.[77:90] ;  12.23.34.[13:25 33:44 56:75]
  ADC_4X = 10.255.78.[50:76]
  ICS_2X = 10.255.89.[5:44]
  CLAWS = 10.255.78.[5:9]

  # example of mixing IP# ranges and both single hostanems and hostname ranges
  MIXED = a64-[002:004] ; 10.255.78.16; 10.255.78.[10:12] ; dabrick ; claw[5:9]

  # for a definition based on a script, the value must be in the form of:
  #   [SCRIPT:"whatever the script is"]
  # with required escaping being embedded in the submitted script
  # (see below for an example in QHOST)
  # the following QHOST example uses the host-local SGE 'qhost' and 'scut' utilities
  # to generate a list of hosts to process and filters only 'a64' hosts
  # which are responsive (don't have ' - ' entries). Returns the list as
  # a space-delimited set of names.
  QHOST = SCRIPT:"qhost |grep a64 | grep -v ' - ' | scut --c1=0 | perl -e 's/\\n/ /gi' -p"

  # Set temporarily dead nodes in here if required.  Keep the "", if no values to insert.
  # separate hostnames or ranges with ';' as above.

  IGNORE = "10.255.78.12 ; 10.255.78.48 ; 12.23.34.[22:25]"
  # or
  IGNORE = ""

[GROUPS]

# GROUPS can be composed of primary IPRANGE groups as well as other
# GROUP groups as long as they have been previously defined.
  ALL_2X = ICS_2X + ADC_2X
  CENTOS = ICS_2X + ADC_2X + ADC_4X
  ADC_ALL = ALL_2X + ADC_4X + CLAWS

# ADC_2X can be specified either with a central negation range:
    ADC_2X = a64-[104:181 -141:-167]

# or as this a chain of 2 separated ranges
# (you must use a ';' as a chain character)
    ADC_2X = a64-[104:140] ; a64-[168:181]

# you can also specify hostgroups with chains of mixed IP #s and hostnames
# they should resolve to each other and both are listed on output
    BOOBOO = 10.255.78.[22:46] ; a64-[168:177]

      --help / -h............. dump usage, tips

      --version / -v ......... dump version #

      --config=/alt/config/file .. an alternative config file.
          On 1st execution, clusterfork will write a template config file
          to ~/.clusterforkrc and exit.  You must edit the template to
          provide parameters for your particular setup.

     --target=[quoted IP_Range or predefined GROUP name]
          where IP_Range -> 12.23.23.[27:45 -33 54:88]
            or  'a64-[00023:35 -25:-28].bduc'
          (Note that leading zeros in the FIRST range specifier will be
          replicated in the output; the above pattern will generate:
            a64-00023.bduc, a64-00024.bduc, a64-00029.bduc, etc)

          where GROUP    -> 'ICS_2X,ADC_4X,CLAWS' (from config file)
          (see docs for longer exposition on IP_Ranges and GROUP definition)

      --hosts=[quoted space-delimited hostnames]
          For those times when you have a random number of hostnames to send.
          ie: --hosts='a64-111  a64-142  a64-183  a64-972'.  This will process
          them in the same way as '--target' above, but without having to spec
          the ranges.

      --delay=#  (or #s, #m, #h)
          introduces a this many (s)econds, (m)inutes, or (h)ours between successive
          commands to nodes to prevent overutilization of resources.  ie between
          initiating a 'yum -y upgrade' to prevent all the nodes from hitting a
          local repository simultaneously.  If no s|m|h is given, seconds are assumed.
          Fractions (0.01h) are fine.

      --listgroup=[GROUP,GROUP..] (GROUPs from config file)
          If no GROUP specified, dumps IP #s for ALL GROUPS.
          This option does not require a 'remote command' and ignores it
          if given.

      --fork (default) Sends 'remote command' to all nodes in parallel
          and saves output from nodes into a dated subdir for later perusal.
          If you submit a command to run in parallel, it must run to completion
          without intervention.  ie: to install on a CentOS node, the 'yum install'
          command must have the '-y' flag as well: 'yum install -y' to signify
          that 'Y' is assumed to be the answer to all questions.

          If you use the --fork command (as above)), instead of producing the
          stdout/err immediately, a new subdir will be created with the name
          of the format:
             REMOTE_CMD-(20 chars of the command)-time_date
          and the output for each node will be directed into a separate file
          named for the IP number or hostname (whichever the input spec was).

      --nofork .... Execs 'remote command' serially to each specified node and
          emits the output from each node as it's generated.  If executed with
          this option, it will produce a list of stanzas corresponding to the
          nodes requested:
          --------------------------------
          a64-101 [192.168.0.10]:
            <output from the command>

           a64-102 [192.168.0.11]:
             <output from the command>
          --------------------------------

      --script .... causes the command to run without any STDOUT until it emits
          the result dir at the end of the run.  The usual STDOUT is saved as:
          <result_dir>/LOG.

      --debug ..... causes voluminous debug messages to spew forth

$ clusterfork -h

clusterfork --config=./this_cf_file  --listgroup

$ clusterfork  --target=CLAWS  'ls -lSh'

$ clusterfork --target=192.18.1.[15:75 -66] \
'cd /sys/devices/system/edac/mc &&  grep [0-9]* mc*/csrow*/[cu]e_count'

$ clusterfork --target=ALL_ADC 'ping -c 1 bduc-login'

clusterfork --target='10.255.78.[12:45] 10.255.35.[101:165]' 'dpkg -l'

$ sudo clusterfork --target=ADC_2X --nofork 'lshw -short -class memory'

$ cf --target=QHOST 'locate libXp.so.6

$ sudo clusterfork --target=a64-[0005:0078 -22 -52 -57].bduc 'ps aux |grep gromacs'

$ sudo clusterfork --target=QHOST 'yum install -y tree'

QHOST = "SCRIPT:qhost |grep a64 | grep -v ' - ' | scut --c1=0 | perl -e 's/\\n/ /gi' -p"

$ clusterfork.pl --target=QHOST 'ls -lSh *gz'
INFO: Creating dir [REMOTE_CMD-ls--lSh--gz-15.28.09_2012-01-01]....success
======================================================
    Processing [QHOST]
======================================================

Host: a64-001 [10.255.89.5]:
Host: a64-002 [10.255.89.6]:
Host: a64-003 [10.255.89.7]:
Host: a64-004 [10.255.89.8]:
Host: a64-005 [10.255.89.9]:
Host: a64-006 [10.255.89.10]:
 ...
Host: a64-182 [10.255.78.91]:
Host: a64-183 [10.255.78.93]:

==========================================
  # of processes at start: [118]


Waiting for [41] hosts @ [0] sec:
        a64-142 a64-143 a64-144 a64-145 a64-146 a64-147 a64-148 a64-149 a64-150 a64-151 a64-152
        a64-153 a64-154 a64-155 a64-156 a64-157 a64-158 a64-159 a64-160 a64-161 a64-162 a64-163
        a64-164 a64-166 a64-167 a64-168 a64-169 a64-170 a64-171 a64-172 a64-173 a64-174 a64-175
        a64-176 a64-177 a64-178 a64-179 a64-180 a64-181 a64-182 a64-183
Waiting for [0] hosts @ [2] sec:

... All slave processes finished!

You can find the results of your command in the dir
        [ REMOTE_CMD-ls--lSh--gz-15.28.09_2012-01-01 ]

(View Summary file in 'less')
Summary of contents for files in /home/hmangala/cf/REMOTE_CMD-ls--lSh--gz-16.46.56_2012-08-07
Each line denotes MD5 identical output; wordcount shows similarity
Command: [ls -lSh *gz]
========================================================================
 line / word / chars | # |  hosts ->
         14 126 1040  107  a64-001 a64-002 a64-003 a64-004 a64-005 a64-006 a64-007 a64-008 ...
         14 126 1040   25  n101 n102 n103 n104 n105 n106 n107 n108 n109 n110 n111 n112 n113 ...
         14 112 1096   10  claw1 claw10 claw2 claw3 claw4 claw5 claw6 claw7 claw8 claw9

Summary of contents for files in /home/hmangala/cf/REMOTE_CMD-df--h--grep--56789-.-16.43.14_2012-08-07
Each line denotes MD5-identical output; wordcount shows similarity
Command: [df -h |grep [56789].%]
========================================================================
 line / word / chars | # |  hosts ->
              1 6 54    1  a64-182
               0 0 0  110  a64-001 a64-002 a64-003 a64-004 a64-005 a64-006 a64-007 ...
              1 6 49    1  claw6
              1 6 51    1  claw10
              1 6 46    1  a64-121
              1 6 46    1  a64-109
              1 6 46    1  claw1
            2 11 112   25  n101 n102 n103 n104 n105 n106 n107 n108 n109 n110 n111 ...
              1 6 46    1  a64-035