PBS Tools

Website:  http://www.osc.edu/~troy/pbstools/
Git repo:  https://github.com/tabaer/pbstools

PBS Tools is a collection of utilities that have been developed at
OSC, NICS, and elsewhere to aid in the administration and management
of PBS variants (including OpenPBS, PBS Pro, and TORQUE).  They have
been developed primarily on Linux clusters, but most of them should be
applicable to virtually any system running PBS.


User tools:
-----------

bin/ja -- Job accounting within a PBS job; similar to "ja" in NQE.

bin/pbsdcp -- Distributed copy command within a PBS job.  This can
optionally include an MPI-based scatter functionality that should
scale better than the default rcp-based approach.

bin/qexec -- Workalike for SGE qlogin/qrsh/qsh

bin/parallel-command-processor -- An MPI program for running an array
of serial jobs in the context of a single parallel job (a.k.a. the poor
man's job arrays).

bin/supermover -- A highly configurable data movement tool that can
wrap scp, globus-url-copy, hsi, and a host of other data movement
utilities.

bin/dmsub -- Submits data movement jobs using a variety of batch and
data movement back-ends; understands the transfer description formats
of RFT, DMover, and Stork.  May rely on supermover, depending on how
it's configured.

bin/dagsub -- A workalike for condor_submit_dag.  This allows the
submission of large, complex sets of dependent jobs using a relatively
simple syntax.  Relies on dmsub for emulating Stork data movement jobs.


Admin tools:
------------

bin/js -- Command line lookup of job scripts in the accounting
database.

bin/jobinfo -- Command line lookup of usage data in the accounting
database.

sbin/reaver -- Find (and optionally clean up) processes on a host
which have not been allocated jobs on that host; requires root
privilege.


Reporting tools:
----------------

sbin/jobstats -- Generate an HTML report summarizing PBS job
statistics over a period of months.  Relies on gnuplot; requires root
privilege.

sbin/find-outlyers -- Extract statistical outlyers from the
jobdist.dat file generated by jobstats.


Database/web tools:
-------------------
Note:  these all assume a MySQL DB back end running on the same server
as pbs_server and PHP-enabled httpd.

etc/create-tables.sql -- Create "pbsacct" database, table, and two
user accounts (pbsacct and webapp) used by the other DB tools.  To
initialize, run "mysql -u root -p passwd <create-tables.sql".

sbin/job-db-update -- Parses PBS job accounting records and inserts
them into DB.

sbin/jobscript-to-db -- Inserts one or more job scripts into the job DB.

sbin/spool-jobscripts -- Spools newly submitted jobscripts to /tmp and
then invokes jobscript-to-db on them.

web/jobinfo.php -- Presents information on a single job based from the
job DB.

web/jobs-by-user.php -- Presents information on all jobs by a given
user (over an optional date range) from the job DB.

web/jobs-by-group.php -- Presents information on all jobs by a given
group (over an optional date range) from the job DB.

web/jobs-by-node.php -- Presents information on all jobs on a given
node (over an optional date range) from the job DB.

web/jobstats*.php -- Presents statistics on all jobs (over an optional
date range) from the job DB.

web/software-usage*.php -- Presents information on software
usage on all jobs (over an optional date range) from the job DB.

web/active-users.php -- Presents a list of the most active users over
a date range.

web/active-groups.php -- Presents a list of the most active groups over
a date range.

web/problem-jobs.php -- Searches for potentially problematic jobs over
a date range.  (Note that what constitutes "problematic" is extremely
site specific.)

web/usage-summary.php -- Presents a usage summary over a date range.

web/site-specific.php -- Site specific settings.

web/sql-term.php -- "Terminal" for running arbitrary SQL queries on
job DB.

examples/db-example -- An example of putting job data from multiple PBS
instances into a single job DB.


Suggestions:
------------

Comments and suggestions on this software are welcomed; please send
them to Troy Baer <troy@osc.edu>.


Copyright:
----------
PBS Tools are copyright 2005-2016 Supercomputer Center and/or
copyright 2008-2015 University of Tennessee, and are licensed under
the terms of the GNU GPL v2.