666 lines
32 KiB
Groff
666 lines
32 KiB
Groff
.\" OpenPBS (Portable Batch System) v2.3 Software License
|
|
.\"
|
|
.\" Copyright (c) 1999-2000 Veridian Information Solutions, Inc.
|
|
.\" All rights reserved.
|
|
.\"
|
|
.\" ---------------------------------------------------------------------------
|
|
.\" For a license to use or redistribute the OpenPBS software under conditions
|
|
.\" other than those described below, or to purchase support for this software,
|
|
.\" please contact Veridian Systems, PBS Products Department ("Licensor") at:
|
|
.\"
|
|
.\" www.OpenPBS.org +1 650 967-4675 sales@OpenPBS.org
|
|
.\" 877 902-4PBS (US toll-free)
|
|
.\" ---------------------------------------------------------------------------
|
|
.\"
|
|
.\" This license covers use of the OpenPBS v2.3 software (the "Software") at
|
|
.\" your site or location, and, for certain users, redistribution of the
|
|
.\" Software to other sites and locations. Use and redistribution of
|
|
.\" OpenPBS v2.3 in source and binary forms, with or without modification,
|
|
.\" are permitted provided that all of the following conditions are met.
|
|
.\" After December 31, 2001, only conditions 3-6 must be met:
|
|
.\"
|
|
.\" 1. Commercial and/or non-commercial use of the Software is permitted
|
|
.\" provided a current software registration is on file at www.OpenPBS.org.
|
|
.\" If use of this software contributes to a publication, product, or service
|
|
.\" proper attribution must be given; see www.OpenPBS.org/credit.html
|
|
.\"
|
|
.\" 2. Redistribution in any form is only permitted for non-commercial,
|
|
.\" non-profit purposes. There can be no charge for the Software or any
|
|
.\" software incorporating the Software. Further, there can be no
|
|
.\" expectation of revenue generated as a consequence of redistributing
|
|
.\" the Software.
|
|
.\"
|
|
.\" 3. Any Redistribution of source code must retain the above copyright notice
|
|
.\" and the acknowledgment contained in paragraph 6, this list of conditions
|
|
.\" and the disclaimer contained in paragraph 7.
|
|
.\"
|
|
.\" 4. Any Redistribution in binary form must reproduce the above copyright
|
|
.\" notice and the acknowledgment contained in paragraph 6, this list of
|
|
.\" conditions and the disclaimer contained in paragraph 7 in the
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
.\"
|
|
.\" 5. Redistributions in any form must be accompanied by information on how to
|
|
.\" obtain complete source code for the OpenPBS software and any
|
|
.\" modifications and/or additions to the OpenPBS software. The source code
|
|
.\" must either be included in the distribution or be available for no more
|
|
.\" than the cost of distribution plus a nominal fee, and all modifications
|
|
.\" and additions to the Software must be freely redistributable by any party
|
|
.\" (including Licensor) without restriction.
|
|
.\"
|
|
.\" 6. All advertising materials mentioning features or use of the Software must
|
|
.\" display the following acknowledgment:
|
|
.\"
|
|
.\" "This product includes software developed by NASA Ames Research Center,
|
|
.\" Lawrence Livermore National Laboratory, and Veridian Information
|
|
.\" Solutions, Inc.
|
|
.\" Visit www.OpenPBS.org for OpenPBS software support,
|
|
.\" products, and information."
|
|
.\"
|
|
.\" 7. DISCLAIMER OF WARRANTY
|
|
.\"
|
|
.\" THIS SOFTWARE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. ANY EXPRESS
|
|
.\" OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
|
|
.\" OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT
|
|
.\" ARE EXPRESSLY DISCLAIMED.
|
|
.\"
|
|
.\" IN NO EVENT SHALL VERIDIAN CORPORATION, ITS AFFILIATED COMPANIES, OR THE
|
|
.\" U.S. GOVERNMENT OR ANY OF ITS AGENCIES BE LIABLE FOR ANY DIRECT OR INDIRECT,
|
|
.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
|
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
|
|
.\" OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
|
|
.\" LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
|
|
.\" NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
|
|
.\" EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
.\"
|
|
.\" This license will be governed by the laws of the Commonwealth of Virginia,
|
|
.\" without reference to its choice of law rules.
|
|
.if \n(Pb .ig Iq
|
|
.TH pbs_mom 8B "" Local PBS
|
|
.so ../ers/ers.macros
|
|
.Iq
|
|
.SH NAME
|
|
pbs_mom \- start a pbs batch execution mini-server
|
|
.SH SYNOPSIS
|
|
pbs_mom [\^\-a\ alarm\^] [\^\-C\ chkdirectory\^] [\^\-c\ config\^] [\^\-D\^] [\^\-d\ directory] [\^\-F\^] [\^\-h\ help] [\^\-H\ hostname] [\^\-L\ logfile] [\^\-M\ MOMport\^] [\^\-R\ RPPport\^] [\^\-p\^|\^\-q\^|\^\-r] [\^\-w]
|
|
[\^\-x]
|
|
.LP
|
|
SH DESCRIPTION
|
|
The
|
|
.B pbs_mom
|
|
command starts the operation of a batch \fBM\fPachine \fBO\fPriented
|
|
\fBM\fPini\-server, MOM, on the local host.
|
|
Typically, this command will be in a local boot file such as
|
|
.I /etc/rc.local .
|
|
To insure that the pbs_mom
|
|
command is not runnable by the general user community, the server
|
|
will only execute if its real and effective uid is zero.
|
|
.LP
|
|
One function of pbs_mom is to place jobs into execution as directed
|
|
by the server, establish resource usage limits, monitor the job's usage,
|
|
and notify the server when the job completes. If they exist, pbs_mom will
|
|
execute a prologue script before executing a job and an epilogue script
|
|
after executing the job.
|
|
The next function of pbs_mom is to respond to resource monitor requests.
|
|
This was done by a separate process in previous versions of PBS but
|
|
has now been combined into one process.
|
|
The resource monitor function is provided mainly for the PBS scheduler.
|
|
It provides information about the status of running jobs, memory available etc.
|
|
The next function of pbs_mom is to respond to task manager requests.
|
|
This involves communicating with running tasks over a tcp socket
|
|
as well as communicating with other MOMs within a job (aka a "sisterhood").
|
|
.LP
|
|
Pbs_mom will record a diagnostic message in a log file for any
|
|
error occurrence. The log files are maintained in the
|
|
.I mom_logs
|
|
directory below the home directory of the server.
|
|
If the log file cannot be opened, the diagnostic message is written
|
|
to the system console.
|
|
.SH OPTIONS
|
|
.IP "\-A alias" 16
|
|
Used with \-m (multi-mom option) to give the alias name of this instance
|
|
of pbs_mom
|
|
.IP "\-a alarm" 16
|
|
Specifies the alarm timeout in seconds for computing a resource.
|
|
Every time a resource request is processed, an alarm is set for the
|
|
given amount of time. If the request has not completed before the
|
|
given time, an alarm signal is generated.
|
|
The default is 5 seconds.
|
|
.IP "\-C chkdirectory" 16
|
|
Specifies the path of the directory used to hold checkpoint files.
|
|
[Currently this is only valid on Cray systems.]
|
|
The default directory is PBS_HOME/spool/checkpoint, see the \-d option.
|
|
The directory specified with the \-C option must be owned by root and
|
|
accessible (rwx) only by root to protect the security of the checkpoint files.
|
|
.IP "\-c config" 16
|
|
Specifies an alternative configuration file, see description below.
|
|
If this is a relative file name it will be relative to PBS_HOME/mom_priv,
|
|
see the \-d option. If the specified file cannot be opened, pbs_mom will abort.
|
|
If the \-c option is not supplied, pbs_mom will attempt to open the default
|
|
configuration file "config" in PBS_HOME/mom_priv. If this file is not
|
|
present, pbs_mom will log the fact and continue.
|
|
.IP "\-h help" 16
|
|
Displays the help/usage message.
|
|
.IP "\-H hostname" 16
|
|
Sets the MOM's hostname. This can be useful on multi-homed networks.
|
|
.IP "\-D" 16
|
|
Debug mode. Do not fork.
|
|
.IP "\-d directory" 16
|
|
Specifies the path of the directory which is the home of the servers
|
|
working files, PBS_HOME.
|
|
This option is typically used along with \-M when debugging MOM.
|
|
The default directory is given by $PBS_SERVER_HOME
|
|
which is typically
|
|
.Ty /usr/spool/PBS .
|
|
.IP "\-F" 16
|
|
Do not fork. Use when running under systemd.
|
|
.IP "\-L logfile" 16
|
|
Specifies an absolute path name for use as the log file.
|
|
If not specified, MOM will
|
|
open a file named for the current date in the PBS_HOME/mom_logs
|
|
directory, see the
|
|
.At \-d
|
|
option.
|
|
.IP "\-m" 16
|
|
Directs the MOM to start in multi-mom mode. In addition to using -m the
|
|
\-M, \-R and \-A options need to be used to properly start a MOM in multi-mom mode.
|
|
For example
|
|
.B pbs_mom -m -M 30002 -R 30003 -A alias-host
|
|
will start pbs_mom with the service port on port 30002, the manager port at 30003 and with
|
|
the name alias-host.
|
|
.IP "\-M port" 16
|
|
Specifies the port number on which the mini-server (MOM) will
|
|
listen for batch requests.
|
|
.IP "\-R port" 16
|
|
Specifies the port number on which the mini-server (MOM) will
|
|
listen for resource monitor requests, task manager requests and inter-MOM
|
|
messages.
|
|
.IP "\-p" 16
|
|
(Default after version 2.4.0) (Preserve running jobs) -- Specifies the
|
|
impact on jobs which were in execution when the mini-server shut-down.
|
|
The \-p option tries to preserve any running jobs when the MOM restarts.
|
|
The new mini-server will not be the parent of any running jobs, MOM
|
|
has lost control of her offspring (not a new situation for a mother).
|
|
The MOM will allow the jobs to continue to run and monitor them
|
|
indirectly via polling. All recovered jobs will report an exit code
|
|
of 0 when they are complete. The \-p option is mutually exclusive with
|
|
the \-r, \-P and \-q options.
|
|
.IP "\-P" 16
|
|
(Terminate all jobs and remove them from the queue) -- Specifies the impact
|
|
on jobs which were in execution when the mini-server shut-down.
|
|
With the \-P option, it is assumed that either the entire system
|
|
has been restarted or the MOM has been down so long that it can no longer
|
|
guarantee that the pid of any running process is the same as the recorded job
|
|
process pid of a recovering job. Unlike the \-p option no attempt is made to
|
|
try and preserve or recover running jobs. All jobs are terminated and removed
|
|
from the queue.
|
|
The \-q option is mutually exclusive with the \-p, \-q and \-r options.
|
|
.IP "\-q" 16
|
|
(Requeue all jobs - This is the default behavior in versions prior
|
|
to 2.4.0) -- Specifies the impact on jobs which were in execution when
|
|
the mini-servershut-down. Do not terminate running processes.
|
|
With the \-q option, it is assumed that either the entire system
|
|
has been restarted or the MOM has been down so long that it can no longer
|
|
guarantee that the pid of any running process is the same as the recorded job
|
|
process pid of a recovering job. No attempt is made to kill job processes.
|
|
The MOM will mark the jobs as terminated and notify the batch server which
|
|
owns the job. Re-runnable jobs will be requeued.
|
|
The \-q option is mutually exclusive with the \-p, \-P and \-r options.
|
|
.IP "\-r" 16
|
|
(Terminate running processes and requeue all jobs) -- Specifies the impact on jobs
|
|
which were in execution when the mini-server shut-down. With the \-r option,
|
|
MOM will kill any processes belonging to running jobs, mark the jobs as
|
|
terminated and notify the batch server that owns the job. Re-runnable jobs are reset
|
|
to a queued state so they can be run again.
|
|
The \-r option is mutually exclusive with the \-p, \-P and \-q options.
|
|
.IP
|
|
If the \-r option is used following a reboot,
|
|
process IDs (pids) may be reused and
|
|
MOM may kill a process that is not a batch session.
|
|
.IP "\-S port" 16
|
|
Specifies the port number on which the pbs_server is listening for requests.
|
|
If pbs_server is started with a \-p option, pbs_mom will need to use the \-S
|
|
option and match the port value which was used to start pbs_server.
|
|
.IP "\-w" 16
|
|
When started with -w, pbs_moms wait until they get their MOM hierarchy file from pbs_server to send their first update, or until 10 minutes pass. This reduces network traffic on startup and can bring up clusters faster.
|
|
.IP "\-x" 16
|
|
Disables the check for privileged port resource monitor connections. This is
|
|
used mainly for testing since the privileged port is the only
|
|
mechanism used to prevent any ordinary user from connecting.
|
|
.LP
|
|
.SH CONFIGURATION FILE
|
|
The configuration file may be specified on the command line at
|
|
program start with the \-c flag. The use of this file is
|
|
to provide several types of run time information to pbs_mom:
|
|
static resource names and values, external resources provided
|
|
by a program to be run on request via a shell escape, and values
|
|
to pass to internal set up functions at initialization
|
|
(and re-initialization).
|
|
.LP
|
|
Each item type is on a single line with the component parts separated by
|
|
white space. If the line starts with a hash mark (pound sign, #),
|
|
the line is considered to be a comment and is skipped.
|
|
.IP "Static Resources"
|
|
For static resource names and values, the configuration file contains a
|
|
list of resource names/values pairs, one pair per line and separated by
|
|
white space. An Example of static resource names and values could be
|
|
the number of tape drives of different types and could be specified by
|
|
.IP
|
|
.Ty "tape3480 4"
|
|
.br
|
|
.Ty "tape3420 2"
|
|
.br
|
|
.Ty "tapedat 1"
|
|
.br
|
|
.Ty "tape8mm 1"
|
|
.IP "Shell Commands"
|
|
If the first character of the value is an exclamation mark (!),
|
|
the entire rest of the line is saved to be executed through the services of
|
|
the \fBsystem\fP(3) standard library routine.
|
|
.IP
|
|
The shell escape provides a means for the resource monitor to yield
|
|
arbitrary information to the scheduler. Parameter substitution is
|
|
done such that the value of any qualifier sent with the query, as
|
|
explained below, replaces a token with a percent sign (%) followed
|
|
by the name of the qualifier. For example, here is a configuration file
|
|
line which gives a resource name of "escape":
|
|
.IP
|
|
.Ty "escape !echo %xxx %yyy"
|
|
.IP
|
|
If a query for "escape" is sent with no qualifiers, the command
|
|
executed would be "echo %xxx %yyy". If one qualifier is sent,
|
|
"escape[xxx=hi there]", the command executed would be "echo hi there %yyy".
|
|
If two qualifiers are sent, "escape[xxx=hi][yyy=there]", the command
|
|
executed would be "echo hi there". If a qualifier is sent with
|
|
no matching token in the command line, "escape[zzz=snafu]", an error
|
|
is reported.
|
|
.IP size[fs=<FS>]
|
|
Specifies that the available and configured disk space in the <FS> filesystem
|
|
is to be reported to the pbs_server and scheduler. NOTE: To request disk space
|
|
on a per job basis, specify the file resource as in 'qsub \-l
|
|
nodes=1,file=1000kb' For example, the available and configured disk space in
|
|
the /localscratch filesystem will be reported:
|
|
.IP
|
|
.Ty "size[fs=/localscratch]"
|
|
.br
|
|
.IP "Initialization Value"
|
|
An initialization value directive has a name which starts with a
|
|
dollar sign ($) and must be known to MOM via an internal table.
|
|
The entries in this table now are:
|
|
.RS
|
|
.IP auto_ideal_load
|
|
if jobs are running, sets idea_load based on a simple expression. The expressions
|
|
start with the variable 't' (total assigned CPUs) or 'c' (existing CPUs), an
|
|
operator (+ \- / *), and followed by a float constant.
|
|
.IP
|
|
.Ty "$auto_ideal_load t-0.2"
|
|
.br
|
|
.IP auto_max_load
|
|
if jobs are running, sets max_load based on a simple expression. The expressions
|
|
start with the variable 't' (total assigned CPUs) or 'c' (existing CPUs), an
|
|
operator (+ \- / *), and followed by a float constant.
|
|
.IP cputmult
|
|
which sets a factor used to adjust cpu time used by a job. This is provided
|
|
to allow adjustment of time charged and limits enforced where the job might
|
|
run on systems with different cpu performance.
|
|
If Mom's system is faster than the reference system, set cputmult to a decimal
|
|
value greater than 1.0. If Mom's system is slower, set cputmult to a value
|
|
between 1.0 and 0.0.
|
|
For example:
|
|
.IP
|
|
.Ty "$cputmult 1.5
|
|
.br
|
|
.Ty "$cputmult 0.75
|
|
.IP configversion
|
|
specifies the version of the config file data, a string.
|
|
.IP check_poll_time
|
|
specifies the MOM interval in seconds. MOM checks each job for updated
|
|
resource usages, exited processes, over-limit conditions, etc. once per
|
|
interval. This value should be equal or lower to pbs_server's job_stat_rate.
|
|
High values result in stale information reported to pbs_server. Low values
|
|
result in increased system usage by MOM. Default is 45 seconds.
|
|
.IP down_on_error
|
|
causes MOM to report itself as state "down" to pbs_server in the event of a
|
|
failed health check. This feature is EXPERIMENTAL and likely to be removed in
|
|
the future. See HEALTH CHECK below.
|
|
.IP enablemomrestart
|
|
enable automatic restarts of MOM. If enabled, MOM will check if its binary has
|
|
been updated and restart itself at a safe point when no jobs are running; thus
|
|
making upgrades easier. The check is made by comparing the mtime of the
|
|
pbs_mom executable. Command-line args, the process name, and the PATH env
|
|
variable are preserved across restarts. It is recommended that this not be
|
|
enabled in the config file, but enabled when desired with momctl (see RESOURCES
|
|
for more information.)
|
|
.IP ideal_load
|
|
ideal processor load. Represents a low water mark for the load average. Nodes
|
|
that are currently busy will consider itself free after falling below ideal_load.
|
|
.IP igncput
|
|
Ignore cpu time violations on this mom, meaning jobs will not be cancelled due to exceeding their limits for cpu time.
|
|
.IP ignmem
|
|
Ignore memory violations on this mom, meaning jobs will not be cancelled due to exceeding their memory limits.
|
|
.IP ignvmem
|
|
If set to true, then pbs_mom will ignore vmem/pvmem limit enforcement.
|
|
.IP ignwalltime
|
|
If set to true, then pbs_mom will ignore walltime limit enforcement.
|
|
.IP job_output_file_mask
|
|
Specifies a mask for creating job output and error files. Values can be specified in base 8, 10, or 16; leading 0 implies octal and leading 0x or 0X hexadecimal. A value of "userdefault" will use the user's default umask.
|
|
.Ty "$job_output_file_mask 027"
|
|
.br
|
|
.IP log_directory
|
|
Changes the log directory. Default is $TORQUEHOME/mom_logs/. $TORQUEHOME default is /var/spool/torque/ but can be changed in the ./configure script. The value is a string and should be the full path to the desired mom log directory.
|
|
.Ty "$log_directory /opt/torque/mom_logs/"
|
|
.br
|
|
.IP logevent
|
|
which sets the mask that determines which event types are logged by pbs_mom.
|
|
For example:
|
|
.IP
|
|
.Ty "$logevent 0x1fff"
|
|
.br
|
|
.Ty "$logevent 255"
|
|
.IP
|
|
The first example would set the log event mask to 0x1ff (511) which enables
|
|
logging of all events including debug events. The second example would set
|
|
the mask to 0x0ff (255) which enables all events except debug events.
|
|
.IP log_file_suffix
|
|
Optional suffix to append to log file names. If %h is the suffix, pbs_mom appends the hostname for where the log files are stored if it knows it, otherwise it will append the hostname where the mom is running.
|
|
.Ty "$log_file_suffix tom = 20100223.tom"
|
|
.br
|
|
.IP log_keep_days
|
|
Specifies how many days to keep log files. pbs_mom deletes log files older than the specified number of days. If not specified, pbs_mom won't delete log files based on their age.
|
|
.IP loglevel
|
|
specifies the verbosity of logging with higher numbers specifying more verbose
|
|
logging. Values may range between 0 and 7.
|
|
.IP log_file_max_size
|
|
If this is set to a value > 0 then pbs_mom will roll the
|
|
current log file to log-file-name.1 when its size is greater than
|
|
or equal to the value of log_file_max_size. This value is
|
|
interpreted as kilobytes.
|
|
.
|
|
.IP log_file_roll_depth
|
|
If this is set to a value >=1 and log_file_max_size is set
|
|
then pbs_mom will continue rolling the log files to
|
|
log-file-name.log_file_roll_depth.
|
|
.
|
|
.IP max_load
|
|
maximum processor load. Nodes over this load average are considered busy (see
|
|
ideal_load above).
|
|
.
|
|
.IP memory_pressure_threshold
|
|
The option is only available, if pbs_mom is enabled to use cpusets.
|
|
If set to a value > 0, a job gets killed if its memory pressure exceeds
|
|
this value, and if $memory_pressure_duration is set.
|
|
The default is 0 (memory pressure recording is off).
|
|
.br
|
|
See cpuset(7) for more information about memory pressure.
|
|
.IP memory_pressure_duration
|
|
The option is only available, if pbs_mom is enabled to use cpusets.
|
|
Specifies the number of subsequent MOM intervals a job's memory pressure must be
|
|
above $memory_pressure_threshold to get killed.
|
|
The default is 0 (jobs are never killed due to memory pressure).
|
|
set
|
|
.br
|
|
See cpuset(7) for more information about memory pressure.
|
|
.
|
|
.IP node_check_script
|
|
specifies the fully qualified pathname of the health check script to run (see
|
|
HEALTH CHECK for more information).
|
|
.IP node_check_interval
|
|
specifies when to run the MOM health check. The check can be either periodic,
|
|
event-driver, or both. The value starts with an integer specifying the number
|
|
of MOM intervals between subsequent executions of the specified health check.
|
|
After the integer is an optional comma-separated list of event names.
|
|
Currently supported are "jobstart" and "jobend". This value defaults to 1 with
|
|
no events indicating the check is run every MOM interval. (see HEALTH CHECK for
|
|
more information)
|
|
.IP
|
|
.Ty "$node_check_interval 0 #Disabled."
|
|
.br
|
|
.Ty "$node_check_interval 0,jobstart #Only runs at job starts"
|
|
.br
|
|
.Ty "$node_check_interval 10,jobstart,jobend"
|
|
.IP
|
|
.IP nodefile_suffix
|
|
Specifies the suffix to append to a host names to denote the data channel network adapter in a multihomed compute node.
|
|
.Ty "$nodefile_suffix i"
|
|
With the suffix of 'i' and the control channel adapter with the name node01, the data channel would have a hostname of node01i.
|
|
.IP nospool_dir_list
|
|
If the job's output file should be in one of the paths specified here, then it will be spooled directly in that directory instead of the normal spool directory.
|
|
.br
|
|
Specified in the format path1, path2, etc.
|
|
.Ty $nospool_dir_list /home/mike/*,/var/tmp/spool/
|
|
.IP pbsclient
|
|
which causes a host name to be added to the list of hosts which will be allowed
|
|
to connect to MOM as long as they are using a privilaged port for the purposes
|
|
of resource monitor requests.
|
|
For example, here are two
|
|
configuration file lines which will allow the hosts "fred" and "wilma"
|
|
to connect:
|
|
.IP
|
|
.Ty "$pbsclient fred"
|
|
.br
|
|
.Ty "$pbsclient wilma"
|
|
.IP
|
|
Two host name are always allowed to connection to pbs_mom, "localhost" and the
|
|
name returned to pbs_mom by the system call gethostname(). These names need
|
|
not be specified in the configuration file. The hosts listed as "clients" can
|
|
issue Resource Monitor (RM) requests. Other MOM nodes and servers do not need
|
|
to be listed as clients.
|
|
.IP pbsserver
|
|
which defines hostnames running pbs_server that will be allowed to
|
|
submit jobs, issue Resource Monitor (RM) requests, and get status updates. MOM
|
|
will continually attempt to contact all server hosts for node status and state
|
|
updates. Like $PBS_SERVER_HOME/server_name, the hostname may be followed by a
|
|
colon and a port number. This parameter replaces the oft-confused $clienthost
|
|
parameter from TORQUE 2.0.0p0 and earlier. Note that the hostname in
|
|
$PBS_SERVER_HOME/server_name is used if no $pbsserver parameters are found
|
|
.IP prologalarm
|
|
Specifies maximum duration (in seconds) which the MOM will wait for the job prolog
|
|
or job job epilog to complete. This parameter default to 300 seconds (5 minutes)
|
|
.IP rcpcmd
|
|
Specify the the full path and argument to be used for remote file copies. This overrides
|
|
the compile-time default found in configure. This must contain 2 words: the full path to
|
|
the command and the switches. The copy command must be able to recursively copy files to
|
|
the remote host and accept arguments of the form "user@host:files" For example:
|
|
.IP
|
|
.Ty "$rcpcmd /usr/bin/rcp \-rp
|
|
.br
|
|
.Ty "$rcpcmd /usr/bin/scp \-rpB
|
|
.IP restricted
|
|
which causes a host name to be added to the list of hosts which will be allowed
|
|
to connect to MOM without needing to use a privilaged port. These names
|
|
allow for wildcard matching. For example, here is a configuration file
|
|
line which will allow queries from any host from the domain "ibm.com".
|
|
.IP
|
|
.Ty "$restricted *.ibm.com"
|
|
.IP
|
|
The restriction which applies to these connections is that only
|
|
internal queries may be made. No resources from a config file
|
|
will be found. This is to prevent any shell commands from being
|
|
run by a non-root process.
|
|
.br
|
|
This parameter is generally not required except for some versions of OSX.
|
|
.IP remote_checkpoint_dirs
|
|
Specifies what server checkpoint directories are remotely mounted. This directive is used to
|
|
tell the MOM which directories are shared with the server. Using remote checkpoint directories eliminates the need to copy the checkpoint files back and forth between the MOM and the server. This parameter is available in 2.4.1 and later.
|
|
.IP
|
|
.Ty "$remote_checkpoint_dirs /var/spool/torque/checkpoint"
|
|
.IP remote_reconfig
|
|
Enables the ability to remotely reconfigure pbs_mom with a new config file.
|
|
Default is disabled. This parameter accepts various forms of true, yes, and 1.
|
|
.IP source_login_batch
|
|
Specifies whether or not mom will source the /etc/profile, etc. type files for batch jobs. Parameter accepts various forms of true, false, yes, no, 1 and 0. Default is True.
|
|
.IP source_login_interactive
|
|
Specifies whether or not mom will source the /etc/profile, etc. type files for interactive jobs. Parameter accepts various forms of true, false, yes, no, 1 and 0. Default is True.
|
|
.IP spool_as_final_name
|
|
If set to true, jobs will spool directly as their output files, with no intermediate locations or steps. This is mostly useful for shared filesystems with fast writing capability.
|
|
.IP status_update_time
|
|
Specifies (in seconds) how often MOM updates its status information to
|
|
pbs_server. This value should correlate with the server's scheduling interval.
|
|
High values increase the load of pbs_server and the network. Low values cause
|
|
pbs_server to report stale information. Default is 45 seconds.
|
|
.IP tmpdir
|
|
Sets the directory basename for a per-job temporary directory. Before job
|
|
launch, MOM will append the jobid to the tmpdir basename and create the
|
|
directory. After the job exit, MOM will recursively delete it. The env
|
|
variable TMPDIR will be set for all pro/epilog scripts, the job script, and TM
|
|
tasks.
|
|
.br
|
|
Directory creation and removal is done as the job owner and group, so the owner
|
|
must have write permission to create the directory. If the directory already
|
|
exists and is owned by the job owner, it will not be deleted after the job. If
|
|
the directory already exists and is NOT owned by the job owner, the job start
|
|
will be rejected.
|
|
.IP timeout
|
|
Specifies the number of seconds before TCP messages will time out. TCP messages include job obituaries, and TM requests if RPP is disabled.
|
|
Default is 60 seconds.
|
|
.IP usecp
|
|
specifies which directories should be staged with cp instead of rcp/scp. If a
|
|
shared filesystem is available on all hosts in a cluster, this directive is used to
|
|
make these filesystems known to MOM. For example, if /home is NFS mounted on
|
|
all nodes in a cluster:
|
|
.IP
|
|
.Ty "$usecp *:/home /home"
|
|
.br
|
|
.IP varattr
|
|
This is similar to a shell escape above, but includes a TTL. The command will
|
|
only be run every TTL seconds. A TTL of \-1 will cause the command to be
|
|
executed only once. A TTL of 0 will cause the command to be run everytime
|
|
varattr is requested. This parameter may be used multiple times, but all
|
|
output will be grouped into a single "varattr" attribute in the request and
|
|
status output. The command should output data in the form of
|
|
.Ty varattrname=va1ue1[+value2]...
|
|
.IP
|
|
.Ty "$varattr 3600 /path/to/script [<ARGS>]..."
|
|
.IP use_smt
|
|
This option is only available, if pbs_mom is enabled to use cpusets.
|
|
It has only effect, if there are more that one logical processor per physical
|
|
core in the system (simultaneous multithreading or hyperthreading is enabled
|
|
via BIOS settings).
|
|
If set to true, all logical processors of allocated cores are added to the cpuset
|
|
of a job. If set to false, only the first logical processor per allocated core
|
|
is contained in the cpuset of a job.
|
|
The default is true.
|
|
.IP wallmult
|
|
which sets a factor used to adjust wall time usage by to job to a common
|
|
reference system. The factor is used for walltime calculations and limits
|
|
the same as cputmult is used for cpu time.
|
|
.RE
|
|
.LP
|
|
The configuration file must be executable and "secure". It must be owned by a user id and
|
|
group id less than 10 and not be world writable. Output from this file must be in
|
|
the format $VAR=$VAL, i.e.,
|
|
.IP
|
|
.Ty dataset13=20070104
|
|
.br
|
|
.Ty dataset22=20070202
|
|
.br
|
|
.Ty viraltest=abdd3
|
|
.LP
|
|
.IP xauthpath
|
|
Specifies the path to the xauth binary to enable X11 fowarding.
|
|
.IP mom_host
|
|
Sets the local hostname as used by pbs_mom.
|
|
.br
|
|
.SH LAYOUT FILE
|
|
There is also an optional layout file for creating multiple moms on one box in a specified layout. In the file, each mom on the single box is given its own hostname, cpu indexes, memory nodes (a linux construct), and memory size. This is useful for NUMA systems. Each line in the file specifies one mom. The file follows the following format:
|
|
.IP "<hostname> cpus=<X> mem=<Y> memsize=<Z>"
|
|
cpus and mem can be comma separated lists, while memsize should be a memory size in the format:
|
|
.IP "<number><units>"
|
|
For example, a file could contain the following line:
|
|
.IP "foohost-1 cpus=1,2 mem=1,2,3,4 memsize=8GB"
|
|
This would specify that foohost-1 has cpus 1 and 2, memory nodes 1-4, and a total of 8 GB of memory.
|
|
.SH RESOURCES
|
|
Resource Monitor queries can be made with momctl's \-q option to retrieve and
|
|
set pbs_mom options. Any configured static resource may be retrieved with a
|
|
request of the same name. These are resource requests not otherwise documented
|
|
in the PBS ERS.
|
|
.IP cycle
|
|
forces an immediate MOM cycle
|
|
.IP status_update_time
|
|
retrieve or set the $status_update_time parameter
|
|
.IP check_poll_time
|
|
retrieve or set the $check_poll_time parameter
|
|
.IP configversion
|
|
retrieve the config version
|
|
.IP jobstartblocktime
|
|
retrieve or set the $jobstartblocktime parameter
|
|
.IP enablemomrestart
|
|
retrieve or set the $enablemomrestart parameter
|
|
.IP loglevel
|
|
retrieve or set the $loglevel parameter
|
|
.IP down_on_error
|
|
retrieve or set the EXPERIMENTAL $down_on_error parameter
|
|
.IP "diag0 \- diag4"
|
|
retrieves various diagnostic information
|
|
.IP rcpcmd
|
|
retrieve or set the $rcpcmd parameter
|
|
.IP version
|
|
retrieves the pbs_mom version
|
|
.SH HEALTH CHECK
|
|
The health check script is executed directly by the pbs_mom daemon under the
|
|
root user id. It must be accessible from the compute node and may be a script
|
|
or compiled executable program. It may make any needed system calls and execute
|
|
any combination of system utilities but should not execute resource manager
|
|
client commands. Also, as of TORQUE 1.0.1, the pbs_mom daemon blocks until the
|
|
health check is completed and does not possess a built-in timeout.
|
|
Consequently, it is advisable to keep the launch script execution time short
|
|
and verify that the script will not block even under failure conditions.
|
|
.LP
|
|
If the script detects a failure, it should return the keyword 'ERROR' to stdout
|
|
followed by an error message. The message (up to 256 characters) immediately
|
|
following the ERROR string will be assigned to the node attribute 'message' of
|
|
the associated node.
|
|
.LP
|
|
If the script detects a failure when run from "jobstart", then the job will be
|
|
rejected. This should probably only be used with advanced schedulers like Moab
|
|
so that the job can be routed to another node.
|
|
.LP
|
|
TORQUE currently ignores ERROR messages by default, but advanced schedulers
|
|
like moab can be configured to react appropriately.
|
|
.LP
|
|
If the experimental $down_on_error MOM setting is enabled, MOM will set itself
|
|
to state down and report to pbs_server; and pbs_server will report the node as
|
|
"down". Additionally, the experimental "down_on_error" server attribute can be
|
|
enabled which has the same effect but moves the decision to pbs_server. It is
|
|
redundant to have MOM's $down_on_error and pbs_server's down_on_error features
|
|
enabled. See "down_on_error" in pbs_server_attributes(7B).
|
|
.LP
|
|
.SH FILES
|
|
.IP $PBS_SERVER_HOME/server_name
|
|
contains the hostname running pbs_server.
|
|
.IP $PBS_SERVER_HOME/mom_priv 10
|
|
the default directory for configuration files, typically
|
|
(/usr/spool/pbs)/mom_priv.
|
|
.IP $PBS_SERVER_HOME/mom_logs 10
|
|
directory for log files recorded by the server.
|
|
.IP $PBS_SERVER_HOME/mom_priv/prologue 10
|
|
the administrative script to be run before job execution.
|
|
.IP $PBS_SERVER_HOME/mom_priv/epilogue 10
|
|
the administrative script to be run after job execution.
|
|
.SH SIGNAL HANDLING
|
|
pbs_mom handles the following signals:
|
|
.IP SIGHUP
|
|
causes pbs_mom to re-read its configuration file, close and reopen the log
|
|
file, and reinitialize resource structures.
|
|
.IP SIGALRM
|
|
results in a log file entry. The signal is used to limit the time taken by
|
|
certain children processes, such as the prologue and epilogue.
|
|
.IP "SIGINT and SIGTERM"
|
|
results in pbs_mom exiting without terminating any running jobs.
|
|
This is the action for the following signals as well: SIGXCPU, SIGXFSZ,
|
|
SIGCPULIM, and SIGSHUTDN.
|
|
.IP "SIGUSR1, SIGUSR2"
|
|
causes MOM to increase and decrease logging levels, respectively.
|
|
.IP "SIGPIPE, SIGINFO"
|
|
are ignored.
|
|
.IP "SIGBUS, SIGFPE, SIGILL, SIGTRAP, and SIGSYS"
|
|
cause a core dump if the PBSCOREDUMP environmental variable is defined.
|
|
.LP
|
|
All other signals have their default behavior installed.
|
|
.SH EXIT STATUS
|
|
If the mini-server command fails to begin operation, the
|
|
server exits with a value greater than zero.
|
|
.SH SEE ALSO
|
|
pbs_server(8B), pbs_scheduler_basl(8B), pbs_scheduler_tcl(8B),
|
|
the PBS External Reference Specification, and the PBS Administrator's Guide.
|
|
.\" turn off any extra indent left by the Sh macro
|