1
This commit is contained in:
ruoyunbai
@@ -0,0 +1,105 @@
|
||||
|
||||
I. Description:
|
||||
We now have resource plugin capability for Torque. Specifically, this plugin provides an API
|
||||
through which arbitrary generic resources, generic metrics, varattrs, and features can be added
|
||||
per node. Additionally, arbitrary resource usage can be added per job.
|
||||
|
||||
The API can be found in trq_plugin_api.h. To implement a plugin, all of these functions must
|
||||
be implemented, even if the function does nothing. An implementation that does nothing may be
|
||||
found in contrib/resource_plugin.cpp. If you wish, you may simply add the desired functionality
|
||||
to this file, build the library, and link it to the mom at build time.
|
||||
|
||||
|
||||
II. Implementing and Building the Plugin:
|
||||
|
||||
i. Implementation Recommendations:
|
||||
Your plugin must execute very quickly in order to avoid causing problems for the pbs_mom daemon.
|
||||
The node resource portion of the plugin has a 5 second time limit, and the job resource usage
|
||||
portion has a 3 second time limit. The node resource portion will execute once each time that the
|
||||
mom sends a status to pbs_server, and the job resource usage portion will once per job at the
|
||||
same interval of time. These block pbs_mom while they are executing, so you want them to execute
|
||||
in a short, deterministic amount of time.
|
||||
|
||||
Remember, you are responsible for plugins, so please design well and test thoroughly.
|
||||
|
||||
ii. Building:
|
||||
If you do not change the name of the .cpp file and you wish to build it, you may execute:
|
||||
|
||||
export TRQ_HEADER_LOCATION=/usr/local/include/
|
||||
g++ -fPIC -I $TRQ_HEADER_LOCATION resource_plugin.cpp -shared -o libresource_plugin.so
|
||||
|
||||
NOTE: Change TRQ_HEADER_LOCATION if you configured torque with --prefix.
|
||||
|
||||
III. Testing your plugin:
|
||||
|
||||
NOTE: You assume all responsibility for any plugins. We desire to assist you in testing the plugins,
|
||||
but this list may not be comprehensive. We do not assume responsibility if these suggested tests
|
||||
do not cover everything.
|
||||
|
||||
i. Basic functionality:
|
||||
Once you've implemented and built your library, you can begin testing. For your convenience, a
|
||||
simple test driver can be found in plugin_driver.cpp. You can build this executable and link it
|
||||
against your library in order to manually verify the output:
|
||||
|
||||
export PLUGIN_LIB_PATH=/usr/local/lib
|
||||
g++ plugin_driver.cpp -I $TRQ_HEADER_LOCATION -L $PLUGIN_LIB_PATH -lresource_plugin -o driver
|
||||
|
||||
Then, execute it and manually inspect the output:
|
||||
./driver
|
||||
|
||||
NOTE: Change PLUGIN_LIB_PATH if you have installed it elsewhere.
|
||||
|
||||
ii. Testing for Memory Leaks:
|
||||
In order to prevent your compute nodes from being compromised for speed or even going down due
|
||||
to out-of-memory conditions, you should run your plugin under valigrind to test that it is
|
||||
correctly managing memory.
|
||||
|
||||
Assuming you've being the driver from the 'Basic functionality' section, you can run:
|
||||
|
||||
valgrind --tool=memcheck --leak-check=full --log-file=plugin_valgrind_output.txt ./driver
|
||||
|
||||
If you aren't familiar with valgrind, a good primer can be found here:
|
||||
|
||||
http://valgrind.org/docs/manual/QuickStart.html
|
||||
|
||||
We recommend that you fix any and all errors reported by valgrind.
|
||||
|
||||
|
||||
IV. Enabling the plugin:
|
||||
Once you've implemented, built, and thoroughly tested (remember that our suggestions may not
|
||||
address everything) your plugin, you'll want to enable it in Torque. It can be linked in at
|
||||
build time:
|
||||
|
||||
./configure <your other options> --with-resource-plugin=<path to your resource plugin>
|
||||
|
||||
NOTE: You'll want to make sure that the path you specify is in $LD_LIBRARY_PATH or can otherwise
|
||||
be found by pbs_mom when you start the daemon.
|
||||
|
||||
Once you build, you can now start the new mom and you should be able to observe the plugin's output
|
||||
in pbsnodes, qstat -f, and in the accounting file.
|
||||
|
||||
Sample pbsnodes Output:
|
||||
|
||||
<normal output>
|
||||
gres:hbmem = 20
|
||||
gmetric:temperature = 76.20
|
||||
varattr:octave = 3.2.4
|
||||
features = haswell
|
||||
|
||||
The keywords at the front let Moab know what each one means so it can use them accordingly.
|
||||
|
||||
Sample Accounting File Entry:
|
||||
<normal entry until resources used> resources_used.cput=0 resources_used.energy_used=0 resources_used.mem=452kb resources_used.vmem=22372kb resources_used.walltime=00:05:00 resources_used.stormlight=2broams
|
||||
|
||||
Your plugin resources reported will appear in the form:
|
||||
resources_used.<name you supplied>=<value you supplied>
|
||||
|
||||
For the above example I added the arbitrary resource stormlight and the value of 2broams.
|
||||
|
||||
Sample qstat -f Output:
|
||||
<normal qstat -f output>
|
||||
resources_used.stormlight = 2broams
|
||||
|
||||
The resources used reported from the plugin will appear at the end of the qstat -f output in the
|
||||
same format it does in the accounting file.
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
|
||||
#include <iostream>
|
||||
#include <fstream>
|
||||
#include <map>
|
||||
#include <set>
|
||||
#include <string>
|
||||
#include <sys/time.h>
|
||||
#include <sys/types.h>
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include <signal.h>
|
||||
|
||||
#include "trq_plugin_api.h"
|
||||
|
||||
// plugin_driver.cpp
|
||||
//
|
||||
// Can be built against the plugin library for testing purposes. It will echo
|
||||
// the output of the plugin library so that it may be manually checked for
|
||||
// correctness.
|
||||
//
|
||||
|
||||
|
||||
void test_usage_information()
|
||||
|
||||
{
|
||||
std::map<std::string, std::string> usage_information;
|
||||
std::string jid("1.roshar");
|
||||
std::set<pid_t> job_pids;
|
||||
pid_t max_pid_id = 0;
|
||||
pid_t random_pid = 0;
|
||||
std::ifstream file;
|
||||
|
||||
file.open("/proc/sys/kernel/pid_max");
|
||||
if (file.good())
|
||||
{
|
||||
file >> max_pid_id;
|
||||
file.close();
|
||||
}
|
||||
|
||||
if (max_pid_id == 0)
|
||||
max_pid_id = 32768;
|
||||
|
||||
// Decrement because we'll add one later to make sure we avoid process 0
|
||||
max_pid_id--;
|
||||
|
||||
struct timeval now;
|
||||
gettimeofday(&now, 0);
|
||||
srand(now.tv_usec);
|
||||
|
||||
// Get one random pid to test. Make changes here if you'd like to test with more than 1
|
||||
// pid, or if you'd like specific pids.
|
||||
do
|
||||
{
|
||||
random_pid = rand() % max_pid_id + 1; // add 1 to avoid process 0
|
||||
} while (kill(random_pid, 0) != 0);
|
||||
|
||||
job_pids.insert(random_pid);
|
||||
report_job_resources(jid, job_pids, usage_information);
|
||||
|
||||
printf("Your plugin reported the following for the random pid %d:\n", random_pid);
|
||||
for (std::map<std::string, std::string>::iterator it = usage_information.begin();
|
||||
it != usage_information.end();
|
||||
it++)
|
||||
printf("%s = %s\n", it->first.c_str(), it->second.c_str());
|
||||
} // END test_usage_information()
|
||||
|
||||
|
||||
|
||||
void test_node_info_plugin()
|
||||
|
||||
{
|
||||
std::map<std::string, unsigned int> gres_map;
|
||||
std::map<std::string, double> gmetric_map;
|
||||
std::map<std::string, std::string> varattr_map;
|
||||
std::set<std::string> features;
|
||||
|
||||
report_node_generic_resources(gres_map);
|
||||
report_node_generic_metrics(gmetric_map);
|
||||
report_node_varattrs(varattr_map);
|
||||
report_node_features(features);
|
||||
|
||||
printf("Your plugin reports the following for this host:\n");
|
||||
|
||||
printf("\tGRES:\n");
|
||||
|
||||
for (std::map<std::string, unsigned int>::iterator it = gres_map.begin();
|
||||
it != gres_map.end();
|
||||
it++)
|
||||
printf("\t\t%s = %u\n", it->first.c_str(), it->second);
|
||||
|
||||
printf("\n\tGMETRICS:\n");
|
||||
|
||||
for (std::map<std::string, double>::iterator it = gmetric_map.begin();
|
||||
it != gmetric_map.end();
|
||||
it++)
|
||||
printf("\t\t%s = %.2f\n", it->first.c_str(), it->second);
|
||||
|
||||
printf("\n\tVARATTRS:\n");
|
||||
|
||||
for (std::map<std::string, std::string>::iterator it = varattr_map.begin();
|
||||
it != varattr_map.end();
|
||||
it++)
|
||||
printf("\t\t%s = %s\n", it->first.c_str(), it->second.c_str());
|
||||
|
||||
printf("\n\tFEATURES:");
|
||||
|
||||
for (std::set<std::string>::iterator it = features.begin(); it != features.end(); it++)
|
||||
printf(" %s", it->c_str());
|
||||
|
||||
printf("\n");
|
||||
} // END test_node_info_plugin()
|
||||
|
||||
|
||||
|
||||
int main(
|
||||
|
||||
int argc,
|
||||
char *argv[])
|
||||
|
||||
{
|
||||
test_usage_information();
|
||||
|
||||
test_node_info_plugin();
|
||||
} // END main()
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
|
||||
#include <string>
|
||||
#include <map>
|
||||
#include <set>
|
||||
|
||||
#include <sys/types.h>
|
||||
|
||||
// Node reporting plug-in for Torque
|
||||
//
|
||||
// Please note that if you use the plugin, you assume all responsibilities for testing
|
||||
// and debugging your plugin. As the developer of the plugin, you assume all liability
|
||||
// for maintaining your plugin.
|
||||
//
|
||||
|
||||
|
||||
// To add a generic resource to a node, add the name as the key to the map, and the desired
|
||||
// integer value as the value.
|
||||
//
|
||||
// To clear a generic resource, report it as 0. If you stop reporting the generic resource
|
||||
// and you are having a scheduler use that generic resource as a factor, then it will result
|
||||
// in undefined behavior.
|
||||
void report_node_generic_resources(
|
||||
|
||||
std::map<std::string, unsigned int> &gres_map)
|
||||
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
// To add a generic metric to a node, add the name of the metric as the key, and the desired
|
||||
// numeric value (stored as a double) as the value.
|
||||
//
|
||||
// To clear a generic metric, report it as 0 (or some otherwise neutral value). If you stop
|
||||
// reporting the generic metric and you are having a schedule use that generic metric as a
|
||||
// factor, then it will result in undefined behavior.
|
||||
void report_node_generic_metrics(
|
||||
|
||||
std::map<std::string, double> &gmetric_map)
|
||||
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
// To add a varattr to a node, add the name of the varattr as the key, and the value of the
|
||||
// metric set as the value.
|
||||
//
|
||||
// To clear a varattr, you may simply stop reporting it.
|
||||
void report_node_varattrs(
|
||||
|
||||
std::map<std::string, std::string> &varattr_map)
|
||||
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
// To add a feature to a node, simply add the feature name to this set
|
||||
//
|
||||
// To clear a feature, simply stop reporting it
|
||||
void report_node_features(
|
||||
|
||||
std::set<std::string> &node_features)
|
||||
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
// Job resource reporting plug-in for Torque
|
||||
// Each string in usage_info should be reported in a name=value format
|
||||
// Only one name and value should be placed in each string
|
||||
//
|
||||
// jobid: the id of the job, provided by pbs_mom. Marked const to show that the plugin cannot
|
||||
// communicate anything by altering this parameter.
|
||||
//
|
||||
// job_pids: a set of the pids in the job. Also marked const to show that the plugin cannot
|
||||
// communicate anything by altering this parameter.
|
||||
//
|
||||
// usage_info: The plugin is responsible for populating usage_info and managing it. The mom
|
||||
// will not clear it between runs or otherwise alter the data - it is completely
|
||||
// up to the plugin how this usage information should be altered each interval.
|
||||
// NOTE: The intervals are right around the setting for $status_update_time in the mom's
|
||||
// config file. This value defaults to 45 seconds if it isn't set manually.
|
||||
void report_job_resources(
|
||||
|
||||
const std::string &jobid,
|
||||
const std::set<pid_t> &job_pids,
|
||||
std::map<std::string, std::string> &usage_info)
|
||||
|
||||
{
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user