Relation to the Job Service

A functioning GRIA Job Service installation is composed of three main parts:

1. The service software running under Tomcat/AXIS.
2. Scripts for running or interacting with application codes on an execution platform, which may be the service host itself, a separate execution server, or a cluster.
3. Some installed applications capable of running on the execution platform, each corresponding to a different Job Service end-point.

The Job Service software (1) supports various bookkeeping operations for assigning job ids, workspace, etc. The core operations are those for actually starting and managing the execution of an application: starting jobs, checking their status, and killing jobs. Each of these has to access the execution platform by running the appropriate script (2), which can start or otherwise interact with the application running on the execution platform (3). For security reasons, we do not usually allow users to upload their own applications, so the service operator must install all the applications.

The Job Service and platform scripts are designed to support a uniform model of application execution, shown in Figure 1:

Figure 1 - Application Model

The workspace for each job is set up by the Job Service when the job is initialised (this is one of the bookkeeping service operations that must precede the call to start the job). The workspace has a standard directory structure so the Job Service and platform scripts can create and find information stored in it, including a working sub-directory where the job will actually run.

When the user starts the job, the Job Service transfers input data files from Data Service URI's into the job’s workspace. It then runs a platform script locally, which in turn submits the application to the execution platform (cluster, etc) where it will run (using the specified command line), possibly after some queuing delay. The platform script saves the job handle to the workspace. The application has to read the input data left in its workspace by the Job Service, and write any outputs to the workspace so that the Job Service can find them and transfer them to output Data Service URI's when the job has finished.

When the user asks for the status of the job, the service runs a second platform script that reads status information (e.g. when the job started or finished, etc), which the application must store in the workspace. This platform script may also run an associated monitoring application (using the specified command line) to gather application-specific status information (e.g. number of iterations completed, convergence plots, etc) from the workspace.

If the user asks for the job to be killed, the service uses a third platform script that reads the job handle from the workspace, and issues a command to kill this job on the execution platform. The Job Service will detect that the job has finished, and will transfer any output produced. Note that the user (or their client-side application) should always check the status of a job to find out if it crashed or was killed, as some incomplete output may appear in this case.

Why are Application Wrapper Scripts Required?

In practice, few legacy applications behave exactly according to the model shown in Figure 1. It is rarely possible to change the application itself to fix this, so instead GRIA uses so-called wrapper scripts that do conform to the application model for starting and managing the application.

In practice, the wrapper scripts can do more than just make the underlying application work as indicated in Figure 1. They can also be used to handle and implement application specific features of the service.

One can also use (optional) wrapper scripts to look for application-specific status information in the working directory of the job. Without such scripts, the platform scripts can only provide basic job status information from the job submission system.

Finally, wrapper scripts also provide a configurable mechanism for dealing with any application-specific security risks, e.g. checking for malicious input that may exploit a feature of the application. Few legacy applications were designed as network-accessible services, and since we can’t change them to remove security loopholes, the use of wrapper scripts is essential to check for any exploits of application vulnerabilities. In the limit, one can configure the wrapper (and platform) scripts to run the application in a sandbox (e.g. chroot), with access only to a working sub-directory of the job workspace, as shown in Figure 2.

Figure 2 - Wrapper Scripts and Security

Application Wrapper Functionality

The startJob.pl application wrapper script is a mandatory script that deals with any application specificity, allowing the Job Service to treat all applications in the same way, and so decoupling the Job Service from the details of the application.

The main functions of the application wrapper script are:

• handling input and output data files;
• creating a consistent environment that is referred to correctly in inputs;
• enforcing any security precautions to protect against loop-holes in the application.

The application wrapper is designed to run on the execution platform, having been submitted by the platform script for starting a job. Prior to submitting the wrapper script on the execution platform, the Job Service will have set up a workspace (directory) for the job, copied input data into it e.g. work/inputs, and created a working sub-directory for the job to run in, e.g. work. The following listing shows a workspace directory structure with two input files and an empty outputs directory.

ff808081-1017450e-0110-174532dd-0001-1
`--work
   |-- inputs
   |   |-- input-0
   |   `-- input-1
   `-- outputs

After changing to the workspace directory (not the working sub-directory), the wrapper will be submitted using the following command line:

app-wrapper <application arguments>

The functionality of the wrapper script should include the following:

1. Parse wrapper arguments, including security checks for illegal input designed to inject malicious commands into the command-line used to launch the application.
2. Move input data files into the working sub-directory, including unpacking any that are compressed archives containing multiple inputs.
3. Create a consistent environment in the working directory, by setting up environment variables and rewriting input data to match the local environment where necessary.
4. Touch the file .app_started in the job workspace (this should be outside the working subdirectory), so recording the time when the application started. This step is optional.
5. Build the command line and run the underlying application, making sure that the standard output channels are directed to .stdout and .stderr in the working directory. This step is optional.
6. When the application has finished, touch the file .app_ended in the job workspace, so recording the time when the application finished. This step is optional.
7. Copy output files from the working directory into the specified positions in the workspace, including packing multiple outputs into compressed archive files where necessary.
8. Exit by returning the exit code of the application.

For simple applications, security can be maintained by checking input parameters during step 1. and if necessary data files during step 3. If the application is too complicated for this to be reliable, it may also be necessary to set up a sandboxed working environment and run the code inside it during step 5.

Some of these steps are considered in more detail below.

Input and Output Data Handling

When unpacking input data, the application wrapper should attend to the following:

1. Create any substructure needed in the job's working sub-directory of the workspace.
2. Copy or unzip input files from the inputs sub-directory into the job's working space.
3. Check that all input needed to run the job is present.

The Job Service knows in advance which output files have to be sent back to outputs directory. The application wrapper has to create these files by:

1. Copying or zipping data to create the required output files in the outputs sub-directory
2. Renaming these files to output-x naming scheme.

When the script finishes, the Job Service will detect this and handle the transfer of output files accordingly.

The number of output files is always fixed, so the wrapper can know what outputs are needed, and how to assemble them. However, the user is allowed to distribute multiple inputs across an arbitrary number of input zip archives, provided they specify all the Data Service URIs where these are stored when they start the job. The application wrapper must be capable of handling this.

Consistent Context Reconstruction

Why do we need context reconstruction? The input data for our application has been created on another system with a different directory structure, environment and possibly even operating system. We have to set up an equivalent (not necessarily identical) environment on our execution platform, and make sure any input data references to the remote user's environment are mapped onto the one we have created, or they will be invalid when the application is started.

When and where should context reconstruction be performed? One should handle it as close as possible to the running application—certainly on the execution platform where the job will actually be run—as this is where the environment is needed. This is why the Job Service doesn't attempt to create the context itself - there is no point doing it at the service host if the job will be executed on a compute node in a Condor cluster. Instead, we leave it to the application wrapper to handle everything in an application specific way on the execution platform itself.

A typical approach to context reconstruction might involve passing an array of named parameters to the Job Service, including environment settings as well as application flags. These will be passed to the wrapper through its argument list. In addition, one can provide settings in an extra input file, intended for the wrapper rather than the job itself, and used to set up the environment prior to running the application code.

The hardest job for the wrapper is to parse and rewrite application input data where necessary to ensure it is consistent with the environment established on the execution platform. If this is not needed, it is usually quite easy to 'wrap' an application to run inside the Job Service. Where it is necessary, the wrapper may become a significant body of code in its own right.

For example, consider the following line of input intended for the rendering application AIR, used with the Job Service to provide a grid-enabled video rendering service:

  Option 'searchpath' 'shader' ['&:e:\AnimalLogic\MaxMan\shaders:C:\Sample\shaders']

The problem here is that the application uses plug-ins to perform part of the rendering calculation, and the search path for these can be specified in the user input. This particular input file has been generated automatically using a graphical environment for video post-production, which has filled in the relevant path based on where the shader libraries were installed on the user's local machine.

The wrapper has to identify which shaders are needed, and substitute the path to them on the local system:

   Option 'searchpath' 'shader' ['&:/export/apps/AnimalLogic/MaxMan/shaders:/export/apps/air/Sample/shaders']

In some cases, it may be possible to infer the meaning of client-side environment references by pattern matching against a list of meaningful terms used by the application. In others (probably in this case), it is necessary for the user to send the install path quoted for specific groups of plug-ins as service arguments or environment settings, so the wrapper can find them and map them onto the equivalent installed groups of components on the execution platform.

In extreme cases, it may be necessary to establish multiple services to run the same application in different ways, allowing a different, specific environment to be set up for each. For example, it probably wouldn't make sense to have a single service to run a computational fluid dynamics (CFD) code capable of simulating coolant flows through automotive engines AND the propagation of drugs in aerosol suspension in human lungs. It would be asking too much of a wrapper developer to differentiate and correctly handle such extreme cases, and instead one should set up two services each with its own wrapper specialised to one of these scenarios.

Security Containment

Why Wrappers have to Bother with Security

The Job Service regards application wrapper scripts as trustworthy, because the service operator can inspect them and make sure they don't do anything strange or foolhardy. However, the applications may be third party, closed source executables that cannot be inspected, and were not designed as network-accessible services in the first place.

Wrapper scripts can protect the service from malicious users in three ways:

1. checking any user input used to create the command line for running the application, to exclude command injection attacks using parameters like 'method=gauss; cd /; rm */*';
2. checking input data known to be used in an unsafe way by the application, e.g. to construct system calls for executing plug-ins or moving files around;
3. confining the application to a sandbox, by first preparing the sandbox and then launching the application in it.

If the application is very simple, or designed to withstand malicious users, or if you have only a small number of users you know well (and trust not to mislay their credentials) then it may be OK to include only the first of these measures.

Legacy applications are quite likely to do things in unsafe ways. Renaming files or testing if they exist are often done via system calls if the application developer wasn't filenames to be sent by a remote user who may have malicious intent. If the application isn't too complex, or if you can check with the developer on what might happen, then it should be OK if you also check the user-supplied input and check filenames and other data that may be sent to unsafe system calls.

In the worst case, one has to assume the application will be unsafe, and attempt to contain any damage caused by malicious (or possibly careless) input by restricting what the application can do and where it can do it. There are several possible ways to achieve such restrictions.

Chroot

On Linux systems, chroot can be used to restrict a sub-process to an arbitrary sub-directory, e.g. a job's working directory. The chroot mechanism was designed for use by operating system developers to allow them to create a pseudo-root within which to test their code. The chroot container doesn't prevent access to low-level devices, it will prevent most legacy applications accessing files outside the specified sub-directory. Chroot is widely used to contain web servers and other network applications to minimise the scope for damage if they are compromised.

To use chroot, it is necessary to create a complete operating system environment inside the job's working directory (which it will see as '/'). One has to copy application binaries, resolve any references to system/application libraries, create devices such as /dev/null, etc. To create a self-sufficient chroot 'jail' environment sufficient to run the application may not be easy, and of course, it would need to be repeated for each individual job. However, it can provide a good safety level as its 'jail' environment is enforced by the operating system itself.

Restricted Shells

Many shells, including bash provide a restriction mechanism, usually invoked by running the shell with the -r switch. Some common features of restricted shells are the ability to prevent a program from changing directories, to only allow the execution of commands using absolute pathnames, and to prohibit executing commands in other subdirectories, using command-line redirection operations, or changing the search path.

Minimal privilege accounts

Another approach is to create a low-privilege account for each jobs. The wrapper script would then have to assign such an account, change the working directory so it is owned by this account, and run the application in that working directory under the same account. Provided the same account is not used for anything else (including running other jobs), the application can be prevented from accessing anything outside the working directory, even if it can be induced to run some unforeseen system call by sending some malicious input.

The two drawbacks with this approach are:

1. ideally one should create a pool of accounts and provide a way for the wrapper to assign them to jobs rather than creating new accounts, but this isn't supported at present;
2. the wrapper would need sufficient privilege to set the account under which a sub-process is run, which may make the wrapper more dangerous if it can be compromised.

The second drawback may not be too bad, given that the wrapper at least can be designed to check all inputs and avoid doing anything unpleasant. At present, the Job Service runs with a normal unprivileged user identity, so it may be better to use other methods to contain individual jobs.

Other Methods

The above list is by no means exhaustive. For example, if the chroot 'jail' is not sufficient, one can create an entire virtual machine on which to run a potentially unsafe application. Software such as VMWare can be used to implement this approach, but users who want to go to these lengths are on their own, at least in this version of the software.

Error Handling

If an error is encountered in the application, the wrapper must report the fact. If this is not done, the Job Service will assume everything is OK, and the users client application will probably attempt to continue, which may not be appropriate if some output from the job is missing, etc.

Return values are passed back to the Job Service middleware via three files in the job workspace:

• .app_wrapper_exit_code
• .app_exit_code (optional)
• .app_exit_status (optional)

The application startJob wrapper should exit with an exit status of zero if the job has completed successfully, or with a non-zero status if the job has failed. This value will be stored in .app_wrapper_exit_code by the platform scripts. Generic clients may stop executing a workflow, for example, if this result is not zero.

The two optional files may be used to provide extra application-specific information to specialised counter-part client applications. The Job service is not expected to understand these codes, but will just pass the values, if set, to the client. The application wrapper must write these files, if they are needed.

The .app_exit_code file should contain the application's exit code.

The .app_exit_status file can contain further information.

An Example Wrapper Script

This example is based on the ImageMagick application which was installed as part of the GRIA installation. To get started, we will create a simple wrapper that runs this application.

1. Create a startJob wrapper script:

   #!/bin/sh
   exec > log 2>&1
   echo Swirl wrapper started
   echo Arguments are: $*
   
   INPUT="$2"
   OUTPUT="$4"
   
   echo Copying inputs to work directory...
   cp "$INPUT" image.jpg || exit 1
   
   echo Run the mogrify command...
   mogrify -swirl 60 image.jpg || exit 1
   
   echo Copying result to output stager...
   cp image.jpg "$OUTPUT" || exit 1
   
   echo Swirl job completed successfully
   

   This performs the following steps:

   1. Redirect all output to the log file.
   2. Print the arguments to the log.
   3. Copy the input image into the work directory.
   4. Run the mogrify command to transform the image.
   5. Copy the result to the output stager.
2. Edit the startJob script to run your command instead of swirl (the command you tested above).
3. Make the script executable:

   $ chmod a+x startJob
   

4. Test the wrapper with this command:

   $ ./startJob -i image.jpg -o output.jpg
   $ cat log
   

Check that the log shows that the command ran correctly. It should only take a few seconds to process the image. It it takes longer, press Ctrl-C and examine the log.

Unlike the wrapper for starting an application, the application wrapper for reading status from the working directory is optional. If no such wrapper is provided, the platform script will create a simple status report by checking for files like .app-start and .app-end that indicate if and when the job started or finished, consulting the execution platform manager (e.g. batch queue system), and appending .stderr (if any) to the result.

If you want the job status report to include application-specific information such as convergence plots, iteration counters, etc., you should create a wrapper script that will be invoked by the client calling the checkJob method.

An application status wrapper is usually a lot simpler to create than the main wrapper because it does not take any user-supplied (and hence potentially malicious) arguments, and does not set up (or run) potentially untrustworthy code. All the status wrapper has to do is to examine the job's working directory, read any files it needs in order to extract the desired status information (in the limit, one could simply copy an application-level log file), and write it to the standard output.

Note: the format of the status information is open and application dependent, however, status information should not include binary data.

An Example Status Wrapper Script

This example is based on the ImageMagick application, which was installed as part of the GRIA installation and follows on from the start job example. In the same directory as startJob, create a script called checkJob:

#!/bin/sh
tail log

This is run each time the client checks the status of the job. This example simply returns the last few lines of the log file, and is executed as follows:

1. Make the script executable:

   $ chmod a+x checkJob

2. Test it with these commands:

   $ ./checkJob > statusfile
   $ cat statusfile
   

   You should find that the contents of log are now in statusfile.

The application specific kill script is a script that will allow an application to be killed in a certain way, e.g. using an application specific mechanism. This script is optional, if not present in the argument list, the platform script will try to kill a job at the resource manager level. When an application specific kill script is provided the platform script will invoke it instead.

The return code of this script should be 0 upon success or any other value on error. The platform script accordingly will decide whether the killing operation was successful or failed. The functionality of this script is application dependent and difficult to be described in a generic way.

For some applications terminating a job, might be as simple as creating a single file in the job workspace. Some other applications are aware of signals that can be passed to them, etc.

Appendix II describes in detail an alternative way to kill jobs gracefully which uses a signalling mechanism for the wrapper to kill the job. The application wrapper can respond to that signal by terminating the application and preparing any useful output data.

An Example Kill Wrapper Script

We are not aware of any particular way that ImageMagick can be killed, therefore we cannot provide a complete example of a kill-wrapper script. In practice, such an example will be very similar to the default job termination mechanism used in the platform kill script, e.g. kill -9 $jobID. In the following paragraphs we provide some hints about possible ways terminating jobs.

If a particular application is aware of a termination file in the job workspace, then the kill-wrapper can be as simple as:

touch $WORK_DIR/.terminate

Many applications are aware of various signals e.g. SIGTERM, SIGALRM, SIGSTOP, etc. The kill-wrapper then can pass the appropiate signal to the application and the application can respond by terminating the job gracefully, for example:

# find the process ID $pid and send a termination signal
kill -SIGTERM $pid

Application description files are XML files containing metadata about applications deployed on GRIA. These files are essential for GRIA users to discover available applications and use them. To create an XML description file of an application you need to use the following schema to identify the application's main features including name, version number, description, and inputs/outputs (if any). For example, The following code describes the Swirl application:

<application>

    <name>http://it-innovation.soton.ac.uk/2005/gria/tutorial/swirl</name>
    <version>1.0.0</version>
    <description>Application to swirl an image</description>

    <application-inputType>
        <name>image.jpg</name>
        <type>jpg</type>
        <description>image file of any type</description>
    </application-inputType>

    <application-outputType>
        <name>image.jpg</name>
        <type>jpg</type>
        <description>swirled image of the same type as input type</description>
    </application-outputType>

</application>

The following code describes the Paint application:

<application>

    <name>http://it-innovation.soton.ac.uk/2005/gria/tutorial/paint</name>
    <version>1.0.0</version>
    <description>Application to render an image into painting</description>

    <application-inputType>
        <name>image.jpg</name>
        <type>jpg</type>
        <description>image file of any type</description>
    </application-inputType>

    <application-outputType>
        <name>image.jpg</name>
        <type>jpg</type>
        <description>painted image of the same type as input type</description>
    </application-outputType>

</application>

Note that you can add as many inputs/outputs as necessary, according to your application.

Every type of application provided by the Job service must be given a unique name. The ensure they are unique, a URI is used. Note that although these names look like web page addresses, they may not necessarily point to web pages if treated as URL. They are simply unique strings.

Job Constraints

The job constraints feature is a new experimental feature. It does not currently integrate with the SLA Management Service.

Job constraints are passed to platform scripts either from the Job Service using the -r command line argument (defined statically) or directly by the client user (using the client API) via a constraints XML file which the Job Service will store in the job session directory as resources.xml. The startJob platform script should parse these constraints and translate them to resource manager directives. Typical job service constraints are expected to describe resource constraints such as WallClockTime, CPUSpeed, PhysicalMemory, DiskSpace, etc.

Constraints passed from the Job Service as command line arguments should follow the form of name=value pairs, for example -r CPUSpeed=1800 (in MHz for a CPU speed constraint), or -r WallClockTime=3600 for a runtime constraint of an hour.

Job Service providers have to specify and advertise which job constraints a user is allowed to apply for a job run. This can be done easily with an apropriate XML schema. Client users can then submit jobs along with their constraints file. An example of a simple user supplied constraints file could be:

<?xml version="1.0"?>
<Resources>
        <CPUArchitecture>amd64</CPUArchitecture>
        <CPUSpeed>180</CPUSpeed>
        <PhysicalMemory>1024</PhysicalMemory>
        <!--DiskSpace>40</DiskSpace-->
        <WallClockTime>210</WallClockTime>
        <IndividualCPUCount>1</IndividualCPUCount>
        <TotalCPUCount>1</TotalCPUCount>
        <FileSizeLimit>200</FileSizeLimit>
</Resources>

A suitable XML schema for these constraints could be:

<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

  <xsd:annotation>
    <xsd:documentation xml:lang="en">
     GRIA resource schema
    </xsd:documentation>
  </xsd:annotation>

  <xsd:element name="Resources" type="ResourcesType"/>

  <xsd:complexType name="ResourcesType">
    <xsd:sequence>
      <xsd:element name="Comment" type="xsd:string" minOccurs="0"/>
      <xsd:element name="CPUArchitecture" type="cpuarchitecture" minOccurs="0" default="x86"/>
      <xsd:element name="CPUSpeed" type="xsd:int" minOccurs="0"/>
      <xsd:element name="PhysicalMemory" type="xsd:int" minOccurs="0"/>
      <xsd:element name="DiskSpace" type="xsd:int" minOccurs="0"/>
      <xsd:element name="WallClockTime" type="xsd:long"/>
      <xsd:element name="IndividualCPUCount" type="xsd:int" minOccurs="0" default="1"/>
      <xsd:element name="TotalCPUCount" type="xsd:int" minOccurs="0" default="1"/>
      <xsd:element name="FileSizeLimit" type="xsd:long" minOccurs="0"/>
    </xsd:sequence>
  </xsd:complexType>

  <xsd:simpleType name="cpuarchitecture">
    <xsd:restriction base="xsd:string">
      <xsd:enumeration value="x86"/>
      <xsd:enumeration value="ia64"/>
      <xsd:enumeration value="amd64"/>
      <xsd:enumeration value="sparc"/>
      <xsd:enumeration value="other"/>
    </xsd:restriction>
  </xsd:simpleType>

</xsd:schema>

Supported Constraints in the Supplied Platform Scripts

The platform (startJob) scripts supplied with GRIA implement the following job constraints:

WallClockTime

Maximum amount of time a job can run in seconds. If the job service and the user both specify this constraint, the minimum of the two is taken.

PhysicalMemory

The minimum amount of required physical memory in MB. If the job service and the user both specify this constraint, the maximum of the two is taken.

CPUSpeed

The minimum CPU speed required in MHz. If the job service and the user both specify this constraint, the maximum of the two is taken.

DiskSpace

The minimum amount of available disk space required in MB. If the job service and the user both specify this constraint, the maximum of the two is taken.

OSName

The Job Service overwrites the user supplied constraint.

Note: GRIA pre-supplied scripts are using XML::Simple perl module to handle the user constraints XML file, which is only capable of handling simple XML documents without attributes.

The following table shows which constraints are implemented with the GRIA pre-supplied platform scripts.

Depending on the platform capabilities Job Service providers should customise section E of the startJob platform script accordingly.