Workflow Tutorial 2 - Job Execution

Introduction

The learning objectives for this exercise are:

1. Understand how to create and manage jobs at the Job service; and
2. Learn how to correctly monitor job execution.

Step 1. Create a client proxy to the remote job service as seen in Exercise 1.

StateRepository repository = new MemoryStateRepository();
RemoteJobService jobService = (RemoteJobService)  
repository.getOrCreateObject(RemoteJobService.class,ConversationID.getEPR(JOB_SERVICE_ENDPOINT));

Step 2. Create a new Job

JobConversation jobConv = jobService.createJob(APPLICATION, null, "");

The APPLICATION parameter is the URI for the application for which we wish to create a new job. In this example we will use the swirl application for simple image processing. The URI for the swirl application is:

http://it-innovation.soton.ac.uk/grid/imagemagick/swirl

Note that we are assuming that the Job Service is free. Therefore a null endpoint reference for the billing argument is used. If instead the service were not free, we would pass an endpoint reference for an SLA or account instead (see Exercise 4).

Step 3. Get the inputs and outputs that for new job.

These are used to upload input files and download the results, respectively.

DataConversation[] inputs = jobConv.getInputs();
DataConversation[] outputs = jobConv.getOutputs();

Step 4. Upload the input image

inputs[0].save(new DataHandler(new FileDataSource(INPUT_FILE)));

Step 5. Submit the job.

Command line arguments can be passed to the remote application in place of the empty string array.

jobConv.submitJob(null, new String[]{});

Step 6. Monitor execution

The checkJob() method makes a remote method invocation on the Job service, so the thread is made to sleep between rounds of polling job status, to reduce resource use.

while(jobConv.checkJob().getInProgress()) {	
	Thread.sleep(2000);
}

Step 7. Check the applications exit status and deal with errors appropriately.

JobStatus jobStatus = jobConv.checkJob();
if(jobStatus.getExitStatus() != 0) {
  // handle the failure
  throw new RuntimeException("Swirl job failed. Here is the log:\n" +   
                             jobStatus.getLogText());
}

Step 8. Download the output.

OUTPUT_FILE should be a String path to the file where you wish the output to be saved.

outputs[0].read(new File(OUTPUT_FILE));

Step 9. Clean up server resources by finishing the job.

jobConv.destroy();

Completed Java

If you need some help here is a completed version of the java file

import java.io.File;
import javax.activation.DataHandler;
import javax.activation.FileDataSource;

import uk.ac.soton.ecs.iam.grid.client.staterepos.MemoryStateRepository;
import uk.ac.soton.ecs.iam.grid.client.staterepos.RemoteJobServiceImpl;
import uk.ac.soton.ecs.iam.grid.comms.client.DataConversation;
import uk.ac.soton.ecs.iam.grid.comms.client.JobConversation;
import uk.ac.soton.ecs.iam.grid.comms.client.RemoteJobService;
import uk.ac.soton.itinnovation.grid.service.types.JobStatus;
import uk.ac.soton.itinnovation.grid.types.ConversationID;
/**
* Simple data and job orchestration example. Edit JOB_SERVICE_ENDPOINT to be the soap endpoint of your job service. */

public class JobExecutionCompleted {
	
	/** Change this Variable to point to your job service!!! */
	private static final String JOB_SERVICE_ENDPOINT = "https://....:8443/gria-basic-app-services/services/JobService";
	
	/** The application to execute, identified by its URI */
	private static final String APPLICATION = "http://it-innovation.soton.ac.uk/grid/imagemagick/swirl";
	
	/** The input file to upload to the job */
	private static final String INPUT_FILE = "data/input-image.png";
	
	/** The ouptut file to download the results to */
	private static final String OUTPUT_FILE = "data/output-image.png";
	
	public static void main(String[] args) throws Exception {

		StateRepository repository = new MemoryStateRepository();
		RemoteJobService jobService = (RemoteJobService)  
			repository.getOrCreateObject(RemoteJobService.class,ConversationID.getEPR(JOB_SERVICE_ENDPOINT));
		JobConversation jobConv = jobService.createJob(APPLICATION, null, "");

		DataConversation[] inputs = jobConv.getInputs();
		DataConversation[] outputs = jobConv.getOutputs();

		inputs[0].save(new DataHandler(new FileDataSource(INPUT_FILE)));

		jobConv.submitJob(null, new String[] {});
		// poll job status
		while(jobConv.checkJob().getInProgress()) {
			System.out.println(".");
			Thread.sleep(2000);
		}

		JobStatus jobStatus = jobConv.checkJob();

		if(jobStatus.getExitStatus() != 0) {
			// handle the failure
			throw new RuntimeException("Swirl job failed. Here is the log:\n" + jobStatus.getLogText());
		}
		outputs[0].read(new File(OUTPUT_FILE));

		System.out.println("Swirl job completed. Output downloaded to: " + OUTPUT_FILE);

		// clean up resources at the server
		jobConv.destroy();
	}
}

This tutorial has covered how to create, manage, execute and monitor jobs at a remote Job Service.