Using a Job Controller to Manage the Job

What is a Job Controller?

A job controller is is a work unit that is delegated responsibility of determining the life span of the job, which means that the job ends as soon as the job controller work has terminated. This is the the only real difference between a job controlling work unit and other work units. However, depending on the computational workflow to be supported, it may be natural to let the job controller take on key tasks, such as dynamic work scheduling and/or result aggregation.

When is it Useful?

The job controller may be useful for both predefined and dynamic workflows. The following describes a few example scenarios when a job controller can be useful.

Incremental Result Aggregation

For predefined workflow, the job controller can be a useful result aggregator that aggregates results from sibling work units as they arrive. If result aggregation is time consuming, this can be a more efficient workflow than adding a reduction step to a ParallelWork to perform the aggregation work. In the former case, aggregation can be done incrementally as soon as part results are available, while in the latter case, the aggregation can not be done until all part results are available. This may represent a bottleneck that can be avoided by letting the job controller do incremental reduction as soon as part results are available.

Map-Reduce

By utilizing the available mechanisms for dynamic workflows and worker messaging, the job controller may be used to implement a full map-reduce workflow by both creating (mapping) the work units of the job and aggregating (reducing) the results received from the created work units.

How to Create a Job with a Job Controller

The job controller is defined by assigning a WorkUnit to the Job.JobControllerWork member.

var job = new Job
{
    ServiceName = "s1",
    JobControllerWork = new WorkUnit(jobControllerInput),
    Work = new ParallelWork(workUnits),
};

How to Implement a Job Controlling Worker

Creation and submission of dynamic work can be done by a worker at any time during execution. This ability is not restricted to the worker processing the job controlling work unit only, but can be done by a worker processing any work unit.

Worker Initialization

To submit work during execution, the worker will need an instance of the IScheduleWork interface. This can be injected through the worker constructor by defining the constructor as follows:

[ImportingConstructor]
public MyWorker(IScheduleWork scheduleWorkService)
{
    this.scheduleWorkService = scheduleWorkService;
}

The scheduleWorkService enables the worker to create and submit any work item to be executed. This work item will then be added to the job and orchestrated by the OneCompute job scheduler.

Dynamic Work Scheduling

A work item can then be created by the worker at any time during execution. The following example shows how to do it from the worker execution method:

public async Task<object> ExecuteAsync(
                                        IWorkerExecutionStatusNotificationService _, 
                                        IWorkUnit workUnit, 
                                        IEnumerable<Result> dependencyResults)
{
    ...
    var dynamicWork = ... // Create the new work item
    await this.scheduleWorkService.ScheduleWorkAsync(workUnit, dynamicWork);
    ...
}

The first argument to ScheduleWorkAsync is the work unit that generated the new work item (the originating work unit). The second argument represents the new work to be added to the job and can be any work item.

Checking the Task Role

The IWorkUnit.TaskRole member enables a single worker to discriminate between work units with different roles. The TaskRole enum identifies the role of the work unit and enables the worker to process a work unit differently, depending on its role. A work unit assigned as JobControllerWork will have role TaskRoles.JobControllerTask.

See also

Concepts:

Types: