Status and Progress for Jobs and Work Items

Background

Work Item status information is recorded by the Worker Hosts as the Worker starts processing, as well as when it stops processing, either by successful completion or by failure. Likewise, work item progress is recorded by the Worker. In addition, while processing the work item, the Worker may post progress information, which subsequently will be recorded by the Worker Host. Job status and progress is aggregated from the work item information by the Job Scheduler as it orchestrates the job.

Once a job has been submitted for processing, the client application may need to retrieve status and progress information about the job. OneCompute provides components for retrieving job status and progress for jobs and constituent work items.

Note

Job status and progress information is recorded to different stores, depending on the Backend environment. Therefore, depending on which Backend environment is used by the application, different components must be used to retrieve this information. For instance, when running with an Azure Batch backend, Azure Storage is used by OneCompute, while other backends may use SQLite for recording job status and progress. See the Storage section for details.

It should be noted that progress, both for the job and the work item, is a number between 0 and 1. Progress starts at 0 and ends at 1 when the job or work item terminates (completes or fails).

Getting the storage connection string

To connect to the job information storage, a storage connection string is needed. For information about how to retrieve the storage connection information for the relevant Backend environment, see the Storage section.

Aggregated Job Status and Progress

The interface for retrieving aggregated job information is IJobStatusService.

Acquiring and connecting the job status service

First, we must acquire the job status service and connect it to the job information storage, using the IStorageServiceFactory implementation provided for the chosen Backend platform. When running with a backend that uses Azure Storage, such as the Azure Batch backend, the DNV.One.Compute.StorageLibrary.Azure library should be referenced and AzureTableStorageServiceFactory can be used:

using DNV.One.Compute.StorageLibrary.Azure.Factory;
...
string storageConnectionString = <storage connection string for the OneCompute deployment>
IStorageServiceFactory storageServiceFactory = new AzureTableStorageServiceFactory();
IJobStatusService jobStatusService = storageServiceFactory.CreateJobStatusService(storageConnectionString);

If running with backends that use the SQLite storage service, the DNV.One.Compute.StorageLibrary.SQLite should be referenced and SQLiteStorageServiceFactory can be used:

using DNV.One.Compute.StorageLibrary.SQLite.Factory;
...
string storageConnectionString = <storage connection string for the OneCompute deployment>
IStorageServiceFactory storageServiceFactory = new SQLiteStorageServiceFactory();
IJobStatusService jobStatusService = storageServiceFactory.CreateJobStatusService(storageConnectionString);

Retrieving Job Information

Once connected to the job information storage, the IJobStatusService.GetJobStatus method must be used:

string jobId = <the id of the job>
var jobInfo = jobStatusService.GetJobInfo(jobId);

The job status and progress can then be retrieved from the returned JobInfo object:

string message = jobInfo.Message;
WorkStatus status = jobInfo.Status;
double progress = jobInfo.Progress; // Number between 0 and 1

Work Item Status and Progress

Acquiring and connecting the work item status service

Status and progress for individual or collections of work items can be retrieved in much the same way as for job status and progress. First, we must acquire the work item status service and connect it to the work item storage, using the IStorageServiceFactory implementation provided for the chosen Backend platform. When running with a backend that uses Azure Storage, such as the Azure Batch backend, the DNV.One.Compute.StorageLibrary.Azure library should be referenced and AzureTableStorageServiceFactory can be used:

using DNV.One.Compute.StorageLibrary.Azure.Factory;
...
string storageConnectionString = <storage connection string for the OneCompute deployment>
IStorageServiceFactory storageServiceFactory = new AzureTableStorageServiceFactory();
IWorkItemStatusService workItemStatusService = storageServiceFactory.CreateWorkItemStatusService(storageConnectionString);

If running with backends that use the SQLite storage service, the DNV.One.Compute.StorageLibrary.SQLite should be referenced and SQLiteStorageServiceFactory can be used:

using DNV.One.Compute.StorageLibrary.SQLite.Factory;
...
string storageConnectionString = <storage connection string for the OneCompute deployment>
IStorageServiceFactory storageServiceFactory = new SQLiteStorageServiceFactory();
IWorkItemStatusService workItemStatusService = storageServiceFactory.CreateWorkItemStatusService(storageConnectionString);

Retrieving Work Item Information for all work items in a job

To retrieve work item information for all work items in a job, the IWorkItemStatusService.GetJobItemInfo method should be used:

string jobId = <the id of the job>
IEnumerable<WorkItemInfo> workItemsInfos = workItemStatusService.GetJobItemInfo(jobId);
foreach(WorkItemInfo info in workItemInfos)
{
    double progress = info.Progress; // Number between 0 and 1
    WorkStatus workItemStatus = info.Status;
}

Retrieving Work Item Information for one work item

To retrieve work item information for one particular work item in a job, the IWorkItemStatusService.GetWorkItemInfo method should be used:

string jobId = <the id of the job>
string workItemId = <the work item id>
WorkItemInfo workItemsInfo = workItemStatusService.GetWorkItemInfo(jobId, workItemId);

double progress = info.Progress; // Number between 0 and 1
WorkStatus workItemStatus = info.Status;

Retrieving Work Item Information for child work items

Work items may have children. For instance, ParallelWork will have children that are scheduled to be processed in parallel. To retrieve status and progress for child items of a given item, the IWorkItemStatusService.GetPartItemInfo method should be used:

string jobId = <the id of the job>
IEnumerable<WorkItemInfo> workItemsInfos = workItemStatusService.GetPartItemInfo(jobId, string parentItemId);
foreach(WorkItemInfo childItemInfo in workItemInfos)
{
    double progress = childItemInfo.Progress; // Number between 0 and 1
    WorkStatus workItemStatus = childItemInfo.Status;
}