Storage Services

OneCompute requires access to a storage service in order to pass input data to the compute server and to receive results back from the server. In addition, a storage service is required for the OneCompute client to keep track of the status and progress of jobs and work units.

Currently OneCompute supports Azure Storage (namely Tables) and SQLite (for local use). However, it is extensible and further storage services are likely to be supported in the future.

Using Azure Storage as the OneCompute Storage

Azure Storage is the most flexible option if you want to run workloads on the cloud. It can be used all of our supported back end servers, Azure Batch, REST & Console.

You can instantiate a Storage Account resource through the Azure Portal or using the Azure CLI.

Further information on this here.

Step 1- Obtain the Azure Storage connection string

OneCompute will use the Azure Storage API to communicate with the data tables. In order to do this, it must authenticate first using a connection string. The simplest way of authenticating against Azure Storage is using Shared Key Credentials. You can obtain your storage account name & key from the Keys blade in Azure as shown below...

OneCompute provides some classes to assist you in configuring the authenticated storage connection that will be used by the client and server libraries.

Use Shared Key Credentials

Using the storage account name and key that you obtained from the Keys blade of the portal, you can use the code below to produce the required connection string.

var storageAccountName = "<enter your storage account name here>";
var storageAccountKey = "<enter your storage account key here>";

var azureStorageConnectionStringResolver = new AzureStorageConnectionStringResolver();
azureStorageConnectionStringResolver.SetStorageAccountNameAndAccessKey(storageAccountName, storageAccountKey);
var storageConnectionString = azureStorageConnectionStringResolver.ResolveStorageConnectionString();

(Advanced) Using Storage Connection string stored in Azure Key Vault

Alternatively, you can utilize Azure Key Vault and Azure AD to retrieve the connection string from a Key Vault secret. This is a more advanced method and requires more complex configuration in the portal. The code below demonstrates how you would obtain the connection string using this method. For a more in depth discussion on this, see the Security section.

var secretUri = "<enter your azure key vault uri to the storage connection string secret>";
var azureADClientId = "<enter your Azure AD client Id>";
var azureADClientSecret = "<enter your Azure AD client secret";

var azureStorageConnectionStringResolver = new AzureStorageConnectionStringResolver();
azureStorageConnectionStringResolver.SetKeyVaultCredentialsByClientSecret(secretUri, azureADClientId, azureADClientSecret);
var storageConnectionString = azureStorageConnectionStringResolver.ResolveStorageConnectionString();

Step 2 - Create the OneCompute storage service instances

Now that you have the connection string, you can use the OneCompute AzureTableStorageServiceFactory class to instantiate the 4 storage services...

// Create storage service factory - for Azure Table storage
var storageServiceFactory = new AzureTableStorageServiceFactory();
// And construct the 4 storage services using the storage connection string
var workItemStatusService = storageServiceFactory.CreateWorkItemStatusService(storageConnectionString);
var jobStatusService = storageServiceFactory.CreateJobStatusService(storageConnectionString);
var workItemStorageService = storageServiceFactory.CreateWorkItemStorageService(storageConnectionString);
var resultStorageService = storageServiceFactory.CreateResultStorageService(storageConnectionString);

Step 3 - Create a OneCompute scheduler instance passing these storage services

These services should be passed to the OneCompute scheduler you create for the compute back-end.

Using SQLite as the OneCompute Storage

If you are using the Console back-end service then you have the option of using SQLite as the storage service instead. SQLite is an embedded database that works directly off a database file. This means that there is no administrative installation for SQLite, instead the SQLite libraries are used in process. The beauty of this scenario is that you can run offline - you don't need any network/cloud access at all. This provides you with the potential to switch your workloads from running completely locally to the cloud and vice-versa.

Note

A useful open-source tool for working with SQLite databases is DB Browser for SQLite. OneCompute does not require you to have any manual interaction with the database but should you want to, this is tool is very helpful.

Step 1- Obtain the connection string

For SQLite the connection string is simply the path to the SQLite database file.

The OneCompute StorageConnectionStringResolver will look in a user environment variable OneComputeStorageConnectionString for the connection string.

storageConnectionString = new StorageConnectionStringResolver().ResolveStorageConnectionString();

Tip

If you add an environment variable when Visual Studio is running, you will need to restart Visual Studio before a debug session sees the new/modified variable.

But you may find it easier just to specify the database path directly using code like this...

storageConnectionString = Path.Combine(Path.GetTempPath(), "MyOneComputeStorage.db");

This will use a SQLite database named MyOneComputeStorage.db in your system temporary directory.

You do not need to create the database yourself, OneCompute does this for you.

Step 2 - Create the OneCompute storage service instances

Now that you have the connection string, you can use the OneCompute SQLiteStorageServiceFactory class to instantiate the 4 storage services...

// Create storage service factory - for SQLite
var storageServiceFactory = new SQLiteStorageServiceFactory();
// And construct the 4 storage services using the storage connection string
var workItemStatusService = storageServiceFactory.CreateWorkItemStatusService(storageConnectionString);
var jobStatusService = storageServiceFactory.CreateJobStatusService(storageConnectionString);
var workItemStorageService = storageServiceFactory.CreateWorkItemStorageService(storageConnectionString);
var resultStorageService = storageServiceFactory.CreateResultStorageService(storageConnectionString);

Step 3 - Create a OneCompute scheduler instance passing these storage services

These services should be passed to the OneCompute scheduler you create for the compute back-end.