Managing Files with OneCompute Platform

All registered users of OneCompute Platform has one blob container that may store files. The main purpose of this container is to provide high capacity persistent storage of files that are either input to or output from OneCompute jobs. This blob container is private and accessible only to the user.

The following methods can be used to get access to the container:

Method	Description
GetContainersAsync	Gets the containers the user has access to. Currently, the user has access to only one container
GetContainerUriAsync	Gets the container URI to the given container. This URI embeds an access token that gives time limited read and write access to the container.
CreateBlobUrisAsync	Create BLOB URIs for the given paths within the container. This gives direct read and write access to the given blobs.

Getting access to cloud file storage

In order to access the user's private blob container and upload files to it or download files from it, a container URI must be retrieved through the OneCompute API. This container URI contains an access token that grants read and write access to the container for a limited time.

// Create the OneCompute Platform client
var apiUrl = "https://..."; // Url to the OneCompute Platform API
var authToken = ... // Authentication token obtained by authenticating with the IDP.
var oneComputePlatformClient = new OneComputePlatformClient(OneComputeApiUrl, authToken);

// Get the name of the user's blob container
var containers = await oneComputePlatformClient.GetContainersAsync();
var container = containers.FirstOrDefault();

// Get container URI
var containerUri = await oneComputePlatformClient.GetContainerUriAsync(container);

Uploading input files to blob storage

Before running jobs that require input files provided by the client application, these input files should be uploaded to the user's blob container. This can be done in two ways:

• Transfer files using the Azure Storage Library for .NET.
• Use the OneCompute BlobStorageFileTransferService

Upload files using the Azure Storage Library

The following code snippet shows how to instantiate a Microsoft.Azure.Storage.Blob.CloudBlobContainer, using the blob container URI obtained above and upload a file to the the container:

var cloudBlobContainer = new Microsoft.Azure.Storage.Blob.CloudBlobContainer(containerUri);

// Get a reference to a blob and upload a file to it
var blobPath = "myblobdirectory/myblob.txt";
var filePath = <file path>;
var blob = cloudBlobContainer.GetBlockBlobReference(blobPath);
await blob.UploadFromFileAsync(filePath);

Upload files using OneCompute BlobStorageFileTransferService

The OneCompute BlobStorageFileTransferService is used by OneCompute Platform to for reliable, high throughput and parallel transfer of files to and from the compute nodes in the Azure Batch backend. Likewise, it can be used by client applications to transfer files to and from the user's blob container.

The following code snippet shows how to upload a set of files to the container using the BlobStorageFileTransferService:

var fts = new DNVGL.One.Compute.StorageLibrary.Azure.FileTransfer.BlobStorageFileTransferService();
var spec = new FileTransferSpecification
{
    DestinationSpecification = new BlobDirectorySpecification
    {
        ContainerUrl = containerUri.ToString(),
        Directory = "mydirectory"
    },
    SourceSpecification = new FileSystemDirectorySpecification { Directory = @"c:\temp" },
    SelectedFiles = new[] { "**/*.*" },
    ExcludedFiles = new[] { "**/*.txt" }
};
var fileTransferResults = await fts.UploadFilesAsync(new[] { spec });
if (fileTransferResults.Any(r => r.TransferStatus == WorkStatus.Faulted))
{
    foreach (var failedTransferResult in fileTransferResults.Where(r => r.TransferStatus == WorkStatus.Faulted))
    {
        Console.WriteLine($"Failed to upload {failedTransferResult.SourceFile}. Error: {failedTransferResult.Message}.");
    }
}

See File Transfer for details and more code examples.

Downloading output files from blob storage

Download files using the Azure Storage Library

The following code snippet shows how to instantiate a Microsoft.Azure.Storage.Blob.CloudBlobContainer, using the blob container URI obtained above and download a file from the the container:

var cloudBlobContainer = new Microsoft.Azure.Storage.Blob.CloudBlobContainer(containerUri);

// Get a reference to a blob and download a file from it
var blobPath = "myblobdirectory/myblob.txt";
var filePath = <file path>;
var blob = cloudBlobContainer.GetBlockBlobReference(blobPath);
await blob.DownloadToFileAsync(filePath, System.IO.FileMode.OpenOrCreate);

Download files using OneCompute BlobStorageFileTransferService

The following code snippet shows how to download a set of files from the container using the BlobStorageFileTransferService:

var fts = new DNVGL.One.Compute.StorageLibrary.Azure.FileTransfer.BlobStorageFileTransferService();
var spec = new FileTransferSpecification
{
    SourceSpecification = new BlobDirectorySpecification
    {
        ContainerUrl = containerUri.ToString(),
        Directory = "mydirectory"
    },
    DestinationSpecification = new FileSystemDirectorySpecification { Directory = @"c:\temp" },
    SelectedFiles = new[] { "**/*.*" },
    ExcludedFiles = new[] { "**/*.txt" }
};
var fileTransferResults = await fts.DownloadFilesAsync(new[] { spec });
if (fileTransferResults.Any(r => r.TransferStatus == WorkStatus.Faulted))
{
    foreach (var failedTransferResult in fileTransferResults.Where(r => r.TransferStatus == WorkStatus.Faulted))
    {
        Console.WriteLine($"Failed to download {failedTransferResult.SourceFile}. Error: {failedTransferResult.Message}.");
    }
}