Managing Files with OneCompute Platform

All registered users of OneCompute Platform has one blob container that may store files. The main purpose of this container is to provide high capacity persistent storage of files that are either input to or output from OneCompute jobs. This blob container is private and accessible only to the user.

The following methods can be used to get access to the container:

Method	Description
GetContainersAsync	Gets the containers the user has access to. Currently, the user has access to only one container
GetContainerUriAsync	Gets the container URI to the given container. This URI embeds an access token that gives time limited read and write access to the container.
CreateBlobUrisAsync	Create BLOB URIs for the given paths within the container. This gives direct read and write access to the given blobs.

Getting access to cloud file storage

In order to access the user's private blob container and upload files to it or download files from it, a container URI must be retrieved through the OneCompute API. This container URI contains an access token that grants read and write access to the container for a limited time.

// Create the OneCompute Platform client
var apiUrl = "https://..."; // Url to the OneCompute Platform API
var authToken = ... // Authentication token obtained by authenticating with the IDP.
var oneComputePlatformClient = new OneComputePlatformClient(OneComputeApiUrl, authToken);

// Get the name of the user's blob container
var containers = await oneComputePlatformClient.GetContainersAsync();
var container = containers.FirstOrDefault();

// Get container URI
var containerUri = await oneComputePlatformClient.GetContainerUriAsync(container);

Uploading input files to blob storage

Before running jobs that require input files provided by the client application, these input files should be uploaded to the user's blob container. This can be done in two ways:

• Transfer files using the Azure Storage Blobs client library for .NET.
• Use the OneCompute BlobStorageFileTransferService

Upload files using the Azure Storage Library

The following code snippet shows how to instantiate a Azure.Storage.Blobs.BlobContainerClient, using the blob container URI obtained above and upload a file to the container:

// Create a client object that references the container
var blobContainerClient = new Azure.Storage.Blobs.BlobContainerClient(containerUri);

// Create a client object that references the BLOB
var blobPath = "myblobdirectory/myblob.txt";
var blobClient = blobContainerClient.GetBlobClient(blobPath);

// Open the file and upload its data
var localFilePath = <file path>;
using var uploadFileStream = File.OpenRead(localFilePath);
await blobClient.UploadAsync(uploadFileStream, true);
uploadFileStream.Close();

Upload files using OneCompute BlobStorageFileTransferService

The OneCompute BlobStorageFileTransferService is used by OneCompute Platform to for reliable, high throughput and parallel transfer of files to and from the compute nodes in the Azure Batch backend. Likewise, it can be used by client applications to transfer files to and from the user's blob container.

The following code snippet shows how to upload a set of files to the container using the BlobStorageFileTransferService:

var fts = new DNVGL.One.Compute.StorageLibrary.Azure.FileTransfer.BlobStorageFileTransferService();
var spec = new FileTransferSpecification
{
    DestinationSpecification = new BlobDirectorySpecification
    {
        ContainerUrl = containerUri.ToString(),
        Directory = "mydirectory"
    },
    SourceSpecification = new FileSystemDirectorySpecification { Directory = @"c:\temp" },
    SelectedFiles = new[] { "**/*.*" },
    ExcludedFiles = new[] { "**/*.txt" }
};
var fileTransferResults = await fts.UploadFilesAsync(new[] { spec });
if (fileTransferResults.Any(r => r.TransferStatus == WorkStatus.Faulted))
{
    foreach (var failedTransferResult in fileTransferResults.Where(r => r.TransferStatus == WorkStatus.Faulted))
    {
        Console.WriteLine($"Failed to upload {failedTransferResult.SourceFile}. Error: {failedTransferResult.Message}.");
    }
}

See File Transfer for details and more code examples.

Downloading output files from blob storage

Download files using the Azure Storage Library

The following code snippet shows how to instantiate a Azure.Storage.Blobs.BlobContainerClient, using the blob container URI obtained above and download a file from the container:

// Create a client object that references the container
var blobContainerClient = new Azure.Storage.Blobs.BlobContainerClient(containerUri);

// Create a client object that references the BLOB
var blobPath = "myblobdirectory/myblob.txt";
var blobClient = blobContainerClient.GetBlobClient(blobPath);

// Download the BLOB's contents and save it to a file
var download = await blobClient.DownloadAsync();
var localFilePath = <file path>;
using (var downloadFileStream = File.OpenWrite(localFilePath))
{
    await download.Content.CopyToAsync(downloadFileStream);
    downloadFileStream.Close();
}

Download files using OneCompute BlobStorageFileTransferService

The following code snippet shows how to download a set of files from the container using the BlobStorageFileTransferService:

var fts = new DNVGL.One.Compute.StorageLibrary.Azure.FileTransfer.BlobStorageFileTransferService();
var spec = new FileTransferSpecification
{
    SourceSpecification = new BlobDirectorySpecification
    {
        ContainerUrl = containerUri.ToString(),
        Directory = "mydirectory"
    },
    DestinationSpecification = new FileSystemDirectorySpecification { Directory = @"c:\temp" },
    SelectedFiles = new[] { "**/*.*" },
    ExcludedFiles = new[] { "**/*.txt" }
};
var fileTransferResults = await fts.DownloadFilesAsync(new[] { spec });
if (fileTransferResults.Any(r => r.TransferStatus == WorkStatus.Faulted))
{
    foreach (var failedTransferResult in fileTransferResults.Where(r => r.TransferStatus == WorkStatus.Faulted))
    {
        Console.WriteLine($"Failed to download {failedTransferResult.SourceFile}. Error: {failedTransferResult.Message}.");
    }
}