Support for Azure Batch container workloads

Background

Azure Batch provides users with the choice to run their applications on Docker compatible containers.

The main difference with this approach is that the execution environment will be confined to the container which will require users to deploy their application as Docker images instead of Application packages. User will have to define their pools with the ability to pull images from the Azure Containers Registry

OneCompute provides mechanisms to allow the users to define their task to take advantage of this feature. More details of Azure Batch containers workload can be found here

Define Azure Batch Pool connected to Azure Container Registry

Before being able to run an application on a Docker container, users need to define the Azure Batch Pool where the tasks are going to be executed. The pool must be able to pull images from at least one Azure Batch Container Registry.

Adding a container compatible Azure Batch Pool

The new pool can be set to connect to more than one Azure Container Registry:

Pool connected to several registries

Define Docker file to create Worker Image

Users must create a Docker image that will be pushed to one of the registries to which the pool has access to.

Details about how to create and publish a Docker image can be found here. Users can take the example below as a guide to produce a valid Docker file to build a worker image:

################################################################
### DNVGL.One.Compute.Samples.Squarer dockerfile
### example build command...
### docker build -t onecompute.azurecr.io/samples.squarer:latest --build-arg PAT=<ENTERYOURPAT> -f src\DNVGL.One.Compute.Samples.Squarer\DNVGL.One.Compute.Samples.Squarer.Worker\Deployment\dockerfile .
################################################################

##### BUILD STAGE
# Prepare a build image
FROM microsoft/dotnet:2.2-sdk AS build
# With build arguments. Must provide a Personal Access Token for the one feed.
ARG FEED_URL=https://dnvgl-one.pkgs.visualstudio.com/_packaging/One/nuget/v3/index.json
ARG PAT

# download and install latest credential provider. Not required after https://github.com/dotnet/dotnet-docker/issues/878
RUN wget -qO- https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh | bash

# Environment variable to enable session token cache. More on this here: https://github.com/Microsoft/artifacts-credprovider#help
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
# Environment variable for adding endpoint credentials. More on this here: https://github.com/Microsoft/artifacts-credprovider#help
# Add "FEED_URL" AND "PAT" using --build-arg in docker build step. "endpointCredentials" field is an array, you can add multiple endpoint configurations.
# Make sure that you *do not* hard code the "PAT" here. That is a sensitive information and must not be checked in.
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS {\"endpointCredentials\": [{\"endpoint\":\"${FEED_URL}\", \"username\":\"ArtifactsDocker\", \"password\":\"${PAT}\"}]}

# In the build container we will work in a src folder.
WORKDIR /src

# We want to do a dotnet restore but want to take advantage of the docker build cache.
# The restore only depends upon the .csproj, .sln and nuget.config files so only copy these files across.
COPY ["src/nuget.config", "src/nuget.config"]
COPY ["src/DNVGL.One.Compute.Samples.Squarer/DNVGL.One.Compute.Samples.Squarer.Common/DNVGL.One.Compute.Samples.Squarer.Common.csproj", "src/DNVGL.One.Compute.Samples.Squarer/DNVGL.One.Compute.Samples.Squarer.Common/"]
COPY ["src/DNVGL.One.Compute.Samples.Squarer/DNVGL.One.Compute.Samples.Squarer.Worker/DNVGL.One.Compute.Samples.Squarer.Worker.csproj", "src/DNVGL.One.Compute.Samples.Squarer/DNVGL.One.Compute.Samples.Squarer.Worker/"]

# And perform the restore
RUN dotnet restore "src/DNVGL.One.Compute.Samples.Squarer/DNVGL.One.Compute.Samples.Squarer.Worker/DNVGL.One.Compute.Samples.Squarer.Worker.csproj"

# Only now copy over the source files
COPY . .
# And build the worker project
WORKDIR "src/DNVGL.One.Compute.Samples.Squarer/DNVGL.One.Compute.Samples.Squarer.Worker/"
RUN dotnet build "DNVGL.One.Compute.Samples.Squarer.Worker.csproj" -c Release --framework netcoreapp3.1 -o /app

##### PUBLISH STAGE
FROM build AS publish
RUN dotnet publish "DNVGL.One.Compute.Samples.Squarer.Worker.csproj" -c Release --framework netcoreapp3.1 -o /app

##### PACKAGE STAGE
# Prepare the final application image using the onecompute azurebatch workerhost image as the base.
FROM onecompute.azurecr.io/workerhost.azurebatch:2.2.0-alpha0011 AS base
WORKDIR /app
EXPOSE 80
FROM base AS package
WORKDIR /app
COPY --from=publish /app .

Note

User created images should be based on the latest version of the onecompute.azurecr.io/workerhost.azurebatch image that is maintained and updated by the OneCompute team.

Define container settings for a specific Task

After the image is built and published to a container registry then any specific WorkUnit can be set to be executed on a container. Users must at least set ImageName to indicate what to use to create the container.

RunOptions are optional. Default value for this property is: "-it --rm"

.....
return new WorkUnit(input)
{
    Properties =
    {
        { ContainerSettingsNames.ImageName, "docker pull onecompute.azurecr.io/samples.squarer:2.2.0-alpha0011" },
        {ContainerSettingsNames.RunOptions, "-dt -env DATA=/data" }
    }
};
.....