CFD,  Cloud computing,  HPC,  OpenFOAM

CFD simulations on Azure Batch – Part 1

August 21, 2021

I have recently used Azure Batch to do CFD simulations using OpenFOAM®, and I summarize my experience and the procedure I followed below. Before entering the details, it is worth saying what Azure Batch is. Azure Batch is a compute management service that lets users to rapidly deploy batch processes either by running multiple tasks on individual nodes, or a single task on multiple nodes, with high-performance communication (Infiniband or, as it is called in the service, RDMA). While Azure Batch is not a full replacement for an HPC cluster with standard queue systems, it is a convenient option for projects that require a defined number of simulations that need to run for a well-established amount of time and on clearly defined hardware resources.

In order to run simulations on Azure Batch, the following steps need to be followed:

  1. Prepare the simulation cases so that they can be executed entirely without human intervention
  2. Install useful tools to work with Azure Batch and Azure Storage
  3. Configure Azure Storage to persist input and output data to be used by Azure Batch
  4. Configure the pool of nodes on Azure Batch
  5. Add jobs to the pool of nodes (A job in Azure Batch is a set of tasks: a task can be seen as a single unit of work. In our case a task will be a simulation) and add tasks to each job

This post is dedicated to the first point, while in the next posts I will show how to configure Azure Storage and Azure Batch as I did in my use case. The configuration may not be the absolute optimal but it is sufficiently simple to use and similar to what would be the workflow on a traditional HPC cluster.

Assumptions

The description below is based on the following assumptions:

  • The considered use case considered a non-isothermal multicomponent non-reactive buoyant flow, with a mesh of some million cells (approximately 5 million in the specific case).
  • The solver used was rhoReactingBuoyantFoam with OpenFOAM v2106.
  • Multiple cases are executed in parallel. This is a typical use case when multiple flow configurations need to be evaluated.
  • The cases were meshed with snappyHexMesh and run on virtual machines with 120 cores (Azure HB120RS V2 instances with 450 GB of RAM).
  • No internode communication was used since the mesh size was well handled on a single node.

Virtual machines were configured with Ubuntu 18.04 LTS, since it is one of the default Linux distributions directly available in Azure Batch. Such a distribution also has the advantage that OpenFOAM is packaged for it, consequently, it requires a minimal amount of configuration to set the working environment up on Azure Batch.

Requirements

In order to execute simulations on Azure Batch, it is required to provide a bash command that needs to be executed. Some important aspects need to be reminded when doing so:

  • A single command line can be specified in Azure Batch.
  • Azure Batch does not automatically execute commands in a shell. Consequently, if it is necessary to execute commands in a bash shell, this must be explicitly specified by executing the desired shell before issuing the command line.
  • The environment variable AZ_BATCH_NODE_MOUNTS_DIR indicates the path to the mounted storage units in each compute node. We will see more details later about how to define a storage unit and mount it so that it becomes accessible from the nodes running the tasks.

Keeping these points in mind, if we want to use the Linux bash shell, enter the directory testData on the mounted disk testDisk, and run the icoFoam command on case1, contained in the directory case1, we have to issue the following instruction:

/bin/bash -c "cd $AZ_BATCH_NODE_MOUNTS_DIR/testDisk/testData/case1 && /usr/bin/openfoam2106 icoFoam -parallel "

The instruction above also shows that we leverage one facility provided by OpenFOAM v2106, the dedicate shell for OpenFOAM, which allows OpenFOAM commands to be run directly without having to configure environment variables explicitly.

In the following, we will create a bash script called Allrun, as commonly done in OpenFOAM. This script will contain all the instructions needed to set up one simulation and will also allow to distinguish cases when the mesh is provided and does not need to be generated, and cases that are being restarted rather than executed for the first time. Such script will be the only instructions that need to be executed; consequently, the command line for Azure Batch will become:

/bin/bash -c "cd $AZ_BATCH_NODE_MOUNTS_DIR/testDisk/testData/case1 && /usr/bin/openfoam2106 ./Allwmake "

Case setup

Each case has to be configured to be executed fully automatically to be able to run on Azure Batch. This is made easy by OpenFOAM, if the the steps below are followed:

  1. Ensure that the applicationName in controlDict is properly set to the name of the OpenFOAM solver that needs to be used to perform the simulations. In this case, the entry is set to rhoReactingBuoyantFoam. Also make sure that startTime is set to latestTime, to avoid changing the settings for restart operations.
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2106                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

application     rhoReactingBuoyantFoam;

startFrom       latestTime;

startTime       0;

// Lines removed for brevity

writeFormat     binary;

In order to preserve accuracy and save space, also set the output format to binary as shown in the example above.

  1. Ensure that all boundary conditions, also those that will be added by snappyHexMesh, are defined in the dictionaries in the 0.orig directory.
  2. Add a generically named processor boundary to all the dictionaries in the 0.orig folder. This step simplifies the process decomposition process since we will be using the -copyZero option. The example below illustrates it for theU field, but the same processor boundary condition can be used for all other fields:
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2106                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       volVectorField;
    object      U;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

dimensions      [0 1 -1 0 0 0 0];

internalField   uniform (0 0 0);

boundaryField
{
    #includeEtc "caseDicts/setConstraintTypes"

    // Processor boundary
    "procBoundary.*"
    {
        type            processor;
        value           $internalField;
    }

    Inlet
    {
        type                  flowRateInletVelocity;
        volumetricFlowRate    0.141584233;        // Volumetric/mass flow rate [m3/s or kg/s]
        value                 uniform (0 0 0);    // Placeholder
    }

    Outlet
    {
        type                flowRateOutletVelocity;
        volumetricFlowRate  1.4536;
        value               uniform (0 0 0);
    }

    "(BottomWall|TopWall|SideWall|Floor)"
    {
        type            noSlip;
    }
}


// ************************************************************************* //
  1. Make sure a valid decomposeParDict dictionary is present in the system folder of the case. For example, to use the scotch decomposition library, the dictionary will look like this:
/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  v2106                                 |
|   \\  /    A nd           | Website:  www.openfoam.com                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 120;  // Number of cores

method          scotch;

// ************************************************************************* //

Bash script to automate case execution

The first time the case under consideration is executed the following commands need to be executed, if the mesh is not provided and needs to be generated:

  • Restore the initial condition folder (0 folder in OpenFOAM) from the original files used for configuration purposes, contained in the 0.orig directory.
  • Create the base mesh for snappyHexMesh using blockMesh
  • Extract features from geometry files used by snappyHexMesh by running surfaceFeatureExtract
  • Decompose the case so that snappyHexMesh can be executed in parallel
  • Check the mesh quality and save the outcome of the mesh test to a log file by executing checkMesh
  • Renumber the mesh to reduce the matrix bandwidth, if possible, by running renumberMesh
  • Reconstruct the mesh for future postprocessing use by running reconstructParMesh
  • Initialize the solution using setFields
  • Execute the rhoReactingBuoyantFoam

If the mesh is provided, the list above for the first run reduces to:

  • Restore the initial condition folder (0 folder in OpenFOAM) from the original files used for configuration purposes, contained in the 0.orig directory.
  • Decompose the case so that snappyHexMesh can be executed in parallel
  • Initialize the solution using setFields
  • Execute the rhoReactingBuoyantFoam

On restart, the only command that needs to be executed is the last one in the list above.

The script checks for the existence of a file called case.foam to decide if the case has been executed before. This file is created by the script itself by executing touch case.foam in the case directory on the first run. A variable called meshProvided is used to decide if the case mesh is already generated (true) or not (false).

#!/bin/sh
cd "${0%/*}" || exit                                # Run from this directory
. ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions        # Tutorial run functions
#------------------------------------------------------------------------------

restartFile=case.foam
meshProvided=false

if [ ! -f "$restartFile" ]; 
then
    echo "First run"
    restore0Dir
    # Creating base mesh
    if [ ! $meshProvided ];
    then
        runApplication blockMesh
        runApplication surfaceFeatureExtract
    fi
    runApplication decomposePar -copyZero
    
    if [ ! $meshProvided ];
    then
        runParallel snappyHexMesh -overwrite
        runParallel checkMesh -allGeometry -allTopology
        runParallel renumberMesh -overwrite
        runApplication reconstructParMesh -constant 
    fi
    # Initializing
    runParallel setFields
    
    # Preparing for postprocessing
    touch $restartFile
else
    echo "Restarting"
   
    # Removing log to allow restart
    mv log.rhoReactingBuoyantFoam log.rhoReactingBuoyantFoam.old
fi

# Running simulation
runParallel $(getApplication)

As shown in the code above:

  1. The script is designed to be executed from the top directory containing the case (where the 0, constant and system folder is).
  2. Commands to execute OpenFOAM applications in serial mode (runApplication) and parallel (runParallel) are used. These commands automatically generate a log file called log.<applicationName>. where <applicationName> is the command line name of the application being executed. If a log file is already present, these commands will not be executed, hence why the log file for rhoReactingBuoyantFoam is renamed on restart.

The steps described in this post show how to automate the execution of each OpenFOAM case that will be launched on Azure Batch. 

In the next posts the actual setup of the Batch service will be discussed.

Stay tuned! 🙂

This offering is not approved or endorsed by OpenCFD Limited, the producer of the OpenFOAM software and owner of the OPENFOAM® and OpenCFD® trade marks. Alberto Passalacqua is neither associated to OpenCFD Ltd nor to Microsoft and did not receive funding from them to write this article. All registered trade marks belong to their legal owners.