Using SCARF’s AMD nodes

Overview

With the release of SCARF20, SCARF now has both Intel and AMD processors. This page documents the steps you’ll need to take to run and compile jobs on SCARF’s AMD nodes.

Introduction

As compilers can optimise code for specific processors we’ve decide to recompile the software we provide on the AMD nodes to ensure that it works. In particular binaries compiled on an Intel host with the Intel Compiler using the -xHost flag will not execute on AMD systems and must be recompiled. Binaries produced by other compilers may work, but you may wish to recompile on the AMD nodes anyway for better performance

Getting Access to AMD compatible software and tools

AMD compiled software and tools are in a separate area of our applications repository, to get access to them, run

module load AMDmodules

A number of toolchains and applications have been made available, please get in touch if you would like additional ones provided.

Running jobs on AMD nodes

Due to the potential incompatibility between Intel compiled binaries and the AMD nodes, we have added a default constraint of “intel” for all jobs that do not specify a constraint. To use AMD nodes, add the parameter below to your submission parameters:

-C amd

If you have verified that your binaries work on Intel nodes and AMD nodes and have no preference to what you run on, then you can specify

-C "[intel|amd]"

If you wish to run on the latest AMD nodes or the latest Intel nodes specify

-C "[scarf18|amd]"

Alternatively you can use the constraint scarf20 in place of amd in the previous examples, this will limit you to running on the current generation of AMD nodes, but would exclude any potential future AMD nodes that SCARF deploys. The combinations noted above are examples, others combinations will also work, please see the available features here

The default constraint of intel will be kept under review and may be changed as the processor mix of SCARF changes or as more software is available for AMD nodes.

Compiler Flag recommendations

For versions of gcc greater than 9.3 use the flag

-march=znver2

to compile for the Epyc2 processors in SCARF 20, For the Intel compiler use the flag

-march=core-avx2

Please see the AMD Compiler Options Quick Reference Guide for AMD’s recomended settings for a number of compilers (note that not all the mentioned compilers and libraries in the linked document are available on SCARF)