Attention

SCARF is migrating to Rocky 9, please see Rocky 9 Migration for details on the Rocky 9 cluster

Rocky 9 Migration

Overview

SCARF is migrating to Rocky 9 as its underlying Linux distribution, away from Centos 7. This page documents some of the changes and what you may need to know

Why change?

Centos 7 has reached end of life and will no longer receive security updates, therefore to comply with organisational security policies we must upgrade.

What’s changed?

In short, quite a lot. Default versions of many tools and libraries on the system have increased, particularly notable is that Python 2 is now unavailable and python scripts must use Python 3. For a fuller list of changes see https://docs.rockylinux.org/release_notes/9_0/

In addition, we decided to make various changes to the system at this time to remove old functionality and add some new.

  • Platform MPI removed

    Platform MPI was historically SCARF’s default MPI library, however the version we have no longer receives commercial support and it is not cost-effective to have a supported version. We have been gradually phasing out Platform MPI - it was not used in software we provided in Centos 7, and with the migration to Rocky 9, it has been removed. OpenMPI is our preferred MPI library.

  • Removing old compiler versions

    We’ve taken the decision to as far as possible use the EasyBuild framework to install and deploy the applications we provide to users. This simplifies the deployment of applications by allowing us to take advantage of community provided build recipes for common applications. However, this does mean that some modules may have changed name or are accessed in a different way. In addition EasyBuild only supports a set of recent compilers and versions, so the older compilers have been removed.

  • Implementing a role based system for module visibility.

    We’ve implemented a new system to make the module list more manageable. This consists of 3 roles:

    • USER - can only see application modules. This is the default role

    • BUILDER - can see toolchains and libraries for building your own software

    • DEVELOPER - can see all modules. This is the role most like the previous system, but note that we are no longer hiding some modules we previously did.

    All roles can see modules under the contrib hierarchy.

    The utility set_scarf_role can be used to change role, taking effect on next login.

  • Upgrade Slurm to a newer version

    We’ve upgraded Slurm, the batch system that schedules and launches jobs to a newer version, however most users will not notice any changes and existing batch scripts should continue to work. The Rocky 9 nodes are implemented as a separate cluster of nodes using a separate controller, but we will keep the configuration in step during migration.

  • Some nodes will not migrate to Rocky 9

    The SCARF18 and SCARF19 nodes will not be migrated to Rocky 9 - these nodes are around 5-6 years old and are due for retirement. Once all the other nodes have migrated they will be turned off and decommissioned. This does mean that for the moment the new Rocky 9 cluster will only have nodes with AMD processors.

How do I access the new system?

Currently ui2.scarf.rl.ac.uk is a Rocky 9 based system which can be used for login. Once all compute nodes are migrated to Rocky 9, ui1 will be migrated to Rocky 9. The home directories and work areas have not changed.

Note that some very old ssh key types have been disabled in Rocky 9 as they are now considered insecure. If you are able to login to ui1 but not ui2 using your ssh key, please generate a new ssh key following the instructions on this page: Generating SSH Key and send the new public key to us, see the bottom of this page for contact details.

ui2 has been reinstalled so it may generate a warning in your ssh client due to its ssh keys changing, the new ssh key fingerprints are:

RSA SHA256:3Kq5yShCoXeStqX7PhMKCJImnk11ju9PcZY1OqGHQ8w
ED25519 SHA256:XHdztFTjHxSdZmMUDgyR8YfCTS1qgjK5EYfv0wA3Kqc
ECDSA SHA256:Slhfj4/KjaNfB10K6kv4vMIBq8Uk2loeUmDKtqrRTAg

(If you need to see the status of jobs across both old and new clusters then ui1 can be used by adding the parameter -M all to your squeue command, but we don’t recommend submitting jobs to the Rocky 9 partitions from ui1 and the remaining Centos 7 systems. ui2 can’t be used to monitor jobs on the Centos 7 cluster due to the slurm upgrade - the old version of the controller handling the Centos 7 nodes doesn’t understand the new clients on the Rocky 9 nodes.)

Where are the modules/software I need?

Firstly, check you have an appropriate role set, see above, as all users will default to the USER role, which only sees top-level applications.

If you are using a contrib module, we have created a fresh, empty contrib area as there many changes and we don’t wish to assume that the software will work as-is. Please ask the maintainer of the contrib area to make the software available again. The old contrib area is available read-only at /oldsw/contrib, but will be removed once the contrib software has been migrated.

We have launched the Rocky 9 service with an initial set of applications, more will be added over time. Please get in touch if you have a need for a particular application.

If you wish to build your own software, we recommend that you start by loading one of the toolchain modules. For example, foss provides a GNU Compiler Collection (GCC) based compiler toolchain, including OpenMPI for MPI support, OpenBLAS (BLAS and LAPACK support), FFTW and ScaLAPACK. You can use module whatis <modulename> to see the description of a module.

I have more questions or need more help

Please get in touch by emailing us at scarf@hpc-support.rl.ac.uk or using the contact form Contact us. We also have a fortnightly office hour on Zoom, the details of which are posted to the SCARF-USERS mailing list.