PHANTOM is an open-source smoothed-particle hydrodynamics code designed for supercomputers, utilising MPI and OpenMP parallelisation frameworks to distribute work across multiple CPU cores. However, up until now, scientific use has primarily relied on the OpenMP code, limiting the available hardware to single nodes on OzSTAR (16 or 32 CPUs) or Gadi (48 CPUs).

The goal of this optimisation project was to eliminate the main bottlenecks in the current MPI implementation. This effort was based on a six-week scoping project conducted by the ADACS team in 2021.

The project achieved several significant outcomes which addressed PHANTOM's parallel performance. Efforts focused on improving efficiency by naming OpenMP critical sections within the MPI implementation to prevent delays from MPI communication calls, previously a significant bottleneck. Dynamic memory reallocation was also implemented, enabling PHANTOM to manage changes in particle counts and load imbalances, thereby improving overall performance, particularly when running simulations with varying geometries. Optimisation efforts identified the optimal balance between MPI tasks and OpenMP threads for hybrid computing modes, departing from the assumption of setting the number of OpenMP threads to equal all of the cores on one node. Additionally, an MPI communication pattern compatible with OpenMP was introduced, allowing concurrent MPI operations by OpenMP threads without blocking.

These improvements were delivered via Pull Requests to the PHANTOM codebase on GitHub and are pending final review before being put into production.

Additionally, the ADACS team parallelised the existing test suite on GitHub Actions, improving rapid detection of issues during the feature branch development workflow. This streamlined the development process for PHANTOM developers, enabling them to focus more on scientific problem-solving rather than manual code troubleshooting.

Looking forward, further work is needed to achieve optimal scaling across larger supercomputing architectures, but this optimisation project represents a significant step by addressing critical bottlenecks and refining the parallelisation strategy. These improvements ensure that PHANTOM will be capable of tackling even more complex simulations in the future.

Project Image: PHANTOM logo.