The gravitational wave research community has greatly benefited from the Parallel Bilby software, known for its impressive capabilities in gravitational wave (GW) inference. Recently, a team of researchers led by Rory Smith discovered that while Parallel Bilby effectively reduces wall time as expected, a significant portion of the total CPU time—over 50%—was spent executing code in kernel space rather than performing calculations.
To address this inefficiency, the researchers sought the assistance of ADACS developers. Their goal was to identify the root cause of this inefficiency and provide guidance on where to focus development and optimisation efforts.
After establishing a set of test cases with the science team, ADACS developers conducted a series of scaling measurements across various hardware configurations and CPU core counts. They then employed a suite of profiling tools to identify bottlenecks in the code.
Initially, it was thought that the cost of MPI communication was causing performance issues. However, profiling revealed that communication time was insignificant relative to computation time, suggesting that optimisations to the MPI communication pattern would not substantially improve overall performance.
Several areas were identified that could benefit from optimisation, including improving load balancing and reducing serial overheads in key functions. A significant finding was that the time spent executing code in kernel space was caused by idle worker tasks due to load imbalance. This issue could be resolved by adjusting the sampling parameters, without requiring major modifications to the code.
These findings were presented to the science team in a detailed optimisation report. The report outlined the profiling work performed and suggested the main areas for development focus to achieve maximum performance improvement.
The collaboration between ADACS developers and Dr Smith’s team has provided valuable insights into optimising Parallel Bilby. In addition to helping them focus their efforts effectively, this work laid the groundwork for a future ADACS optimisation project.
Project Image: Bilby logo
Check out some of our other projects.
ADACS significantly enhanced the NaCo pipeline by parallelising key functions within the VIP library, reducing execution times by a factor of six and enabling more efficient processing of direct imaging data from the ESO archive and beyond.
ADACS improved TESS-Atlas by optimising scalability, reducing website construction time, and automating workflows with a custom Job Controller for handling new exoplanet candidates efficiently.
The HORMONE simulation code makes use of a novel self-gravity solver. ADACS implemented MPI parallelisation into the code, enabling it to scale across multiple compute nodes, facilitating larger and more detailed simulations.