The ASKAP data processing pipeline is developed in a modern Dev/Ops framework and is in general fit for the current purpose. But many of the tasks - such as regression testing and deployment are performed manually. Although these processes are aided by considerable scripting resources and have a moderately static configuration, it still takes too much time and manpower to test and release new versions of the processing package. It is often the case that new requested functionality is developed quickly, but released slowly. This has the effect of greatly increasing the time to delivery of new features. The development team reached out to ADACS to perform an audit of their current system of testing and releasing new features to identify ways in which the process can be better automated so that less human interaction is needed to approve the newly developed features. The ultimate goal is to be able to test and release new features at the same cadence as they are able to be developed by the software team.
During the 2025A semester, ADACS engaged with the ASKAP software developers to map out the path from new features being developed and unit tested, to having these features tested as part of the larger ASKAP workflow and passing the final science verification. As expected there were many different technologies used throughout the test and release processes, and no one person had custody of the entire system. After many discussions, much code review, and proof of concept workflows, ADACS was able to make many recommendations that should result in a more robust and automated test and release process, while requiring much less human interaction.
Check out some of our other projects.
ADACS improved the performance of the PHANTOM SPH code by improving its MPI communication patterns and optimising memory reallocation. Additionally, these upgrades prepare the code for future optimisations.
ADACS automated weather modeling with Nextflow for the DFN (Desert Fireball Network) team, improving efficiency and enabling parallel processing of meteorite fall data.
ADACS significantly enhanced the NaCo pipeline by parallelising key functions within the VIP library, reducing execution times by a factor of six and enabling more efficient processing of direct imaging data from the ESO archive and beyond.