NextFlow training 2022B

In the 2022B semester two groups requested training in the workflow orchestration language of NextFlow.

The first group, led by Sammy McSweeny at Curtin University, are from the Southern-sky MWA Rapid Two-meter (SMART) survey team. They have an existing pipeline that is implemented in NextFlow and require training in order to continue develop and maintain the pipeline as they continue to expand their survey, and incorporate new processing components. The SMART survey team use both OzStar and Pawsey to process their data.

The second group, led by Kieran Luken at Western Sydney University, currently have a range of software that they are deploying on a range of small clusters, and are interested in training in the use of HPC systems (see this workshop), as well as how to use NextFlow to create their workflows.

The NextFlow training was delivered on the 16th and 23rd of November face to face at Curtin Institute for Radio Astronomy (CIRA), and online for remote participants. The syllabus included:

• An introduction to workflows, pipelines, and automation
• How to running NextFlow workflows
• How to develop NextFlow workflows including an example of how to turn a flow diagram into a workflow.
• A lesson on containers (docker and singularity), what they are, how to use them, and how to build them.
• How to configure NextFlow to make use of containers, and interact with a queue scheduler such as SLURM
• An overview of NextFlow best practices

The content of the workshop can be found at the GitHub link below.