Rapidly and Optimally Identifying Gravitational-Wave Optical Counterparts for GOTO

CI: – Dr. Kendall Ackley

We propose to replace the current cron-based job scheduler used for the GOTO image processing pipeline with the asynchronous Celery-style job queue and to optimize the interaction of our pipeline with our databases. We identify these two areas: the job queue and database operations, to be the major limitations for the GOTO pipeline. As we seek to improve and scale up the pipeline for an upcoming additional 4 to 28 telescopes, we are limited in our efficacy as the database I/O and job queue overheads dominate our overall processing time. The job scheduling will be updated to a Celery-style system. We will move to a SSD storage system for the database server and optimize the process of simultaneously storing and querying 10100 thousands of sources per image via simplifying the pipeline to database calls. This proposal will take an estimated 12 weeks: 6 weeks for the database optimizations; 4 weeks for job scheduler implementations; and 2 weeks of overhead; and will significantly increase the speed and efficiency of the scientific reporting with GOTO to the community. We expect our latencies to be reduced from 20-200 minutes per exposure to single minutes with these improvements alone.