DEVILS and GAMA data Access via Data Central
Dr Luke Davies, a WAVES Project Scientist at the International Centre for Radio Astronomy Research (ICRAR), is currently a core member of the Galaxy and Mass Assembly Survey (GAMA – http://www.gama-survey.org) and principal investigator of the Deep Extragalactic VIsible Legacy Survey (DEVILS – devilsurvey.org).
Dr Davies’ current research is in galaxy evolution. He works with the large galaxy surveys GAMA and DEVILS, which study the properties of 100,000s to millions of galaxies. Using these surveys Dr Davies and his team try to understand how these galaxies have changed from those that were formed soon after the Big Bang, right up to the galaxies we see in the Universe at the present day. More specifically, he studies the rate at which stars form in galaxies and how this rate is affected by the galaxy’s local environment and its recent history.
The GAMA and DEVILS surveys consist of a global network of over 100 astronomers spanning 6 continents. All of these groups need easy and intuitive access to these datasets. The majority of the GAMA data is now public and hence can be used for many diverse science projects around the globe. Serving this data to the public in an easily usable manner is essential to the legacy of the project and the continued exploitation of this diverse dataset.
The Challenge
Dr Davies’ team requires constant access to large observational imaging and spectral datasets from the GAMA survey, which can not feasibly be stored locally (>3Tb). The group also need efficient access to a large number of tables of source properties – including on the fly table matching and sample selection. These images/tables contain the properties of 300,000 galaxies which make up the GAMA sample. In order to facilitate many of the science programs, Dr Davies and his team must be able to dial-up individual sources to explore the available data products and compare to other sources in the sample. They must also be able to identify specific galaxy populations based on a desired set of selection criteria and explore this sample dynamically. Finally, the team also need to be able to identify other (non-GAMA) products that exist for the target sources.
For DEVILS, in addition to all of the above, an efficient method for data transfer from the Anglo Australian Telescope (AAT) site in Siding Spring to Perth for data reduction through the observational pipeline is also needed. Within DEVILS Dr Davies and his group have developed a ‘nightly feedback’ observational mode, where all data observed in a night is processed, reduced, analysed and observing target catalogues are updated for the next night. Given this process, an extremely quick turnaround of data transfer and pipeline running has been required (~12h).
The Solution
For both surveys, a hosting solution for large datasets that can be easily accessed by both international team members and the worldwide general astronomy community is required. This also necessitates that there are distinct user logins which have access to different data sets and all data products need to be well documented, with the documentation being easily found and easily updatable.
Furthermore, a number of tools are required to facilitate the access and exploration of this data and to ensure the correct data products are used by the community. These tools consist of things such as positional on-sky matches to identify target sources in a particular location, functionality to extract target imaging data for sources across multiple observing bands, quick and easy catalogue accesses and catalogue query/matching services, a 1D spectrum viewer, etc. These sources then must easily be matched against other surveys to identify additional data products which may be used in the analysis. Furthermore, the DEVILS team also needs to be able to easily access the raw data files from the AAT at any time in order to test and run new and bespoke data reduction techniques.
The lack of direct external access to the AAT database due to firewall constraints complicates the process of running the DEVILS observational pipeline. As such, an easy way to transfer data from the telescope database to Perth on a nightly basis is rewuired. This must allow an observer – either at Siding Spring or at a remote location – to trigger a data sync to the Perth DEVILS database each night, passing through the AAT firewall, and then returning an observational target list to the telescope for the following night.
The Outcomes
The AAO Data Central archiving and services team provide solutions to all of the above issues:
- Data Central is currently hosting GAMA and DEVILS data in the form of both imaging and catalogue data.
- This data is easily an intuitively accessed by Data Central users (more below about services)
- Publicly available parts of these datasets are accessible to the wider community, with proprietary data only accessible to team members.
- Data Central provides a content management system to teams with hosted data to add documentation dynamically which can be viewed by all users. This allows teams to quickly add/amend documentation to facilitate science with their data products.
- The Data Central services provide the tools required to explore the GAMA and DEVILS data.
-
- The cone search service allows user to identify GAMA/DEVILS sources based on a positional match,
- The cutout service allows direct access to the imaging data and has multiple options to produce publication-ready images, and
- The query service allows easy access to the catalogue data, as well as the selection of sub-populations and catalogue matching.
- A spectrum viewer is not currently part of the Data Central suite of services, however, this is in development and will be available soon.
-
- The Data Central query and cone search service also allows matching the GAMA/DEVILS samples to other survey data products which are hosted by Data Central.
- The Data Central AAT archive query search provides easy access raw data files from observations in order to perform subsequent new data reductions in the future.
- The Data Central team have facilitated the access to DEVILS data from the AAT to Perth on a nightly basis. They developed as system where by the observer can upload data to the Data Central OwnCloud in a direct clone of the DEVILS database in Perth. This data is then automati cally synced to the Perth database nightly. This also allows a back-up clone of the DEVILS database to be hosted on the Data Central system. This process is essential to the DEVILS observations and reduction pipeline. This process is now also under development for the GALAH survey team.