Minimal re-computation for exploratory data analysis in astronomy
Accepted version
Peer-reviewed
Repository URI
Repository DOI
Change log
Authors
Abstract
We present a technique to automatically minimise the re-computation when a data processing program is iteratively changed, or added to, as is often the case in exploratory data analysis in radio astronomy. A typical example is flagging and calibration of demanding or unusual observations where visual inspection suggests improvement to the processing strategy. The technique is based on memoization and referentially transparent tasks. We describe a prototype implementation for the CASA data reduction package. This technique improves the efficiency of data analysis while reducing the possibility for user error and improving the reproducibility of the final result.
Description
Keywords
Methods, Data analysis, Functional languages
Journal Title
Astronomy and Computing
Conference Name
Journal ISSN
2213-1337
2213-1345
2213-1345
Volume Title
25
Publisher
Elsevier BV
Publisher DOI
Sponsorship
European Commission (283393)
European Commission Horizon 2020 (H2020) Research Infrastructures (RI) (653477)
European Commission Horizon 2020 (H2020) Research Infrastructures (RI) (653477)