Optimized Data Loading for a Multi-Terabyte Sky Survey Repository (original) (raw)
Related papers
Monetdb/sql meets skyserver: the challenges of a scientific database
Scientific and Statistical …, 2007
This paper presents our experiences in porting the Sloan Digital Sky Survey (SDSS)/SkyServer to the state-of-the-art open source database system MonetDB/SQL. SDSS acts as a well-documented benchmark for scientific database management. We have achieved a fully functional prototype for the personal SkyServer, to be downloaded from our site. The lessons learned are 1) the column store approach of MonetDB demonstrates a great potential in the world of scientific databases. However, the application also ...
AstroPortal: a science gateway for large-scale astronomy data analysis
TeraGrid Conference, 2006
The creation of large digital sky surveys presents the astronomy community with tremendous scientific opportunities. However, these astronomy datasets are generally terabytes in size and contain hundreds of millions of objects separated into millions of files—factors that make many analyses impractical to perform on small computers. To address this problem, we have developed a Web Services-based system, AstroPortal, that uses grid computing to federate large computing and storage resources for dynamic analysis of ...
2006
We present a summary of the major contributions to the Special Session on Data Management held at the IAU General Assembly in Prague in 2006. While recent years have seen enormous improvements in access to astronomical data, and the Virtual Observatory aims to provide astronomers with seamless access to on-line resources, more attention needs to be paid to ensuring the quality and completeness of those resources. For example, data produced by telescopes are not always made available to the astronomical community, and new instruments are sometimes designed and built with insufficient planning for data management, while older but valuable legacy data often remain undigitised. Data and results published in journals do not always appear in the data centres, and astronomers in developing countries sometimes have inadequate access to on-line resources. To address these issues, an 'Astronomers Data Manifesto' has been formulated with the aim of initiating a discussion that will lead to the development of a 'code of best practice' in astronomical data management.