Case Study of Handling Scientific Queries on Very Large Datasets: The SDSS Science Archive

Abstract.

The SDSS Science Archive (SX) was designed to enable scientific data mining and interactive data exploration on the terabyte scale. It consists of a distributed object-oriented database that is accessible via a client-server interface. The lightweight SX GUI client can be run on any platform. SDSS queries are formulated in SXQL, an SQL-like query language with some object-oriented and astronomy extensions. The SX server combines a fully multithreaded query engine with a distributed parallel architecture, splitting the data among multiple hosts and allowing for parallel, scalable I/O and parallel data analysis. Each query is parsed into a query execution tree which is executed in parallel. Data on remote partitions are accessed in parallel locally by remote slave servers. This distributed and multithreaded design allows query execution to be optimized and dynamically load-balanced for any type of multi-processor architecture, from SMP machines to Beowulf-type clusters.