Barbara A. Mikulski Archive for Space Telescopes

Karen Levay,, with contributions from the MAST team

The Barbara A. Mikulski Archive for Space Telescopes (MAST) is NASA’s data repository for astronomy missions in the ultraviolet–optical–near-infrared wavelength range, including both active and legacy missions. MAST supports the astronomical community by facilitating access to its collections, offering expert user support, and providing software for calibration and analysis.

As of April 1, 2012, the volume of MAST’s data holding is over 185 TB. Over the past two years, the archive ingests on average 1.6 TB of data and distributes over 7.5 TB of data each month.


Infrastructure upgrade for the Hubble archive

The server infrastructure for archiving and data processing recently underwent a long-overdue upgrade.  The upgrade improved the speed of the average Hubble retrieval by a factor of 11 (from about 90 minutes to less than 8).  The new configuration provides us with a stable environment with room for future growth in storage and processing capacity.

Since 2003, the servers for the Hubble Data Archive and Distribution System (HST-DADS), and Operational Pipeline Unified System (OPUS) had been running on a SunFire cluster with Solaris, and utilized Sybase database servers. That configuration had also been used for Kepler data. In the upgrade, we deployed new Dell PowerEdge servers that run the Red Hat Enterprise Linux operating system and use MicroSoft SQL database servers.

The initial migration of the DADS software to the new configuration was completed in summer 2010, for the latest installation of DADS for the Webb science integration and test data system.  The transfer of HST-DADS and OPUS to the new configuration was complicated. The migration of 20+ years of Hubble data required months of testing by the instrument teams to verify that the new software would produce identical scientific results to the old configuration.

Another large effort involved the verification and validation of the database servers. This involved verifying data content and query performance between the two systems.  The testing ensured that the data coming from the new systems provided results with the same data quality, and confirmed that the performance from the new systems was significantly improved.

In a final testing phase, we operated the new and old systems in parallel for approximately six months, verifying their identical functionality and minimizing the impact to our customers. In September 2011, the old Hubble system was shut off and decommissioned; the Kepler system came online shortly thereafter, with all work completed in October 2011.