Astronomy of the Cosmos – the last frontier, and the data that supports our incessant desire to obtain knowledge about the universe is causing a surge in data storage, replication, and the need for more. How will this community of scientists and engineers deal with what to keep, what not to keep, and how to keep it when it comes to the vast array of data required our use?

An Astronomical Surge of Data
Degradation in performance should not be the expected course when saving such data. One cannot simply expect that by adding infrastructure as usage increases (common in commercial business enterprises) the problem will be solved. Because astronomy archives generally operate on limited budgets that are fixed for several years, any changes in computer architecture would have to be foreseen and budgeted years early. More so, who do you plan for new discoveries? The current data-access and computing model used in astronomy will be insufficient for future use at the rate it is being collected. Currently, data is downloaded from archives to a local machine to be analyzed. This is done on a very large scale in order to be made accessible to end users. Data discovery, access, and processing are likely to be distributed across several archives (an archive of data warehousing). Given that the maximum that science return will yield from this “federation of data” i.e. multiple archives, a broad wavelength range will be required. Astronomy data is collected and archived in Petabytes (PB). A Petabyte is 1024 Terabytes, or in more simple terms, a million billion bytes. Over time, astronomers have collected data on the stars and their alignments, planets (that have come and gone), and terrestrial beginnings. The common thread in this data is the fact that historical data must be maintained and is the most important factor in the archival capabilities for astronomy as a whole. They are often looking for the next big thing, but must be able to compare it to “what is” or “what was” in order to do that. Subtle changes in astronomy occur sometimes over centennials, not decades. For example, Haley’s Comet is a periodic comet in that every 75 years it appears viewable to us on Earth, but without a history of the comet ever having been seen, we wouldn’t not have been able to track it, thereby knowing it was recurring.
With astronomy, the more years of comparable data you have, the more accurate your predictions become, no matter what the focus of the prediction. This is the way Astronomers have done business. But with the amount of data being captured and kept and analyzed, the more powerful processors we need to fulfill the desired effect. Let’s talk Memory and Storage
Imagine the impact of archiving and processing a PB of data. The astronomical community has been doing just that. A summit yielded a report that recognized important infrastructure failures and their inability to effectively manage the amount and level of data being collected and archived. In 2010, the 2010 Decadal Survey of Astronomy and Astrophysics, commissioned by the National Academy of Sciences, stated a list of priorities for the astronomy community to look at within the next 10 years. This was referred to as the Flash Memory Summit.
It was here that it was noted that the amount of data was rapidly outgrowing the infrastructure currently in place. From storage to processors… we can’t seem to keep up. Hard disk drives are no match for PB of data. Archiving them in data warehouses and the cloud may be possibility, but this amount of data would surely break the bank of a cloud storage or SAAS (storage as a service) System. Gary Gentry, Seagate’s Senior Vice President stated "You can take that as a crisis; we see it as an opportunity”. Gentry was referring to the fact that data storage requirements in 2020 would total around six (6) zettabytes. Just another data note – a zettabyte is a thousand Exabyte, which is a thousand petabytes, which is a thousand terabytes, get it? Think about it in terms of movies, medical artifacts, big-business and corporate data, automotive, banking, and security related information. Not to mention intelligence data and military files. Now can you being to fathom how much data is out there? Now add the astronomical data on to that and you’re beginning to understand the issue of what we need to store and access.

Future Data Storage and Processing
National cyber infrastructure programs must be partnered with businesses that use and develop system resources in order to come up with a working solution. One industry can’t do it alone. There’s just too much to consider to take a chance of missing something important. An investigation of emergent technologies and solutions must be in the forefront. As usual, money is always front and center of problem vice the solution.
Data solutions cost money. Although we get more for the money when it comes to storage today, the cost for research and implementation of new technologies is astronomical to say the least (no pun intended). We are continually limited by our pocketbooks budget, especially with a solution that is just being vetted and has not yet proven its worth. The question is how to you put a price on storage of such important and necessary information? How can you budget for a new discovery?
Two newer technologies being considered are the Graphical Processing Units (GPU) and cloud computing. A GPU (Graphics Processing Unit) is a specialized circuit designed to accelerate the image output in a frame buffer intended for output to a display. GPUs have the ability to manipulating computer graphics in such an efficient manner that they can be more effective than a standard Central Processing Unit (CPU) for calculations and algorithms when processing large blocks of data. GPUs follow where CPUs left off. As more graphic intensive applications are developed, regular CPUs are being phased out in order to allow the heavy lifting of mathematical algorithms to the GPU.
Brian Schmidt, an astronomer at the Research School of Astronomy and Astrophysics at the Australian National University did a lot of research on “big data” in the industry. He noted that new telescopes like, Skymapper, are “creating massive amounts of data, a terabyte of data each night” and that “interdisciplinary groups of researchers to work together to meet these challenges”. He reiterated the fact that “Having digital access to research and reference material through services like the Astrophysics Data System, there are substantial information challenges we are facing that I think libraries, archives and museums could help with.” (2013)

Discovering Data Innovations
Discovering the direction on how to manage the anticipated growth vice not overloaded the system is one of the many technological challenges we face. The VAO (Virtual Astronomical Observatory is in the process of developing an indexing scheme that supports fast, scalable access to massive databases of astronomical data sets. These schema, referred to as R-trees index multidimensional information across archived data warehouses, thereby speeding up the access time. Custom solutions such as these may be more expensive to implement, but are definitely required.

Commercial entities taking a genuine interest in finding and funding such solutions also help to keep the cost down. Taking advantage of commercial cloud solutions are best for memory-intensive applications, of which astronomy definitely is. The ability to process large quantities of image data still needs to be married up with the cost to transfer such data to make it available when needed or for ad hoc. Currently, renting mass storage space on the Amazon cloud is more expensive than purchasing it outright.
The design of cloud computing and SAAS offers a cheaper and much greener alternative to earlier designs. Using the GPU vice CPU as a centrally based GPU cluster is also a step in the right direction. Who knows what data lies beyond the southern sky (or what I can compare it to when looking at images of the southern sky from centuries past)? The truth is, because we know not what we may discover, we need to be prepared to hold on to it for future generations, as those before us did. This need to acquire new technology will continue to bring forth new frontiers, indeed.

Reference
G. BRUCE BERRIMAN, S. L. (2011). DATABASES. Retrieved 2 20, 2014, from HOW WILL ASTRONOMY ARCHIVES SURVIVE THE DATA TSUNAMI?: https://blackboard.strayer.edu/bbcswebdav/pid-10823189-dt-content-rid-58879684_4/institution/CIS/512/1128/Week6-1128/Week6-CaseStudy2-Tsunami.pdf
Hemsoth, N. (2011). Canada explores new frontiers in astroinformatics. Retrieved 2 20, 2014, from In HPC in the Cloud: http://www.hpcinthecloud.com/hpccloud/2011-01-17/canada_explores_new_frontiers_in_astroinformatics.html
Nereus overview. (n.d.). http://www-nereus.physics.ox.ac.uk/about_overview.html.
Rik Myslewski. (2013 , August 14). Seagate: Storage industry ill-prepared for onrushing big data tsunami. Retrieved 2 20, 2014, from The Register: http://www.theregister.co.uk/2013/08/14/storage_industry_unprepared_for_coming_data_needs/
Astronomical Data and Astronomical Digital Stewardship: An interview with Brian Schmidt, published November 14, 2013, Accessed 20 Feb 2014, Author: Trevor Owens, http://blogs.loc.gov/digitalpreservation/2013/11/astronomical-data-and-astronomical-digital- stewardship-an-interview-with-Brian-Schmidt/