Oak Ridge National Laboratory (ORNL) and the National Oceanic and Atmospheric Administration (NOAA) have completed upgrades to the latter’s climate research supercomputer, Gaea, allowing the agency to advance its climate modeling and improve our understanding of climate variability and change.

Gaea is now capable of more than 1,100 trillion calculations each second, or 1.1 petaflops. The system’s 40 cabinets contain 7,520 16-core AMD Interlagos processors, giving it a total of 120,320 processing cores. It also contains 240 terabytes of memory and manufacturer Cray’s resilient Gemini interconnect, which cuts downtime to a minimum.

Gaea is housed at ORNL’s National Center for Computational Sciences (NCCS). It is the largest of three NOAA research and development systems.

“This is a significant improvement in capacity,” noted David Michaud, NOAA’s deputy director for high-performance computing and communications. “With the increased compute capacity, we’re able to run more complex models at much higher resolution. This increased level of detail and fidelity in the models will provide state and local decision makers more actionable information.”

NOAA uses Gaea to study the planet’s notoriously complex climate from a variety of angles. Among other things, Gaea is powering research into the relationship between climate change and extreme weather such as hurricanes. It is enabling scientists to get a better understanding of the relationship between the atmosphere’s chemical makeup and climate. And it is helping unlock the climate role played by the oceans that cover nearly three-quarters of the globe.

“There are three strategies that we can explore when we expand our computing capability,” noted Brian Gross, deputy director of NOAA’s Geophysical Fluid Dynamics Laboratory.

“First is resolution; we’re able to run models with higher resolution. Another is complexity; we’re able to add processes such as atmospheric chemistry, aerosol effects, and ocean biogeochemistry to our coupled climate models. The last is increasing ensemble members, or parallel model simulations, to help us quantify the uncertainty in our models.”

Data from Gaea is used primarily to develop peer-reviewed journal articles, Gross said, but it also produces model output for the latest Climate Model Intercomparison Project, orchestrated by the Program for Climate Model Diagnosis and Intercomparison, located at Lawrence Livermore National Laboratory. This simulation data will be used in producing the Fifth Assessment Report on climate change of the Intergovernmental Panel on Climate Change.

The latest upgrade is the second for Gaea since it was first installed at ORNL in 2010. That first system, capable of up to 260 trillion calculations a second, or 260 teraflops, contained 30,912 processor cores in 14 cabinets. It ranked 32 on the November 2010 TOP500 List of the world’s most powerful supercomputers.

Twenty-six Cray XE6 cabinets were added in January 2012, with 78,336 processing cores, 157 terabytes of memory, and a peak performance of 721 teraflops. With the latest project, the original Gaea installation has been upgraded so that the entire system carries the same processor specifications.

Gaea grew out of a 2009 interagency agreement between NOAA and the Department of Energy Office of Science, which is the primary sponsor of scientific activities at ORNL. A subsequent work-for-others agreement directly with ORNL allowed NOAA to take advantage of ORNL’s expertise in acquiring, deploying, and operating world-class supercomputers. It also provided additional opportunities for collaborative scientific work on climate change and variability.

The two partners are pleased with the collaboration. NCCS Director of Operations James Rogers noted that the current Gaea system is 20 times more powerful than NOAA’s most powerful resource before Gaea.

“This work-for-others agreement has been successful in creating an external resource that the NOAA community can use,” he said. “This is a very important model where multiple agencies can work together toward a common goal.”

Gross agreed.

“ORNL has a longstanding history of operating high-performance computers,” he said. “They’ve owned leadership-class systems for many years. Now we’ve been able to leverage that expertise in NOAA’s first petascale system. This relationship is growing to include scientific collaborations on key questions regarding climate change, such as how changing land use impacts the carbon cycle and its effects on terrestrial ecosystems.”