Oak Ridge National Laboratory (ORNL) and the National Oceanic and Atmospheric Administration (NOAA) have completed upgrades to the latter’s climate research supercomputer, Gaea, allowing the agency to advance its climate modeling and improve our understanding of climate variability and change.

Gaea is now capable of more than 1,100 trillion calculations each second, or 1.1 petaflops. The system’s 40 cabinets contain 7,520 16-core AMD Interlagos processors, giving it a total of 120,320 processing cores. It also contains 240 terabytes of memory and manufacturer Cray’s resilient Gemini interconnect, which cuts downtime to a minimum.

Gaea is housed at ORNL’s National Center for Computational Sciences (NCCS). It is the largest of three NOAA research and development systems.

“This is a significant improvement in capacity,” noted David Michaud, NOAA’s deputy director for high-performance computing and communications. “With the increased compute capacity, we’re able to run more complex models at much higher resolution. This increased level of detail and fidelity in the models will provide state and local decision makers more actionable information.”

NOAA uses Gaea to study the planet’s notoriously complex climate from a variety of angles. Among other things, Gaea is powering research into the relationship between climate change and extreme weather such as hurricanes. It is enabling scientists to get a better understanding of the relationship between the atmosphere’s chemical makeup and climate. And it is helping unlock the climate role played by the oceans that cover nearly three-quarters of the globe.

“There are three strategies that we can explore when we expand our computing capability,” noted Brian Gross, deputy director of NOAA’s Geophysical Fluid Dynamics Laboratory.

“First is resolution; we’re able to run models with higher resolution. Another is complexity; we’re able to add processes such as atmospheric chemistry, aerosol effects, and ocean biogeochemistry to our coupled climate models. The last is increasing ensemble members, or parallel model simulations, to help us quantify the uncertainty in our models.”

Data from Gaea is used primarily to develop peer-reviewed journal articles, Gross said, but it also produces model output for the latest Climate Model Intercomparison Project, orchestrated by the Program for Climate Model Diagnosis and Intercomparison, located at Lawrence Livermore National Laboratory. This simulation data will be used in producing the Fifth Assessment Report on climate change of the Intergovernmental Panel on Climate Change.

The latest upgrade is the second for Gaea since it was first installed at ORNL in 2010. That first system, capable of up to 260 trillion calculations a second, or 260 teraflops, contained 30,912 processor cores in 14 cabinets. It ranked 32 on the November 2010 TOP500 List of the world’s most powerful supercomputers.

Twenty-six Cray XE6 cabinets were added in January 2012, with 78,336 processing cores, 157 terabytes of memory, and a peak performance of 721 teraflops. With the latest project, the original Gaea installation has been upgraded so that the entire system carries the same processor specifications.

Gaea grew out of a 2009 interagency agreement between NOAA and the Department of Energy Office of Science, which is the primary sponsor of scientific activities at ORNL. A subsequent work-for-others agreement directly with ORNL allowed NOAA to take advantage of ORNL’s expertise in acquiring, deploying, and operating world-class supercomputers. It also provided additional opportunities for collaborative scientific work on climate change and variability.

The two partners are pleased with the collaboration. NCCS Director of Operations James Rogers noted that the current Gaea system is 20 times more powerful than NOAA’s most powerful resource before Gaea.

“This work-for-others agreement has been successful in creating an external resource that the NOAA community can use,” he said. “This is a very important model where multiple agencies can work together toward a common goal.”

Gross agreed.

“ORNL has a longstanding history of operating high-performance computers,” he said. “They’ve owned leadership-class systems for many years. Now we’ve been able to leverage that expertise in NOAA’s first petascale system. This relationship is growing to include scientific collaborations on key questions regarding climate change, such as how changing land use impacts the carbon cycle and its effects on terrestrial ecosystems.”

Simulation of global hurricane climatology and response to global warming in a new global High Resolution Atmospheric Model (HiRAM) at 50-kilometer resolution. Images courtesy: Ming Zhao, Isaac Held, Shian-Jian Lin, Gabe Vecchi, NOAA/Geophysical Fluid Dynamics Laboratory

In 1969 scientists from the Geophysical Fluid Dynamics Laboratory (GFDL) in Princeton, New Jersey, published results from the world’s first climate model. Though the model gave scientists their first look at how the ocean and atmosphere interact to influence climate, it covered only one-sixth of the surface of the Earth and did not take into account any human-made, or anthropogenic, causes of climate change.

Forty years later, using an ancestor of this pioneering model, a team led by GFDL climate scientist Venkatramani Balaji is using the Cray XT5 known as Jaguar—housed in the OLCF—to simulate and assess both natural and anthropogenic causes of climate change at exceptional resolutions.

“Resolving our models to 50 kilometers, you can see states and counties. That’s what people are really asking for [from climate science],” said Balaji, head of the Modeling Systems Group at GFDL, a branch of NOAA devoted to developing and using mathematical models and computer simulations to improve understanding of the Earth’s atmosphere, ocean, and climate. The group’s models are higher in resolution than required by the Intergovernmental Panel on Climate Change Fourth Assessment Report (IPCC AR4).

Climate models from which data was gleaned for the IPCC AR4, released in 2007, were in the 100-kilometer range for both ocean and atmosphere. Accurate regional climate forecasting is in-
valuable to areas such as the American Southwest, which experiences persistent droughts that are
rapidly depleting water reservoirs. If climate models can prove that anthropogenic causes are mostly
to blame, stakeholders can amend policies and practices to make the most of scarce resources.

Causes of climate change

Natural variations in global climate arise from phenomena including changes in solar activity, periodic alterations in the Earth’s orbit, and sulfate aerosols from volcanic eruptions. However, the AR4 claims, “It is extremely unlikely (less than 5 percent) that the global pattern of warming during the past half century can be explained without external forcing, and very unlikely (less than 10 percent) that it is due to known natural external causes alone.”

Humanity’s effect on climate is broad. Of most concern to climate scientists and environmentalists are increasing carbon dioxide levels in the Earth’s atmosphere. According to the IPCC AR4, two-thirds of the anthropogenic carbon dioxide emissions come from fossil fuel combustion, while the remaining third comes from land-use changes such as irrigation, deforestation, and ranching. Carbon dioxide is naturally present in the atmosphere along with water vapor, nitrous oxide, methane, ozone, and other trace gases collectively referred to as greenhouse gases. As their name suggests, these gases warm the Earth by trapping energy from sunlight just as a greenhouse does. Ever since the Industrial Revolution, greenhouse-gas emissions have increased, warming the Earth and affecting regional weather patterns.

The terms “climate” and “weather” describe the atmosphere through different timeframes. Climate refers to average, or statistical, behavior of the global atmosphere over a long period of time—say, year to year—whereas weather is a measure of day-to-day local atmospheric conditions. Through cutting-edge models, Balaji’s project, dubbed CHiMES (for Coupled High-Resolution Modeling of the Earth System), aims to explore the relationship between climate and weather.

CHiMES is a collaborative effort between DOE and NOAA. If climate can be predicted at all, the project endeavors to forecast change over time periods of interest to resource managers (e.g., decades) and to understand the responses of phenomena like tropical storms to a warming climate.

Anatomy of a climate model

Climate models are composed of mathematical sets of instructions called differential equations. These equations assess interactions among various components that influence the Earth’s climate—atmosphere, oceans, land, and ice. In short, climate models calculate the solar energy our planet absorbs and the radiation it emits, some of which escapes into space while some remains trapped by Earth’s atmosphere. Differential equations that govern the system are solved in accordance with different boundary conditions, such as specifying sea surface temperature distributions, carbon dioxide concentrations, and atmospheric aerosol distributions.

Balaji’s model—GFDL Flexible Modeling System—is actually two models built upon a flexible framework that allows different components of the climate system to be modeled by multiple scientists and code developers and assembled in a variety of ways. The atmospheric model, Finite Volume Cubed-Sphere, runs at a resolution of 25 kilometers. The oceanic model MOM4 (short for Modular Ocean Model) is what climate scientists call a quarter-degree model, which translates into resolution of roughly 25 kilometers as well. The resolution of the models can be increased
in specific areas, and Balaji and colleagues increased the granularity to 10-kilometer resolution in areas of particular interest.

The equations of any climate model are calculated at individual points on a grid covering the Earth. Traditionally, this grid has been dictated by lines of longitude and latitude in which each cell of the grid is defined as the intersection of a latitude line and a longitude line. Though easy to manage, a latitude/longitude-based grid creates problems for climate scientists. The distance between meridians (lines of longitude) decreases as they draw closer to the poles, requiring the dynamical algorithms to use exceedingly short time steps to keep the solutions stable in the polar regions. Although this stabilizes the calculation in the polar regions, it constrains the time step throughout the global domain, decreasing the overall efficiency of the calculation.

Balaji’s team conquers this issue in its coupled model by using a cubed-sphere grid that projects a three-dimensional cubed grid onto the Earth, removing the clustered points around the poles and providing quasi-uniform resolution at each grid cell. Using a tripolar grid for the ocean model overcomes this “pole problem” by placing three “poles” over landmasses on the grid: one at the South Pole (as there is no ocean there), one over the Asian continent, and one over North America.

The atmospheric and oceanic models employ separate grids and solve different algorithms. The majority of the time, the models run independently but concurrently. A third grid is used to exchange data between the two models every 2 hours.

“Many other climate models only exchange data once a day or not at all, so anything that happens faster than a 24-hour timeframe will be missed,” said Balaji. “Peaks in wind can change the ocean’s circulation [in less than 24 hours], and we are able to resolve that.”

Coupling climate models is computationally expensive, but the Jaguar XT5—the fastest supercomputer in the world at 2.33 quadrillion calculations per second and featuring nearly a quarter of a million processors—gave Balaji’s team the power and speed it needed to frequently link its climate models.

Future forecasts

The most exciting results from the CHiMES project come from the team’s study of tropical storm response to global climate change. In 2009 the team used 20 million processor hours on Jaguar to run approximately 500 years’ worth of coupled-model simulations through an INCITE program allocation. Scaling its high-resolution models from 60,000 to 100,000 cores, Balaji’s team was able to realistically duplicate year-to-year behavior of hurricanes, accurately simulating their seasonal peak in September.

The team is also resolving an issue that has plagued coupled climate models since their inception. The issue pertains to an area around the equator where winds originating in the northern and southern hemispheres meet, affecting the wet and dry seasons of many equatorial nations. The region is called the Intertropical Convergence Zone (ITCZ), and it appears in coupled climate models as two peaks in rainfall. In reality the ITCZ has only one peak in rainfall. As the team continues to increase the resolution of its models, it is slowly coming closer to eliminating the second “peak.”

The team also hopes to use the ocean as a way to forecast climate on the scale of decades. The ocean influences the climate on a longer scale than do Earth’s ice, land, and atmosphere. If models can capture the long-term behavior of the oceans, they can also capture the short-term behavior of Earth’s other climate-influencing components.

While Balaji is unsure of whether his models are capable of forecasting climate on the decadal scale, the team has been awarded another 20 million hours on Jaguar through INCITE for 2010. Balaji’s team plans to complete runs to determine if decadal predictability is possible, but it also hopes to bring even higher-resolution models online—ones capable of resolving fine-scale weather such as cloud convection and shifting winds.

Ultimately Balaji and his colleagues hope to demonstrate that global climate change is not a remote concept, but a tangible problem with significant consequences that the world’s experts must solve together.

“Local changes will be far more intense; you might get warming here and cooling there,” he explained. “The term global warming seems like a temperature problem, but precipitation patterns change—some areas will get wetter, some dryer. We have to make [climate change] more relevant to people by bringing it down to the regional scale. In order to help people adapt, we want to tell them what to be prepared for.”

—by Caitlin Rockett

The Gaea supercomputer runs simulations at the National Climate-Computing Research Center, which supports NOAA research of the Earth system. Image courtesy Jim Rogers.

A supercomputer installed to crunch numbers for the National Atmospheric and Oceanic Administration (NOAA) and its research partners has begun climate simulations at Oak Ridge National Laboratory (ORNL). A mural of snow-capped mountains and hovering clouds adorns the front of the high-performance computing (HPC) system and reflects its mission—assessing climate variability and change in the Earth system. The supercomputer is the premier resource for scientists collaborating at NOAA’s new National Climate-Computing Research Center (NCRC).

“The name of the machine is Gaea, or Mother Earth, from Greek mythology,” says Jim Rogers, director of operations at ORNL’s National Center for Computational Sciences, which houses the new Cray XT6 machine. Rogers directs NOAA’s Climate Modeling and Research System (CMRS) project at ORNL to support NCRC activities. The CMRS is owed by DOE and operated by ORNL’s managing contractor, UT-Battelle, on behalf of the NOAA customer.

Gaea will occupy the same half-acre computer room as the world’s fastest supercomputer, Jaguar, a Cray XT5 system run by ORNL and funded by the Department of Energy’s (DOE’s) Office of Science, and Kraken, the fastest academic supercomputer, a Cray XT5 system run by the University of Tennessee and ORNL and funded by the National Science Foundation.

Cray will deliver the HPC system through a series of upgrades that will culminate in a petascale system by the end of 2011.

In June 2010, installation concluded for a 260-teraflop (trillion calculations per second) Cray XT6 system with 2,576 AMD “Magny-Cours” 12-core, 2.1 GHz processors. After passing a series of acceptance tests, Gaea was released to early users. In September, nearly a dozen users began ramping up their data production.

In June 2011, a 720-teraflop Cray XE6 system will be added to Gaea. It will employ the next-generation AMD Interlagos 16-core processor. After the installation of that second system, the original 260-teraflop system will be upgraded with the same AMD Interlagos processor to achieve 386 teraflops.

The aggregate Gaea system will have a total memory size of 248 terabytes and a peak calculating capability of 1.1 petaflops (quadrillion floating point operations per second), bringing the number of petascale systems at ORNL, the world’s most powerful computing complex, to three.

The next-generation HPC system is liquid-cooled using Cray’s ECOphlex technology, which employs a refrigerant to remove most of the 2.2 MW heat load. The technology is significantly more energy-efficient than the air-cooling systems typically found in other leading-edge HPC systems.

The CMRS includes two separate file systems, both founded on the Lustre parallel file system, to handle data sets that will be among the world’s largest. A high-capacity file system based on DataDirect Networks SFA10000 can stage up to 3.6 petabytes of information. Meanwhile, a high-speed file system with more than a petabyte of storage provides fast scratch space.

NOAA research partners access data remotely through speedy interconnections. Two 10-gigabit (billion bit) lambdas, or optical waves, pass data to NOAA’s national research network through peering points at Atlanta and Chicago.

—by Dawn Levy

The U.S. Department of Energy (DOE) has awarded Cray Inc. a $47 million subcontract to provide high-performance computing (HPC) and scientific services for climate modeling collaborations between Oak Ridge National Laboratory (ORNL) and the National Oceanic and Atmospheric Administration (NOAA). At ORNL, Cray will deliver the Climate Modeling and Research System (CMRS), which includes a supercomputer named “Gaea” for simulating climate. The resources support activities at the newly established National Climate-Computing Research Center (NCRC).

“NOAA is recognized as a world leader in understanding and predicting the Earth’s environment through advanced modeling capabilities, climate research, and real-time weather products,” said Joe Klimavicz, CIO and director of HPC and communications at the Department of Commerce/NOAA. “High-performance computing is absolutely necessary for weather forecasts and climate predictions, and we are excited to have the computational resources of a next-generation Cray supercomputer to further our research.”

The research collaboration is the result of a “work for others” agreement between DOE and the Department of Commerce, which oversees NOAA. The contract with Cray provides hardware, software, third-party applications and system maintenance, support, and operation through 2014, with two one-year options to extend the contract through 2016.

Gaea is a next-generation Cray XT6 system. Occupying about 1,600 square feet, it has only one-fifth the physical footprint of Jaguar, the world’s most powerful supercomputer, which resides in the same room.
Funded through the American Recovery and Reinvestment Act (ARRA), the CMRS will provide a dedicated HPC resource for NOAA and its research partners. The work for others agreement includes additional ARRA funding of approximately $26 million for HPC operations staff, management, electric power, space and cooling.

“Making these computational resources available to the NOAA climate modeling community will transform the way the community will be able to approach their scientific mission due to the sheer magnitude of the increase in capability,” says James Hack, who heads climate research at ORNL and directs the National Center for Computational Sciences, which houses CMRS. The initial capability, which was delivered in June 2010, is a Cray XT6 system that is five times more powerful than NOAA’s current vanguard computational resource.

In 2011, addition of a Cray XE6 system will quadruple the computing capacity. Additional upgrades are planned for 2012. The final aggregate system will have a total memory size of 248 terabytes and a peak calculating capability of 1.1 petaflops (quadrillion floating point operations per second). The climate community will make use of this computational power to acquire knowledge about the Earth system that can inform policymakers and planners.

“The CMRS facility allows the climate community to study systems of greater complexity and in higher resolution, and in the process hopefully improve the fidelity of modeling tools,” Hack says.
—by Dawn Levy