Sophisticated Education Technology Centers (with Related Video)
Aug 1, 2011 12:00 PM, By Jim McEnteggart
With more education institutions looking to expand their research capabilities, high-performance computing facilities are becoming more prevalent.
Nicknamed “Jaguar,” the Cray XT5 System is capable of more than 2 thousand trillion calculations per second. Image courtesy of The National Center for Computational Studies, Oak Ridge National Laboratory
Academic and government research centers are, to date, the exclusive domains of high-performance computing (HPC) because of supercomputers’ ability to perform sophisticated mathematical modeling. Opportunities for research are expanding, as is evidenced by the advent of "P4" (predictive, preventive, personalized, participatory) medicine. P4 medicine will require HPC analysis of each individual’s genome, bringing about a sea change in medicine, including patient care and the application of medical technology. For example, the Ohio State University (OSU) Medical Center is engaged in researching and applying P4 medicine.
HPCs were introduced in 1976, and the first Cray-1 supercomputer was installed at Los Alamos National Laboratory. Designed by Seymour Cray, regarded as the "father of supercomputing," the Cray-1 clocked a speed of 160 megaFLOPS, or 160 million floating-point operations (FLOPS) per second. Last year, Cray Inc. installed the world’s fastest supercomputer at Oak Ridge National Laboratory (ORNL). Named Jaguar, the XT5 System has a clock speed of 1.8 PetaFLOPs, or 1,800,000,000,000,000 FLOPS per second. This surpassed the IBM "Roadrunner" system, installed in 2008, which was the first computer to pass the PetaFLOP barrier.
Academic institutions normally do not house the fastest HPCs; these generally are housed in government research facilities, such as ORNL. Nevertheless, according to a www.top500.org list of the 500 fastest HPCs in the world, 77 sites are at academic institutions. The University of Tennessee, No. 4, has a Cray XT5 system with an 831 TeraFLOP speed, which is housed in the same data center at ORNL that houses the Jaguar system. The University of Alaska, number 435, uses a Cray XT5 platform with a clock speed of 26.31 TeraFLOPs.
A Dramatic Difference
Requests for increased processing capability come from various academic programs, with requirements for the size or scale of the system. Focusing on this requirement is essential before design can begin. The planning team must consider the fact that an HPC facility will require significant increases in power and cooling capacity compared with a typical data center. These massively paralleled networks of specialized servers have a load density between 700 and 1,650 watts per square foot (W/sf), while most data centers have a load density in the range of 100 to 225 W/sf.
To deal effectively with these issues, a university’s HPC planning team should include representatives from the academic programs that will use the HPC facility, as well as representatives from facilities and IT. There also likely will be representatives from the university’s architectural planning committee, especially if the site is to be on a campus that requires a traditional architectural design. The university’s project manager should be a person with decisionmaking authority, knowledge of the budget, and familiarity with design and construction disciplines.
Academic institutions typically hire an architect to form a project A/E team. However, in technical projects of this nature, the architecture must respond to the technical requirements of the space. Moreover, in the case of an HPC facility, the electrical and mechanical trades represent roughly 75 to 80 percent of the construction cost. Therefore, it is advantageous to hire an engineer as the lead member of an E/A team with responsibility for subcontracting the architect. In particular, the E/A team should have experience designing HPC spaces, which narrows the field significantly. The E/A team will include civil and structural engineers, and IT design firms. If the site is to be LEED-certified, the team will need a LEED consultant, as well as a commissioning agent.
Technical Requirements Drive Design
The size and capacity of any HPC facility must respond to the technical requirements of the system above all else. These can be identified by working with the HPC system vendor. In particular, the power and cooling requirements are likely to exceed any other facility on campus; therefore, an institution must plan for a much larger infrastructure area than any other facility. In turn, power densities of this nature drive the infrastructure areas. It is not uncommon for infrastructure areas to be significantly larger than the HPC space. In fact, these can exceed the size of the HPC space by as much as four to six times based on the required capacity and redundancy of support systems.
For example, a hypothetical entry-level HPC is a 24.2 TeraFLOP (24.2 trillion FLOPS) system composed of two Cray XT6 cabinets, requiring roughly 70 square feet of raised floor. This system will require 90 kW of power, and will reject 307,000 BTUs/hr (26 tons) of heat to water. A robust system—a 700-TeraFLOP (700 trillion FLOPS) computer cluster using the same Cray XT6 platform—will fill two rows of 20 cabinets. At an average of 35 square feet per cabinet, this will require 1,400 square feet of raised floor space. This system will require 1,800 kW of power, and will reject 6.14 million BTUs/hr (512 tons) of heat to water.
The spaces quoted above are for the computational elements only; data storage devices also will be required that can meet the high-speed communications of the HPC. It is not unusual for the data storage element to match, or exceed, the size of the computational element. The good news: The power density for data storage is much lower than for the HPC.
An important issue is the expected availability of the system: will it be available strictly during business hours or 24x7x365 days a year? Will it be used strictly to support the institution’s own academic research, or will it lease time to local businesses, academic institutions or third-party research entities? Users’ requirements for availability will play a large role in decisions about support systems redundancy levels.
Acceptable Use Policy blog comments powered by Disqus
