TMCnet Feature Free eNews Subscription
November 13, 2013

Amazon Partner Builds 150,000 Core Supercomputer in the Sky

By Doug Barney, TMCnet Editor at Large

When it comes to high performance computing (HPC), the cloud is a double edge sword. On the one hand, end user performance is only as fast as the slowest part of the entire network, including the WAN. On the other hand, with IaaS and other techniques, one can build massive, supercomputer style systems in the cloud.



That is what HPC vendor Cycle Computing did when is used the Amazon cloud to host a cluster with 156,314 cores and peak processing of 1.21 petaflops.

Meanwhile the world’s fastest supercomputers have been clocked at over 20 petaflops, but these are large, dedicated on-premises clusters and machines.

Engineers and technical users have long been amongst the major consumers of CPU cycles. For many, going to the cloud means losing the responsiveness of their workstations. Applications like interactive 3-D modeling and simulation are not suited for the cloud, at least during the actual design work, because the cloud introduces latency.

But much of this work is designed, then sent out for processing such as rendering and simulation. And here is where the cloud shines, as Cycle Computing demonstrated. An end user can create a model, send it to the cloud for processing, and meanwhile keep working on another project. The workstation isn’t bogged down in the processing, if it was even capable of the processing job in the first place.

Cycle Computing built its cluster to suit this very need, in this case taking care of an application for USC chemistry professor Mark Thompson who was investigating solar energy. “For any possible material, just figuring out how to synthesize it, purify it, and then analyze it typically takes a year of grad student time and hundreds of thousands of dollars in equipment, chemicals, and labor for that one molecule," Cycle Computing CEO Jason Stowe wrote in a blog.

Simulation was the key technique used. Here the computer takes the model and simulate what the materials would be like if they were actually built. Simulation is insanely processor intensive, and has been the domain of high-end multicore, multiprocessor workstation, or for bigger jobs, HPC clusters.

HPC a Technical Computing and Engineering Savior

When workstations or dedicated separate workstations or servers configured to run compute-intensive jobs run out of steam, a shared computing environment such as a cluster should do the trick. And this can be installed on-premises, or as Cycle Computing did, build it in the cloud using Amazon.

With this utility-style computing resource, the demands of multiple users can all be addressed at the same time. Set up properly this resource appears as a large pool of computing power and capacity, even though it is in fact based on multiple bits of hardware. Here even an on-premises installation appears as a private cloud. It is just that the shop itself has to manage all this infrastructure, a point companies such as Cycle are sure to point out.

Scheduling Not a Problem

Whether you are building your cluster in-house or in the cloud like Cycle Computing, scheduling, also known as workload management is essential. Scheduling has to accommodate two opposite forces, the demand side for jobs to be processed, and the supply side which is the raw computing resource.

In Cycle’s case, its scheduler is called Jupiter.

With workload management software, policies are set that determine which demands will be met with what amount of supply, and what type of resources will be allocated, all of which can be controlled by a priority-based schedule. And related workloads can be coordinated for faster processing and keeping projects in sync.

Some users might think that a cluster with 80 percent utilization is fine. It isn’t. You are paying for hardware and computing cycles you aren’t using, and perhaps worse, not using potential computer power to do more simulations, or process more projects. By increasing that utilization to, say, 95 percent, it is as if you are getting extra high-end shareable computing resources essentially for free.

The good thing about cloud HPC, is it is in the interest of providers such as Cycle to fully utilize their infrastructure, and because HPC is their business, they can afford to implement scheduling tools. And a volume utility provider like Amazon can make it even more affordable. 


How Cycle Schedules




Edited by Ryan Sartor
» More TMCnet Feature Articles
Get stories like this delivered straight to your inbox. [Free eNews Subscription]
SHARE THIS ARTICLE

LATEST TMCNET ARTICLES

» More TMCnet Feature Articles