THERE are few events in the universe more complex than the violent deaths of massive stars. Their explosions are so huge that they can be seen thousands of light years away, and they leave behind mind-bending remnants like black holes and neutron stars.
Making detailed simulations of these colossal supernovae is currently beyond the reach of even the world’s most powerful computers, yet Bronson Messer is determined to try. “It’s the biggest bucket of physics you can imagine,” says Messer, an astrophysicist at Oak Ridge National Laboratory (ORNL) in Tennessee.
Advertisement
Messer is building simulations for Aurora, due to come online in the early 2020s, which could be the first machine capable of one exaflop – a billion billion “floating point operations” per second – making it millions of times faster than your trusty laptop. Exascale computers will provide unprecedented power, perhaps enough to unlock fundamental goals in a broad range of scientific disciplines. Predicting the weather, simulating entire brains, recreating the cosmos and tailoring drugs to individuals could all become possible. That’s why the US, China and Japan are racing to build one first.
This is a multibillion-dollar development effort, fuelled by the need to power such a beast of a machine without bringing the grid to its knees, and to create a communication network that can coordinate all of its many parts. But lurking in the background is a provocative question with no clear answer: once we build such a machine, how useful will it be?
Need for speed
The accepted way of ranking the speed of a computer is by how many calculations it can perform per second. These ordinarily involve representing numbers in a format known as floating-point, which allows you to efficiently encode numbers both large and small. The calculator in your desk drawer probably boasts a handful of floating operations per second (FLOPS), the cutting-edge phone in your pocket several billion and the world’s fastest machines – collectively known as supercomputers – millions of times more.
Although the first machine to tick over the exaflop mark should arrive some time in the next three years, it isn’t going to solve all the world’s problems overnight. “There’s nothing magical about certain numbers,” says Simon McIntosh-Smith, a high-performance computing researcher at the University of Bristol, UK. Reaching that milestone, however, will push current technology to the limit, and going much beyond may require rethinking the paradigms that got us this far.
For more than half a century, the key to increasing computing power has been to squeeze ever more transistors onto a chip. Since the 1960s the density has doubled roughly every 18 months, producing an exponential increase in performance known as Moore’s law after chip company Intel’s co-founder Gordon Moore, who was the first person to notice it. In 1974, computer scientist Robert Dennard made another important prediction, noting that as chips shrank, their power consumption stayed in proportion with their area, even as transistor density increased. That meant the power consumed by a set number of transistors halved at every cycle of Moore’s law. Engineers used this headroom for decades to make chips run faster without significantly increasing power consumption.
But that trend broke down in the mid-2000s: Dennard’s assumptions ignored the baseline power that even the smallest transistor must use. That forced a major rethink. Chip-makers realised they could put multiple processors – the silicon brains that carry out instructions from software – on a single chip. Each of these mini processors is called a core. Splitting programs into chunks and running them in parallel on these cores let computers solve problems faster without boosting the speed of the individual parts, which would also boost power consumption. Even the most basic laptop processor today features two cores, but the world’s fastest machine, Summit, which ORNL turned on in June, has more than 2 million.
“Your laptop probably has two cores – the world’s fastest machine has over 2 million”
More recently, the drive for energy efficiency has also fuelled the use of accelerators – specialised devices that carry out certain tasks much more efficiently than general-purpose processors. GPUs, originally designed to render graphics but now widely used for artificial intelligence, are one popular variety. They excel at running parallel tasks at low power and are increasingly finding their way into supercomputers – Summit has more than 27,000 of them.
But all this can only take us so far. “If for some magical reason you could actually build an exascale machine from today’s technology, nobody could afford to power it,” says McIntosh-Smith. Such a behemoth would need hundreds of megawatts, he says. As a rule of thumb, “one megawatt’s about a million dollars for a year”.
Even if chips are more energy efficient, having more of them means you rack up bills elsewhere: specifically, sending signals from one part of the machine to another. “Moving data within a large-scale computer system consumes over an order of magnitude more energy than is needed to compute the data,” says Mike Vildibill, who leads the exascale programme at computer manufacturer Hewlett Packard Enterprise.
Because power consumption rises rapidly with distance when communicating over electrical cables, fibre optics have become the go-to approach for moving large volumes of data between distant clusters of processors within supercomputers. But at the exascale, the cumulative cost of signalling over even millimetre lengths will be a problem.
One potentially revolutionary technology, known as silicon photonics, is allowing us to integrate tiny lasers directly onto chips. That could make communication essentially distance independent, says photonics researcher Keren Bergman at Columbia University in New York. Commercial products of this type have just started appearing, but it is unclear whether they will feature in the first exascale machines. “We can probablyjust reach over the finish line with the technologies we have on hand,” she says. “But to have any legs beyond that, photonics is clearly going to be an integral part.”
For all its importance, energy efficiency is only part of the communication problem. Having more processors is of little use if they are starved of data, and increases in flops have been outstripping data movement speeds for some time. Silicon photonics could boost the volume of data that can be shuttled per second by at least a factor of five, says Bergman, but faster conventional optics are needed in the meantime. The memory chips could also be stacked vertically, an arrangement that can boost both speed and power efficiency, but typically costs more and comes in smaller capacities, presenting a tricky balancing act for computer designers.
Parallel universe
Nonetheless, most experts say the roadmap for exascale hardware is fairly well laid out. What may be trickier is building software and algorithms to run across potentially billions of cores. “Every generation of supercomputers leaves some users behind,” says William Gropp, director of the US National Center for Supercomputing Applications in Illinois. “That’s been a constant problem and I think we’re going to see it again.”
The trouble is an inherent weakness in parallelism, noted in 1967 by computer scientist Gene Amdahl. His eponymous law says that any program’s speed will always be limited by its least parallel part. Gropp compares it to transporting a planeload of passengers from Chicago to New York. Even if you replace the flight with instant teleportation, it will still take each passenger 2 hours to clear security.
In reality there are workarounds. As you increase the number of parallel tasks on the go, time spent on the slowest part represents ever less of a limitation. But there are hard limits to how many chunks some problems can be split into, says Gropp, and the complexity of massively parallel models can be difficult to understand.
“We are running out of parallelism,” says Messer. “Mr Amdahl pokes his head up every once in a while and we try our best to push him back down, but he never goes away.” Messer is confident that would-be supernova simulators have enough tricks up their sleeve to get their models working on exascale machines, and he thinks most developers with experience of the world’s biggest supercomputers will adapt too. But further progress may require disruptive overhauls of code and programmers’ attitudes.
Messer thinks one way forward may be to split problems into the smallest possible sub-tasks and let the machine decide how to divvy them up between processors. But that can make it hard to follow what the program is actually doing, which makes tracking bugs tricky and can present fundamental challenges when running physical simulations. A program that reproduces a specific phenomenon but with underlying steps we struggle to understand is unlikely to provide much insight. “A black box that just gives you an answer is hardly a useful scientific exercise,” he says.
And no matter how perfectly parallel your application is, it still needs to occasionally compare notes between processors, meaning the speed restrictions due to data movement rear up again. One way to tame the problem is to find extra work to do while information is being shuffled around.
Messer’s supernova simulations are a case in point. They split the star into chunks that can be simulated separately, but calculating the impact of gravity at each time step requires every processor to broadcast the mass of its chunk to all the rest. While waiting for this to finish, Messer and his group realised that each processor could happily turn to chunk-specific problems that do not depend on gravity, like the rate at which lighter elements are fusing into heavier ones.
Brain simulations pose similar problems because each neuron connects to so many others. At the Jülich Institute of Neuroscience and Medicine (INM) in Germany, researchers have built models that send every simulated neuron’s output across the entire network. This approach works fine at the petascale – millions of billions of flops – but would simply devour too much time and memory at the exascale. So the team has developed a way to determine at the outset which neurons need to share activity data, a shortcut which should let them run full-brain simulations on exascale machines. “Then it becomes a really useful research tool for scientists and not only a technology demonstration,” says Markus Diesmann, director of INM’s computational neuroscience division.
For other simulations, the transition to the exascale could be much more of a struggle. At the European Centre for Medium-Range Weather Forecasts (ECMWF) in Reading, UK, two petascale machines provide 22 countries with 15-day weather forecasts. The software stretches to millions of lines of code built up over decades, and the forecasting models rely on algorithmic building blocks that deal with very different physical processes. Some are inherently hard to parallelise; others are faster than parallel alternatives at petascale, but will start lagging behind once the upgrade to exascale kicks in.
Peter Bauer at the ECMWF thinks flexibility will be key to revamping the code. If, hypothetically, you can get more regional detail at the cost of your model’s overall accuracy, that trade-off may be worth making. He says they will also need new software tools that seamlessly adapt models to run on the unprecedented diversity of hardware expected in exascale machines.
The booming field of artificial intelligence is offering further workarounds. Neural network calculations run at half the precision of conventional supercomputer code, but can produce usable approximations faster and more efficiently. Rick Stevens at Argonne National Laboratory in Illinois is developing deep learning models for the Aurora supercomputer to predict how drugs affect different cancers, something that could turbocharge drug development and personalised medicine. Stevens says machine learning is particularly promising in areas with reams of data but little theory to help develop traditional simulations. “I think we are just starting to understand how broadly applicable it is,” he says.
But there is a deeper question to confront in the exascale race: whether a focus on flops is always the best way to improve performance. Many simulations use less than a tenth of a supercomputer’s peak capacity. Meanwhile the ability to move data is largely ignored in ranking the world’s top-performing machines, even though it is often the limiting factor for science applications.
Supercomputing’s emergence as the 21st century’s equivalent of the space race could be exacerbating the problem. “My concern here is that the preoccupation with the race – the geopolitical posturing here – may ultimately lead to exascale systems that are not the best for what researchers need to do their jobs,” says supercomputing consultant Bob Sorenson.
Satoshi Matsuoka, who leads the development of Japan’s “exascale” machine, says the country has deliberately set different goals. The objective is to run applications up to 100 times faster than on their existing K supercomputer – a 10-petaflop machine – by focusing development on data movement rather than raw flops. He says candidly that the post-K computer will not hit 1 exaflop, but he is also confident it will run science applications faster than competing first-generation exascale machines.
Doug Kothe at ORNL, who leads the US Exascale Computing Project, doesn’t think the exascale race will compromise the real-world performance of the machines that emerge. Although the US is chasing 1 exaflop, it is also targeting a 50 times speed-up in applications over its 17.6-petaflop Titan supercomputer. With the right priorities, says Kothe, “the first wave of exascale platforms will not be ‘stunt systems'” that can only run a limited number of niche applications.
As for what might come after exascale, things look wide open. In an era where Moore’s law is starting to falter, many predict ever more specialised hardware. Future supercomputers could contain a broader variety of processors and architectures than ever before, or even look entirely different (see “What about quantum?”). If Messer wants to keep watching stars explode, his job is going to get a whole lot more complicated.
What about quantum?
Even the world’s fastest computers (see main story) might one day be obsolete. Enter quantum computers. Instead of encoding information in the same way ordinary computers do, these machines could harness weird quantum effects to let them quickly solve problems that would take millions of years for even the fastest classical computers.
The same quantum quirks may limit the tasks they can perform, but within five years they could compete with their classical counterparts on problems like molecular simulation, materials modelling and optimisation, says John Morton, a quantum computing researcher at University College London. Morton thinks they will probably solve their first problem beyond the reach of conventional computers within 12 months.
While this first test is likely to be designed to favour the quantum computer, he says, “this milestone will mark the start of a new era where quantum software, tailored to the capabilities of early stage quantum hardware, can begin to tackle useful problems.”
This article appeared in print under the headline “Big black box”
