SPEAKING at the University of Cambridge in 1980, Stephen Hawking considered the possibility of a theory of everything that would unite general relativity and quantum mechanics – our two leading descriptions of reality – into one neat, all-encompassing equation. We would need some help, he reckoned, from computers. Then he made a provocative prediction about these machines’ growing abilities. “The end might not be in sight for theoretical physics,” said Hawking. “But it might be in sight for theoretical physicists.”
Artificial intelligence has achieved much since then, yet physicists have been slow to use it to search for new and deeper laws of nature. It isn’t that they fear for their jobs. Indeed, Hawking may have had his tongue firmly in his cheek. Rather, it is that the deep-learning algorithms behind AIs spit out answers that amount to a “what” rather than a “why”, which makes them about as useful for a theorist as saying the answer to the question of life, the universe and everything is 42.
Advertisement
Except that now we have found a way to make deep-learning algorithms speak physicists’ language. We can leverage AI’s ability to scour vast data sets in search of hidden patterns and extract meaningful results – namely, equations. “We’re moving into the discovery phase,” says at the University of Washington in Seattle.
Which isn’t to say Hawking was right. Far from facing extinction, theoretical physicists might have found the ultimate collaborators. Their challenge now is to figure out which aspects of the human theorist’s playbook should be enshrined in machine counterparts so they don’t get stuck in the same ways we have.
Symbolic regression
Data-driven science began in the 16th century with Danish astronomer Tycho Brahe’s meticulous observations of the motions of planets and stars. Poring over Brahe’s notebooks, Johannes Kepler spotted underlying patterns in the data to come up with three simple laws that describe planetary motion – and confirmed the sun’s place at the centre of the universe.
Kepler’s strategy was trial and error. He painstakingly worked through myriad orbital shapes to see which best fit the solar system. Eventually, he uncovered a precise mathematical harmony between the paths of planets and the time it took them to orbit the sun. It was a triumph that put mathematical equations at the heart of our understanding of the universe, where they have remained.
“The great book of nature is written in mathematical language,” wrote Kepler’s contemporary and fellow heliocentrist Galileo Galilei. These days, the question of whether the universe is inherently mathematical or mathematical patterns are something we impose upon it is far from settled. But given the amazing triumphs of mathematics in capturing what appear to be abstract truths in simple equations, it makes sense that modern physicists would try to teach AIs to do something similar.
This was the thinking behind symbolic regression, a method originally developed in the 1970s by Patrick Langley, then at Carnegie Mellon University in Pennsylvania, and later revived and finessed by others, including Hod Lipson and Michael Schmidt, then at Cornell University in New York. It works by methodically running through equations that feature various mathematical symbols, or operations, like addition or multiplication, and combinations of physical variables, such as position or velocity. If one of these equations closely fits the data, say from observations of a planet’s orbital track, it is rewarded and then mutated by, for example, switching a plus for a minus. The new expression is then tested and compared with its predecessor, and so on.
In this way, weaker equations are gradually weeded out in a process akin to natural selection. “The network is able to figure out the law by itself, without our intervention,” says at the Flatiron Institute in New York.
The trouble is that symbolic regression struggles to find equations in “higher-dimensional” data sets, meaning those containing many possible physical variables. These happen to be the bread and butter of many physicists today. Astrophysicists and cosmologists, for example, are enjoying a deluge of data from powerful new telescopes like the James Webb Space Telescope and the European Space Agency’s Gaia space observatory. That is a problem for symbolic regression because when you have more variables, the number of possible equations that your algorithm has to test explodes – to the point that it is too much even for today’s beefiest computers.
Deep-learning algorithms make light work of big data sets. This explains why, a few years ago, , in collaboration with Ho, attempted to – deep learning’s pattern-finding prowess and symbolic regression’s easy-to-interpret outputs.
They fed a deep-learning neural network real-world NASA data of planets and moons orbiting in our solar system – the same sort of things Kepler would have worked with. After the neural network had found patterns in the data, it was frozen. A series of numbers were then fed into the neural network and the outputs formed a new data set, one that reflected these patterns and was suitable for symbolic regression, allowing it to find equations that fit. Sure enough, the researchers’ symbolic regression algorithm, known as PySR, “rediscovered” Isaac Newton’s law of gravity using little more than raw data.
But this was just a proof of principle. Applied to the glut of new astrophysical data by Cranmer and many others, PySR is already being used to discover equations that describe diverse and interrelated features of the cosmos. From the deduced from gravitational wave detections to , symbolic regression is suddenly offering astrophysicists new ways to find mathematical order in the universe. “They are actually driving brand new discoveries purely from the data, it’s really beautiful work,” says Brunton, who, in 2016, co-created another popular symbolic regression algorithm called SINDy.
Thought experiments
Take dark matter, the mysterious source of gravitation that keeps galaxies from flying apart and makes up 80 per cent of all matter in the universe. In cosmology, the absence of dark matter is called a void and the distribution and characteristics of voids can be related to universe-wide constants. But finding these equations is hard because there are many voids and each one is described differently, so there are many variables. In May 2021, Cranmer, along with Ho and other collaborators, used deep learning in conjunction with PySR to discover and what fraction of the universe’s total energy is present as mass.
Then, in March 2022, and her collaborators used PySR to discover an – a clump of dark matter – from other properties, such as how quickly stars form in galaxies. Surprisingly, this equation worked accurately across almost all galaxies in the history of the universe. “It was spectacular,” says Francisco Villaescusa-Navarro at the Simons Foundation in New York, who was a co-author. “It’s probably because [the neural network] has found some really fundamental relation.”
Fundamental relations are exactly what physicists are looking for because they are “generalisable”, meaning they can describe unusual physical systems just as well as the original data set that the relation was discovered from – and, as such, are a hallmark of understanding. Newton’s famous second law of motion, for example, which says that the force acting on an object is equal to the mass of the object multiplied by its acceleration, works just as well for a falling apple as it does for landing rockets on the moon.
Symbolic regression programs tend to churn out equations that are more generalisable than deep-learning neural networks deployed on their own, which often grind out nonsense when applied to scenarios outside their comfort zone. “They don’t suffer from this tendency to pay attention to small details of one particular data set,” says Shao.
Even so, all of PySR’s successes so far are empirical equations. In other words, they are descriptive and good at replicating experimental data rather than directly offering a theoretical explanation, or the deeper “why”, that physicists want. Kepler’s law, for example, is an empirical equation. It fits Brahe’s reams of data surprisingly well, yet Kepler didn’t know why this was. Only later, when Newton thought deeply about the nature of gravity, did this law make sense as part of Newton’s law of gravitation.
We know the fitting of mathematical symbols to data isn’t the only way to make sense of the world. Albert Einstein came to his theory of general relativity, which superseded Newton to say that gravity is the result of mass warping space-time, through a series of imaginative thought experiments. Observations, like the unusual movement of Mercury in the night sky, only confirmed general relativity rather than inspiring the theory. Meanwhile, in quantum mechanics, an empirical equation called Planck’s law – which describes the emission of radiation from objects – was the precursor to deeper insights about the microscopic world.
Cranmer and Ho see empirical equations as more of a stepping stone to deeper truths than truths in themselves. They hesitate to even call these new equations “discoveries”. “We’re definitely not at that point yet,” says Cranmer, speculating that hybrids of symbolic regression and deep learning could bring major scientific discoveries – such as what dark matter is, or whether it really exists at all – within a decade.
The idea is that these symbolic expressions will give direction to physicists, says Ho, helping them to make bigger leaps. “When you express your deep-learning model in this language, then immediately you can see connections to existing theory,” says Cranmer. Scientists still play a crucial role in studying these equations, understanding their form and how they connect to each other and physics as a whole – at least for now.
at the University of Montreal, Canada, who was involved in the rediscovery of Newton’s law of gravity, suspects that symbolic regression could soon weigh into debates about the nature of dark matter and also dark energy, another mysterious entity thought to be driving the expansion of the universe. “We’re clearly a bit stuck,” he says. “Maybe this can shed some light on how we should be looking at the data.”
Fundamental laws
Cranmer and others are already starting to think about how they might develop AIs capable of finding fundamental laws of nature on their own. One strategy is to embed cultural norms from the physicist’s playbook. Already ingrained in PySR and other symbolic regression programs is a penchant for simplicity – enshrined for centuries as Occam’s razor, the principle of shaving off needlessly complicated explanations or, when it comes to equations, excessive symbols.
With symbolic regression, there is usually a trade-off between accuracy and simplicity. If the equations you come up with using this method are very complicated, then you can often match the data you are using more closely, as there are more dials you can twiddle. This is called “overfitting” and it increases the risk that your mathematical expression is less accurate outside of its test data set. In other words, it isn’t generalisable. With simpler expressions, however, there is “a much better chance that you’re actually capturing a mechanism”, says Brunton. Cranmer suspects this compromise is why PySR fell upon Newton’s law of gravity, rather than Einstein’s equations of general relativity, when he applied it to NASA’s orbital data.
Symmetry, as defined by physicists, is another guiding light for physicists seeking universal laws of nature. It is the property of being able to transform something and end up where you started. There are already AIs that can make physics problems more symmetric. And now Bruton, alongside J. Nathan Kutz, also at the University of Washington, have “baked in” knowledge of many types of symmetry into their SINDy program, which extracts equations to describe the complex behaviour of flowing fluids. “You almost always get better models from less data that are more accurate and can handle more noise,” says Brunton.
Then there is mathematical “beauty”, or “elegance”, which is hard to define, but which many physicists still strive for when developing new theories. It is possible that all such methods and ideals could be enshrined in AIs searching for new equations. But at the Massachusetts Institute of Technology points out that though our intuitions about reality have been useful in the past, they have also led us astray.
As a result, Thaler warns that in trying to make AIs too human-like, we risk missing out on their most revolutionary promise: the ability to offer a new perspective. Lemos takes a similar view. “You don’t want to give it too much human intuition because your end goal is to use it to find equations that humans cannot find. Maybe it will get stuck in the same way as we’re getting stuck,” he says.
Thaler remembers setting an AI to work on a problem he had been stuck on for a decade. Quickly, it came back with a solution, which he then scrutinised. “I was embarrassed that I had not thought of it myself,” he says. “I realised I had a prejudice about the way the problem should be solved, which imposed a constraint on my own human thinking that I hadn’t given to the computer.”
Rather than forcing them to think just like humans, AIs need the freedom to search over all possible approaches to problems and every combination of symbols as solutions. In fact, Thaler reckons physicists need to start thinking differently. “I was trained to think about what are the laws of nature in our universe. Now it’s ‘what is the space of all possible rules?'” This is what is required, he says, if we are serious about AIs making big leaps in fundamental physics.
For now, symbolic regression is winning over AI sceptics. “You can have some very senior theorist who hates machine learning, but then you introduce them to symbolic regression and they often like it, because it gives you expressions you can write down on paper and interpret,” says Cranmer. In that sense, symbolic regression is just another tool to add to telescopes, computers and calculus. Computers aren’t naturally curious, so there has always got to be a human there to ask the question in the first place. “Humans still have to specify the sandbox in which you operate, and then the computer can look at every single grain of sand and inspect it,” says Thaler.
But now that physicists and artificial intelligence are finding a common language, there is a new kind of dialogue. This could radically shift how they work towards a genuinely collaborative relationship, says Thaler. “It feels to me like the beginning of a sea change.”
Article amended on 8 December 2022
We have corrected Tycho Brahe’s nationality
