IT IS almost impossible to go a day without interacting with an algorithm. They help direct the whole of our online experience, recommending what we should buy, read, watch and listen to. Some 74 per cent of adults in the US use Facebook at least once a day – and what they see is decided entirely by an algorithm. Offline, they are increasingly used to help us make tricky decisions, screening job applications, moderating exam results and even directing which crimes police investigators focus on.
As they have become ubiquitous, algorithms have generated a mixture of hype and concern. On the one hand, we are regularly told that they can be opaque and biased. On the other, we hear that they can be incredibly handy, pulling off tasks that humans can struggle with, from optimising complex trade logistics to spotting the earliest signs of disease in medical scans.
Advertisement
So what’s the truth about algorithms? It helps to understand that the word can mean quite different things (see “What is an algorithm?“). It also helps to get to know some of the algorithms that shape our lives – so that’s what we’ll do over the next few pages.
1 SOCIAL MEDIA
Facebook’s news feed
Few algorithms wield as much power as those under the bonnet of Facebook. The social media giant’s algorithms control which updates its 2.8 billion monthly users see from which friends and what headlines they read on their news feed.
When we speak of the “Facebook algorithm”, we’e actually referring to dozens of pieces of software that are based on a range of technologies and are constantly being tweaked. This software analyses what the firm calls the “inventory”: the collection of posts from those people, pages or groups that a user follows. It then uses neural networks, a form of artificial intelligence, to score those posts on various factors. As far back as 2014, Facebook employees reported that the news feed was taking 100,000 factors into account. Eventually, it combines these scores into a single ranking for each post. This is used to curate what a user sees.
Facebook rarely talks publicly about exactly how its algorithms work. In truth, it doesn’t know itself, at least not at the level of individual users. Nick Clegg, Facebook’s vice president of global affairs, recently wrote that .
This complexity could spell a reckoning for Facebook in the coming years. There has long been tension between the company’s business model and what is perceived as good for society at large. Facebook wants to keep users on its site for longer so it can sell more ads. This has caused concern that its algorithms show people more posts about controversial topics, such as Brexit, because they tend to prompt higher engagement.
of adults in the US use Facebook at least once a day
The company has long used human moderators to help decide whether controversial content should be removed or not. Recently, it set up the independent Oversight Board to review especially difficult cases. The board is composed of 20 journalists, politicians, campaigners and academics, one of whom is technology law specialist at Queensland University of Technology in Brisbane, Australia. “There are big problems to address, such as whether the way that news is amplified on social media aligns with our social aims,” says Suzor. “These problems cannot be addressed until Facebook becomes more transparent about how the algorithms work.”- Matthew Sparkes
2 WEATHER FORECASTING
The Unified Model
The British obsession with the weather may baffle the rest of the world, but one good thing to come of it is a first-class weather forecasting service. At the heart of the Met Office’s forecasts is an algorithm called the . It works by sucking in data on the state of the atmosphere from weather stations and satellites, then extrapolating to calculate how it will change. The results determine whether we see an icon for sunshine or snow.
The algorithm is built around the Navier-Stokes equations, our best description of how liquids and gases flow. These are how we predict the flow of dry and moist air in the atmosphere, a major factor in determining the weather. It isn’t possible to solve the equations exactly, but you can do so approximately if you take a series of incremental changes in the state of the atmosphere and add them together. This is what the Unified Model does.
of Met Office next-day temperature forecasts are accurate to within 2°C
For decades, computer processors have been getting quicker, allowing algorithms like the Unified Model to process measurements made at ever shorter increments and so produce better, faster predictions. It can seem like weather forecasts get it wrong all too often, but the Met Office’s next-day temperature forecasts are accurate to within 2°C on 92 per cent of occasions. So, while it can’t forecast every shower with total precision, it is a pretty reliable algorithm.
To get even better, the Unified Model is going to have to change. “We are no longer getting faster processors, we’e just getting more processors,” says at the Met Office. He and his colleagues are now redesigning the model so it can be run on parallel processors. – Anna Demming
3 DIGITAL SHARING
JPEG
If you recognise the acronym JPEG, you might think of it as merely a type of file. In reality, it is an algorithm used to compress the amount of data in an image. The blistering speed with which we can share pictures online is partly thanks to compression algorithms like this one.
JPEG is named after the Joint Photographic Experts Group, which devised this standard back in 1992. The algorithm itself is quite sophisticated, but it is based on simple insights about human vision. “A lot of research has been put into what humans can perceive in images and sound, and it is a lot less than the files are capable of transmitting,” says at Sheffield Hallam University in the UK. “So, for efficiency’s sake, compression algorithms focus on what’s important for an eye or ear to perceive.”
Cone cells in the human eye, which register colour, are 500 to 1000 times less sensitive to light than the eye’s rod cells, which are most sensitive to brightness. Among other things, JPEG exploits this by dropping some of the information about colour, while retaining the brightness more precisely.
images are shared online every minute
There are situations where you wouldn’t want to use a loss-inducing format like JPEG, such as in medical imaging or digital art. With compression algorithms, it is always a trade-off between quality and speed of transmission. Data speeds have increased drastically since 1992, but demand has skyrocketed too, so compression is still important. Other kinds of compression algorithms can squeeze down other types of file – MP3 is often used for music files, for instance.
There are plenty of would-be successors to JPEG designed for even more efficient data transmission. One of these is JPEG XL, which at Google is working on. “Images are the heart of the internet,” he says. “We are trying to introduce a new way to model colours.” The idea is that this will produce images that are even better tuned to human vision. – Chelsea Whyte
4 WORLD WIDE WEB
PageRank
Back in 1998, two entrepreneurs filed a patent for an algorithm called PageRank. They were Larry Page and Sergey Brin, the founders of a medium-sized internet search engine called Google. From there, Google quickly grew to be the clear leader in online search – and it was PageRank that made this happen.
Before Page and Brin filed that patent, search algorithms tended to rely on analysing the words on a web page. This determined which websites a user would be served when they entered a search query. PageRank worked differently: it assigned each link a rank based on the number and authority of any websites linking to it. If site A links to site B, site B will inherit some of A’s PageRank, boosting its position on Google’s results page. This turned the search market on its head, negating the tricks and workarounds that website owners were using to game the rankings. It provided better, more accurate results.
searches are processed by Google every second
The PageRank algorithm no longer has the influence it once did. Instead, Google now uses a series of different algorithms to analyse hundreds of different factors that decide where websites rank, drawing clues from all of Google’s many products. You may not realise it, but your location, the time of day and the device you are searching from – plus many other subtle factors, not all of which Google reveals – affect your search results. As with so many other parts of the online experience, algorithms personalise what you see in a way that’s hard to fully understand (see “Facebook’s news feed”).
It is difficult to overstate how much this algorithmic indexing has changed the world. Google’s supremacy allows it to sell targeted advertising that nets it vast profits: $46.2 billion of revenue in the third quarter of 2020 alone. But it has also sucked support from other industries, not least the media, which once enjoyed reliable streams of revenue from adverts. We will surely be grappling with the legacy of PageRank for years to come. – Alexander McNamara
5 FINANCE
Trading algorithms
Financial trading has always been an arena ripe for automation. It isn’t just that humans frequently lose their nerve and make mistakes. There is also decades of price information ready to be analysed and used to develop rules on when to buy, when to hold and when to sell. Today, algorithms control a huge portion of the global economy.
“Good traders follow good rules, but as a human, you let emotions come in and you maybe panic or get too greedy,” says , a business economist at the University of Oxford. “Algorithms are the ultimate way to be systematic.”
One breed of finance algorithm carries out what is called high-frequency trading, where computers make vast numbers of trades at incredible speed. Each transaction is likely to generate only a tiny profit, but this adds up. These algorithms have been accused of causing financial crashes in the past. But they are only a small part of the market.
More common are algorithms that lay out the circumstances under which particular financial assets, such as shares and bonds, are bought and sold. It isn’t so much about being quick, but about minimising human bias. Such digital rule books are mainly developed and used by funds handling huge pots of money and their inner workings aren’t public. An increasing number of global financial decisions are made according to these algorithms. “It’s growing all the time,” says Vulkan. “Hedge funds have secrets, so nobody can tell for sure, but I would guess it’s about half of the market now.”
We do know that trading algorithms are becoming more complex. Some funds are going beyond simple rules and using artificial intelligence and machine learning. Systems based on these can use not only stock market data as an input variable, but also things like the number of positive and negative words in a CEO’s media interview. In time, this might give firms with the smartest algorithms an advantage. – Leah Crane
6 ENCRYPTION
The RSA algorithm
Ever used the internet? Then you have used RSA cryptography, a combination of algorithms and protocols that makes it possible to send information privately between computers. Those secrets could be anything from an email to your bank details.
Methods for sending secrets rely on scrambling the message in such a way that only the intended recipient can unscramble it. A simple encryption method would be to move each letter in a message one along in the alphabet. You send “Ij!” and then the recipient knows to decrypt it by reversing the process to get the original “Hi!”.
Trouble is, the first time you send a message, you must overtly tell the recipient how to decrypt it, which weakens the system. But here is a cunning trick: if you use an encryption key that comes in two parts, one public and one private, you can encrypt your message with the recipient’s public key but they alone can decrypt it with their private key.
The way to pull this off is to have a process that is easy to do in one direction (encrypting the message), but hard to do in the other (decrypting the message). The RSA algorithm does this with a quirk of prime numbers. Multiplying primes together is simple, but factorising the number produced to get back the original primes takes a lot more time.
In real life, the RSA algorithm deals in truly gigantic primes and the system is practically impossible to break – for now, at least. Factorising numbers into primes is only hard for classical computers. For a big enough quantum computer, it will be a breeze. But fear not – work is under way on cryptography that relies on other, more quantum-resistant mathematics. – Timothy Revell
7 HEALTH
Triage algorithms
Imagine you start feeling a crushing chest pain. You quickly phone the emergency services for help. In many countries, it will be an algorithm on the end of the line – well, sort of. The call handler will take you through a series of questions governed by an algorithm to work out if you are having a heart attack or something else. The outcome determines if an ambulance needs to be sent and how quickly. “If you put everything as a priority, then nothing is a priority,” says Richard Webber at the UK’s College of Paramedics.
Elsewhere in medicine, algorithms are beginning to be used without much, if any, human intervention. In the online version of NHS 111, the UK’s non-emergency healthcare triage service, users are taken through a series of completely automated questions to help direct them. There are also several “symptom checker” apps available. These ask about your medical problems and then, based on an algorithm, suggest a probable diagnosis. However, the UK’s Royal College of General Practitioners has said that such apps can’t yet replace doctors’ abilities to make decisions based on their breadth of experience, their training and, sometimes, their gut instincts.
More sophisticated machine-learning algorithms are being used too. There are several systems that can read medical images, spotting signs of a broken bone in an X-ray, for instance, or the onset of diabetes-related blindness in an eye scan. Algorithms like this are developed by training the software on thousands of images that have been categorised by doctors. In some cases, these systems have outperformed trained professionals and some hospitals are already using them, usually double-checked by a doctor, to help make diagnoses far more quickly. – Clare Wilson
8 INTERNET
The internet protocol suite
When you fire off an email or type into a browser, you are asking your computer to exchange messages with another one somewhere in the vast digital expanse of the internet. This miraculous connection is governed by a sprawling set of algorithms and rules known as the internet protocol suite.
The purpose of these rules is to make sure traffic gets where it needs to go. To that end, the algorithms don’t specify exact routes through the cables that make up the internet in case parts of it get disrupted. Instead, the message is chopped into packets and the algorithms help them find a way by any viable route, with each packet basing its trajectory on feedback from the one in front. Once at the destination, the packets are put back together.
This algorithmic rule book was written by computer scientists Vint Cerf and Robert Kahn and implemented in 1974. It has formed the backbone of the internet ever since – but it has needed a few tweaks along the way.
One reason for this is that it works hand-in-glove with the internal protocol (IP) address system, which gives each connected device, from servers to smartphones, a unique, machine-readable identifier. As the world became ever more connected, the number of available IP addresses dwindled, and in 2017 we all but ran out. This was because IP version 4, the first publicly used version of the internet, was structured such that only 4.3 billion IP addresses were available.
Thankfully, the internet got an upgrade. The new address system, IPv6, has more possible addresses than there are atoms on the surface of Earth. So, even with trillions of future devices to be connected to the internet, we won’t run out of addresses any time soon. – Donna Lu
9 SCIENCE
Monte Carlo algorithms
The scientists developing the atomic bomb during the second world war needed to understand nuclear chain reactions – and there was no room for error. The problem was, the physics was far too complex to analyse with conventional methods. So mathematician Stanisław Ulam devised a set of algorithms that could solve the problem in a new way, harnessing the nascent computers of the time.
The idea was to fake the physics many times over and see what tended to happen on average, producing not a direct answer but the most probable outcomes. Ulam named the algorithm after the card game Monte Carlo – sometimes called solitaire – because he first tested it by modelling outcomes of the game.
These days, Monte Carlo algorithms are employed incredibly widely, from forecasting stock market trends to the output of wind farms. “They are important because they can be used to understand complex situations where there is no analytical formula that would allow one to directly compute the result,” says computer scientist at the Massachusetts Institute of Technology.
They are especially useful for scientists wrestling with complex phenomena. For instance, they are routinely used to figure out what shapes of molecule can best glom on to important proteins, providing a lead on compounds that may be useful drugs. Many of the medicines we rely on wouldn’t exist without Monte Carlo algorithms.
We are still finding new uses for them. One devilish problem in physics is understanding the behaviour of materials in which lots of electrons whizz around and mutually affect each other. The maths is nigh-on impossible, which means we can’t predict how materials we haven’t yet made will behave. at the Perimeter Institute in Waterloo, Canada, has found a shortcut. He uses Monte Carlo algorithms to do the work instead, applying them to hunt exotic new states of matter. – Daniel Cossins
What is an algorithm?
Ask a computer scientist this question and they will tell you it is a sequence of instructions that takes an input, performs some repeatable computation on it and provides an output. Think of an algorithm like a super-precise recipe, usually written in the cold logic of a programming language.
A simple example is the bubble sort, which arranges a list of numbers in ascending order. It begins by comparing the first two numbers. If the first is greater than the second, it swaps them. Otherwise, it moves on to the next pair. It cycles through the list again and again until it passes through without any swaps needed, at which point it outputs an ordered list. If you are shopping online and filter products by price, the bubble sort algorithm is kicking into gear behind the scenes.
These days, popular use of the word algorithm is morphing: it is increasingly used to describe almost anything that a computer accomplishes. That includes the realms of artificial intelligence (AI) and machine learning, where the steps in the recipe aren’t always quite so clearly laid out.
Take neural networks, a type of AI system that mimics the human brain in that it can be trained to perform a task based on looking at examples of correct and incorrect results. Such “algorithms” can be incredibly powerful, but it is usually hard to look inside and determine how they really work.
There are those who find the loosening of the term algorithm to include AI unhelpful. “Now people use ‘algorithm’ to mean almost anything,” says at the University of Leeds, UK. “I”ve become so annoyed at people misusing it.”
Dyer warns that, in future, we may increasingly lean on machine learning as an “easy way out” – a route to solve problems without fully understanding them ourselves.
He says we ought to apply the right kind of algorithm in the right context. There are times when a rigid set of predictable steps is desirable and times when highly capable but ambiguous AI can be beneficial. “It’s fine if it gets wrong whether you like this book or not, but it’s not fine if it crashes your car,” says Dyer. – Matthew Sparkes
