IT SHOULD not have taken Cambridge Analytica to remind us that algorithms can have an insidious influence. Arguments rumble on about what privacy rules were broken, if any, and whether the company’s mass profiling of Facebook users swung the 2016 US Presidential Election and the UK’s Brexit vote. What we are clear on is something we had been warned about: give an algorithm a load of data about ourselves, and in return it assumes power over our lives.
Facebook and Google’s artificial-intelligence algorithms, learning from the data we feed them, already control what we read on the web. Similar machine-learning algorithms determine the interest we pay on a loan and, in some places, the chances the police will stop and search us on our way home. Soon they could be driving cars, helping to make life-or-death decisions in the operating theatre and deciding fates on the battlefield.
Advertisement
Sometimes these algorithms blunder, discriminate, or overstep the line – so we need to be able to hold them to account. The European Union has fired the first salvo, giving its citizens the right to an explanation for why an algorithm did something that affects their lives. The trouble is, the techniques behind the AI boom are by their very nature a black box. Even the people who create these machine minds don’t understand their reasoning.
That’s alarming enough given their current reach. But if AI is going to fulfil its promise and take an ever-more important role in society, we need to find a way to trust it. The question now is: how?
Algorithms are not intrinsically mysterious. They are simply sets of instructions that tell a computer how to perform a task. Even so, many in use today are proprietary because the companies behind them want to protect their intellectual property – and that has already raised some troubling scenarios.
Perhaps the most notorious case is that of Eric Loomis. In 2013, he was convicted of fleeing from the police and operating a vehicle without its owner’s consent in La Crosse, Wisconsin. Sentencing him to six years in prison, the judge cited the “high risk” Loomis posed to the local community – a risk determined in part by his score on the COMPAS assessment, a proprietary algorithm designed to predict the likelihood that someone will reoffend.
Loomis challenged the ruling on the grounds that the judge, by considering the outcome of an algorithm whose workings are not transparent, had violated due process. But in June 2016, the Wisconsin Supreme Court rejected his appeal – a verdict handed down just a month after the non-profit news organisation ProPublica discovered that the COMPAS system was .
Equivant, the company that developed the system, disputes that analysis. But COMPAS is not the only algorithm under scrutiny. In fact, examples of algorithmic discrimination have stacked up over the past few years, and it’s not hard to see why it happens. If you feed an algorithm data from the real world, it will reproduce the biases that already exist there.
“Forcing AIs to explain themselves could in many cases hold them back”
Now governments are under pressure to ensure that algorithms are fair and transparent. Provisions for algorithmic accountability are baked into the EU’s wide-ranging General Data Protection Regulation, which comes into force next month (see “Take back control!”). It is a laudable aim. But there are question marks over whether it is even possible.
In many cases, the companies involved could plausibly be forced to give up their code to a government watchdog, which would go through it line by line to understand the decisions it makes. But for the growing number of systems reliant on machine learning, the collection of techniques underpinning the most sophisticated AIs today, that would be impossible.
“These things think in a very foreign way,” says at the US Defense Advanced Research Projects Agency (DARPA), which is interested in AI’s potential to supercharge reconnaissance, among other things. “They use bizarre mathematical logic that it is very alien to us.”
With traditional computer programs, the machine gets line-by-line instructions. With machine learning, however, the computer must work out how best to solve the problem. The result is a machine that essentially programs itself.
Imagine instructing a robot to make soup. The conventional approach would be to write out a precise recipe for SoupBot to follow. First peel the onion, then cut the onion. But a SoupBot based on machine learning would instead work out what to do on its own, perhaps by watching thousands of videos of people making soup and trying to come up with its own soup-like recipe, or by attempting to make soup again and again and learning from feedback on the results of each attempt.
In the case of SoupBot, the conventional approach would be most efficient. But simple recipes don’t exist in many scenarios. There isn’t one for recognising words in a sound recording, say, or for verifying a face to unlock a phone. And this is where machine learning comes into its own. By working out how to quickly spot patterns in vast amounts of data, an AI can master exceedingly complex tasks.
Open the box
This is usually thanks to an underlying technique called deep learning – one of the most successful ways to get machines to learn for themselves. It involves a vast, layered network of connections, inspired by neurons in the brain. With every example the system sees, and sometimes there are billions, the network tweaks the pattern and strength of its connections to reflect the new information, in a similar way to how neurons in the brain reinforce connections when learning something new.
The most famous deep-learning system is AlphaGo, an AI created by Google-owned DeepMind for playing the ancient Chinese board game Go. It had no strategies directly programmed into it, not even the rules of the game. But after viewing thousands of hours of human play, and then refining its technique by playing against itself, AlphaGo became the best Go player in the world.
Just like our brains, however, deep learning is deeply mysterious. Once the network is up and running, not even its creators can know what it is doing. For a long time, this black box problem was AI’s dirty little secret. But these days it is out in the open, and researchers are trying to figure out the best solution.
For at the Massachusetts Institute of Technology, the answer lies in making AIs that can explain themselves. “Transparency helps build confidence,” she says.
The first steps in that direction have already been taken. A team led by at the University of California, Berkeley, took a machine-learning system designed to identify bird species in photographs and bolted on another with the sole purpose of explaining how it arrives at its conclusions. For example, it correctly identified a picture of a white pelican because, it explained, “this bird has a white body, long neck, and large orange bill”.
Barzilay and her team have done something similar in a medical setting, working with an AI designed to predict the type of cancer a person has from their medical records. Here, the explanation doesn’t come in the form of a line of text, but in a nod to the parts of the report that led the AI to its conclusion.
Training the system wasn’t easy: the team had to manually annotate thousands of reports, which were then fed into the algorithm to teach the system to process documents itself. But for Barzilay, the efforts will be worth it if her system can convince doctors that AIs can improve diagnosis. “AI is not used very much in medicine yet, because for doctors it is a foreign tool,” she says. “They need it to explain why it makes predictions.”
But prising open the black box in this way means making trade-offs, says Gunning, who leads DARPA’s multimillion-dollar Explainable AI project. “The highest-performing system will be the least explainable,” he says. This is because machines can create far more complicated and intricate models of the world than most humans can comprehend. Ultimately, if this technology is going to be most useful when it goes beyond what humans can do, forcing it to explain itself could in many cases hold it back.
But perhaps AIs don’t have to explain themselves. “You don’t have to crack open the black box to demonstrate fairness,” says at the Alan Turing Institute in London. Instead of explaining why something happens, Russell and his colleagues use a “counterfactual” approach: they tweak the inputs to demonstrate what would have to change to alter an AI’s decision. Say you were denied a loan, for example, you might find that if your salary were £30,000 rather than £25,000, the loan would have been approved.
“What people want is to understand the decision, so that they can either challenge it or have an indication of what would need to change to alter it,” says at the Oxford Internet Institute in the UK, who worked with Russell to develop the technique.
at Carnegie Mellon University in Pittsburgh, Pennsylvania, is using a similar approach to root out biased and discriminatory AIs. He and his colleagues propose testing them by tweaking inputs such as gender or ethnicity, and seeing whether the outcome changes. For example, if two people who differed only in ethnic origin weren’t given the same likelihood of committing a crime in the future, that would indicate that the system may be biased.
Leaps of faith
The technique could form part of a certification system that every algorithm must go through before it is released, says Datta. “It can also be used on systems already in use,” he adds, so biased AI can be exposed and challenged under relevant laws.
The trouble with the counterfactual approach is that it works best when reasonably simple bits of information are used to make a decision – a few personal details, say. It is a lot trickier when there is an almost continuous stream of data to analyse, as in the case of an AI behind the wheel of a self-driving car.
But some argue that even in life-or-death scenarios, we may not always need AI to show its workings. Last year, Kilian Weinberger of Cornell University asked his audience at the Neural Information Processing Systems conference in Los Angeles to imagine they had a heart disease that required surgery. There is a 10 per cent fatality rate if a human performs the procedure, but only a 1 per cent fatality rate if a robot does it. If the surgeon makes a mistake, they can explain it: sorry, I cut the wrong artery. But the robot can’t because it uses machine-learning software. “Which one would you pick?” asked Weinberger.
Assuming the error rates are accurate, you would trust the robot, he argued. We take these leaps of faith all the time. We have been using aspirin for thousands of years, initially in the form of willow bark, but didn’t understand how it worked until the 1970s. “You don’t have to understand why a drug works to get it approved by the regulators,” said Weinberger. “You just have to show that it does.”
That said, it is not only a trust issue – it is also about legal responsibility. The death of a person hit by a self-driving car in Arizona in March has brought into sharp focus the question of how an AI can be held to account in the same way a human would be. This stuff is no longer hypothetical.
As the Cambridge Analytica story shows, the stakes are high for all of us. “Society needs to understand what’s happening, so that we can ask about what kind of world we want,” says , at the University of Cambridge.
And there’s the rub. If AI is to enhance our lives rather than dictate them according to arbitrary, incomprehensible rules in some sort of Kafkaesque scenario, we need to be clear about exactly what we expect of it.
Take back control!
For the first time in two decades, the European Union is sprucing up its data protection laws. From 25 May 2018, the General Data Protection Regulation will come into effect across the EU. Here’s what its citizens will gain.
Consent Companies will no longer get away with a check box and thousands of pages of terms and conditions. They will have to make it clear how they will use your data, and who they will sell data to, in a concise manner. You will be able to withdraw your consent at any time.
Freedom Rather than your data being tied into one platform forever, you will be able to demand that a company extracts all the data they hold about you and sends the information to another company. You will also be able to delete the original records, all free of charge.
Explanation You will have to be informed when automated decisions are made that affect your life, and you can challenge the outcomes. If things go wrong, companies will have to give you meaningful information about the decision. Some call this a right to an explanation, but it is not clear how informative the explanations will be.
This article appeared in print under the headline “Computer says “no comment””
