Who should we listen to when deciding whether a criminal will reoffend: a sophisticated algorithm, or random people on the internet? Trick question 鈥� it turns out they both produce the same results, according to a new analysis that demonstrates the danger of handing over control of our lives to the machines.
Such predictive algorithms are already well-entrenched, informing and making crucial decisions. But for many of them, evidence they are accurate and fair is lagging behind. This needs to change.
鈥淲e have a tendency to immediately trust them because we consider them 鈥榮mart鈥� or 鈥榠ntelligent鈥�,鈥� says at the Oxford Internet Institute. But perhaps we should be more critical.
Advertisement
As a case in point, a new paper published in Science Advances this week looked at a well-established tool called (COMPAS), which is used by courts across the US to predict the likelihood of someone with a conviction reoffending. It uses 137 different features about the defendant鈥檚 case to put together its score, and is used by judges to inform sentencing decisions.
Tip of the iceberg
Despite its ubiquity, until now COMPAS had not been tested against human decision-making. The results show it is no better at predicting reoffenders than untrained people on the internet.
In the study, 400 crowdsourced participants on average correctly predicted whether someone would reoffend within two years in 67 per cent of cases, using just seven features including sex, age, and previous criminal history, but not race. In comparison, using 137 features, COMPAS was only accurate 65 per cent of the time 鈥� a statistically insignificant difference. Equivant, which makes the software, did not respond to requests for comment.
This is not the first time that COMPAS has come under criticism. In 2016, news organisation that although COMPAS does not use an individual鈥檚 race as part of its assessment, it was twice as likely to incorrectly predict a black person would reoffend than a white person.
鈥淭hough COMPAS has flaws, it鈥檚 just the tip of the iceberg,鈥� says Hany Farid at Dartmouth College in New Hampshire, co-author of the study. Predictive algorithms are frequently involved in everything from music recommendations and advertising, to university admissions and loan decisions. Police are using systems to predict where crimes are most likely to occur, and judges are using them to inform decisions about who should be granted bail or for how long someone should go to prison. For the individuals involved, these decisions are life-changing, yet the algorithms are nearly always opaque, and rarely proven to be completely accurate and unbiased.
Show your working
Part of the problem is that the machine learning algorithms behind these systems are tricky to dissect. Rather than following a simple procedure to make a decision, they are trained to pick up on patterns within thousands or millions of examples and apply these to future assessments.
Although this method can be very successful, showing that it will always produce fair and valid outcomes is very difficult. There aren鈥檛 simple steps that can be retraced to check the logic was sound, and even if there were, companies often want to keep their algorithms under lock and key, afraid of losing their competitive edge.
There are ways round this. We don鈥檛 always know how medicines work, but regulators require evidence of their efficacy, through rigorous controlled trials, before they are sold. We could do the same for algorithms 鈥� trusted independent third parties could assess and evaluate these systems to avoid trade secrets becoming public.
Humans are often biased, illogical, and unfair. Computers can remove these pitfalls from many important decisions, but they also have the potential to entrench them. Let鈥檚 not give algorithms the benefit of the doubt. A system that鈥檚 good enough should be able to stand the scrutiny.
Science Advances
