In the late 18th century, mathematician and physicist Joseph-Louis Lagrange made a shocking discovery: his star student, a Monsieur Le Blanc, was actually a woman.

Lagrange taught at France’s École Polytechnique, which allowed students to��receive lecture notes and submit work without attending the university in person. This was particularly beneficial to Sophie Germain, who longed to study mathematics despite objections from her parents. She took up the identity of a lapsed student and might have got away with it, but Lagrange noticed the vast and sudden improvement in Le Blanc’s work and demanded to meet in person.

Germain isn’t the only person to note how the name we use changes the way we are perceived. As psychologist Keon West explains here, experiments using identical job applications show that those with names assumed to belong to a Black person are��less successful than those with names thought to belong to a white person.

In recent years, many organisations have adopted measures to combat the biases that lead to these outcomes, such as��removing names from job applications. These measures fall under the umbrella of diversity, equity and inclusion (DEI). Now, however, US President Donald Trump has��ordered government agencies to dismantle��DEI programmes, promising in his 20 January inauguration speech that society would be “merit-based”.

Trump's approach to diversity, equity and inclusion is unlikely to produce a meritocracy

Some DEI initiatives have firmer grounding in evidence than others. As the résumé test demonstrates, merit��alone isn’t enough to overcome people’s biases, and a number of studies��have shown that for disadvantaged groups. On the other hand, unconscious bias training, in the form of one-off sessions that aim to make employees aware of snap judgements they may make about people based on their race and gender, has .

Trump’s heavy-handed approach to DEI, based in ideology rather than evidence,��is unlikely to produce his desired outcome of a meritocracy. Instead of developing an organisation where the best people are encouraged to flourish, the current efforts seem to be fostering a culture of fear, with being warned of “adverse consequences” for failing to identify and end DEI work.

Thankfully for Germain, there were no��such consequences. Lagrange accepted��her for who she was and championed her mathematical development. Despite this, she still used the Le Blanc pseudonym in some correspondence, most notably with mathematician Carl Friedrich Gauss, who, on discovery of her true identity, wrote that she had “nobler courage, quite extraordinary talents, and superior genius”. If we want more Germains to��flourish, we must acknowledge and address the barriers they face, not pretend��that they don’t exist.

��

Commercial AI chatbots demonstrate racial prejudice toward speakers of African American English – despite expressing superficially positive sentiments toward African Americans. This hidden bias could influence AI decisions about a person’s employability and criminality.

“We discover a form of covert racism in [large language models] that is triggered by dialect features alone, with massive harms for affected groups,” said at the Allen Institute for AI, a non-profit research organisation in Washington state, in a . “For example, GPT-4 is more likely to suggest that defendants be sentenced to death when they speak African American English.”

Hofmann and his colleagues discovered such covert prejudice in a dozen versions of large language models, including OpenAI’s GPT-4 and GPT-3.5, that power commercial chatbots already used by hundreds of millions of people. OpenAI did not respond to requests for comment.

The researchers first fed the AIs text in the style of African American English or Standard American English, then asked the models to comment on the texts’ authors. The models characterised African American English speakers using terms associated with negative stereotypes. In the case of GPT-4, it described them as “suspicious”, “aggressive”, “loud”, “rude” and “ignorant”.

When asked to comment on African Americans in general, however, the language models generally used more positive terms such as “passionate”, “intelligent”, “ambitious”, “artistic” and “brilliant.” This suggests the models’ racial prejudice is typically concealed beneath what the researchers describe as a superficial display of positive sentiment.

The researchers also showed how covert prejudice influenced chatbot judgements of people in hypothetical scenarios. When asked to match African American English speakers with jobs, the AIs were less likely to associate them with any employment, compared with Standard American English speakers. When the AIs did match them with jobs, they tended to assign roles that do not require university degrees or were related to music and entertainment. The AIs were also more likely to convict African American English speakers accused of unspecified crimes, and to assign the death penalty to African American English speakers convicted of first-degree murder.

Free newsletter

Sign up to The Weekly

Sign up to newsletter

The researchers even showed that the larger AI systems demonstrated more covert prejudice against African American English speakers than the smaller models did. That echoes previous research showing how bigger AI training datasets can produce even more racist outputs.

The experiments raise serious questions about the effectiveness of AI safety training, where large language models receive human feedback to refine their responses and remove problems like bias. Such training may superficially reduce overt signs of racial prejudice without eliminating “covert biases when identity terms are not mentioned”, says at Brown University in Rhode Island, who was not involved in the study. “It uncovers the limitations of current safety evaluation of large language models before their public release by the companies,” he says.