The most sophisticated AI models in existence today have scored poorly on a new benchmark designed to measure their progress towards artificial general intelligence (AGI) – and brute-force computing power won’t be enough to improve, as evaluators are now taking into account the cost of running the model.

There are many competing definitions of AGI, but it is generally taken to refer to an AI that can perform any cognitive task that humans can do. To measure this, the ARC Prize Foundation previously launched a test of reasoning abilities called ARC-AGI-1. Last December, OpenAI announced that its o3 model had scored highly on the test, leading some to ask if the company was close to achieving AGI.

But now a new test, ARC-AGI-2, has raised the bar. It is difficult enough that no current AI system on the market can achieve more than a single-digit score out of 100 on the test, while every question has been solved by at least two humans in fewer than two attempts.

In announcing ARC-AGI-2, ARC president Greg Kamradt said the new benchmark was required to test different skills from the previous iteration. “To beat it, you must demonstrate both a high level of adaptability and high efficiency,” he wrote.

The ARC-AGI-2 benchmark differs from other AI benchmark tests in that it focuses on AI models’ abilities to complete simplistic tasks – such as replicating changes in a new image based on past examples of symbolic interpretation – rather than their ability to match world-leading PhD performances. Current models are good at “deep learning”, which ARC-AGI-1 measured, but are not as good at the seemingly simpler tasks, which require more challenging thinking and interaction, in ARC-AGI-2. OpenAI’s o3-low model, for instance, scores 75.7 per cent on ARC-AGI-1, but just 4 per cent on ARC-AGI-2.

The benchmark also adds a new dimension to measuring an AI’s capabilities, by looking at its efficiency in problem-solving, as measured by the cost required to complete a task. For example, while ARC paid its human testers $17 per task, it estimates that o3-low costs OpenAI $200 in fees for the same work.

Free newsletter

Sign up to The Daily

Sign up to newsletter

“I think the new iteration of ARC-AGI now focusing on balancing performance with efficiency is a big step towards a more realistic evaluation of AI models,” says at the University of Bath, UK. “This is a sign that we’re moving from one-dimensional evaluation tests solely focusing on performance but also considering less compute power.”

Any model that is able to pass ARC-AGI-2 would need to not just be highly competent, but also smaller and lightweight, says Imperial – with the efficiency of the model being a key component of the new benchmark. This could help address concerns that AI models are becoming more energy-intensive – sometimes to the point of wastefulness – to achieve ever-greater results.

However, not everyone is convinced that the new measure is beneficial. “The whole framing of this as it testing intelligence is not the right framing,” says at the University of Staffordshire, UK. Instead, she says these benchmarks merely assess an AI’s ability to complete a single task or set of tasks well, which is then extrapolated to mean general capabilities across a series of tasks.

Performing well on these benchmarks should not be seen as a major moment towards AGI, says Flick: “You see the media pick up that these models are passing these human-level intelligence tests, where actually they’re not; what they are doing is really just responding to a particular prompt accurately.”

And exactly what happens if or when ARC-AGI-2 is passed is another question – will we need yet another benchmark? “If they were to develop ARC-AGI-3, I’m guessing they would add another axis in the graph denoting [the] minimum number of humans – whether expert or not – it would take to solve the tasks, in addition to performance and efficiency,” says Imperial. In other words, the debate over AGI is unlikely to be settled soon.

The UK’s technology secretary, Peter Kyle, has asked ChatGPT for advice on why the adoption of artificial intelligence is so slow in the UK business community – and which podcasts he should appear on.

This week, Prime Minister Keir Starmer said that the UK government should be making far more use of AI in an effort to increase efficiency. “No person’s substantive time should be spent on a task where digital or AI can do it better, quicker and to the same high quality and standard,” .

Now, New Scientist has obtained records of Kyle’s ChatGPT use under the Freedom of Information (FOI) Act, in what is believed to be a world-first test of whether chatbot interactions are subject to such laws.

How this moment for AI will change society forever (and how it won’t)

There is no doubt that the latest advances in artificial intelligence from OpenAI, Google, Baidu and others are more impressive than what came before, but are we in just another bubble of AI hype?

These records show that Kyle asked ChatGPT to explain why the UK’s small and medium business (SMB) community has been so slow to adopt AI. ChatGPT returned a 10-point list of problems hindering adoption, including sections on “Limited Awareness and Understanding”, “Regulatory and Ethical Concerns” and “Lack of Government or Institutional Support”.

The chatbot advised Kyle: “While the UK government has launched initiatives to encourage AI adoption, many SMBs are unaware of these programs or find them difficult to navigate. Limited access to funding or incentives to de-risk AI investment can also deter adoption.” It also said, concerning regulatory and ethical concerns: “Compliance with data protection laws, such as GDPR [a data privacy law], can be a significant hurdle. SMBs may worry about legal and ethical issues associated with using AI.”

“As the Cabinet Minister responsible for AI, the Secretary of State does make use of this technology. This does not substitute comprehensive advice he routinely receives from officials,” says a spokesperson for the Department for Science, Innovation and Technology (DSIT), which Kyle leads. “The Government is using AI as a labour-saving tool – supported by clear guidance on how to quickly and safely make use of the technology.”

Free newsletter

Sign up to The Daily

Sign up to newsletter

Kyle also used the chatbot to canvas ideas for media appearances, asking: “I’m Secretary of State for science, innovation and technology in the United Kingdom. What would be the best podcasts for me to appear on to reach a wide audience that’s appropriate for my ministerial responsibilities?” ChatGPT suggested The Infinite Monkey Cage and The Naked Scientists, based on their number of listeners.

As well as seeking this advice, Kyle asked ChatGPT to define various terms relevant to his department: antimatter, quantum and digital inclusion. Two experts New Scientist spoke to said they were surprised by the quality of the responses when it came to ChatGPT’s definitions of quantum. “This is surprisingly good, in my opinion,” says at Imperial College London. “I think it’s not bad at all,” says at Heriot-Watt University in Edinburgh, UK.

New Scientist made the request for Kyle’s data following his recent , in which the politician was described as “often” using ChatGPT. He said that he used it “to try and understand the broader context where an innovation came from, the people who developed it, the organisations behind them” and that “ChatGPT is fantastically good, and where there are things that you really struggle to understand in depth, ChatGPT can be a very good tutor for it”.

DSIT initially refused New Scientist’s FOI request, stating: “Peter Kyle’s ChatGPT history includes prompts and responses made in both a personal capacity, and in an official capacity”. A refined request, for only the prompts and responses made in an official capacity, was granted.

The fact the data was provided at all is a shock, says Tim Turner, a data protection expert based in Manchester, UK, who thinks it may be the first case of chatbot interactions being released under FOI. “I’m surprised that you got them,” he says. “I would have thought they’d be keen to avoid a precedent.”

This, in turn, poses questions for governments with similar FOI laws, such as the US. For example, is ChatGPT more like an email or WhatsApp conversation – both of which have historically been covered by FOI based on past precedent – or the results of a search engine query, which traditionally have been easier for organisations to reject? Experts disagree on the answer.

“In principle, provided they could be extracted from the department’s systems, a minister’s Google search history would also be covered,” says Jon Baines at UK law firm Mishcon de Reya.

“Personally, I wouldn’t see ChatGPT as being the same as a Google search,” says , an FOI expert. That is because Google searches don’t create new information, he says. “ChatGPT, on the other hand, does ‘create’ something based on the input from the user.”

With this uncertainty, politicians might want to avoid using privately developed commercial AI tools like ChatGPT, says Turner. “It’s a real can of worms,” he says. “To cover their own backs, politicians should definitely use public tools, provided by their own departments, as if the public might end up being the audience.”