Exams aren’t just for humans VCG via Getty Images
The results are in. An artificial intelligence has gone to the top of its class after passing an English exam. Though it can’t beat more able human students, it achieved the best mark yet for a machine.
Hai Zhao at Shanghai Jiao Tong University in China and his colleagues trained their AI on more than 25,000 English reading comprehension tests.
Each contained a 200 to 300-word story followed by a series of related multiple-choice questions. The tests were sourced from English proficiency exams aimed at Chinese students aged from 12 to 18 years.
Advertisement
While some answers could be directly found in the text, over half of them required a degree of reasoning. For example, one of the questions asked you to choose the best headline for a story from four options.
After the training, the AI sat a finalexam consisting of 1400tests ithadn’t seen before. It achieved an overall score of 74per cent, better than .
Zhao’s AI uses a system that can identify parts of the story that are relevant to the question, then selects the answer that is most similar in meaning and logic.
The nextbest was a system made byTencent, aleading Chinese technology firm, which scored 72percent on the same exam. Tencent’s AI learned to compare the information carried by each option and use their differences as cues to look for evidence in the text.
Despite topping the leader board, Zhao is determined to improve his system’s abilities. “What our AI got is very average, a C+ at most,” he says. “For students who want to get into good universities in China, they will aim for 90per cent.”
To increase its score, the team will try to modify the AI so that it can understand information embedded in sentence structure and feed it with more data to expand its vocabulary.
Understanding human language is a major headache for AI, as it is often imprecise and involves hidden contextual and societal clues that machines struggle to pick up on.
It is unclear what rules AIs follow when they learn our languages, saysGuokun Lai at Carnegie Mellon University in Pennsylvania, who originally collated the tests in 2017 forAI research. “They seem to be ableto [understand our logic] afterreading tonnes of sentences andstories.”
Reference: arXiv,arxiv.org/abs/1901.09381
Topics:
