The supply of high-quality language data used to train machine-learning artificial intelligence models may run out in three years, leading AI advancement to stagnate.
Machine learning powers AI programs like text-prompted image generator Midjourney and OpenAI鈥檚 chat-based text generator ChatGPT. Such models train on vast reams of human-created data from the internet to learn, for instance, when asked to draw a banana that it should be yellow or green and curved.
Now, Pablo Villalobos at , a collection of researchers studying the development of AI, and his colleagues have analysed how quickly models have churned through existing data. They believe that high-quality language data used to train models like ChatGPT will run out by 2026 and the advance in collective knowledge of AI could come to a halt soon after.
Advertisement
鈥淭hat language is only produced by humans, and higher-quality data like books or scientific papers are costly to generate,鈥� says Villalobos. 鈥淚t鈥檚 economically expensive to produce it.鈥�
The researchers believe there will still be plenty of lower-quality text on which to train machine-learning models, such as blog posts and website text, for several decades to come. However, using lower-quality training data means the gains in knowledge will become more incremental.
鈥淎 guiding assumption of contemporary AI is scale,鈥� says at the London School of Economics. 鈥淢achine learning finds sophisticated patterns in data, and, so far, by having more and more of it, the results have become better.鈥�
Until now, the language data sets used by AI models for training have grown in size by around 50 per cent each year, while the total stock of language data available to train on is only growing by 7 per cent a year, say the researchers, and so won鈥檛 be able to keep up with demand.
High-quality language data is produced even more slowly and so will run out faster. 鈥淲e assume that there will be no big increases in data efficiency, and also that there鈥檚 no widespread use of self-training on self-generated data,鈥� says Villalobos. 鈥淚f that鈥檚 the case, then it seems likely that we will see a slowdown in progress over the next decades as we exhaust this training data.鈥�
OpenAI didn鈥檛 respond to a request for comment about the potential shortage of new training data.
Image data used to train generators such as Midjourney will last longer, Villalobos reckons, but the availability of both types could theoretically be extended by training models on data generated by AIs rather than humans. 鈥淚t鈥檚 probably one of the most promising ways to circumvent this problem,鈥� he says.
鈥淚n many machine-learning training techniques, data is routinely modified to get more mileage out of an existing data set,鈥� says independent researcher .
Such augmentation can include creating data sets by translating text data into a different language and then back again or by replacing words in the original text with synonyms.
Yet using modified or AI-generated data sets comes with risks: they often include errors or biases, which can be compounded in the same way that inbreeding can affect future generations鈥� genes.
However, Hundt doesn鈥檛 think Epoch鈥檚 findings are too alarming. 鈥淒ata limitations are routine in machine learning,鈥� he says, and the projection for running out of data will mainly affect just a few methods that use particularly high-quality data.
Reference:
