Google announces astonishing speed of new AI training technology

CoinDesk Report:
Researchers at Google’s DeepMind have announced a new method to accelerate artificial intelligence training, significantly reducing the computational resources and time required to complete this task. According to a recent research paper, this new method for the typical energy-intensive process can make AI development faster and cheaper, which could be good news for the environment.
The study stated, “Our method – Jointly Enhanced Sample Selection for Multi-Modal Contrastive Learning (JEST) – requires 13 times fewer iterations and 10 times less computation than state-of-the-art models.”
The AI industry is known for its high energy consumption. Large-scale AI systems like ChatGPT require a significant amount of processing power, which in turn requires a lot of energy and water to cool these systems. For example, due to the increase in AI computing demands, Microsoft’s water usage surged by 34% from 2021 to 2022, with ChatGPT being accused of consuming nearly half a liter of water every 5 to 50 prompts.
The International Energy Agency (IEA) predicts that electricity consumption in data centers will double from 2022 to 2026, comparing the power demand of AI with the often criticized energy situation of cryptocurrency mining.
However, methods like JEST can provide a solution. Google stated that by optimizing data selection for AI training, JEST can significantly reduce the number of iterations and required computational power, thereby lowering overall energy consumption. This approach aligns with efforts to improve AI technology efficiency and mitigate its environmental impact.
If proven effective on a large scale, trainers of AI models will only need a small fraction of the power required to train their models. This means they can create more powerful AI tools with the resources currently used and consume fewer resources to develop new models.
JEST Working Principle
JEST maximizes the learnability of AI models by selecting complementary batches of data. Unlike the traditional method of selecting individual examples, the algorithm considers the composition of the entire set.
For example, if you are learning multiple languages, instead of learning English, German, and Norwegian separately, you might find it more effective to learn them together in a way that one knowledge supports another, perhaps in order of difficulty.
Google has adopted a similar approach and achieved success.
The researchers stated in the paper, “We demonstrate that jointly selecting batches of data is more effective than independently selecting examples.”
To do this, Google researchers used “Multi-Modal Contrastive Learning,” the JEST process, to identify dependencies between data points. This method enhances the speed and efficiency of AI training while requiring less computational power.
Google pointed out that the key to this method is starting from a pre-trained reference model to guide the data selection process. This technology allows the model to focus on high-quality, carefully curated datasets, further optimizing training efficiency.
The paper explained, “In addition to the overall quality of independently considered data points, the quality of batches is also a function of their composition.”
Experiments in the study showed steady performance improvements across various benchmark tests. For instance, training on the general WebLI dataset using JEST demonstrated a significant improvement in learning speed and resource efficiency.
The researchers also found that the algorithm quickly identified highly learnable sub-batches, accelerating the training process by focusing on specific data segments that “match” together. This technique, known as “data quality bootstrapping,” prioritizes quality over quantity, proving to be more suitable for AI training.
The paper stated, “A reference model trained on a small curated dataset can effectively guide a much larger curated dataset, allowing for training a model of significantly higher quality than the reference model on many downstream tasks.”
Edited by Ryan Ozawa