The artificial intelligence industry faces a critical challenge as OpenAI cofounder Ilya Sutskever declared at the December Neurips conference that AI has reached “peak data” - meaning all useful internet data has already been used to train AI models. This pre-training process, which powered breakthrough systems like ChatGPT, is slowing down significantly, threatening trillions of dollars in AI investments and stock market valuations.
However, Google DeepMind researchers have proposed a promising solution called test-time or inference-time compute. This innovative technique allows AI models to “think” through complex problems by breaking queries into smaller tasks, creating a chain of reasoning where each component must be solved correctly before moving forward. OpenAI’s o1 model, released in September 2024, pioneered this approach, followed quickly by Google and Chinese AI lab DeepSeek with their own reasoning models.
The breakthrough potential lies in creating a self-improvement loop. These reasoning models generate higher-quality outputs, particularly excelling at mathematics and tasks with definitive answers. These superior outputs can then be fed back as training data for new AI models, potentially circumventing the data scarcity problem. Google DeepMind researchers, including Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar, published research in August 2024 proposing this technique as a way to maintain AI progress despite data limitations.
Charlie Snell, who co-authored the research while interning at Google DeepMind, explained that if AI models can use extra inference-time compute to improve outputs, “that’s a way for it to generate better synthetic data.” This creates a valuable new training data source that could surpass existing internet content in quality.
Microsoft CEO Satya Nadella expressed optimism about this approach, describing inference-time compute as “another scaling law” that creates a feedback loop where test-time sampling generates tokens for pre-training, producing increasingly powerful models. Early evidence suggests this strategy is already being deployed - DeepSeek’s V3 model reportedly may have used outputs from OpenAI’s o1 to train its own system, achieving impressive benchmark performance and potentially becoming “the first ones to reproduce o1,” according to Snell.
While questions remain about how well test-time compute generalizes beyond clear-answer problems to subjective tasks like essay writing, 2025 will serve as a crucial testing ground for this approach’s viability in sustaining AI advancement.
Key Quotes
We’ve achieved peak data and there’ll be no more
OpenAI cofounder Ilya Sutskever made this stark declaration at the Neurips conference in December 2024, warning that the era of pre-training AI models on internet data “will unquestionably end.” This statement highlighted the existential challenge facing the AI industry’s continued growth.
In the future, we envision that the outputs of applying additional test-time compute can be distilled back into the base LLM, enabling an iterative self-improvement loop
Google DeepMind researchers wrote this in their August 2024 research paper, proposing the theoretical framework for how test-time compute could solve the peak data problem by creating a feedback loop where AI-generated outputs become training data for future models.
If you can get an AI model to use extra inference-time compute and improve its outputs, that’s a way for it to generate better synthetic data. That’s a useful new source of training data. This seems to be a promising way to get around these pretraining data bottlenecks
Charlie Snell, who co-authored the Google DeepMind research paper while interning there, explained the core insight behind test-time compute in an interview. Now back at UC Berkeley, Snell articulated how this technique could circumvent data scarcity limitations.
So you have pretraining, and then you have effectively this test-time sampling that then creates the tokens that can go back into pretraining, creating even more powerful models that then are running on your inference. That’s I think a fantastic way to increase model capability
Microsoft CEO Satya Nadella expressed enthusiasm about test-time compute on a recent video podcast, describing it as “another scaling law” and demonstrating confidence that AI progress will continue despite the peak data challenge.
Our Take
The emergence of test-time compute as a potential solution to AI’s data crisis reveals how quickly the industry adapts to existential challenges. What’s particularly striking is the convergence of major players - OpenAI, Google, and DeepSeek - all rapidly implementing similar reasoning approaches, suggesting this isn’t speculative research but a critical strategic pivot.
The self-improvement loop concept raises profound questions about AI development’s future trajectory. If models can generate superior training data than humans produce naturally, we’re approaching a threshold where AI advancement becomes increasingly decoupled from human input. This could accelerate progress unpredictably.
However, Snell’s caveat about generalization beyond clear-answer problems is crucial. Much of AI’s commercial value lies in subjective, creative, and nuanced tasks where “correctness” isn’t binary. Whether test-time compute can crack these challenges will determine if this approach truly solves the scaling problem or merely delays the inevitable plateau. The 2025 testing period will be revelatory for the industry’s long-term viability.
Why This Matters
This development represents a pivotal moment for the AI industry’s future trajectory. With an estimated trillions of dollars invested in AI development and countless businesses banking on continued model improvements, the peak data problem threatened to halt progress and potentially trigger a market correction. The test-time compute solution offers a lifeline that could sustain the AI boom.
The implications extend beyond technical capabilities. If AI models can generate training data superior to human-created internet content, we’re witnessing a fundamental shift toward AI systems that improve themselves with minimal human intervention. This self-improvement loop could accelerate AI advancement exponentially, impacting every sector from healthcare to finance.
For businesses, this means AI capabilities may continue improving despite data constraints, justifying continued investment. For workers, it signals that AI systems will likely become more capable at complex reasoning tasks. The competitive dynamics are also shifting - companies like DeepSeek demonstrating they can rapidly replicate advanced models suggests the AI race remains wide open, with potential for disruption from unexpected players challenging established leaders like OpenAI and Google.
Recommended Reading
For those interested in learning more about artificial intelligence, machine learning, and effective AI communication, here are some excellent resources: