Dwarkesh Podcast ·

Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements

Breaking down 6 years of pretraining progress into data vs model improvements

Lead Source

Dwarkesh Podcast — Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements ↗

How this story grew

Coverage · 0 Discussion · 0

Sep 9Sep 10

Discussion

Rohin Govindarajan

@dwarkesh_sp @arimorcos @dorialexander @sarahookr

Related stories

South China Morning Post: China says its AI compute capacity rose 177% YoY to 2,185 eflops by the end of June, and is targeting 9,800 eflops by 2030 via ~$532B in IT infrastructure spend

The Register: OpenAI says new monitoring and security safeguards will add 20% compute overhead to monitored inference workloads; costs will not be passed on to customers

Counterpoint Research: In Q2 2026, server-led eSSDs reached 48% of NAND flash shipments as AI workloads shifted from training to inference, driving 5x YoY industry revenue growth

@fi56622380: Unlike internet's one-dimensional, flat-fee growth, AI grows across two exponentials, user penetration and tokens per user, allowing AI labs' economics to work

Bloomberg: Microsoft plans to expand its data center capacity from 12GW today to 38GW+ by 2032, with about a third of the 38GW centered on AI-specific chips

Discuss with AI

ChatGPT ↗ Claude ↗

← Back to TechSnif