THE FINDING
● 2019 to 2025: roughly 3.24x more compute-efficiency gain from data than from model improvements.
● The two are largely independent — they do not compound with each other.
● Small models benefit most from data quality. Large models less so.
Why this is counterintuitive
The public story of AI progress is architectural — transformers, mixture-of-experts, reasoning models. Those are visible, nameable and get papers written about them.
Data work is invisible. Nobody announces a better deduplication pipeline. But if this analysis holds, that invisible work contributed more to efficiency over six years than the things everyone can name.
IT ALSO EXPLAINS WHY SMALL LABS CAN STILL COMPETE
Magic claims its pretraining recipe is more than ten times more compute-efficient than leading open-weight base models — the argument being that without large compute, efficiency is the only lever available.
If most efficiency gain comes from data rather than scale, that lever is real rather than a consolation prize.
What it means practically
| If you are... | The read |
|---|
| Fine-tuning a small model | Spend your effort on data quality. That is where the gain is concentrated at this size |
| Choosing between model sizes | A well-trained small model may close more of the gap than the parameter count suggests |
| Evaluating a lab | Ask what they did to the data, not only what they did to the architecture |
| Watching for disclosure | Almost nobody publishes their corpus. K2 Horizon is the exception this year |
That last row connects two things that are usually discussed separately. If data is where the gains are, then a lab publishing its training data is disclosing its actual advantage — which is why almost none of them do it.
The caveat
This is one analysis over one period, and the conclusion that data and model improvements are independent is the part most likely to be contested. Treat it as a well-argued position rather than a settled finding, and watch whether it replicates.
FAQ
Does this mean bigger models do not help?
No. It means that over 2019 to 2025, data improvements contributed roughly three times more compute-efficiency gain than model improvements did. Both mattered.
Why do small models benefit most?
The analysis suggests data quality gains matter less as model size increases. A small model has less capacity to compensate for a weak corpus.
How does this relate to open weights?
If data is the differentiator, publishing training data means publishing your advantage. That is why K2 Horizon releasing weights, code, data, checkpoints and logs is unusual.