Smaller models, larger results.
What we teach becomes what we build. We study which data a model needs, in what mixture, and in what order.
We could count the tokens. Not which ones deserved to be there.
We are a small group of researchers who worked on foundation models at large AI labs.
A training run had a deadline. More data was ready to go in. There was time to increase the volume, but rarely enough time to understand the mixture: what was duplicated, what was unreliable, what was overrepresented, and what the model actually needed next.
Eventually, we wanted to give that question our full attention. So we left, and started this lab.
What it encounters matters. So does when.
We think about training a model a little like raising a child: what it encounters, how often, and what it learns to trust.
A clean beginning means knowing where the data comes from, why it belongs, and what happens when we remove it. It leaves room for difficult examples, conflicting evidence, and the disorder of the real world.
How little is enough?
Everything that was ready to go in. The volume we could report; the mixture we hadn’t yet understood.
Remove what is duplicated.
Repetition looks like evidence. Often it is the same lesson, taught again.
Set aside what is unreliable. Rebalance what is overrepresented.
A model learns what to trust from what it is shown. Some sources crowd out others simply by being large.
Keep what the model needs next.
Then evaluate on tasks it has not seen. Sometimes more data will be the right answer. We want the experiment to tell us when.
Small, controlled training runs.
We compare mixtures, remove redundant material, investigate failures, and evaluate on tasks the model has not seen.
Our ambition is to build capable models with as little training data as we can—and make the reasons they work easier to understand.
Time a release schedule rarely leaves.
These questions need experiments whose most useful result might be:
- Left outA dataset we decide not to use
- Made lighterA training recipe that becomes smaller
- UnlearnedAn assumption we stop believing
If we learn which data earns its place, each run becomes more informative and each iteration less expensive—giving smaller teams more room to experiment, and a clearer account of how a model acquired its abilities.
We do not yet know how far this approach can take us. We started this lab to find out.