You could have trained the Kimi K1.5
The internet was a one-time subsidy.
I've been thinking about this a lot, and I was recently reminded of John Schulman's point that many breakthroughs are surprisingly simple at their core.
So to challenge myself, I decided to look at some older papers and demystify how these models are trained.
Today I'm looking into a lesser-known model, Kimi k1.5, which was released on the same day as DeepSeek R1 on January 20, 2025:
https://arxiv.org/pdf/2501.12599
Despite being one of the earlier multimodal reasoning models, Kimi k1.5 was largely overshadowed by DeepSeek's breakthrough in cold-start reinforcement learning.
The Recipe
A key theme driving this paper was data scarcity and the resulting shift toward reinforcement learning. The paper itself opens with:
Language model pretraining with next-token prediction has proved effective for scaling compute but is limited to the amount of available training data.
Intentionally steering away from more complicated techniques like value functions and process reward models, Moonshot opted for better data, environment selection, and giving the model more room to think and grow.
What this looks like in practice is:
- A good set of problems
- Reliable environments and evaluations
- Enough room for a model to think
- Repetition, repetition, repetition
None of these ideas is particularly difficult on its own. The tricky part was making all four work together at scale.
Overthinking
After reinforcement-learning training, a model's answers tend to become much longer.
There's a fairly intuitive reason for this. Reinforcement learning is not that different from being quizzed in class and seeing a problem set for the first time.
From my experience, there are normally two important moments:
- Through arcane methods unknown to man, you somehow arrive at the right answer. Congratulations! Reward +1. Problem set done. Now you can go play games.
- Your friend asks you how to solve the problem. Your explanation is long and sparse because, deep down, you're not really sure how you got there, but you just know that you're right.
Often, buried somewhere in that long response, there is a kernel of reasoning that explains why the answer is correct. The explanation itself could probably have been 90% shorter.
A representative MATH500 run on a smaller internal long-CoT model: accuracy generally improves during reinforcement learning, while response length rises alongside it. Source: Kimi k1.5 technical report, Figure 5.
What you know is that you'll need more practice and will have to dig deeper. This idea is very close to the training process Moonshot used to compress long chains of thought into shorter ones.
Long2Short
So, how did Moonshot cut down on this?
There were a few approaches.
1. Model Merging
By taking a checkpoint of the model that reached the answer quickly and combining it with a version that reached the answer slowly, you can produce a model that tends to generalize better.
Intuitively, the best answer lies somewhere between extremely short and extremely deep.
A good question is: Why not simply use the model that reached the answer faster, with less thinking?
The reason is that the goal is generalization, to uncover and preserve the principle behind the correct answer.
By using only the shorter version, you lose some of the deep-reasoning priors that made the longer version effective. This hybrid approach gets you closer to the short model's efficiency while retaining more of the long model's reasoning.
The hybrid is also easy to produce. You simply average the weights of the two models.
2. Shortest Rejection Sampling
Models tend to generate correct answers with large differences in response length.
Based on this observation, the researchers selected the shortest correct answers and reused them for supervised fine-tuning, essentially direct quizzing.
This is similar to how teachers improve over time. They repeatedly teach students the same subject and eventually discover the shortest correct way to explain it.
I suspect this is also why most past university exam questions look similar. The core concepts behind them don't change.
3. Direct Preference Optimization
Direct Preference Optimization, or DPO, is a method for training a model to prefer one answer over another without running a full reinforcement-learning process.
So, how do you teach preferences?
The answer is very similar to the previous approach: examples.
You can think of it like grading a student. For each question, you collect several answers, keep the shortest correct one as the gold standard, and mark down the longer ones. You then quiz the student on these pairs until they reliably choose the short, correct answer over the longer one.
These positive and negative pairs are what DPO trains on, gradually instilling a preference for the shortest correct response over longer but still correct alternatives.
4. Long2Short Reinforcement Learning
After standard reinforcement-learning training, the researchers selected a checkpoint that balanced performance and token efficiency.
They then put it through a separate Long2Short RL phase with stronger length penalties and a much shorter maximum response length.
The final model isn't just capable of long reasoning. It can also deliver those insights within a tight token budget.
Moonshot's Long2Short methods produced several different balances between response length and accuracy. The orange Kimi variants generally achieved higher accuracy for the number of tokens they used than the competing short models. Source: Kimi k1.5 technical report, Figure 7.
Data Mix
It's clear from the data mix that, even in early 2025, Moonshot was positioning itself for more agentic use cases than other labs were.
It's important to remember that, at this point, language models were largely confined to the chatbox. AutoGPT had appeared in 2023 as a self-prompting and acting harness, but it didn't work particularly well and eventually faded into the background.
"Agents" still seemed far away.
What's even more interesting is that, at the time, there was a perception within AI circles that Moonshot was falling behind on language models.
The Kimi k1.5 technical report was Moonshot's first public model paper and an answer to that perception.
A few things stand out in the training mix.
Vision Data
There was a strong focus on developing Kimi k1.5's real-world visual-reasoning capabilities.
The data mix included science questions across grade levels that required graphical reasoning and comprehension, along with tasks involving complex charts and other types of visual data.
Location-guessing tasks were included as well. That initially seemed a little confusing, but they may have been intended to help the model generalize across real-world visual-reasoning problems.
In this respect, I think Moonshot was quite ahead of its time.
Kimi k1.5's long-CoT model scored 74.9 on MathVista and 70.0 on MMMU, making it competitive across both visual-reasoning benchmarks. Source: Moonshot AI's Kimi k1.5 benchmark results.
At that point, I could imagine models needing to understand charts or interact with computers. It was much harder to imagine them needing broader, higher-level graphical reasoning beyond those use cases.
A Lack of Coding Data?
The only surprising part was Kimi's approach to coding data.
The paper specifically states:
Since test cases are not available for many coding problems from the web, we design a method to automatically generate test cases that serve as a reward to train our model with RL.
I found this surprising because GitHub contains an enormous amount of code and test data.
That said, repository tests are not necessarily clean, self-contained test cases that can verify solutions to standalone coding problems. Converting them into reliable RL rewards is its own data-engineering problem.
¯\_(ツ)_/¯
Given the timeframe, however, Claude Code hadn't been released, and Cursor wasn't very good yet.
Not finding a way to fully leverage GitHub for Kimi k1.5 therefore didn't matter much in the grand scheme of things.
Conclusion
The Kimi k1.5 paper was fascinating to read. None of the individual ideas were particularly mysterious, and there is a certain kind of beauty in that.
What also stood out was Moonshot's discipline in searching for simple, effective ways to scale reinforcement learning.
To me, the focus on vision data is where the paper feels prescient. Before working with models through terminals had become commonplace, Moonshot was already investing heavily in vision.
I think they were anticipating a much more agentic future, beyond chatboxes and maybe even beyond terminals.
If you enjoyed reading this, follow me on Twitter for more. My DMs are also open if you'd like to chat about papers, and other ideas.