Mid-training is essential for LLM reasoning, IBM study shows

For years, the basic recipe for building a capable large language model was straightforward: train a model on mountains of text, then teach it to respond in a helpful, humanlike way through reinforcement learning. At some point, an intermediate training phase was added in, with a heavy focus on math, code, and science, and the reasoning capabilities of LLMs seemed to take a giant leap.

This stage is now referred to as mid-training. Today it’s a routine, if mysterious, step in training today’s reasoning models to do things like rooting out mistakes in complex code bases, lengthy contracts, or financial statements. A new IBM study explains why mid-training so effective, in the first large-scale, systematic look at mid-training in open-source LLMs.

Through more than 500 controlled experiments, IBM researchers found that mid-training boosted overall reasoning capabilities in models of varying sizes and architectures by 3 to 4 times, while preserving knowledge gained during…