We train the target MoE architecture from the beginning. This is a recipe choice for the long training horizon considered in this report, not a general conclusion that upcycling is ineffective. In sparse upcycling, a pretrained dense checkpoint is converted into an MoE, typically by copying or mapping its feed-forward weights into multiple experts, initializing a router, and then continuing training (Komatsuzaki et al. 2023). This can provide a useful head start when a suitable dense checkpoint already exists. When dense pretraining and sparse continuation share one fixed compute budget, however, the dense prefix also consumes tokens that could have trained the target sparse model directly.
The published OLMoE study tested this trade-off. It converted an OLMo-1B checkpoint trained for 2T tokens into a top-2-of-8 MoE, then compared continued training with a sparse model trained from scratch. The upcycled model was better early in the continuation, but the scratch model caught up after roughly 500B tokens of sparse training and began to outperform it near 600B tokens (Muennighoff et al. 2024). That crossover matters for our setting: like the 5T-token plan of the published OLMoE recipe, our training horizon is far longer than the roughly 500B tokens over which the upcycled model led. A dense checkpoint was therefore not a compelling prerequisite for the training run.
Upcycling also couples the target model to the source architecture and its initialization. In the OLMoE study, for example, the source checkpoint lacked query-key normalization and used normal initialization, and the authors report that both hurt stability. More generally, copying one dense multilayer perceptron into several routed experts starts those experts from the same function; the router and subsequent gradients must then create specialization. Olmo-core includes prototype conversion paths for copying the dense MLP into routed or shared branches and for the virtual-group initialization that He et al. (2024) introduce for upcycling into fine-grained experts, but changing the mapping does not remove the underlying budget decision. For a new model whose sparse architecture and training recipe are known in advance, direct sparse training lets the router and independently initialized experts adapt together over the entire run.
This decision is conditional. Drop-Upcycling shows that partially reinitializing copied experts can improve specialization and long-run learning relative to direct copying (Nakamura et al. 2025). Scaling-law results likewise show that the preferred path depends on how much compute has already been spent on the dense checkpoint and how much sparse continuation remains (Liew et al. 2025). Upcycling may therefore be attractive when a compatible dense checkpoint already exists and the continuation is short. For our long training horizon and target architecture, the available evidence did not justify making it the default.