Skip to report text

Findings and OutlookAcknowledgments

Contents

Supercharging Olmo-core for Efficient and Scalable MoE Training

Cover and abstract
About the report
Statement of AI Assistance
Speed-Read (With Video)
Part IIntroduction01–02
Why the Dense Training Assumptions Break
Olmo-core DDP Training Stack
Part IIParallelization03–07
Parallelism Methods Divide Different Work and Storage
Distributed Data Parallel
Expert Parallelism
Pipeline Parallelism
Mapping Parallelism to Hardware
Part IIIEfficient Training08–15
Synchronization-Free Training
Grouped GEMM
Routing, Capacity, and Load Balancing
MXFP8 Training
Topology-Agnostic Checkpoint Saving and Loading
Activation Recompute
Production Training Throughput
Optimization Attempts and Lessons
Part IVFindings and Outlook16–24
Learning Rate Scaling with Global Batch Size
Scaling Expert Learning Rate with Sparsity Did Not Help
Why We Do Not Upcycle
GEMM Timing Depends on Operand Values
One Shared Expert Is All You Need
Merging Routed and Shared Expert Activations
Beyond Pretraining
Related MoE Systems Make Different Trade-Offs
Conclusion
Acknowledgments
Appendices
Kernel Interfaces and Profiling Reference
Reproducible Experiment Configurations and Metrics
Glossary of Parallelism, Routing, and Training Storage
System Method Trade-Off Matrix
Expert-Merge Normalization in Public MoE Models
Context Parallelism and MoE Parallel Folding
Author Contributions
Reference material
References

Findings and Outlook

Acknowledgments

1 min read

About the report
Statement of AI Assistance
Speed-Read (With Video)
Part IIntroduction
01Why the Dense Training Assumptions Break
02Olmo-core DDP Training Stack
Part IIParallelization
03Parallelism Methods Divide Different Work and Storage
04Distributed Data Parallel
05Expert Parallelism
06Pipeline Parallelism
07Mapping Parallelism to Hardware
Part IIIEfficient Training
08Synchronization-Free Training
09Grouped GEMM
10Routing, Capacity, and Load Balancing
11MXFP8 Training
12Topology-Agnostic Checkpoint Saving and Loading
13Activation Recompute
14Production Training Throughput
15Optimization Attempts and Lessons
Part IVFindings and Outlook
16Learning Rate Scaling with Global Batch Size
17Scaling Expert Learning Rate with Sparsity Did Not Help
18Why We Do Not Upcycle
19GEMM Timing Depends on Operand Values
20One Shared Expert Is All You Need
21Merging Routed and Shared Expert Activations
22Beyond Pretraining
23Related MoE Systems Make Different Trade-Offs
24Conclusion
Acknowledgments
Appendices
AKernel Interfaces and Profiling Reference
BReproducible Experiment Configurations and Metrics
CGlossary of Parallelism, Routing, and Training Storage
DSystem Method Trade-Off Matrix
EExpert-Merge Normalization in Public MoE Models
FContext Parallelism and MoE Parallel Folding
GAuthor Contributions
Reference material
§References

This material is based upon work supported by the National Science Foundation under Award No. 2413244.

← Previous · 24Conclusion Next · A →Kernel Interfaces and Profiling Reference

§

Supercharging Olmo-core for Efficient and Scalable MoE Training

October 2026 · Read the original PDF

Search the full report

Search words or phrases

Search across sections, appendices, and references.

Enter a term to find passages. Esc closes search.