Dense Olmo pretraining uses the FSDP-oriented path described by Team Olmo et al. (2025). The Olmo-core DDP training stack is a separate path centered on distributed data parallel (DDP) and built for MoE. It coordinates expert placement, token movement, gradient synchronization, optimizer updates, memory use, and low-precision execution across the training step.1
Sparse routing adds work throughout the training step. It adds data movement, irregular expert matrix multiplications, dynamic shapes, and CPU launch work. DDP provides the synchronization foundation; expert parallelism (EP) divides the expert parameters, pipeline parallelism (PP) divides layers, and the distributed optimizer shards the FP32 main weights and optimizer states.
Figure 5 shows the scope of this design. Configuration and train-module orchestration sit above the MoE model, parallel runtimes, and supporting systems. A low-level software, kernel, communication, and transport layer provides their common execution substrate.
Figure 5Interactive · conceptual map, not an execution order
Six layers, from configuration down to the interconnects
Highlight a layer
Low Level Support
Reading map
Choose a subsystem, or a layer above, to see what Section 2 says about it and where the report covers it.
- §4 · App. A
- Sections and appendices that cover a subsystem
- Pink dot
- Pinned subsystem; choose it again to clear
- ← → ↑ ↓
- Move between subsystems; Esc clears
Layergreen in the figure
Configuration
“Configuration and train-module orchestration sit above the MoE model, parallel runtimes, and supporting systems.”
Subsystem in this layer
Layerpurple in the figure
Train-module orchestration
“Configuration and train-module orchestration sit above the MoE model, parallel runtimes, and supporting systems.”
Subsystem in this layer
Layeryellow in the figure
MoE model
Subsystem in this layer
Layerblue in the figure
Parallel runtimes
3 subsystems in this layer
Layerorange in the figure
Supporting systems
6 subsystems in this layer
Layergray in the figure
Libraries, kernels, communication, interconnects
“A low-level software, kernel, communication, and transport layer provides their common execution substrate.”
5 subsystems in this layer
Configurationgreen in the figure
Configuration And Launch Surface
“Configuration and train-module orchestration sit above the MoE model, parallel runtimes, and supporting systems.”
Where the report covers it
- Appendix BReproducible Experiment Configurations and MetricsAppendices
Train-module orchestrationpurple in the figure
Train Module Orchestration
“Configuration and train-module orchestration sit above the MoE model, parallel runtimes, and supporting systems.”
Covered across the report
No single section is named for this subsystem.
MoE modelyellow in the figure
MoE Model Architecture
Where the report covers it
Parallel runtimesblue in the figure
Multi-Group DDP Runtime
“DDP provides the synchronization foundation”
Where the report covers it
Parallel runtimesblue in the figure
Expert Parallel Runtime
“expert parallelism (EP) divides the expert parameters”
Where the report covers it
Parallel runtimesblue in the figure
Pipeline Parallel Runtime
“pipeline parallelism (PP) divides layers”
Where the report covers it
- Section 6Pipeline ParallelismParallelization
Supporting systemsorange in the figure
Precision And FP8 Runtime
“low-precision formats are integrated with activation storage and transport rather than applied only to isolated GEMMs”
Where the report covers it
Supporting systemsorange in the figure
Optimizer And State Layout
“the distributed optimizer shards the FP32 main weights and optimizer states”
Where the report covers it
- Section 4.3.1Distributed Optimizer Shards FP32 Main Weights and Optimizer StatesParallelization
Supporting systemsorange in the figure
Activation Control
Where the report covers it
Supporting systemsorange in the figure
torch.compile and Graph Execution
Where the report covers it
Supporting systemsorange in the figure
Symmetric Memory Backend And Lifetime Management
Where the report covers it
Supporting systemsorange in the figure
Checkpointing And Export
Where the report covers it
Libraries, kernels, communication, interconnectsgray in the figure
PyTorch
“PyTorch provides the foundation, but its standard operators cannot remove every overhead introduced by sparse routing.”
Covered across the report
No single section is named for this subsystem.
Libraries, kernels, communication, interconnectsgray in the figure
Custom CUDA/C++/Triton
“Olmo-core therefore combines PyTorch with custom CUDA, C++, and Triton kernels”
Where the report covers it
Libraries, kernels, communication, interconnectsgray in the figure
CUTLASS Grouped GEMM
Where the report covers it
Libraries, kernels, communication, interconnectsgray in the figure
NCCL / NVSHMEM
Where the report covers it
Libraries, kernels, communication, interconnectsgray in the figure
NVLink / InfiniBand / PCIe
“NVLink within a node, InfiniBand between nodes, and PCIe between each GPU and its host.”
Where the report covers it
Original report figure
PyTorch provides the foundation, but its standard operators cannot remove every overhead introduced by sparse routing. Olmo-core therefore combines PyTorch with custom CUDA, C++, and Triton kernels (Tillet et al. 2019); NCCL and NVSHMEM communication (NVIDIA 2026n, 2026s); and CUTLASS grouped GEMM (NVIDIA 2026i). These components run over the hardware links beneath them: NVLink within a node, InfiniBand between nodes, and PCIe between each GPU and its host. InfiniBand GPUDirect Async (IBGDA) lets GPUs initiate InfiniBand transfers directly. We design token layouts together with communication and expert computation, and low-precision formats are integrated with activation storage and transport rather than applied only to isolated GEMMs. A fast kernel is insufficient if it requires an extra permutation, a device-to-host synchronization, or a gap between launches elsewhere in the step.
The preliminary capacity sweep in Figure 4 motivates this design. One composed DDP+EP8 configuration retains most of its throughput as stored expert parameter size grows, whereas the FSDP-oriented baseline slows. The result does not isolate the contribution of DDP, EP, or any particular kernel, and it measures throughput rather than time to quality.
The same design extends to trillion-parameter configurations. Because EP, PP, and the distributed optimizer each divide a different part of the model, the amount stored by each GPU stays bounded as the global model grows. We have run 296B-total-parameter BF16 and 1.200T-total-parameter MXFP8 configurations on 512 B300 GPUs, and an optional DeepEP v2 backend (Zhao, Zhou, et al. 2025) extends the demonstrated envelope to a 2.380T-total-parameter configuration. The 2.380T figure comes from a short nine-step run and describes reachable capacity, not a sustained-throughput or time-to-quality result at that scale; Section 14 reports production throughput separately.
Measured throughput spans scales. Section 14 reports achieved useful-model TFLOP/s/GPU for five model scales, which we name Tiny, Small, Medium, Large, and Ultra in order of size, together with the active and total parameter counts, batch, EP/PP topology, precision, and recomputation policy needed to interpret each number. The selected headline values use random routing as a systems reference and are not a controlled scaling curve or a time-to-quality comparison.
Part II covers model placement, DDP synchronization, EP, PP, and their physical-network mapping. Part III covers synchronization-free execution, grouped GEMM, routing, MXFP8, topology-agnostic checkpointing, and recompute. The later findings examine training recipes and interactions observed in the complete stack.
Selected achieved throughput
B300 NVL8
| Size | Active @ total parameters | Activation | Experts | Configuration | Useful-model TFLOP/s/GPU | MFU |
|---|---|---|---|---|---|---|
| Tiny | 1.59B @ 12.9B | 12.3% | 64 | BF16 EP1 PP1 | 903 | 40.1% |
| Small | 4.29B @ 40.9B | 10.5% | 64 | BF16 EP8 PP1 | 841 | 37.4% |
| Medium | 7.38B @ 137B | 5.38% | 128 | BF16 EP8 PP4 | 843 | 37.5% |
| Large | 15.1B @ 296B | 5.12% | 128 | BF16 EP8 PP4 | 768 | 34.1% |
| Ultra | 58.4B @ 1.20T | 4.86% | 128 | MXFP8 EP8 PP8 | 858 | 38.1% |
The percentage under each size is the parameter activation ratio \(\alpha_P\) (active over total parameters). MFU uses the B300 dense BF16 peak (2,250 TFLOP/s/GPU), also for MXFP8; it is not FP8-peak utilization (Equation 95). Full configurations and evidence limits: Section 14.
Olmo-core continues to support MoE through its earlier FSDP-oriented Transformer stack. We call that implementation the legacy MoE path only to distinguish it from the Olmo-core DDP stack presented here.↩︎