MoE systems differ in which tensors they move, which weights remain in memory between uses, how long buffers remain valid, and which numerical representations determine the optimizer update. We compare those choices here, including systems that use the same terms—expert parallelism, distributed optimization, FP8, or overlap—for different implementations. Table 39 summarizes four design axes: weight materialization, route representation, update precision, and overlap resource control. Buffer lifetime and replica groups are discussed in Sections 23.3 and 23.4. The comparison covers the mechanism families used in this report rather than an exhaustive survey of MoE implementations. Relative performance requires matched experiments; another system’s reported results are not performance measurements for Olmo-core. The quantitative results discussed here apply to their stated models, software, and hardware; they do not establish portability or scaling beyond those settings.
| System | Weight materialization | Route representation | Update tensor precision | Overlap resource control |
|---|---|---|---|---|
| Olmo-core DDP (this report) | Rank-local compute weights stay resident across microbatches; experts partitioned by EP | Fixed-capacity, route-indexed rowwise movement with device-side agreement inside a node; optional DeepEP v2 across nodes; no host split lists | FP32 main weights, optimizer states, and gradients are the source of truth; BF16 and MXFP8 are derived caches and payloads | Phase-sequential by default; wave and two-batch overlap measured and not adopted |
| Megatron-Core MoE | Distributed optimizer: resident local compute weights; FSDP is a separate sharded-weight option | Dispatch as a backend choice, collective and optimized, drop-and-pad or dropless, including a sync-free dropless path | Configurable; example: FP32 main weights and gradients, BF16 moments; FP32 update arithmetic | Hiding conditioned on sufficient independent work; SM carve-out with a reported GEMM cost |
| PyTorch FSDP | Full-reshard policy: gather weights before forward and backward; release gathered copies after each | Not MoE-specific | FSDP2: original-dtype parameter shards for optimizer; compute and reduction dtypes configurable | Parameter prefetch; not an MoE dispatch concern |
| DeepSpeed-MoE | Gather/retention policy not specified in cited MoE paper | MoE-specific collectives at scale | Main-weight, gradient, and moment dtypes not specified in cited MoE paper | Not its emphasis |
| MegaBlocks | Gather/retention policy not specified in cited paper | Block-sparse expert kernels keep every assignment without capacity padding | Megatron-LM mixed precision; update-tensor dtypes not specified in cited paper | Not its emphasis |
| DeepSeek-V3 and DeepEP | V3 training: ZeRO-1 retains full local compute weights; experts partitioned by EP | V3 training: optimized dispatch and combine kernels; DeepEP v2 lowers communication’s SM footprint | V3 training: FP32 master weights and accumulation gradients; BF16 AdamW moments | V3 training: communication SMs assigned explicitly; cache and compute interference acknowledged |
23.1 Conditional Compute Shifts Work to Systems
Sparse activation separates stored expert parameter size from per-token computation. Sparsely gated MoE layers established this construction (Shazeer et al. 2017); Switch Transformers simplified routing to top-1 (Fedus et al. 2022); GShard demonstrated automatic sharding of conditional computation (Lepikhin et al. 2021); and GLaM applied sparse activation to large-scale language models (Du et al. 2022).
Inactive experts still contribute parameter, optimizer, and checkpoint storage, and selected tokens must reach the ranks holding their experts. Conditional computation reduces active compute while adding expert placement, token movement, and distinct gradient-reduction and optimizer requirements.
23.2 Token and Weight Movement Costs Differ
PyTorch FSDP is a general answer to model-memory pressure. It shards parameters, gradients, and optimizer state and materializes parameters for computation (Zhao et al. 2023; PyTorch Contributors 2026e). Large dense matrix multiplications can amortize those gather and reshard operations. For routed experts, however, the stored pool may be much larger than the weights used by any token. Olmo-core’s DDP path keeps assigned weights in device memory across microbatches to avoid repeatedly materializing those expert weights.
ZeRO established the broader optimizer-state and gradient-partitioning design space (Rajbhandari et al. 2020). DeepSpeed-MoE then combined these ideas with expert parallelism and MoE-specific communication at scale (Rajbhandari et al. 2022). Olmo’s distributed optimizer belongs to this family, while its step ordering and multi-group reducer are specific to the implementation described here.
Keeping local compute weights materialized increases the persistent training-state footprint, so FSDP can remain useful for memory-constrained MoEs. EP can reduce that footprint by partitioning experts, but then selected token rows must travel to the ranks that hold them. How long full weights remain materialized and where experts are placed are separate decisions, which can in principle be composed in several ways. The current measurements do not fully isolate the cost of keeping assigned weights materialized from expert placement, so they establish neither a universal wall-time advantage nor a universal memory advantage for resident weights over full-reshard FSDP.
Megatron-Core’s parallel folding allows dense and sparse portions of a layer to use different parallel mappings. Its MoE technical report (Yan et al. 2026) describes this design in Megatron-LM (NVIDIA 2026l), together with its interactions with dispatch, expert compute, load balance, memory, distributed optimization, checkpointing, reduced precision, and overlap.
Olmo-core represents the same dense–sparse distinction with a dense mesh and an MoE mesh with EP-DP and EP-MP axes. Comparing these mappings requires following their effects on gradient reduction, optimizer state, runtime caches, and checkpoint views.
23.3 Route Representations Trade Compactness for Regularity
A routed layer must represent a data-dependent number of token assignments. One design family packs tokens and launches variable-size collectives described by per-rank counts. This represents the accepted payload compactly, but a host-facing collective or grouped-GEMM interface can make GPU route counts part of the CPU launch dependency. Another family reserves bounded capacity and keeps a GPU-side map from each accepted route to its destination. That removes the split-list dependency, but requires capacity, route metadata, agreement, and a precisely defined overflow policy.
Tutel studies adaptive MoE dispatch and parallel placement across different cluster and model configurations (Hwang et al. 2023). MegaBlocks instead uses block-sparse kernels to preserve token assignments without padding every expert to the same capacity (Gale et al. 2023). These systems illustrate that token layout, communication, and expert-kernel shape must be considered together; the collective’s name alone determines none of their costs.
Megatron-Core presents token dispatch as a first-class backend choice, including collective and optimized alternatives (Yan et al. 2026; NVIDIA 2026l). The Megatron-Core MoE report also describes a sync-free dropless path, in which device-initiated grouped GEMM and dispatch into preallocated upper-bound buffers remove the host synchronization on token counts. Olmo-core’s rowwise backend uses fixed-capacity route-indexed movement and GPU-side agreement, with communication payloads in preallocated symmetric buffers managed by the rowwise runtime. DeepEP, which Olmo-core can also select as its inter-node EP backend (Section 5.8), supplies another optimized dispatch design and, in its v2 path, explicitly reduces the SM footprint devoted to communication (Zhao, Zhou, et al. 2025).
A compact variable payload may expose a host dependency, while a bounded payload may reserve unused rows or drop overflow; an optimized GPU transport can also compete with expert compute for device resources. Comparing these representations requires holding accepted work and expert shape fixed, then accounting for host gaps, bytes, drops, capacity, and maximum-rank time.
Dynamic shape also raises a buffer-reuse question that transport APIs alone do not answer. A dispatched activation saved for Wgrad may outlive the call that moved it, especially under recomputation or a pipeline schedule. Olmo makes that dependency explicit with autograd-visible leases on symmetric payloads (Section 5.7). Keeping more dispatched activations for backward requires more simultaneous symm buffers. Reusing a buffer sooner saves memory, but is safe only after every consumer operator of its previous contents has finished.
23.4 Placement Determines Replica Groups
Once experts are partitioned, a local model part contains different kinds of replication. Dense parameters may agree over the folded dense group, while a local expert shard agrees only with copies of that shard. A system can encode that distinction in module decomposition, parallel mappings, reducer metadata, optimizer state, or some combination. The choice changes both the stored tensors and the gradient traffic for each rank.
Megatron-Core includes distributed optimization and checkpointing inside its MoE system boundary (Yan et al. 2026). Olmo-core uses a per-parameter process-group selector in its DDP reducer, carries the same parameter families into high-precision optimizer state, and exposes topology-independent checkpoint views. This path saves without assembling any full expert tensor on one GPU; in one controlled MXFP8 development run of Section 12, save-time peak memory stayed at its pre-save values (a save-side measurement only).
GShard and DeepSpeed-MoE likewise couple expert placement to the process groups that synchronize or shard model state (Lepikhin et al. 2021; Rajbhandari et al. 2022). Like Olmo-core, these systems derive their synchronization and sharding groups from the chosen parallel placement; Olmo-core differs in the particular reducer, optimizer-state, and checkpoint interfaces described here.
23.5 Conversion Costs and Communication–Compute Interference
Reduced precision lowers arithmetic, communication, or storage cost only by introducing another encoding of each model value. Megatron-Core treats reduced precision and recomputation as parts of its integrated MoE performance problem (Yan et al. 2026). In Olmo-core, FP32 main weights, optimizer states, and accumulated gradients determine the training update. BF16 compute weights and oriented MXFP8 weight caches are derived from those main weights; MXFP8 activation payloads are derived runtime representations. Conversion, cache refresh, transport, memory, and loss behavior are therefore all part of the BF16-versus-MXFP8 comparison; raw GEMM time alone does not characterize the training trade-off.
FP8 training and the later OCP microscaling formats provide the numerical and hardware context for this design (Micikevicius et al. 2022; Rouhani et al. 2023; Open Compute Project 2023), and DeepSeek-V3 is the prominent production example of FP8 training with fine-grained block scaling (DeepSeek-AI et al. 2024). The Olmo-core-specific questions are where quantization occurs, which representations cross operator or communication boundaries, and whether the subsequent GEMM or transfer saves more time than the conversion takes.
DeepSeek-V3 and Megatron-Core issue token movement beside independent expert computation, but both expose the resource budget behind the overlap: DeepSeek-V3 assigns communication SMs and discusses cache/compute interference, while Megatron-Core conditions hiding on sufficient independent work and reports a GEMM cost from its SM carve-out (DeepSeek-AI et al. 2024; Yan et al. 2026). DeepEP v2 lowers communication’s SM footprint (Zhao, Zhou, et al. 2025); fewer communication SMs do not guarantee that concurrent kernels retain standalone speed.
COMET makes the same concern explicit: coarse waves and separate-stream scheduling can reduce expert-compute efficiency, motivating finer-grained fused resource control (Zhang et al. 2025). A broader distributed-training study observes compute stretch under communication while still finding overlap beneficial on average in its own suite (Lee et al. 2025). These results depend on how resources are allocated for the measured shape, transport, and platform.
Lancet schedules MoE communication and computation jointly, while MegaScale-MoE combines several parallel and communication optimizations for large training runs (C. Jiang et al. 2024; Jin et al. 2025). Both treat overlap as a scheduling and resource-allocation problem, with explicit coordination between communication and compute.
Olmo evaluates overlap in two separate studies. A model-level two-batch overlap (TBO) study evaluates two lanes in flight. A separate same-batch EP-wave study compares wave-overlapped execution with an unsplit no-wave baseline; it lacks the wave-sequential control that would separate smaller GEMMs from concurrency (Section 5.9). The unsplit baseline is needed because an overlapped timeline can still lose to a larger, more efficient unsplit GEMM.
23.6 Expert Placement Connects Token Movement and Updates
Olmo-core DDP combines resident rank-local compute weights with route-indexed token movement and replica-specific updates. EP determines which experts a rank stores and which token rows it receives; the reducer synchronizes each expert’s gradients only with replicas of that expert. Keeping those choices consistent is what lets the resident-weight design extend from forward execution to optimizer updates and checkpoint export.