Olmo-core’s DDP stack reduces the parameter movement that slowed our early MoE baselines and removes the per-block host synchronization of the original MoE path. In the preliminary capacity sweep (one eight-GPU node, an eight-layer model, random routing, one run per point), the composed DDP and EP stack keeps throughput within a modest margin of the dense reference as the expert pool grows. Separate production-scale runs cover a 12.9-billion-parameter model on 16 GPUs through a 1.2-trillion-parameter model on 512 GPUs, with a short 2.38-trillion-parameter extension. Those headline rates use random routing and different operating configurations; they demonstrate the stack’s supported model sizes and throughput, not a controlled scaling curve or a time-to-quality result.
24.1 Reducing Costs That Repeat Across the Step
Sparse routing limits the compute per token, but it does not by itself limit weight movement, gradient traffic, or optimizer memory. Keeping rank-local model partitions resident removes repeated weight materialization; EP reduces the expert parameters held by each rank and routes token rows to them instead. The trillion-parameter recipes also use sharded optimizer state, MXFP8 activations and EP payloads, and per-layer recompute to fit in device memory. These changes affect both how fast a configuration runs and whether it fits at all.
An operation’s total cost scales with how often it runs. Routing runs for every token in every MoE layer, buffer allocation and reuse must accommodate each microbatch, and kernel launches repeat on every rank. In the original MoE path, all-to-all split sizes and grouped-GEMM group sizes both required routing counts on the CPU. Copying those counts back to the host repeatedly interrupted submission of GPU work. Device-resident metadata removes this recurring wait without changing the routing calculation.
A bandwidth budget can also rule out an optimization without an end-to-end benchmark. For the routed block analyzed in the offload study, the estimated activation production rate is about six times the theoretical host-link bandwidth. Scheduling can shift when copies occur, but cannot sustain full offload at that production rate. This is a bandwidth budget for the stated workload and link, not a claim that all forms of activation offload are ineffective.
24.2 What the Measurements Do and Do Not Establish
Several optimizations looked promising in isolation but offered little or no end-to-end benefit. Communication and expert compute can overlap on a timeline while still slowing each other. Quantizing attention projections alone gave only a modest gain, because conversion and layout work offset much of the GEMM savings; it remains in the adopted recipe, where the MLP and EP path supplies nearly all of the benefit. A weight-gradient kernel that preserves the natural MXFP8 scale layout avoids a conversion, yet in a B200 kernel-level prototype it was slower than two global dequantization kernels followed by tuned BF16 grouped GEMM. Component timings explain these outcomes; they cannot replace a measurement of the full execution path.
Even a kernel benchmark needs more than shapes and dtypes to be reproducible. In one B300 run, we measured different GEMM times with identical shapes, layouts, and precision when the operand values changed. The effect was measured only at the kernel level, but it changes how we construct and interpret microbenchmarks.
A related measurement problem arises in load balancing. The Switch-style scalar is a cross-term between soft router scores and hard expert counts. It can decrease as soft score mass shifts even when the requested routes remain imbalanced. A lower scalar therefore does not establish better load balance. Requested and kept route counts must be inspected directly; the scalar’s value, its gradient, and the resulting routing behavior are distinct parts of the analysis.
24.3 Precision, Memory, and Scheduling Interact
MXFP8 affects more than GEMM time: it reduces EP payloads and selected saved activations, slightly increases persistent memory through its two weight caches, and changes the operand layouts required by the GEMMs. Recomputing a routed block repeats dispatch and combine as well as expert compute. Concurrent dispatch and expert kernels share SMs, caches, and memory bandwidth; the measurements show the resulting slowdown but do not identify which shared resource causes it. These interactions explain why the combined recipe must be measured even when each component has already been benchmarked.
Pipeline scheduling adds a correctness constraint. Activations from several microbatches can remain live for backward, along with the communication payloads they reference. Those buffers can be reused only after the last consumer has finished, even if their addresses are fixed for graph replay (Sections 5.7 and 15).
These dependencies need to be explicit at component interfaces. A quantizer must produce a layout the transport and GEMM can consume without avoidable conversion. A buffer pool must be sized for the pipeline schedule’s maximum concurrency, not just one forward pass. Device memory, SM capacity, network bandwidth, and host submission time all constrain the same training step. We report unsuccessful optimizations as well as adopted ones because a small timing gain may not justify the additional scheduling or memory complexity.
24.4 What Carries Forward
The measured crossovers are specific to the hardware and configurations in this report. A different fabric, expert shape, or GEMM implementation can change whether overlap helps. Faster activation production would make full offload harder if host-link bandwidth stayed fixed; a different link or a more selective offload policy changes that calculation. Reusing a result requires checking those conditions, including the frequency of each cost and the maximum number of live microbatches.
The training mechanisms also offer a starting point for the learner in supervised fine-tuning, preference training, and reinforcement learning. Those workloads add different combinations of frozen models, generation, and data movement; the pretraining measurements do not establish their end-to-end performance. Routing, expert shapes, and operand values also matter for the surrounding inference systems; this report makes no claim about inference performance.
The result of this work is a training stack with substantially less repeated parameter movement, GPU-resident routing metadata, and memory policies that support trillion-parameter configurations. Its performance depends on how these mechanisms are combined. The configurations, profiles, and negative results in this report document those dependencies so that subsequent changes can be evaluated against the complete training step.
