When the same distributed MoE learner is used for post-training, parameter placement and gradient synchronization are still needed for a model whose total parameter size exceeds one device’s memory, but supervision, additional model copies, and rollout generation change the workload. We map those requirements onto the sparse learner and identify the systems needed around it. This section is a design mapping: we report no integrated MoE post-training run, throughput, or quality result.
Ai2’s open-instruct codebase provides implementations of several post-training workloads, while the Tülu 3 report documents an earlier actor-critic reinforcement-learning recipe (Ai2 2026; Lambert et al. 2024). This section considers four workloads: supervised fine-tuning (SFT), Direct Preference Optimization (DPO), Group Relative Policy Optimization (GRPO) (Shao et al. 2024), and Proximal Policy Optimization (PPO) with a critic (Schulman et al. 2017). The open-instruct implementations use different backends: cached-reference DPO uses Accelerate and one GRPO path uses Olmo-core HSDP, while open-instruct’s SFT support includes OLMoE models. Reusing a post-training objective does not require copying its original distributed backend. Table 38 decomposes each workload into its components and marks what this report’s mechanisms could contribute to each. Every workload contains a learner. Preference and reinforcement-learning workloads add reference policies or other model copies, and reinforcement learning also needs a rollout inference system.
| Component in workload | ||||||
|---|---|---|---|---|---|---|
| Component | SFT | DPO | GRPO | PPO | Transfer | What transfers |
| Policy learner: the trained MoE | ✓ | ✓ | ✓ | ✓ | Learner mechanisms | The mechanisms in Parts II and III apply to the learner: distributed data parallel (DDP) synchronization and the sharded optimizer (Section 4), expert parallel (EP) placement and sync-free rowwise movement (Section 5), pipeline parallel (PP) layer placement (Section 6), grouped GEMM (Section 9), MXFP8 (Section 11), recompute (Section 13), and topology-agnostic checkpoints (Section 12). Response-only SFT changes the learner’s label tensor and loss denominator while retaining these mechanisms. |
| Reference policy: frozen, forward only | – | ✓ | ✓ | ✓ | Forward path only | The forward path only: sync-free dispatch, grouped GEMM, and the MXFP8 forward caches. DPO can cache reference log-probabilities before updates, as open-instruct does, avoiding a separate resident reference model. A resident reference adds parameter storage and forward activations, but no gradient buffers or optimizer states. |
| Critic: trained value model | – | – | – | ✓ | Second learner | A trained value model. If it uses the same MoE backbone as the policy, the learner mechanisms apply to both; its actual architecture determines the added memory. GRPO removes this model by using group-relative advantages. |
| Reward: verifier or reward model | – | – | \(\circ\) | \(\circ\) | None needed, or forward path only | A CPU-based checker needs no learner GPU, but verification cost depends on the task and execution environment. A learned reward model adds a separate forward-only workload. |
| Rollout generation: inference engine | – | – | \(\circ\) | \(\circ\) | Insight only, not code | An inference system outside this report’s measurements. At small decode batches, rows per expert can fall below the grouped-GEMM crossover (Section 9); batching and routing determine that count. Host-visible route metadata (Section 8), load imbalance (Section 10), and operand-dependent GEMM time (Section 19) are relevant costs, but the training measurements do not quantify them in a serving engine. |
| Weight transfer: learner to inference | – | – | \(\circ\) | \(\circ\) | Substrate only | The topology-agnostic checkpoints of Section 12 are the substrate, since they already describe every saved chunk by its place in the global tensor, independent of the EP and PP layout that produced it; their latency inside a rollout loop is not measured here. |
22.1 Response-Only Supervision Uses Label Masking
Supervised fine-tuning (Wei et al. 2022; Ouyang et al. 2022) trains on supervised input–target examples, such as instructions paired with demonstrations.
Here we consider a response-only objective: the prompt conditions the prediction, while only response tokens contribute direct loss terms. Olmo’s label-mask path can express this choice by marking prompt labels as ignored. The DDP train module excludes those labels from the loss denominator and identifies them through label_ignore_index (default \(-100\)); it derives per-batch labels when the data loader does not supply them. A supervised batch therefore differs from a pretraining batch in the contents of the label tensor and in the resulting denominator, not in the forward path, the routed expert computation, or the gradient reduction.
Masked positions still execute the MoE layer. Packed, padded, and interleaved fixed-sequence-length datasets are available in the data layer, and the choice among them determines how many real tokens each rank microbatch carries. This matters more under sparse routing than under dense training. Every position present in the microbatch is routed, dispatched to its selected experts, computed, and combined, whether or not its label contributes to the loss. Padding and masked prompt positions consume dispatch bandwidth, expert GEMM rows, and activation memory even when they have no direct loss term. Under EP, masked rows still cross the interconnect during dispatch and combine. Sequence packing reduces this overhead by reducing padding. Post-training throughput should therefore count supervised tokens (the useful tokens of Table 51) separately from routed rows before it is compared with the useful-model rates of Section 14.
Instance-level filtering keeps removed tokens in the denominator. A separate, optional instance filter, which Olmo’s pretraining recipes enable and its SFT recipes do not, removes whole sequences from the objective but adds every label position of each removed sequence back into the loss denominator. Those positions therefore contribute zero loss over a nonzero count, lowering the reported loss relative to an exact denominator over the retained instances; the bias depends on how many instances the filter removes. Loss curves from a filtered and an unfiltered stage are not directly comparable.
22.2 Preference Training Scores Two Sequences Against a Reference
DPO scores paired sequences under a policy and a reference model. DPO computes its loss from a chosen and a rejected continuation of the same prompt, scored under both the trained policy and a frozen reference policy (Rafailov et al. 2023). One logical example becomes two sequences. Their combined lengths determine the routed token count, so the effective rank microbatch must be re-derived; the memory headroom measured for pretraining does not transfer unchanged. The paired sequences also share a prompt prefix whose routed computation is performed twice unless the implementation deliberately shares it. Reference log-probabilities require either a second resident model, which uses memory alongside the learner, or a precomputation pass whose outputs are cached and read back. Precomputation removes the resident model’s memory cost but fixes the reference policy for the run.
The cached-reference DPO implementation in open-instruct scores the dataset with the initial policy before updating it, then reads those scores during training (Ai2 2026; Lambert et al. 2024). This avoids allocating a second resident reference copy. Its existing trainer uses Accelerate; a sparse DPO adaptation could instead supply the objective to the learner described here. The paired-sequence and reference accounting would still apply, but this report makes no throughput or quality claim for that adaptation.
22.3 RLVR Adds Systems Outside the Learner
RLVR surrounds the learner with a rollout and verification system. Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al. 2024) generates rollouts from the current policy, verifies them against a programmatic checker, and updates the policy from the resulting rewards. GRPO and the actor-critic PPO configuration considered here differ inside the learner: PPO trains a value model alongside the policy, whereas GRPO estimates advantages from a group of rollouts (Schulman et al. 2017; Shao et al. 2024). A sparse GRPO policy therefore needs one learner; PPO adds value-model training, with a cost set by the chosen critic architecture. DeepSeek-R1 is a public example applied to an MoE policy (Guo et al. 2025), and HybridFlow makes the boundary between the rollout system and the learner explicit (Sheng et al. 2025). The learner described in this report is one component of that arrangement. The additional components—rollout generation at inference precision and batch shape, reward verification, actor orchestration, and repeated transfer of updated weights from the learner to the inference engine—are a separate systems problem whose costs are not addressed by any measurement in this report.
Online weight transfer requires a separate latency budget. Olmo-core can already export a trained MoE checkpoint into a Hugging Face model definition, and it contains a generation module and sampling path. Those provide an offline route from a training checkpoint to an inference engine for evaluation. Online reinforcement learning requires repeated, low-latency propagation of updated expert weights from the training topology into a serving topology whose expert placement is chosen for inference rather than for training. The topology-agnostic checkpoint representation of Section 12 is the natural substrate for that transfer, since it already describes every saved chunk by its place in the global tensor, independent of the EP and PP layout that produced it. Whether it is fast enough to sit inside a rollout loop is untested here.
22.4 Supervised and Routed Token Rates Measure Different Work
Comparisons across stages should state the base checkpoint, learner configuration, batch construction, and token accounting. The same number of supervised tokens can require different numbers of routed tokens because prompts, padding, and preference pairs still execute the model. Reporting supervised tokens per second alongside routed tokens per second makes that difference visible.