Section 2 introduced the complete Olmo-core DDP stack. Part II examines its parallel methods in terms of what ranks divide, what one rank keeps during computation, and when ranks communicate (Table 3).
A rank-local model partition is the part of the model assigned to one rank after model-parallel placement; without model parallelism, it is the full model. Model-parallel methods, namely expert parallelism (EP), pipeline parallelism (PP), and tensor parallelism (TP), decide which parameters the partition contains. Ranks that are assigned the same partition and process different training examples are data-parallel (DP) ranks. Data-parallel methods decide how those ranks store and synchronize the partition they share: distributed data parallel (DDP) keeps a complete copy on every DP rank, whereas fully sharded data parallel (FSDP) splits it across the DP ranks and gathers each piece when it is needed. Context parallelism (CP) divides the sequence rather than the model, while a distributed optimizer shards the FP32 main weights and optimizer states without changing forward or backward model placement.
For data-parallel and optimizer-sharding methods, we follow the standard definitions of DDP (Li et al. 2020), ZeRO-style distributed optimization (Rajbhandari et al. 2020), FSDP, and hybrid sharded data parallel (HSDP) (Zhao et al. 2023). For model and activation partitioning, we follow the standard definitions of EP (Lepikhin et al. 2021; Rajbhandari et al. 2022), PP (Huang et al. 2019; Narayanan et al. 2019), TP (Shoeybi et al. 2019), and CP (Jacobs et al. 2023; Liu et al. 2024).
| Method | What it divides |
What one rank keeps during computation |
When ranks communicate |
|---|---|---|---|
| Data-parallel and optimizer-sharding methods | |||
| DDP | Input examples across DP ranks | One complete copy of the rank-local model partition | Accumulated gradients synchronize before each optimizer update |
| Distributed optimizer | FP32 main weights and optimizer states within a replica group | Compute weights and the local optimizer shard | Gradient intake, local shard update, and compute-weight reconstruction once per optimizer step |
| FSDP / HSDP | Persistent training state—parameters, gradients, and optimizer data—across a shard group; HSDP repeats shard groups | Full weights for the current wrapped unit; other units remain sharded | Gather weights before each forward and backward use; free them after each use and reduce-scatter gradients in backward |
| Model and activation partitioning | |||
| EP | Routed experts | The assigned expert shard; dense and shared weights follow the other axes | Dispatch and combine in each routed layer and microbatch; synchronize expert copies once per optimizer step |
| PP | Transformer layers across physical and virtual stages | The layers assigned to the local stages | Hidden activations and their gradients cross stage boundaries for every microbatch |
| TP | Individual tensors and matrix operations | One tensor or operator shard | Activation or partial-result collectives inside each sharded layer |
| CP | Sequence positions | A local sequence slice; attention also needs keys and values from the other slices | Ring attention passes key/value blocks around the CP ranks during attention; Ulysses exchanges sequence and head slices with all-to-all before and after attention |
DDP and FSDP determine when compute weights move. We start with this choice because it is the data-parallel foundation beneath EP and PP. FSDP remains a data-parallel method even though it shards stored parameters: its ranks process different data while executing the same model. EP and PP instead assign different experts or layers to different ranks.
In the FSDP configuration used by Section 4, each wrapped unit is gathered before its forward pass, freed afterward, gathered again before backward, and followed by gradient reduce-scatter (Zhao et al. 2023; PyTorch Contributors 2026e). DDP instead keeps one static copy of the rank-local model partition at stable GPU locations across all microbatches that contribute to an optimizer step, then synchronizes the accumulated gradients. Here, static describes placement, not value: the optimizer still updates the weights after every successful step. This difference sets both the memory cost and the communication frequency developed in the next section. Appendix B.3 discusses other FSDP reshard policies.
EP and PP make the DDP partition fit. Olmo’s distributed optimizer shards FP32 main weights and optimizer states, EP divides routed experts, and PP divides layers. Optional CP divides long sequences when activation memory or attention work becomes limiting. Attention mixes token positions, so it is the layer CP must restructure. The routed experts process each position independently and can treat CP ranks as additional DP ranks. Olmo does not use TP. Every expert in our configurations fits on one GPU, and TP’s per-layer collectives would compete with EP for each node’s NVLink domain (Section 7).
The rest of Part II follows this dependency. Section 4 first compares DDP with FSDP. Section 5 then explains how EP divides experts, and Section 6 explains how PP divides layers. Section 7 finally maps their composition onto the physical network.
Appendix F covers the optional CP path in detail. Olmo uses the Ulysses form of CP, which changes attention from a local sequence slice with all heads to the full sequence for a subset of heads, then reverses that layout after attention (Jacobs et al. 2023). The MoE layers view the same ranks along two axes, developed in Section 5: ranks along the expert-sharding axis (EP-MP) hold different routed experts, while ranks along the expert-replica axis (EP-DP) hold copies of the same experts and synchronize their gradients. The appendix also explains how Olmo maps the dense DP\(\times\)CP view and this MoE view onto the same ranks through MoE parallel folding (D. Liu et al. 2025). Appendix C collects the symbols and terms used across these sections.