Checkpoint tensors are independent of rank placement. A checkpoint should identify the model and optimizer tensors required to resume a particular training step, without requiring the next run to use the same rank layout. Topology-independent checkpoint tensors and load-time resharding are also explicit goals of Universal Checkpointing and PyTorch Distributed Checkpoint (Lian et al. 2025; PyTorch Contributors 2026h). Olmo saves FP32 main weights, FP32 optimizer states, and the small amount of optimizer and trainer metadata needed to continue the run. In MoE, dense tensors are sharded over DP, expert tensors over EP-MP and EP-DP, and layers over PP. Changing those degrees changes which slice each rank holds, not the underlying tensor values.
| Saved item | Format | Size | 166.7B example (GiB) |
What it restores |
|---|---|---|---|---|
| FP32 main weights | FP32 | \(4P\) bytes | 621.10 | The source-of-truth parameter values updated by the optimizer. |
| FP32 first moment | FP32 | \(4P\) bytes | 621.10 | Adam’s exponential moving average of gradients. |
| FP32 second moment | FP32 | \(4P\) bytes | 621.10 | Adam’s exponential moving average of squared gradients. |
| Optimizer bookkeeping | FP32 scalars | Negligible | < 0.04 | Per-parameter step counters and rolling windows of recent losses and gradient norms. |
| Trainer and run metadata | Mixed | Negligible | < 0.04 | Step, token and epoch counters; data-loader position; random-number-generator state; callbacks; configuration; and data manifest. |
| Total | \(\boldsymbol{\approx 12P}\) bytes | 1,863.33 predicted 1,863.29 |
BF16 compute weights and MXFP8 compute-weight caches are reconstructed after loading. |
In Table 27, the complete 166.7B-parameter checkpoint occupies 1,863.33 GiB by file size, within 0.04 GiB of three times its FP32 array size. All remaining checkpoint contents fit in that difference. Topology changes alter the files and chunks written by each rank, but not the dominant \(12P\) payload.
12.1 Checkpoint Tensors Are Topology-Independent
Olmo identifies each global checkpoint tensor by a stable key, global shape, dtype, tensor role (for example, main weight or optimizer moment), canonical element order, and offsets within the global tensor. The physical checkpoint files may still contain chunks shaped by the save topology; topology independence comes from describing where those chunks belong in the global tensor. For example, changing from EP8 to EP4 changes the number of experts held by each EP-MP rank, not the expert identities or their order. FP32 main weights and optimizer states are persisted, while BF16 compute weights, MXFP8 compute-weight caches, communication buffers, and process groups are derived for the active run.
12.2 Low-Memory Saving and Load-Time Resharding
During saving, dense tensors are already exposed in their checkpoint DTensor layouts (PyTorch Contributors 2026i). Expert tensors use a checkpoint-only flat view that orders the local EP-MP expert shards and records EP-DP sharding or replication. In the normal path this view aliases the live optimizer storage. The distributed checkpoint writer stages each rank’s local chunks on CPU and serializes them with their global coordinates, writing each replicated chunk once from its lowest rank, without first gathering the complete expert tensor on GPU. PP contributes globally named layer keys from each stage rather than encoding PP rank IDs in those keys. In one controlled MXFP8 development run, save-time peak allocated and reserved memory remained at their pre-save values of 93.1 and 121.4 GiB; this is a save-side measurement only.
During loading, the destination run first constructs the shards required by its requested topology (PyTorch Contributors 2026h). The reader stages saved chunks on CPU, copies the intersections needed by those targets, and restores the FP32 main weights maintained by the distributed optimizer. A direct training load then reconstructs the BF16 compute weights and MXFP8 compute-weight caches. Unlike saving, this step all-gathers the DP and EP-DP shards of the main weights on GPU, because each rank needs complete compute weights; this report does not measure load-time memory.
12.3 Topology Changes and Model Conversion
Topology-agnostic loading changes placement, not the model schema. The source and destination must agree on parameter names, shapes, tensor roles, and canonical ordering. Changing DP, EP-MP, EP-DP, or PP is a topology change that redistributes those same global tensors; changing the number of experts, the model or expert width, or the layer structure is a model conversion.
As a result, one checkpoint can resume or resize an experiment under a different DP, EP-MP, EP-DP, or PP layout, and it can be inspected or exported to another format without a separate full-checkpoint GPU job. The evidence in this section covers the mechanism and the save path; this report includes no measured restore under a changed topology.