The same object can have a model meaning, a runtime placement, and a checkpoint representation. This glossary fixes the terms used for those layers.
C.1 Parallel Dimensions and Mesh Axes
| Term | Meaning |
|---|---|
| \(P\) / PP | Pipeline degree. PP partitions layers and changes execution order, not which data-parallel ranks replicate each parameter. |
| \(D\) / DP | Dense data-parallel degree. With CP, ordinary dense gradients reduce over the folded DP\(\times\)CP replica pool; dense optimizer state is sharded or replicated over DP and duplicated across CP. |
| \(C\) / CP | Context-parallel degree. CP shards sequence before the PP schedule and participates in the folded expert pool. |
| \(D_E\) / EP-DP | Expert replica degree. Ranks on this axis hold the same local EP-MP expert shard and reduce its ordinary gradients together. |
| \(M_E\) / EP-MP | Expert-sharding degree. Ranks on this axis hold different routed-expert shards; it is not a gradient all-reduce group. |
| EP | Expert parallelism, used generically for distributing routed experts and moving token rows to ranks that store the selected experts. |
| TP | Tensor parallelism, which partitions individual parameter tensors and their operations. |
| dense mesh | Device-mesh view (pp?, dp, cp?). |
| MoE mesh | View of the same ranks as (pp?, ep_dp, ep_mp). Within a PP stage, \(DC=D_E M_E\). |
C.2 Model Size and Routing Quantities
| Symbol or term | Meaning |
|---|---|
\(d\) / d_model |
Transformer model width. |
| \(h_e\) | Routed expert intermediate width. |
| \(m_s\) | Total shared-expert intermediate width (Section 20); distinct from the microbatch count \(m\). |
\(N\) / num_experts |
Total routed-expert count in an MoE layer. |
\(K\) / top_k |
Number of routed experts selected per token. |
| \(S\) | Sequence length before context-parallel sharding. |
| \(T\) | Number of input tokens before top-\(K\) expansion and capacity handling. |
| \(B\) / \(m\) | Useful tokens processed per rank in one optimizer step / number of microbatches in that step, when used in the cost ledger before model parallelism. |
num_local_experts |
Routed experts stored on one EP-MP rank, normally \(N/M_E\). |
| active parameters \(P_{\mathrm{active}}\) | Parameters used by one token; routed expert work scales with \(K\), not \(N\). |
| total parameters \(P_{\mathrm{total}}\) | Full persistent parameter pool; optimizer tensors and checkpoint size scale with this quantity. |
| \(X\text{B}@Y\text{B}\) | Model-size shorthand for \(X\) billion active and \(Y\) billion total parameters (active@total), for example \(3.21\text{B}@23.10\text{B}\). |
| \(P_{\mathrm{local}}\) | Parameter size held by one rank after PP and EP placement. This is neither the model-wide total parameter count nor the per-token active parameter count. |
| \(P_E\) / \(P_{\mathrm{dense}}\) | Routed-expert and dense (non-expert) parameter counts of the whole model; one rank holds about \(P_E/M_E\) of the expert part. |
| expert activation ratio \(\alpha_E\) | Fraction of routed experts selected for one token, \(K/N\). |
| parameter activation ratio \(\alpha_P\) | Fraction of total model parameters active for one token, \(P_{\mathrm{active}}/P_{\mathrm{total}}\). It generally differs from \(\alpha_E\) because part of the model is always active. |
| \(\Gamma_{\mathrm{capacity}}\) | Model-wide capacity-to-active gap, \(P_{\mathrm{total}}/P_{\mathrm{active}}=1/\alpha_P\). It is not itself an overhead or a conversion from bytes or launches to elapsed time. |
| routed experts | Sparse branch selected by the router and partitioned over EP-MP. |
| shared expert | Always-active branch that remains in the dense parameter family. |
| router | Projection and policy producing selected expert indices, route weights, and load-balancing metadata. |
| expert offsets/counts | GPU or host metadata delimiting unequal per-expert row ranges for grouped computation. |
C.3 Runtime Buffers and Reuse Terms
| Term | Meaning |
|---|---|
| persistent training state | Compute weights, accumulated gradients, FP32 main weights, and optimizer states that must remain available across microbatches or update boundaries. |
| execution working memory | Activations, routing and communication buffers, and workspaces used by execution. A preallocated fixed-address buffer can still belong to this class. |
| static placement / shape | A stable placement, buffer identity or address, capacity, shape, or graph structure. Static values need not be immutable. |
| symmetric memory | Collectively established storage that ranks in one team can address through one-sided operations. |
| Olmo symmetric memory | Current rowwise backend built around Olmo’s CUDA/NVSHMEM wrapper rather than the legacy Torch symmetric-memory path. |
| dispatch / combine | Movement from token order to expert-buffer order / return and router-weighted accumulation into token order. |
| lease | Reservation of a symmetric payload that must remain valid for a later backward. Autograd releases it after the consumer finishes. |
| shared transient slot | Bounded scratch reused across blocks or lanes; its existence does not indicate when a saved payload may be overwritten. |
| fixed per-block cache | Stable cached storage used when sharing is disabled or unsafe; a cache hit alone does not show that backward has finished with the previous contents. |
| high-water | Maximum simultaneous live payload count predicted or observed for a stage and storage family. |
| qdata–scale payload | One MXFP8 value represented by quantized data and its block scales. Both components must remain valid and be released together. |
| BF16 anchor | Temporary initialization value and cache seed for an FP8WeightStore-backed parameter. It may be released after optimizer setup and is not an independent checkpoint source. |
| MXFP8 compute-weight cache | Derived orientation-specific qdata/scale compute weight, refreshed from the FP32 main weight. |
| MXFP8 runtime payload | Derived qdata/scale representation for activation or communication data. |
C.4 Optimizer and Checkpoint Terms
| Term | Meaning |
|---|---|
MultiGroupDistributedDataParallel |
Olmo reducer assigning a process group per parameter family and keeping persistent bucket views. |
| dense bucket | Gradient bucket reduced over the folded DP\(\times\)CP replica group. |
| expert bucket | Routed-expert bucket reduced over EP-DP, the group holding replicas of the same expert shard. |
OLMoDDPOptimizer |
Olmo optimizer that manages FP32 main weights, optimizer states, gradient intake, global gradient norm, copy-back, and checkpoint views. |
| compute weight | Weight representation consumed by forward or backward computation: a BF16 tensor for an ordinary parameter or a derived, orientation-specific MXFP8 qdata/scale cache for a selected linear. |
| FP32 main weight | FP32 parameter value updated by the optimizer and used to refresh its compute weights. |
main_grad |
Optimizer-facing high-precision gradient slice. |
| copy-back | Reconstruction of compute weights after a successful update or load. |
FP8WeightStore |
Trainable FP8-weight interface with high-precision gradients and derived orientation-specific MXFP8 compute-weight caches; not an FP8 nn.Parameter. |
| checkpoint-view DTensor | Global tensor view exposed to distributed checkpointing, normally without a full GPU expert tensor. |
| topology portability | Ability to restore complete model and optimizer tensors under a changed DP, EP-MP, EP-DP, or PP parameter-placement map. |