Section 21 motivates Olmo’s routed/shared merge from its dense-MLP decomposition and initialization scale. This appendix places that choice beside public model and framework implementations (Table 52). The comparison is based on the cited reports, released configurations, and source code.
For consistency with Equation 80, let \(p_i\) denote selected routed scores when they are renormalized to sum to one, let \(\alpha\) be any additional routed scale, so that the routed coefficients are \(r_i=\alpha p_i\), and let \(s(x)\) be the shared-expert coefficient. A rule that instead retains the top-\(K\) probabilities \(q_i(x)\) of a softmax over all \(N\) experts has token-dependent routed mass \(m_K(x)=\sum_{i\in\mathcal{S}(x)}q_i(x)<1\). These are separate design choices: normalizing the selected routed scores does not determine either \(\alpha\) or \(s(x)\).
| Model or implementation | Routed coefficients | Shared branch | Interpretation |
|---|---|---|---|
| Olmo, this report | Selected normalization followed by \(\alpha=K\); hence \(\sum_i r_i=K\). | Unit gain, \(s=1\). | Unit-average selected coefficients; matches the additive dense-block reference under uniform selected routing. |
| Megatron-Core default, when shared experts are enabled | Post-top-\(K\) softmax with \(\alpha=1\) by default; \(\alpha\) is configurable. | Unit gain by default; an optional learned sigmoid gate is available. | A configurable framework rather than one prescribed model recipe (Yan et al. 2026; NVIDIA 2026l). |
| DeepSeek-V3 released reference | Selected sigmoid affinities are renormalized and multiplied by \(\alpha=2.5\). | Unit gain. | A model-specific routed scale between one and \(K=8\) (DeepSeek-AI et al. 2024; DeepSeek-AI 2025a). |
| Qwen1.5-MoE and Qwen2-MoE released models | Full softmax over all experts followed by top-\(K\) without selected renormalization; routed mass is \(m_K(x)<1\). | Learned sigmoid gate. | An upcycled recipe in which routed mass and shared gain are both token dependent (Qwen Team 2024a, 2024b, 2024c; Yang et al. 2024; Qwen Team et al. 2024). |
| Qwen3.5 MoE public implementation | Selected top-\(K\) probabilities are renormalized to sum to one. | Learned sigmoid gate. | Unit routed mass with a token-dependent shared contribution (Qwen Team 2026b; Hugging Face 2026). |
| Mixtral | Softmax over the selected logits; routed mass is one. | No shared expert. | A routed-only weighted-average convention (A. Q. Jiang et al. 2024; Mistral AI 2026). |
| OLMoE-1B-7B | Full softmax over 64 experts followed by top-8 without selected renormalization; routed mass is below one. | No shared expert. | The merge rule in this report is not inherited from the earlier OLMoE model (Muennighoff et al. 2024). |
| MegaBlocks options | Selected route weights may remain unnormalized or receive an \(L_p\) normalization. | Direct addition by default; an option uses routed coefficient \(K/(K+1)\) and shared coefficient \(1/(K+1)\). | Exposes normalization as a configurable recipe choice rather than a fixed framework identity (Databricks 2026). |
Megatron-Core illustrates the distinction between a framework default and a model recipe particularly clearly. Its current examples use routed scales of 2.5 for DeepSeek-V3, 2.827 for Kimi K2, and either 2.5 or 5.0 for the surveyed Nemotron 3 variants, rather than deriving one scale from \(K\) or expert width. The examples establish that the mechanism supports model-specific choices; they are not a controlled ablation showing that one value is preferable (NVIDIA 2026l).
The released DeepSeek-V3 reference requires a similar qualification. Its inference code applies the routed scale 2.5, while the DeepSeek-V3 report’s routing equations describe normalized selected sigmoid affinities and an additive shared branch. The original large-scale training implementation is not public, so the released inference setting does not establish the exact pretraining merge formula.
Across these systems, routed mass may be below one, equal to one, multiplied by a model-specific constant, or equal to \(K\); the shared contribution may be absent, have unit gain, or use a learned gate. Expert widths and initialization histories also differ, and Qwen’s upcycling changes the relevant starting point. The initialization-scale rationale in Section 21 applies to Olmo’s architecture. These public implementations do not establish a universal or uniquely optimal merge normalization.