The preceding sections measured individual mechanisms and controlled combinations. Here we report complete training configurations, from Tiny to Ultra, with their model scale, batch, topology, precision, recompute policy, and useful-model TFLOP/s/GPU. These runs establish feasible operating points; their different configurations limit comparisons between them.
14.1 Production Run Matrix
The headline runs use random routing. Random routing removes the changing expert-load distribution of a learned router and gives the training system a reference that stabilizes quickly once a run is long enough. It does not measure model quality or the throughput ultimately reached by a learned router, so every row in Table 30 is labeled as a systems reference rather than a time-to-quality result. All runs use sequence length 8192, top-4 routing, one shared expert, NVIDIA B300 GPUs, and eight GPUs per node. Tiny uses 16 GPUs, Small and Medium use 128, and Large and Ultra use 512.
Each run contributes a representative rate read from its throughput curve (for the nine-step Ultra-256E run, its best step), and we select the highest observed rate across each configuration’s runs. Some configurations were launched more than once; environmental problems were suspected in the lower-rate repeats, but their causes were not established. These readings are neither repetition averages nor estimates of the hardware ceiling, and they do not come from one synchronized timing window.
We report BF16-reference MFU (Equation 95), using the B300’s dense BF16 peak of 2,250 TFLOP/s/GPU (NVIDIA 2026q). These percentages are derived from the throughput readings, not separately measured. We keep the same denominator for MXFP8, whose execution mixes precisions; its percentage is therefore a BF16 reference, not utilization of the FP8 peak. The table also gives the parameter activation ratio \(\alpha_P = P_{\mathrm{active}}/P_{\mathrm{total}}\) defined in Equation 3.
These measurements use an eight-GPU NVLink domain. Each node provides one eight-GPU NVLink domain; communication beyond those eight GPUs crosses the inter-node fabric (Section 7.4). The Megatron-Core MoE report evaluates GB200 on NVL72, where a much larger EP group can remain inside one NVLink domain. It also reports GB300 NVL results, but does not specify a 72-GPU domain for those measurements (Yan et al. 2026). Some of its reported per-GPU throughput values are higher than ours, but the different GPU platform, NVLink domain, model, batch, topology, and software recipe prevent an apples-to-apples framework comparison.
| Operating point | Active @ total Parameter activation |
GBS | EP / PP | Precision | Recompute | TFLOP/s/GPU MFU (BF16 ref.) |
|---|---|---|---|---|---|---|
| Tiny-64E | 1.59B @ 12.91B \(\alpha_P=12.3\%\) |
8 Mi | 1 / 1 | BF16 | No | 903 40.1% |
| Small-64E | 4.29B @ 40.86B \(\alpha_P=10.5\%\) |
24 Mi | 8 / 1 | BF16 | No | 841 37.4% |
| Medium-64E | 7.37B @ 70.22B \(\alpha_P=10.5\%\) |
16 Mi | 8 / 2 | BF16 | No | 853 37.9% |
| Medium-96E | 7.38B @ 103.75B \(\alpha_P=7.11\%\) |
24 Mi | 8 / 2 | BF16 | No | 855 38.0% |
| Medium-128E | 7.38B @ 137.27B \(\alpha_P=5.38\%\) |
24 Mi | 8 / 4 | BF16 | No | 843 37.5% |
| Large-128E | 15.14B @ 295.99B \(\alpha_P=5.12\%\) |
32 Mi | 8 / 4 | BF16 | No | 768 34.1% |
| Ultra-128E | 58.36B @ 1.200T \(\alpha_P=4.86\%\) |
64 Mi | 8 / 8 | MXFP8 | Yes | 858 38.1% |
| Ultra-256E | 58.41B @ 2.380T \(\alpha_P=2.45\%\) |
64 Mi | 32 / 8 | MXFP8 | Yes | 689\(\ddagger\) 30.6% |
14.2 Throughput Across Model Sizes
Stored expert parameter size grows faster than active compute. The three Medium configurations hold active parameters nearly constant while increasing total parameter size from 70.22B to 137.27B. Their selected observed throughputs remain in a narrow 843–855 TFLOP/s/GPU band, but their global batches and PP degrees are not all matched. The result shows feasible operating points at each total parameter size; it does not isolate the cost of adding experts.
Learned routing runs below the random reference. The learned-routing companions in Appendix B.10 show a gap of about one percent at Tiny without EP, and about nine percent at Small with EP8, where a learned router’s changing expert loads can reduce throughput. These companions show how optimistic the random-routing reference can be; Section 10 examines the associated load imbalance.
Memory capacity limits the largest operating points. Large-128E runs at the smallest rank microbatch of any BF16 operating point in the matrix, a choice made under memory pressure. At Ultra scale, MXFP8 also saves activation and EP-buffer memory, and the recipe still requires per-layer recomputation (Section 13). Recomputing those activations adds forward work, so despite MXFP8, Ultra-128E’s 858 TFLOP/s/GPU stays close to the smaller BF16 points. Its 64-Mi-token global batch and PP8 topology also differ from Large-128E’s 32-Mi-token batch and PP4 topology, preventing a precision-only comparison. Ultra-256E further changes EP from 8 to 32 and uses the DeepEP v2 path; its nine measured steps establish only that the configuration ran at the reported rate during a short, still-changing window.
Across these configurations, the stack supports 12.9 billion to 1.2 trillion total parameters, with the 2.38-trillion-parameter extension demonstrated in the short run described above.