Abnar, Samira, Harshay Shah, Dan Busbridge, Alaaeldin El-Nouby, Joshua M. Susskind, and Vimal Thilak. 2025. “Parameters Vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models.” Proceedings of the 42nd International Conference on Machine Learning, Proceedings of machine learning research, vol. 267: 204–30. https://proceedings.mlr.press/v267/abnar25a.html.
Cited in §1
Ai2. 2026. open-instruct: Ai2’s Post-Training Codebase. GitHub repository. https://github.com/allenai/open-instruct/tree/b9269782a3d2c81f2e43ce14ba5290417923f4c0.
Cited in §22
Ainslie, Joshua, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.” Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 4895–901. https://aclanthology.org/2023.emnlp-main.298/.
Cited in App. B
Allen Institute for AI. 2024. Dolma2 Tokenizer. https://huggingface.co/allenai/dolma2-tokenizer/blob/5292e5d6c0f40b67cc765fe41bec991cf4345b5c/README.md.
Cited in App. B
Ansel, Jason, Edward Yang, Horace He, et al. 2024. “PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation.” Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 929–47. https://doi.org/10.1145/3620665.3640366.
Cited in §6
Austin, Jacob, Sholto Douglas, Roy Frostig, Anselm Levskaya, et al. 2025. All about Rooflines. How To Scale Your Model, Part 1. https://jax-ml.github.io/scaling-book/roofline/.
Bhalachandra, Sridutt, Brian Austin, Samuel Williams, and Nicholas J. Wright. 2022. “Understanding the Impact of Input Entropy on FPU, CPU, and GPU Power.” arXiv Preprint arXiv:2212.08805, ahead of print. https://doi.org/10.48550/arXiv.2212.08805.
Cited in §19
Chen, Tianqi, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. “Training Deep Nets with Sublinear Memory Cost.” arXiv Preprint arXiv:1604.06174. https://arxiv.org/abs/1604.06174.
Cited in §13
Chowdhery, Aakanksha et al. 2023. “PaLM: Scaling Language Modeling with Pathways.” Journal of Machine Learning Research 24 (240): 1–113. https://jmlr.org/papers/v24/22-1144.html.
Clark, Aidan, Diego De Las Casas, Aurelia Guy, et al. 2022. “Unified Scaling Laws for Routed Language Models.” Proceedings of the 39th International Conference on Machine Learning, Proceedings of machine learning research, vol. 162: 4057–86. https://proceedings.mlr.press/v162/clark22a.html.
Cited in §1
Dai, Damai, Chengqi Deng, Chenggang Zhao, et al. 2024. “DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models.” Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1280–97. https://doi.org/10.18653/v1/2024.acl-long.70.
Dao, Tri, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.” Advances in Neural Information Processing Systems 35. https://proceedings.neurips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html.
Cited in App. A
Databricks. 2026. MegaBlocks Source Code. GitHub. https://github.com/databricks/megablocks/tree/952db33d6eac334d22c61e47a0d5d41446298784.
Cited in App. E
Databricks AI Research Team. 2024. Introducing DBRX: A New State-of-the-Art Open LLM. https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm.
Cited in §1
DeepSeek-AI et al. 2024. DeepSeek-V3 Technical Report. Technical Report arXiv:2412.19437v2. DeepSeek-AI. https://doi.org/10.48550/arXiv.2412.19437.
DeepSeek-AI. 2025a. DeepSeek-V3 Reference Implementation. GitHub. https://github.com/deepseek-ai/DeepSeek-V3/tree/9b4e9788e4a3a731f7567338ed15d3ec549ce03b.
DeepSeek-AI. 2025b. DualPipe: Bidirectional Pipeline Parallelism for Computation–Communication Overlap. GitHub. https://github.com/deepseek-ai/DualPipe/tree/030ce4325f4ebeb437da4ebc6d00a70469dd58ae.
DeepSeek-AI. 2026. DeepEP: Legacy Low-Latency and Two-Batch-Overlap Documentation. https://github.com/deepseek-ai/DeepEP/blob/60d44037a702f651a6e18bd4aea65ed8409051c2/docs/legacy.md.
Cited in §15
DeepSeek-AI et al. 2026. “DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence.” arXiv Preprint arXiv:2606.19348, ahead of print. https://doi.org/10.48550/arXiv.2606.19348.
Du, Nan et al. 2022. “GLaM: Efficient Scaling of Language Models with Mixture-of-Experts.” Proceedings of the 39th International Conference on Machine Learning, Proceedings of machine learning research, vol. 162: 5547–69. https://proceedings.mlr.press/v162/du22c.html.
Cited in §23
Fedus, William, Barret Zoph, and Noam Shazeer. 2022. “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.” Journal of Machine Learning Research 23 (120): 1–39. https://www.jmlr.org/papers/v23/21-0998.html.
Gale, Trevor, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2023. “MegaBlocks: Efficient Sparse Training with Mixture-of-Experts.” Proceedings of Machine Learning and Systems 5: 288–304. https://proceedings.mlsys.org/paper_files/paper/2023/hash/5a54f79333768effe7e8927bcccffe40-Abstract-mlsys2023.html.
Gemma Team. 2026. Gemma 4 Model Card. https://ai.google.dev/gemma/docs/core/model_card_4.
Cited in §1
GLM-5 Team et al. 2026. “GLM-5: From Vibe Coding to Agentic Engineering.” arXiv Preprint arXiv:2602.15763, ahead of print. https://doi.org/10.48550/arXiv.2602.15763.
Cited in §1
Gregersen, Theo, Pratyush Patel, and Esha Choukse. 2024. “Input-Dependent Power Usage in GPUs.” SC24-w: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, 1872–77. https://doi.org/10.1109/SCW63240.2024.00235.
Cited in §19
Guo, Daya et al. 2025. “DeepSeek-R1 Incentivizes Reasoning in LLMs Through Reinforcement Learning.” Nature 645: 633–38. https://doi.org/10.1038/s41586-025-09422-z.
Cited in §22
Hägele, Alexander, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, and Martin Jaggi. 2024. “Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations.” Advances in Neural Information Processing Systems 37: 76232–64. https://proceedings.neurips.cc/paper_files/paper/2024/hash/8b970e15a89bf5d12542810df8eae8fc-Abstract-Conference.html.
He, Ethan, Abhinav Khattar, Ryan Prenger, et al. 2024. Upcycling Large Language Models into Mixture of Experts. 2410.07524. arXiv. https://arxiv.org/abs/2410.07524.
Cited in §18
Hoffer, Elad, Itay Hubara, and Daniel Soudry. 2017. “Train Longer, Generalize Better: Closing the Generalization Gap in Large Batch Training of Neural Networks.” Advances in Neural Information Processing Systems 30. https://proceedings.neurips.cc/paper/2017/hash/a5e0ff62be0b08456fc7f1e88812af3d-Abstract.html.
Cited in §16
Hu, Shengding, Yuge Tu, Xu Han, et al. 2024. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. 2404.06395. arXiv. https://arxiv.org/abs/2404.06395.
Cited in App. B
Huang, Howard, Will Constable, Ke Wen, et al. 2024. Training with Zero-Bubble Pipeline Parallelism. PyTorch Forums, Distributed with TorchTitan. https://discuss.pytorch.org/t/distributed-w-torchtitan-training-with-zero-bubble-pipeline-parallelism/214420.
Cited in §6
Huang, Yanping, Youlong Cheng, Ankur Bapna, et al. 2019. “GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism.” Advances in Neural Information Processing Systems 32. https://proceedings.neurips.cc/paper/2019/hash/093f65e080a295f8076b1c5722a46aa2-Abstract.html.
Hugging Face. 2026. Qwen3.5-MoE Implementation in Transformers. GitHub. https://github.com/huggingface/transformers/blob/01dd2fd817a51a67eeb9126233faf9cbb380a17c/src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py.
Cited in App. E
Hwang, Changho, Wei Cui, Yifan Xiong, et al. 2023. “Tutel: Adaptive Mixture-of-Experts at Scale.” Proceedings of Machine Learning and Systems 5: 269–87. https://proceedings.mlsys.org/paper_files/paper/2023/hash/5616d34cf8ff73942cfd5aa922842556-Abstract-mlsys2023.html.
Jacobs, Robert A., Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. “Adaptive Mixtures of Local Experts.” Neural Computation 3 (1): 79–87. https://doi.org/10.1162/neco.1991.3.1.79.
Cited in §20
Jacobs, Sam Ade, Masahiro Tanaka, Chengming Zhang, et al. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. Technical Report arXiv:2309.14509v2. Microsoft. https://doi.org/10.48550/arXiv.2309.14509.
Jain, Paras, Ajay Jain, Aniruddha Nrusimha, et al. 2020. “Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization.” Proceedings of Machine Learning and Systems 2: 497–511. https://proceedings.mlsys.org/paper_files/paper/2020/hash/0b816ae8f06f8dd3543dc3d9ef196cab-Abstract.html.
Cited in §13
Jiang, Albert Q., Alexandre Sablayrolles, Antoine Roux, et al. 2024. “Mixtral of Experts.” arXiv Preprint arXiv:2401.04088, ahead of print. https://doi.org/10.48550/arXiv.2401.04088.
Jiang, Chenyu, Ye Tian, Zhen Jia, Shuai Zheng, Chuan Wu, and Yida Wang. 2024. “Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping.” Proceedings of Machine Learning and Systems 6: 74–86. https://proceedings.mlsys.org/paper_files/paper/2024/hash/339caf45a6fa281cae8adc6465343464-Abstract-Conference.html.
Jin, Chao, Ziheng Jiang, Zhihao Bai, et al. 2025. “MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production.” arXiv Preprint arXiv:2505.11432. https://arxiv.org/abs/2505.11432.
Cited in §23
Jordan, Keller, Yuchen Jin, Vlado Boza, et al. 2024. Muon: An Optimizer for Hidden Layers in Neural Networks. https://kellerjordan.github.io/posts/muon/.
Cited in §4
Jordan, Michael I., and Robert A. Jacobs. 1994. “Hierarchical Mixtures of Experts and the EM Algorithm.” Neural Computation 6 (2): 181–214. https://doi.org/10.1162/neco.1994.6.2.181.
Cited in §20
Kimi Team. 2026. Kimi K3: Open Frontier Intelligence. https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf.
Cited in §1
Kingma, Diederik P., and Jimmy Ba. 2015. “Adam: A Method for Stochastic Optimization.” International Conference on Learning Representations. https://arxiv.org/abs/1412.6980.
Cited in §4
Komatsuzaki, Aran, Joan Puigcerver, James Lee-Thorp, et al. 2023. “Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints.” International Conference on Learning Representations. https://openreview.net/forum?id=T5nUQDrM4u.
Cited in §18
Korthikanti, Vijay Anand, Jared Casper, Sangkug Lym, et al. 2023. “Reducing Activation Recomputation in Large Transformer Models.” Proceedings of Machine Learning and Systems 5: 341–53. https://proceedings.mlsys.org/paper_files/paper/2023/hash/80083951326cf5b35e5100260d64ed81-Abstract-mlsys2023.html.
Cited in §13
Kwon, Woosuk, Zhuohan Li, Siyuan Zhuang, et al. 2023. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 611–26. https://doi.org/10.1145/3600006.3613165.
Cited in §22
Lambert, Nathan, Jacob Morrison, Valentina Pyatkin, et al. 2024. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. 2411.15124. arXiv. https://arxiv.org/abs/2411.15124.
Cited in §22
Lange, Robert Tjarko, Yuki Imajuku, and Edoardo Cetin. 2026. “ShinkaEvolve: Towards Open-Ended and Sample-Efficient Program Evolution.” The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=lKEdGCoDNC.
Cited in §10
Lee, Seonho, Jihwan Oh, Junkyum Kim, Seokjin Go, Jongse Park, and Divya Mahajan. 2025. Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications. Technical Report arXiv:2507.03114v1. arXiv. https://doi.org/10.48550/arXiv.2507.03114.
Lepikhin, Dmitry, HyoukJoong Lee, Yuanzhong Xu, et al. 2021. “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.” International Conference on Learning Representations. https://openreview.net/forum?id=qrwe7XHTmYb.
Lewis, Mike, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. “BASE Layers: Simplifying Training of Large, Sparse Models.” Proceedings of the 38th International Conference on Machine Learning, Proceedings of machine learning research, vol. 139: 6265–74. https://proceedings.mlr.press/v139/lewis21a.html.
Cited in §10
Li, Shen, Yanli Zhao, Rohan Varma, et al. 2020. “PyTorch Distributed: Experiences on Accelerating Data Parallel Training.” Proceedings of the VLDB Endowment 13 (12): 3005–18. https://doi.org/10.14778/3415478.3415530.
Lian, Xinyu, Sam Ade Jacobs, Lev Kurilenko, et al. 2025. “Universal Checkpointing: A Flexible and Efficient Distributed Checkpointing System for Large-Scale DNN Training with Reconfigurable Parallelism.” 2025 USENIX Annual Technical Conference (USENIX ATC 25), 1519–34. https://www.usenix.org/conference/atc25/presentation/lian.
Cited in §12
Liew, Seng Pei, Takuya Kato, and Sho Takase. 2025. “Scaling Laws for Upcycling Mixture-of-Experts Language Models.” Proceedings of the 42nd International Conference on Machine Learning, Proceedings of machine learning research, vol. 267: 37682–704. https://proceedings.mlr.press/v267/liew25a.html.
Cited in §18
Liu, Dennis, Zijie Yan, Xin Yao, et al. 2025. MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core. Technical Report arXiv:2504.14960v3. NVIDIA. https://doi.org/10.48550/arXiv.2504.14960.
Liu, Hao, Matei Zaharia, and Pieter Abbeel. 2024. “RingAttention with Blockwise Transformers for Near-Infinite Context.” The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=WsRHpHH4s0.
Cited in §3
Liu, Jingyuan, Jianlin Su, Xingcheng Yao, et al. 2025. Muon Is Scalable for LLM Training. 2502.16982. arXiv. https://arxiv.org/abs/2502.16982.
Cited in §4
Liu, Yuxi. 2024. Mixture of Experts. Yuxi on the Wired. https://yuxi.ml/essays/mixture-of-experts.
Cited in §10
Loshchilov, Ilya, and Frank Hutter. 2019. “Decoupled Weight Decay Regularization.” International Conference on Learning Representations. https://openreview.net/forum?id=Bkg6RiCqY7.
Lucas, Jan, and Ben Juurlink. 2016. “ALUPower: Data Dependent Power Consumption in GPUs.” 2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, 95–104. https://doi.org/10.1109/MASCOTS.2016.21.
Cited in §19
Ludziejewski, Jan, Jakub Krajewski, Kamil Adamczewski, et al. 2024. “Scaling Laws for Fine-Grained Mixture of Experts.” Proceedings of the 41st International Conference on Machine Learning, Proceedings of machine learning research, vol. 235: 33270–88. https://proceedings.mlr.press/v235/ludziejewski24a.html.
Cited in §1
Lv, Minxuan, Zhenpeng Su, Leiyu Pan, et al. 2025. “DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs.” Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 19711–22. https://doi.org/10.18653/v1/2025.emnlp-main.997.
Cited in §20
Ma, Jiaqi, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. 2018. “Modeling Task Relationships in Multi-Task Learning with Multi-Gate Mixture-of-Experts.” Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1930–39. https://doi.org/10.1145/3219819.3220007.
Cited in §20
Malladi, Sadhika, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. 2022. “On the SDEs and Scaling Rules for Adaptive Gradient Algorithms.” Advances in Neural Information Processing Systems 35: 7697–711. https://proceedings.neurips.cc/paper_files/paper/2022/hash/32ac710102f0620d0f28d5d05a44fe08-Abstract.html.
Cited in §16
Markthub, Pak, Jim Dinan, Sreeram Potluri, and Seth Howell. 2022. Improving Network Performance of HPC Systems Using NVIDIA Magnum IO NVSHMEM and GPUDirect Async. NVIDIA Technical Blog. https://developer.nvidia.com/blog/improving-network-performance-of-hpc-systems-using-nvidia-magnum-io-nvshmem-and-gpudirect-async/.
Cited in §7
McCandlish, Sam, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. 2018. An Empirical Model of Large-Batch Training. Technical Report arXiv:1812.06162. OpenAI. https://doi.org/10.48550/arXiv.1812.06162.
Cited in §16
Merrill, Will, Shane Arora, Dirk Groeneveld, and Hanna Hajishirzi. 2025. “Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training.” Advances in Neural Information Processing Systems 38: 116936–59. https://proceedings.neurips.cc/paper_files/paper/2025/hash/a99f732df9b668284b449da0214a3286-Abstract-Conference.html.
Cited in §16
Meta AI. 2025. The Llama 4 Herd: The Beginning of a New Era of Natively Multimodal AI Innovation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/.
Cited in §1
Micikevicius, Paulius, Sharan Narang, Jonah Alben, et al. 2018. “Mixed Precision Training.” International Conference on Learning Representations. https://openreview.net/forum?id=r1gs9JgRZ.
Cited in §4
Micikevicius, Paulius, Dusan Stosic, Neil Burgess, et al. 2022. “FP8 Formats for Deep Learning.” arXiv Preprint arXiv:2209.05433. https://arxiv.org/abs/2209.05433.
Mishra, Asit, Dusan Stosic, and Simon Layton. 2025. Recipes for Pre-Training LLMs with MXFP8. Technical Report arXiv:2506.08027v1. arXiv. https://doi.org/10.48550/arXiv.2506.08027.
Cited in §11
Mistral AI. 2026. Mistral Inference. GitHub. https://github.com/mistralai/mistral-inference/tree/9eaeb91c17450e09021b6065a1d5cc69876507c8.
Mistral AI Team. 2023. Mixtral of Experts. https://mistral.ai/news/mixtral-of-experts/.
Moonshot AI. 2026. Kimi-K2.5 Model Card. https://huggingface.co/moonshotai/Kimi-K2.5/blob/4d01dfe0332d63057c186e0b262165819efb6611/README.md.
Cited in §1
Muennighoff, Niklas, Luca Soldaini, Dirk Groeneveld, et al. 2024. “OLMoE: Open Mixture-of-Experts Language Models.” arXiv Preprint arXiv:2409.02060, ahead of print. https://doi.org/10.48550/arXiv.2409.02060.
Nakamura, Taishi, Takuya Akiba, Kazuki Fujii, Yusuke Oda, Rio Yokota, and Jun Suzuki. 2025. “Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-Initialization.” International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2025/hash/d24b7366d714b09a977946ef0d9bf3ad-Abstract-Conference.html.
Cited in §18
Narayanan, Deepak, Aaron Harlap, Amar Phanishayee, et al. 2019. “PipeDream: Generalized Pipeline Parallelism for DNN Training.” Proceedings of the 27th ACM Symposium on Operating Systems Principles, 1–15. https://doi.org/10.1145/3341301.3359646.
Narayanan, Deepak, Mohammad Shoeybi, Jared Casper, et al. 2021. “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM.” Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 1–15. https://doi.org/10.1145/3458817.3476209.
Cited in §6
NVIDIA. 2025a. NVIDIA RTX PRO Blackwell GPU Architecture. NVIDIA Corporation. https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/pdf/NVIDIA-RTX-Blackwell-PRO-GPU-Architecture-v1_1.pdf.
Cited in §11
NVIDIA. 2025b. Parallel Thread Execution ISA 8.8. https://docs.nvidia.com/cuda/archive/12.9.1/parallel-thread-execution/index.html.
Cited in §11
NVIDIA. 2026a. Blackwell SM100 GEMMs. https://docs.nvidia.com/cutlass/latest/media/docs/cpp/blackwell_functionality.html.
NVIDIA. 2026b. Components. NVIDIA HGX AI Factory Enterprise Reference Architecture. https://docs.nvidia.com/enterprise-reference-architectures/hgx-ai-factory/latest/components.html.
NVIDIA. 2026c. Compressible Memory. https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/virtual-memory-management.html\#compressible-memory.
Cited in §19
NVIDIA. 2026d. cuBLAS: 16/32-Element 1D Block Scaling for FP8 and FP4 Data Types. https://docs.nvidia.com/cuda/cublas/#element-1d-block-scaling-for-fp8-and-fp4-data-types.
Cited in §11
NVIDIA. 2026e. CUDA GPUDirect RDMA. https://docs.nvidia.com/cuda/gpudirect-rdma/index.html.
Cited in §7
NVIDIA. 2026f. CUDA Programming Guide: Asynchronous Execution. https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/asynchronous-execution.html.
Cited in §8
NVIDIA. 2026g. CUDA Programming Guide: CUDA Graphs. https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html.
NVIDIA. 2026h. CUTLASS SM120 Grouped GEMM Example. https://github.com/NVIDIA/cutlass/blob/da5e086dab31d63815acafdac9a9c5893b1c69e2/examples/79_blackwell_geforce_gemm/79d_blackwell_geforce_nvfp4_grouped_gemm.cu.
Cited in App. A
NVIDIA. 2026i. CUTLASS: Grouped Kernel Schedulers. https://docs.nvidia.com/cutlass/4.3.5/media/docs/cpp/grouped_scheduler.html.
NVIDIA. 2026j. Hopper Tuning Guide: Inline Compression. https://docs.nvidia.com/cuda/hopper-tuning-guide/index.html#inline-compression.
Cited in §19
NVIDIA. 2026k. Megatron Core TransformerConfig API. Developer documentation. https://docs.nvidia.com/megatron-core/developer-guide/latest/apidocs/core/core.transformer.transformer_config.html.
Cited in §10
NVIDIA. 2026m. Megatron-LM: Buffered Global Load-Balancing Loss Implementation. Source code. https://github.com/NVIDIA/Megatron-LM/tree/addc601f57ed539506183b704bb9d08f459d7f50/megatron/core/transformer/moe.
Cited in §10
NVIDIA. 2026n. NCCL Collective Operations. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html.
NVIDIA. 2026o. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. Technical Report arXiv:2606.15007. NVIDIA. https://doi.org/10.48550/arXiv.2606.15007.
Cited in §1
NVIDIA. 2026p. NVIDIA H100 Tensor Core GPU Product Specifications. https://www.nvidia.com/en-us/data-center/h100/.
Cited in §11
NVIDIA. 2026q. NVIDIA HGX B200/B300 Platform Specifications. https://www.nvidia.com/en-us/data-center/hgx/.
NVIDIA. 2026r. NVSHMEM Memory Model. https://docs.nvidia.com/nvshmem/api/gen/mem-model.html.
NVIDIA. 2026t. Quantum-X800 InfiniBand Platform. https://www.nvidia.com/en-us/networking/products/infiniband/quantum-x800/.
Cited in §7
NVIDIA. 2026u. tcgen05 MMA Programming Guide. https://docs.nvidia.com/cutlass/latest/media/docs/pythonDSL/guides/mma/tcgen05_programming.html.
Cited in App. A
NVIDIA. 2026v. TensorRT Support Matrix: GPU Architecture and Precision Support. https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/support-matrix.html.
Cited in §11
NVIDIA. 2026w. Transformer Engine: MXFP8 Training. https://docs.nvidia.com/deeplearning/transformer-engine/features/low_precision_training/mxfp8/mxfp8.html.
Open Compute Project. 2023. OCP Microscaling Formats (MX) Specification, Version 1.0. Open Compute Project Foundation. https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf.
Cited in §11
OpenAI. 2025. “gpt-oss-120b & gpt-oss-20b Model Card.” arXiv Preprint arXiv:2508.10925, ahead of print. https://doi.org/10.48550/arXiv.2508.10925.
Cited in §1
Ouyang, Long, Jeffrey Wu, Xu Jiang, et al. 2022. “Training Language Models to Follow Instructions with Human Feedback.” Advances in Neural Information Processing Systems 35. https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html.
Cited in §22
Patarasuk, Pitch, and Xin Yuan. 2009. “Bandwidth Optimal All-Reduce Algorithms for Clusters of Workstations.” Journal of Parallel and Distributed Computing 69 (2): 117–24. https://doi.org/10.1016/j.jpdc.2008.09.002.
Cited in §4
PyTorch Contributors. 2026a. CUDA Grouped Matrix Multiplication Backend. https://github.com/pytorch/pytorch/blob/1c23e463efdf735913fabb00c48f0d0e5f6e2282/aten/src/ATen/native/cuda/GroupedBlas.cpp.
Cited in App. A
PyTorch Contributors. 2026b. CUDA Semantics. https://docs.pytorch.org/docs/main/notes/cuda.html.
PyTorch Contributors. 2026c. Distributed Communication Package: All_to_all_single. https://docs.pytorch.org/docs/stable/distributed.html.
Cited in §5
PyTorch Contributors. 2026d. Distributed Data Parallel Notes. https://docs.pytorch.org/docs/stable/notes/ddp.html.
Cited in §4
PyTorch Contributors. 2026e. fully_shard: FSDP2 Documentation. https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html.
PyTorch Contributors. 2026f. Grouped Matrix Multiplication Fallback. https://github.com/pytorch/pytorch/blob/1c23e463efdf735913fabb00c48f0d0e5f6e2282/aten/src/ATen/native/GroupedMMUtils.h.
Cited in App. A
PyTorch Contributors. 2026g. Torch.cuda.set_sync_debug_mode. https://docs.pytorch.org/docs/stable/generated/torch.cuda.set_sync_debug_mode.html.
Cited in App. B
PyTorch Contributors. 2026h. Torch.distributed.checkpoint. https://docs.pytorch.org/docs/stable/distributed.checkpoint.html.
Cited in §12
PyTorch Contributors. 2026i. Torch.distributed.tensor (DTensor). https://docs.pytorch.org/docs/stable/distributed.tensor.html.
Cited in §12
PyTorch Contributors. 2026j. torch.empty Documentation. https://docs.pytorch.org/docs/stable/generated/torch.empty.html.
Cited in §19
PyTorch Contributors. 2026k. Torch.nonzero. https://docs.pytorch.org/docs/stable/generated/torch.nonzero.html.
Cited in §8
PyTorch Contributors. 2026l. Torch.utils.checkpoint. https://docs.pytorch.org/docs/stable/checkpoint.html.
Cited in §13
Qi, Penghui, Xinyi Wan, Nyamdavaa Amar, and Min Lin. 2024. “Pipeline Parallelism with Controllable Memory.” Advances in Neural Information Processing Systems 37: 46539–66. https://papers.nips.cc/paper_files/paper/2024/hash/527dad0b9159805289906d5740a0bdd3-Abstract-Conference.html.
Cited in §6
Qi, Penghui, Xinyi Wan, Guangxing Huang, and Min Lin. 2024. “Zero Bubble (Almost) Pipeline Parallelism.” International Conference on Learning Representations. https://openreview.net/forum?id=tuzTN0eIO5.
Cited in §6
Qiu, Zihan, Zeyu Huang, Bo Zheng, et al. 2025. “Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models.” Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Vienna, Austria), July, 5005–18. https://doi.org/10.18653/v1/2025.acl-long.249.
Cited in §10
Qwen Team. 2024a. Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters. https://qwenlm.github.io/blog/qwen-moe/.
Cited in App. E
Qwen Team. 2024b. Qwen1.5-MoE-A2.7B Released Model Configuration. https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B/blob/1a758c50ecb6350748b9ce0a99d2352fd9fc11c9/config.json.
Cited in App. E
Qwen Team. 2024c. Qwen2-57B-A14B Released Model Configuration. https://huggingface.co/Qwen/Qwen2-57B-A14B/blob/f29d6a182178ddf39f68f403361779b2d37fdce8/config.json.
Cited in App. E
Qwen Team. 2026a. Qwen3.5-35B-A3B Model Card. https://huggingface.co/Qwen/Qwen3.5-35B-A3B.
Cited in §1
Qwen Team. 2026b. Qwen3.5-397B-A17B Model Card. https://huggingface.co/Qwen/Qwen3.5-397B-A17B.
Qwen Team, Alibaba Group, and Hugging Face Team. 2024. Qwen2 MoE Implementation in Transformers. https://github.com/huggingface/transformers/blob/573565e35a5cc68f6cfb6337f5a93753ab16c65b/src/transformers/models/qwen2_moe/modeling_qwen2_moe.py.
Rafailov, Rafael, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model.” Advances in Neural Information Processing Systems 36: 53728–41. https://proceedings.neurips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html.
Cited in §22
Raffel, Colin, Noam Shazeer, Adam Roberts, et al. 2020. “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.” Journal of Machine Learning Research 21 (140): 1–67. https://jmlr.org/papers/v21/20-074.html.
Cited in §17
Rajbhandari, Samyam, Conglong Li, Zhewei Yao, et al. 2022. “DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale.” Proceedings of the 39th International Conference on Machine Learning, Proceedings of machine learning research, vol. 162: 18332–46. https://proceedings.mlr.press/v162/rajbhandari22a.html.
Rajbhandari, Samyam, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.” Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 1–16. https://doi.org/10.1109/SC41405.2020.00024.
Rhu, Minsoo, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W. Keckler. 2016. “vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design.” 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 1–13. https://doi.org/10.1109/MICRO.2016.7783721.
Cited in §15
Rouhani, Bita Darvish et al. 2023. “Microscaling Data Formats for Deep Learning.” arXiv Preprint arXiv:2310.10537. https://arxiv.org/abs/2310.10537.
Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. “Proximal Policy Optimization Algorithms.” arXiv Preprint arXiv:1707.06347. https://arxiv.org/abs/1707.06347.
Cited in §22
Shallue, Christopher J., Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. 2019. “Measuring the Effects of Data Parallelism on Neural Network Training.” Journal of Machine Learning Research 20 (112): 1–49. https://www.jmlr.org/papers/v20/18-789.html.
Cited in §16
Shao, Zhihong, Peiyi Wang, Qihao Zhu, et al. 2024. “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” arXiv Preprint arXiv:2402.03300. https://arxiv.org/abs/2402.03300.
Cited in §22
Shazeer, Noam. 2020. “GLU Variants Improve Transformer.” arXiv Preprint arXiv:2002.05202. https://arxiv.org/abs/2002.05202.
Cited in §20
Shazeer, Noam, Azalia Mirhoseini, Krzysztof Maziarz, et al. 2017. “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.” International Conference on Learning Representations. https://openreview.net/forum?id=B1ckMDqlg.
Sheng, Guangming, Chi Zhang, Zilingfeng Ye, et al. 2025. “HybridFlow: A Flexible and Efficient RLHF Framework.” Proceedings of the Twentieth European Conference on Computer Systems, 1279–97. https://doi.org/10.1145/3689031.3696075.
Cited in §22
Shoeybi, Mohammad, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.” arXiv Preprint arXiv:1909.08053. https://arxiv.org/abs/1909.08053.
Snowflake AI Research. 2024. Snowflake Arctic: The Best LLM for Enterprise AI — Efficiently Intelligent, Truly Open. https://www.snowflake.com/en/blog/arctic-open-efficient-foundation-language-models-snowflake/.
Cited in §1
Su, Jianlin. 2025. MoE Travelogue 2: On Load Imbalance. Scientific Spaces. https://spaces.ac.cn/archives/10735.
Cited in §10
Tang, Hongyan, Junning Liu, Ming Zhao, and Xudong Gong. 2020. “Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations.” Fourteenth ACM Conference on Recommender Systems, 269–78. https://doi.org/10.1145/3383313.3412236.
Cited in §20
Team Olmo et al. 2025. “Olmo 3.” arXiv Preprint arXiv:2512.13961, ahead of print. https://doi.org/10.48550/arXiv.2512.13961.
Tillet, Philippe, H. T. Kung, and David Cox. 2019. “Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations.” Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 10–19. https://doi.org/10.1145/3315508.3329973.
Cited in §2
Vaswani, Ashish, Noam Shazeer, Niki Parmar, et al. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
Cited in §1
Wang, Lean, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. 2024. “Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts.” arXiv Preprint arXiv:2408.15664. https://arxiv.org/abs/2408.15664.
Cited in §10
Wei, Jason, Maarten Bosma, Vincent Y. Zhao, et al. 2022. “Finetuned Language Models Are Zero-Shot Learners.” International Conference on Learning Representations. https://openreview.net/forum?id=gEZrGCozdqR.
Cited in §22
Xiaomi MiMo Team. 2026. MiMo-V2.5-Pro Model Card. https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro.
Cited in §1
Yan, Zijie, Hongxiao Bai, Xin Yao, et al. 2026. Scalable Training of Mixture-of-Experts Models with Megatron Core. Technical Report arXiv:2603.07685v2. NVIDIA. https://doi.org/10.48550/arXiv.2603.07685.
Yang, An et al. 2024. “Qwen2 Technical Report.” arXiv Preprint arXiv:2407.10671, ahead of print. https://doi.org/10.48550/arXiv.2407.10671.
Cited in App. E
Yang, An et al. 2025. “Qwen3 Technical Report.” arXiv Preprint arXiv:2505.09388, ahead of print. https://doi.org/10.48550/arXiv.2505.09388.
Cited in §1
Zhang, Shulai, Ningxin Zheng, Haibin Lin, et al. 2025. “COMET: Fine-Grained Computation-Communication Overlapping for Mixture-of-Experts.” Proceedings of Machine Learning and Systems 7. https://proceedings.mlsys.org/paper_files/paper/2025/hash/e27ea0cd50b798ff8942caf9203f0992-Abstract-Conference.html.
Zhang, Zhengyan, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. “MoEfication: Transformer Feed-Forward Layers Are Mixtures of Experts.” Findings of the Association for Computational Linguistics: ACL 2022, 877–90. https://doi.org/10.18653/v1/2022.findings-acl.71.
Cited in §20
Zhao, Chenggang, Zhean Xu, Liang Zhao, et al. 2025. DeepGEMM: Clean and Efficient BLAS Kernel Library on GPU. Https://github.com/deepseek-ai/DeepGEMM; GitHub.
Zhao, Chenggang, Shangyan Zhou, Liyue Zhang, et al. 2025. DeepEP: An Efficient Expert-Parallel Communication Library. Https://github.com/deepseek-ai/DeepEP; GitHub.
Zhao, Yanli, Andrew Gu, Rohan Varma, et al. 2023. “PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel.” Proceedings of the VLDB Endowment 16 (12): 3848–60. https://doi.org/10.14778/3611540.3611569.
Zhou, Yanqi, Tao Lei, Hanxiao Liu, et al. 2022. “Mixture-of-Experts with Expert Choice Routing.” Advances in Neural Information Processing Systems 35: 7103–14. https://proceedings.neurips.cc/paper_files/paper/2022/hash/2f00ecd787b432c1d36f3de9800728eb-Abstract-Conference.html.
Cited in §10
Zoph, Barret, Irwan Bello, Sameer Kumar, et al. 2022. ST-MoE: Designing Stable and Transferable Sparse Expert Models. 2202.08906. arXiv. https://arxiv.org/abs/2202.08906.
Cited in §10