The methods in this report solve different parts of the MoE systems problem. Figure 76 collects their direct first-order effects in one place. The matrix separates a general parallelism mechanism from its implementation or schedule. General Expert Parallelism (Gen. EP), for example, describes expert sharding and its unavoidable token movement; one-dimensional all-to-all (1D-A2A) and rowwise all-to-all (Row-A2A) describe two implementations of that movement. Similarly, general Pipeline Parallelism (Gen. PP) describes layer sharding, while Interleaved-1F1B (I-1F1B) and 1F1B-V describe two schedules.
The remaining headers abbreviate Distributed Data Parallel (DDP), Fully Sharded Data Parallel (FSDP), distributed optimizer (Dist Opt), Context Parallelism (CP), activation recomputation (Recomp), grouped general matrix multiplication (Grp GEMM), microscaling FP8 (MXFP8), synchronization-free execution (Sync-Free), CUDA Graph execution (CUDA Graph), communication–compute overlap (Overlap), hardware-aware topology mapping (HW Map), topology-agnostic checkpointing (Topo-Agst Ckpt), CPU activation offload (CPU Offload), and the MegaMoE megakernel prototype (MegaMoE; Section 15.4).
A green plus means that the method directly reduces the named bottleneck. A red minus means that it introduces or increases that cost. An amber triangle marks an effect that depends materially on model shape, topology, schedule, hardware, or implementation. An empty cell means that the method has no direct first-order effect; it does not mean that the two are independent in a complete training run. The matrix is a qualitative design guide, not a performance ranking, and combinations of methods can change the individual effects.