PyTorch 2.15 Enhances Distributed Training and Performance

What happened

PyTorch 2.15 features a rewritten NCCL backend that is now the default for all nccl process groups. This change aims to streamline distributed training processes.

The release also includes a flight recorder feature that captures data during collective operations when they hang, enabling post-mortem analysis for debugging.

Why it matters

These updates are important for users working with large-scale distributed training, as they can improve efficiency and reduce potential bottlenecks. Enhanced debugging tools also help in diagnosing and resolving issues more effectively.

Key details

PyTorch 2.15 adds one-sided transports for torch.distributed, using a NIXL backend for data movement. Additionally, FSDP2 provides more precise control over memory and communication, with support for fp32 for precision-sensitive parameters.

Apple Silicon now supports FP8 with new operators like float8_e4m3fn and _scaled_mm, improving decode performance. The grouped_mm operator also includes a decode fast path, increasing MoE decode throughput by 1.32–1.34x.