PyTorch Introduces Zephon for Deterministic Data Loading

What happened

PyTorch Blog announced the release of Zephon, a new data loading tool that maintains consistent data order regardless of the number of GPUs used in training. This ensures that the training process remains deterministic even when scaling across different GPU configurations.

Zephon addresses the issue where changing the number of GPUs can lead to divergent loss curves and evaluation scores when using standard data loaders. By maintaining a consistent data order, Zephon helps prevent these inconsistencies.

Why it matters

Training experiments often involve multiple GPU configurations, which can affect data order and model performance. Traditional data loaders are sensitive to GPU count, leading to different data orders and model outcomes. Zephon ensures that the data order remains consistent across different GPU configurations, maintaining elastic determinism.

Key details

Zephon supports GPU counts ranging from 8 to 64 and reduces variance in evaluation scores to 0.05 points for DCLM Core v1 and 0.011 points for FineWeb. It also allows users to define datasets, a WorkSource for mixing and ordering data, and a pipeline of operators for processing samples into training tensors.

The tool performs tasks like tokenization, packing, shuffling, and mixing on the fly during training, eliminating the need for preprocessed data copies. It also enables parallelism in preprocessing steps, such as tokenization, by assigning multiple workers to handle different parts of the data processing pipeline.