NVIDIA 如何用自动化工作负载恢复保障分布式训练持续运行
Keep Distributed Training Running with Automated Workload Recovery
随着分布式 AI 训练规模扩大，硬件与软件依赖故障会中断训练进度，NVIDIA 提出自动化工作负载恢复方案应对。单个训练 rank 停滞可能触发 NCCL 超时，GPU 节点可能不再响应调度器，慢节点则会拖慢整个任务。该方案针对这些各自可控的故障实现快速可靠恢复。
As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery...
As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery essential. A stalled training rank can trigger an NVIDIA Collective Communications Library (NCCL) timeout, a GPU node can stop responding to the scheduler, or a straggler can slow the entire job. Each disruption is manageable on its own…
来源：NVIDIA Developer Blog · developer.nvidia.com