用 NVIDIA Attribution 诊断分布式 AI 工作负载故障
Diagnose Distributed AI Workload Failures with NVIDIA Attribution
NVIDIA Attribution 可诊断分布式 AI 工作负载故障，解决单一 GPU 或网络事件在其他 rank 上表现为 NCCL 错误、调度器状态变化和大量下游消息的级联问题。报错异常只显示工作负载停止的位置，而非故障起点。文章介绍如何在示例工作负载证据上运行 NVIDIA Attribution。
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state...
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state changes, and thousands of downstream messages. The reported exception shows where the workload stopped, not where the failure began. In this post, you’ll learn how to run NVIDIA Attribution on sample workload evidence…
来源：NVIDIA Developer Blog · developer.nvidia.com