用 NVIDIA Attribution 诊断分布式 AI 工作负载故障
Diagnose Distributed AI Workload Failures with NVIDIA Attribution
NVIDIA 介绍如何用 NVIDIA Attribution 诊断分布式 AI 工作负载故障：单个 GPU 或网络结构事件会在其他 rank 上表现为 NCCL 错误、调度器状态变化和大量下游消息，报出的异常只显示工作负载停在哪里，而非故障起点。文章演示在示例工作负载证据上运行 NVIDIA Attribution。
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state...
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state changes, and thousands of downstream messages. The reported exception shows where the workload stopped, not where the failure began. In this post, you’ll learn how to run NVIDIA Attribution on sample workload evidence…
来源：NVIDIA Technical Blog · developer.nvidia.com