NVIDIA 如何通过协同设计最大化 AI 工厂中每一字节的价值
Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory
NVIDIA 提出五种协同设计方法，以最大化 AI 工厂中 HBM 每一字节的有效产出。在 AI 工厂中，HBM 是最快、最贵且竞争最激烈的内存层级，冗余权重副本、空闲会话的 KV cache 和重算前缀都会挤占用于生成 token 的字节。核心问题已从 GPU 拥有多少内存转向每字节能交付多少有效工作。
In an AI factory, high bandwidth memory (HBM)is the fastest, most expensive, and most contested memory tier. Every byte spent on a redundant weight copy, an...
In an AI factory, high bandwidth memory (HBM)is the fastest, most expensive, and most contested memory tier. Every byte spent on a redundant weight copy, an idle session’s KV cache, or a recomputed prefix is a byte that is not being used to produce tokens. The question is no longer how much memory a GPU has, but how much useful work each byte delivers. The generational data in Figure 1 below…
来源：NVIDIA Developer Blog · developer.nvidia.com