---
格式版本: 2
标题: "How NVIDIA’s Inference Software Stack Powers the Lowest Token Cost | NVIDIA Blog"
原文链接: "https://blogs.nvidia.com/blog/inference-software-lowest-token-cost/"
发布日期: "2026-06-30"
发布时间校准状态: "found"
发布时间来源: "llm:strict_original_body"
发布时间证据: "time class=related-news-date nvidia-article-date datetime=2026-06-30T08:00:57-07:00: Jun 30, 2026"
发布时间校准原因: "该日期来自HTML metadata中的article-date字段，位于标题附近，符合文章发布时间的特征。"
发布时间校准置信度: "100"
发布时间候选数量: 36
发布时间严格候选数量: 12
发布时间原页读取状态: "原页面来自已抓取 HTML"
发布时间未找到原因: "候选日期无效或 LLM 未确认"
发布时间校准时间: "2026-07-20T11:12:21+08:00"
发现时间: "2026-07-20T09:24:42+08:00"
入库时间: "2026-07-20T03:20:39.871Z"
来源平台: "NVIDIA Blog 搜索"
搜索渠道: "source_template"
搜索词: "https://blogs.nvidia.com/?s=NVL72"
匹配关键词:
  - "NVL72"
  - "GPU"
  - "Nvlink"
相关厂家:
  - "NVIDIA"
相关专家:
  []
内容类型: "网页"
抓取工具: "AgentKey Scrape"
清洗工具: "AgentKey Markdown + LLM 正文裁剪"
原始附件:
  []
AI优质: "否"
AI打分: 72
AI分档: "召回候选"
AI质检状态: "不通过"
AI打分理由: "NVIDIA官方发布，提及GB300 NVL72及推理软件栈优化，但核心聚焦软件层token成本与框架生态，缺乏超节点硬件架构、供电散热互连等物理层细节，商业部署信息较泛。"
AI质检模型: "qwen3.6-plus"
AI质检时间: "2026-07-20T11:20:39+08:00"
AI主题相关性: 12
AI来源权威性: 14
AI新颖性: 16
AI技术细节: 10
AI商业部署信号: 10
AI完整性: 10
图片摘要:
  - "★ ./assets/img-31744db1.jpg | chart | SemiAnalysis数据图显示，通过软件优化，GB300 NVL72系统在一个月内将每百万Token成本降低5倍，同时提升交互性。"
  - "★ ./assets/img-26961be8.jpg | diagram | 架构图对比传统同质化请求与Agentic分布式架构，展示Prompt、编排、上下文、推理、行动及工具调用的复杂交互流程。"
  - "★ ./assets/img-d68d6ccf.jpg | diagram | NVIDIA推理软件栈全景架构图，涵盖从基础设施访问、内核库、运行时（SGLang/vLLM/TensorRT-LLM）到生产运营（Dynamo/Orches…"
  - "★ ./assets/img-3ebe4a34.jpg | chart | 柱状图展示NVIDIA Blackwell吞吐量随软件优化叠加（解耦服务、大EP、NVFP4、MTP）从基准1x提升至20x的过程。"
  - "★ ./assets/img-52dbe4a7.jpg | chart | PyTorch下载量增长曲线及NVIDIA协同开发时间轴，标注了cuDNN、Tensor Cores、FlashAttention等关键技术支持的集成节点。"
  - "✓ ./assets/img-2af8bdc3.png | chart | 柱状图显示软件优化使NVIDIA Blackwell性能在一个月内提升5倍。"
  - "✗ ./assets/img-83d2f103.jpg | photo | 文章底部推荐阅读/相关文章缩略图，非正文内容配图"
  - "✗ ./assets/img-1f6575f1.png | diagram | 文章底部推荐阅读/相关文章缩略图，非正文内容配图"
  - "✓ ./assets/img-3511966c.jpg | other | "
  - "✓ ./assets/img-e47e99dc.png | other | "
采集批次: "2026年7月20日9点23分34秒"
采集批次ID: "20260720-092334-062"
去重键: "https://blogs.nvidia.com/blog/inference-software-lowest-token-cost"
---

As organizations move from AI pilots to production AI factories, infrastructure decisions have shifted from peak chip specifications to cost per [token](https://blogs.nvidia.com/blog/ai-tokens-explained/): how many useful tokens they can deliver per dollar, per watt and within required latency targets.

Codesigned with NVIDIA GPUs, CPUs, networking and systems, and strengthened by a broad open source ecosystem, NVIDIA’s full-stack inference software continuously improves hardware performance. On the [NVIDIA Blackwell](https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture/) platform, the software stack has already reduced token costs by up to 5x on the DeepSeek V4 model in just one month.

![图片](./assets/img-31744db1.jpg)

SemiAnalysis InferenceX results comparing token cost and interactivity for NVIDIA GB300 NVL72 systems with SGLang and the NVIDIA Dynamo inference framework.

Leading companies and inference providers are already seeing the compounding value of NVIDIA’s inference software stack on Blackwell:

- [Baseten](https://www.baseten.co/products/model-apis/) used the NVIDIA TensorRT-LLM open source library to serve DeepSeek V4 Pro on Blackwell GPUs for reasoning, coding and long-context workloads, applying proprietary runtime optimizations to deliver up to 50% more tokens per second.
- [Cognition](https://cognition.com/blog/swe-1-6) is using the NVIDIA Dynamo inference framework to manage inference GPUs, giving its team a ready-made path to scale reinforcement learning workloads without needing to build that infrastructure from scratch.
- [Deep Infra](https://deepinfra.com/blog/deepinfra-nvidia-inference-stack) uses the NVIDIA inference software stack to serve frontier open source models performantly on Blackwell from day zero, including DeepSeek V4.
- DigitalOcean helped Hippocratic AI use NVIDIA inference software on Blackwell GPUs to serve healthcare AI faster and more efficiently, increasing inference throughput by 30% while maintaining a sub-half-second time to first response across 10 million patient calls.
- [Together AI](https://youtu.be/10Kb3IB0d70) used NVIDIA TensorRT-LLM on Blackwell to help Cursor accelerate the path from model optimizations to production endpoints for its real-time coding experience.

## Why Software Matters for Inference Economics

Traditional web, search and software-as-a-service workloads were relatively predictable: A user might load a page, refresh a feed or update a business record. These requests typically followed similar software paths, reading from or writing to a database, and scaled by adding more of the same servers.

Agentic AI is different.

![图片](./assets/img-26961be8.jpg)

Agentic AI runs distributed, stateful workflows that span LLMs, tools, memory, security, networking and accelerated computing across the data center.

Agents can reason, plan, call tools, spin up specialist subagents and manage massive context across multi-turn workflows. They turn a single request into a distributed computing problem that can span hundreds of subagents, thousands of tasks and multiple large language models, running across GPUs, CPUs, DPUs and storage systems.

The software stack determines whether that complexity turns into wasted capacity or lower [cost per token](https://blogs.nvidia.com/blog/lowest-token-cost-ai-factories/).

Lower cost per token comes from turning individual optimizations into system-level performance. NVIDIA’s inference software stack does this by connecting three layers:

- **Production Operation:** Coordinates distributed serving, orchestration, autoscaling and memory management so inference can run across the right compute and storage resources.
- **Application Acceleration:** Runs models with high performance while giving developers room to tune and customize, using runtime optimizations such as overlapping compute and communication and kernel fusion.
- **Infrastructure Access:** Exposes NVIDIA GPU, networking, memory and system capabilities without requiring developers to manage every device instruction set or data-transfer protocol directly.
![图片](./assets/img-d68d6ccf.jpg)

The NVIDIA software stack spans model serving, runtime scheduling, kernels, communication libraries and hardware-aware optimizations, enabling rapid performance gains and lower serving costs as improvements compound across layers.

When these layers work as one system, individual optimizations compound.

Disaggregated serving, large expert parallelism over [NVIDIA NVLink](https://www.nvidia.com/en-us/data-center/nvlink/) interconnect technology, NVFP4 precision and multi-token prediction each deliver meaningful gains on their own. Combined, they increase throughput by up to 20x.

The chart below shows the result. Capturing that gain in production is complex, requiring coordination across the full inference stack — from production operations and model runtimes to kernels, communication libraries and hardware access. NVIDIA’s inference software stack is designed to make those layers work together so each optimization can build on the others.

![图片](./assets/img-3ebe4a34.jpg)

Stacking software optimizations compounds performance gains, increasing NVIDIA Blackwell token throughput per GPU from baseline to up to 20x with disaggregated serving, large expert parallelism (Large EP), NVFP4 and multi-token prediction (MTP).

## Open Source Amplifies the Full-Stack Advantage

That same full-stack foundation is amplified by the open source ecosystem. Many of today’s most widely used open source AI frameworks and inference projects are built natively on [NVIDIA CUDA](https://developer.nvidia.com/cuda), which means new research and software optimizations run with leading performance on NVIDIA GPUs from day zero.

PyTorch is a leading example. Launched in 2016 with native CUDA support, PyTorch has coevolved with NVIDIA’s architecture, giving developers access to innovations such as Tensor Cores, Transformer Engine and NVFP4 directly through a familiar framework.

When breakthroughs such as [DFlash speculative decode](https://developer.nvidia.com/blog/boost-inference-performance-up-to-15x-on-nvidia-blackwell-using-dflash-speculative-decoding/), which delivers up to 15x more throughput on existing hardware, or [FastVideo](https://haoailab.com/blogs/fastvideo_realtime_1080p/), which generates 1080p videos in less than five seconds, land in PyTorch, they can run instantly on NVIDIA, helping AI factories convert research progress into lower token costs.

![图片](./assets/img-52dbe4a7.jpg)

NVIDIA and PyTorch codevelopment helps bring new AI software innovations to developers, helping turn CUDA-native advances into production performance as PyTorch adoption grows.

The same open source momentum is why when a new frontier open model like DeepSeek V4 is released, leading inference frameworks like vLLM and SGLang have [day-zero deployment recipes](https://developer.nvidia.com/blog/build-with-deepseek-v4-using-nvidia-blackwell-and-gpu-accelerated-endpoints/) for the NVIDIA Blackwell architecture — making the model accessible across millions of Blackwell GPUs. It’s also why DeepSeek V4 performance on Blackwell improved by up to 5x within about a month across vLLM and [SGLang](https://pytorch.org/blog/serving-deepseek-v4-on-gb300-with-sglang-5x-higher-throughput-at-the-same-interactivity-since-day-0/) frameworks, cutting token costs to roughly one-fifth of previous levels.

![图片](./assets/img-2af8bdc3.png)

SemiAnalysis InferenceX results comparing token throughput at same interactivity for NVIDIA GB200 NVL72 systems with vLLM and the NVIDIA Dynamo inference framework.

That’s the open source flywheel: more developers optimize CUDA-native inference paths, more production deployments feed back into the ecosystem and each software improvement increases delivered token output while lowering cost per token over time.

*Explore how software multiplies hardware performance in this* [*NVIDIA AI Podcast on tokenomics*](https://www.youtube.com/watch?v=zNuOOMM20Tk) *and this* [*inference solutions page*](https://www.nvidia.com/en-us/solutions/ai/inference/)*.*

![图片](./assets/img-31744db1.jpg)SemiAnalysis InferenceX results comparing token cost and interactivity for NVIDIA GB300 NVL72 systems with SGLang and the NVIDIA Dynamo inference framework.

![图片](./assets/img-26961be8.jpg)Agentic AI runs distributed, stateful workflows that span LLMs, tools, memory, security, networking and accelerated computing across the data center.

![图片](./assets/img-d68d6ccf.jpg)The NVIDIA software stack spans model serving, runtime scheduling, kernels, communication libraries and hardware-aware optimizations, enabling rapid performance gains and lower serving costs as improvements compound across layers.

![图片](./assets/img-3ebe4a34.jpg)Stacking software optimizations compounds performance gains, increasing NVIDIA Blackwell token throughput per GPU from baseline to up to 20x with disaggregated serving, large expert parallelism (Large EP), NVFP4 and multi-token prediction (MTP).

![图片](./assets/img-52dbe4a7.jpg)NVIDIA and PyTorch codevelopment helps bring new AI software innovations to developers, helping turn CUDA-native advances into production performance as PyTorch adoption grows.

![图片](./assets/img-2af8bdc3.png)SemiAnalysis InferenceX results comparing token throughput at same interactivity for NVIDIA GB200 NVL72 systems with vLLM and the NVIDIA Dynamo inference framework.

![NVIDIA Unlocks AI Compute at Scale, Inviting Partners to Power the AI Infrastructure Buildout](./assets/img-3511966c.jpg)

![How NVIDIA’s Inference Software Stack Powers the Lowest Token Cost](./assets/img-e47e99dc.png)