--- 格式版本: 2 标题: "Scaling AI Inference Across Multiple GPUs Using NVIDIA TensorRT with Multi-Device Inference Support | NVIDIA Technical Blog" 原文链接: "https://developer.nvidia.com/blog/scaling-ai-inference-across-multiple-gpus-using-nvidia-tensorrt-with-multi-device-inference-support/" 发布日期: "2026-06-25" 发布时间校准状态: "found" 发布时间来源: "llm:strict_original_body" 发布时间证据: "div class=post-info: Jun 25, 2026" 发布时间校准原因: "日期位于标题下方的 post-info 区域，符合文章发布时间的典型位置特征。" 发布时间校准置信度: "100" 发布时间候选数量: 8 发布时间严格候选数量: 2 发布时间原页读取状态: "原页面来自已抓取 HTML" 发布时间未找到原因: "候选日期无效或 LLM 未确认" 发布时间校准时间: "2026-07-20T11:49:32+08:00" 发现时间: "2026-07-20T11:40:00+08:00" 入库时间: "2026-07-20T03:56:31.054Z" 来源平台: "固定入口" 搜索渠道: "fixed_url" 搜索词: "https://developer.nvidia.com/blog/" 匹配关键词: - "GPU" - "Nvlink" 相关厂家: - "NVIDIA" 相关专家: [] 内容类型: "网页" 抓取工具: "AgentKey Scrape" 清洗工具: "AgentKey Markdown + LLM 正文裁剪" 原始附件: [] AI优质: "否" AI打分: 48 AI分档: "非优质" AI质检状态: "不通过" AI打分理由: "文章聚焦TensorRT软件层面的多GPU推理并行策略与NCCL通信，未涉及超节点/AI Rack/机柜级硬件架构、供电散热或互连硬件细节，商业部署信号弱，不符合本项目对机柜级AI基础设施的评分要求。" AI质检模型: "qwen3.6-plus" AI质检时间: "2026-07-20T11:56:31+08:00" AI主题相关性: 8 AI来源权威性: 14 AI新颖性: 12 AI技术细节: 8 AI商业部署信号: 2 AI完整性: 4 图片摘要: - "✗ ./assets/img-a52839cc.jpg | decorative | 装饰性图片，无实质技术内容" - "✓ ./assets/img-fa7bfd11.webp | diagram | 展示列并行与行并行投影的矩阵乘法示意图，解释多GPU并行计算的基础概念" - "★ ./assets/img-d1636e90.webp | diagram | 展示Context Parallelism架构，通过AllGather KV策略在GPU间partition token sequence并聚合注意力输入" - "★ ./assets/img-bcce06d1.webp | diagram | 展示Ring Attention架构，通过GPU间传递部分KV块并更新attention output，降低峰值内存" - "★ ./assets/img-08219572.webp | diagram | 展示DeepSpeed Ulysses架构，通过All-to-all通信在注意力块前后交换序列长度与注意力头维度" - "★ ./assets/img-b6ac731b.webp | chart | Cosmos 3端到端延迟对比图，显示AllGather KV和Ulysses策略在1至8 GPU下的延迟随GPU数量增加而降低" - "★ ./assets/img-1b19adaf.webp | chart | Cosmos 3 Backbone加速比图，显示AllGather KV和Ulysses策略在1至8 GPU下的线性加速效果" - "★ ./assets/img-d9b00bbc.webp | chart | Flux端到端延迟对比图，显示AllGather KV、Ulysses和Ring Attention在1至8 GPU下的延迟表现，Ulysses在N=8时最低" - "✓ ./assets/img-239ce909.webp | other | " - "✓ ./assets/img-dff275fe.webp | other | " - "✓ ./assets/img-bab70b95.webp | other | " - "✓ ./assets/img-9d3a9443.webp | other | " 采集批次: "2026年7月20日11点39分58秒" 采集批次ID: "20260720-113958-114" 去重键: "https://developer.nvidia.com/blog/scaling-ai-inference-across-multiple-gpus-using-nvidia-tensorrt-with-multi-device-inference-support" --- Generative AI workloads are rapidly outgrowing the memory and compute budget of single GPUs. For inference developers building media generation pipelines, the challenge is scaling across multiple devices without sacrificing the critical optimizations—like kernel fusions, memory planning, and quantization—that NVIDIA TensorRT delivers for production deployments. Multi-device inference support, a new feature introduced in TensorRT 11.0, brings native high-performance multi-GPU inference to the TensorRT runtime, enabling multi-device production deployments targeting edge devices. Combining the multi-device inference support in TensorRT with [Torch-TensorRT](https://docs.pytorch.org/TensorRT/), developers can convert and deploy massive PyTorch models out-of-framework, shattering single-device memory and compute limits. Download TensorRT 11.0 with multi-device inference support from [NVIDIA Developer Portal](https://developer.nvidia.com/tensorrt/download) to unlock native, high-performance multi-device acceleration for your models. ## NVIDIA NCCL: The transport layer for distributed inference The NVIDIA Collective Communications Library (NCCL) provides high-performance multi-GPU and multi-node collective operations powering large-scale model training across thousands of GPUs. NCCL automatically selects the optimal transport for a given topology, abstracting NVIDIA NVLink, NVIDIA NVSwitch, PCIe, and InfiniBand behind a uniform interface. By integrating directly with NCCL, TensorRT inherits this transport optimization for inference workloads, when running multi-device inference. For more information on NCCL, see [https://developer.nvidia.com/nccl](https://developer.nvidia.com/nccl). The new multi-device feature covers the full set of NVIDIA NCCL distributed collectives: [AllReduce](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#allreduce), [Broadcast](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#broadcast), [Reduce](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#reduce), [AllGather](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#allgather), [ReduceScatter](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#reducescatter), [AlltoAll](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#alltoall), [Gather](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#gather), and [Scatter](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#scatter). ## Parallelism strategies for distributed inference Distributed inference can be expressed using several parallelism strategies, each with different trade-offs between memory savings, compute scaling, and communication overhead. The most common strategies are tensor parallelism and context parallelism. ### Tensor parallelism In tensor parallelism, the weights of a single layer are partitioned across GPUs. Each GPU computes a shard of the layer’s matrix multiplication and then combines partial results through a collective to produce the full output. This reduces per-device memory weight, making it the natural (and often the only) choice when an individual layer’s weights exceed the memory of a single GPU, independent of the input sequence length or batch size. In a transformer block, column-parallel projections (for example, QKV and the MLP up-projection) are paired with row-parallel projections (the attention output and the MLP down-projection) so that each block requires only a single AllReduce, keeping communication overhead bounded. ![An image showing the side-by-side comparison of column-wise and row-wise parallel projections.](./assets/img-fa7bfd11.webp) Figure 1. Column-wise and row-wise parallel projections ### Context parallelism In context parallelism, the input sequence is partitioned across GPUs along the sequence dimension. Each GPU processes only a slice of the sequence, while collective operations make the global sequence available where needed, such as during attention. Context parallelism is particularly effective for long-sequence workloads, where attention’s quadratic scaling with sequence length makes it the dominant consumer of compute and memory. It is also an especially natural fit for diffusion and DiT models, whose bidirectional attention sidesteps the load-imbalance issues that arise with causal masks. Read the [Context Parallelism for Scalable Million-Token Inference](https://arxiv.org/abs/2411.01783) article for additional details on context parallelism. NVIDIA TensorRT 11.0 introduces support for the \`IDistCollectiveLayer\` primitives required by the various parallelization strategies. The remainder of this post focuses on context parallelism, which directly addresses the dominant cost in modern generative media pipelines: long-sequence attention. #### Context parallelism for generative media Diffusion-based image and video generation pipelines spend a large fraction of their compute and memory budget inside attention blocks operating over long token sequences. A high-resolution image latent or a multi-frame video clip can produce sequences of tens of thousands of tokens per block, and attention scales quadratically with sequence length. #### AllGather KV Context parallelism partitions the sequence across GPUs. Each rank processes a slice of the queries (Q) corresponding to its sequence partition. A straightforward way to implement context parallelism is the AllGather KV approach, where ranks exchange their key (K) and value (V) shards through an AllGather collective before computing local attention, enabling each rank to attend over the full sequence. The result is a per-rank attention output covering the full sequence at the cost of one additional collective per attention block, while the local Q × Kᵀ matrix multiplication shrinks proportionally to the number of ranks. For video and high-resolution image diffusion, this trade-off compounds favorably across denoising steps. Communication overhead per step remains bounded by the sequence-dimension AllGather, while compute and memory savings apply to every attention layer in every step. ![An image showing that context parallelism partitions a token sequence across GPUs along the sequence dimension. Each rank processes a slice of Q locally. Before attention, ranks run AllGather on K and V so every slice can attend over the full sequence.](./assets/img-d1636e90.webp) Figure 2. AllGather KV strategy for context parallelism #### Ring Attention Context parallelism can be implemented in various ways, each presenting distinct trade-offs. One potential improvement over the AllGather KV method is Ring Attention, where communication and computation are overlapped. This enables each GPU to process its local Q simultaneously as the K and V continuously stream past in a ring topology. Ring Attention also reduces the memory footprint: using an online softmax, the full-size K and V tensors do not need to be materialized on any GPU. Read the [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://arxiv.org/pdf/2310.01889) article to learn more about Ring Attention. ![An image showing an alternative implementation of context parallelism, using Ring Attention. The image shows that only partial-sized K and V are present on any given GPU, resulting in peak memory reduction.](./assets/img-bcce06d1.webp) Figure 3. Ring Attention strategy for context parallelism #### DeepSpeed Ulysses For long context (tens of thousands of tokens), an alternative context parallelism implementation approach is DeepSpeed Ulysses. It initially partitions individual samples along the sequence dimension across participating GPUs. Before the attention computation, it employs an all-to-all communication collective on the partitioned Q, K, and V. This ensures that each GPU receives the full sequence length, but only for a non-overlapping subset of the attention heads, enabling them to compute attention in parallel. Finally, a second all-to-all communication gathers the results across the attention heads while repartitioning them along the sequence dimension. Read more about context parallelism for long context in the article [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://arxiv.org/pdf/2309.14509). ![An image showing an alternative implementation of context parallelism, using DeepSpeed Ulysses. The image illustrates the dual all-to-all communication steps that swap the parallelism dimension from the sequence length to the attention heads right before the attention block, and then back to the sequence length immediately afterward.](./assets/img-08219572.webp) Figure 4. DeepSpeed Ulysses strategy for context parallelism ## Benchmarks: Media generation with context parallelism in C++ The following benchmarks evaluate multi-device TensorRT inference for media generation workloads intended for C++ production deployment. Two representative generative AI pipelines are used: a video generation pipeline based on NVIDIA Cosmos 3 and an image generation pipeline based on FLUX.1. These pipelines were first authored in PyTorch, then converted out of the framework using [Torch-TensorRT](https://docs.pytorch.org/TensorRT/) to produce NVIDIA TensorRT engines suitable for deployment in C++ inference applications. This workflow enables developers to retain PyTorch as the model development environment while deploying optimized TensorRT engines in production systems. The benchmarks compare end-to-end latency across different context parallelism strategies: AllGather KV, Ring Attention, and Ulysses. All results were collected on a single node with 8 GPUs. ### Video generation with NVIDIA Cosmos 3 The [NVIDIA Cosmos](https://github.com/nvidia/cosmos) model platform is a world foundation model platform, and the [Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano) model can generate images, video, audio, and other formats based on multimodal inputs, including text, images, and video. We used the [example prompt file](http://huggingface.co/nvidia/Cosmos3-Nano/blob/main/assets/example_t2v_prompt.json) for our benchmarks. Based on these benchmarks, Ulysses is the clear winner when a diffusion model has excessively long context lengths (in the order of tens of thousands of input tokens). ![A column chart showing the latency of different context parallelism strategies on 1,2,4 and 8 GPUs. The image shows that the latency scales well using both the Ulysses and the AllGather KV strategy.](./assets/img-b6ac731b.webp) Figure 5. NVIDIA Cosmos 3 E2E latencies i n milliseconds on N GPUs with different CP strategies ![A column chart showing the scaling of different context parallelism strategies on 1,2,4 and 8 GPUs.](./assets/img-1b19adaf.webp) Figure 6. NVIDIA Cosmos 3 backbone speedup on GPUs with different context parallelism strategies Figure 7. Sample outputs of the NVIDIA Cosmos 3 model on 8 GPUs with different CP strategies ### Image generation with Flux.1 The [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) model from Black Forest Labs can generate images from text descriptions. We used the prompt: “a beautiful photograph of Mt. Fuji during cherry blossom” for our benchmarks. Based on the benchmarks, the Ulysses strategy is the winner in the case of image generation as well, but it’s worth noting that Ring Attention also scaled well to 4 GPUs. ![A column chart showing the latency of different context parallelism strategies on 1,2,4 and 8 GPUs. The image shows that the lowest latency is achieved by applying the Ulysses strategy.](./assets/img-d9b00bbc.webp) Figure 8. Flux E2E latencies in milliseconds on N GPUs with different CP strategies ![A column chart showing the scaling of different context parallelism strategies on 1,2,4 ,and 8 GPUs.](./assets/img-239ce909.webp) Figure 9. Flux backbone speedup on GPUs with different CP strategies ![A side-by-side comparison of the outputs of AllGather KV, Ring Attention ,and Ulysses CP strategies. The output images look very similar to each other. All 3 output images show Mt. Fuji with a cherry tree branch in the foreground.](./assets/img-dff275fe.webp) Figure 10. Sample outputs of the Black Forest Lab Flux.1 model on 8 GPUs with different CP strategies ## Getting started using TensorRT with the multi-device feature TensorRT supports multi-device inference, enabling a single network to execute across multiple GPUs through integrated distributed communication primitives. The core workflow is similar to that of single-device TensorRT. The difference is that the network can now include distributed communication layers. In this guide, it’s assumed that the same network is deployed on all GPU ranks, but this isn’t a strict requirement, and, in theory, each rank can run a different model. A working sample is provided in the [TensorRT repository](https://github.com/NVIDIA/TensorRT/blob/main/samples/sampleDistCollective/sampleDistCollective.cpp). The following guide provides a step-by-step description of how to use the new multi-device feature. **Prerequisites** - Download TensorRT 11 from the [NVIDIA Developer Portal](https://developer.nvidia.com/tensorrt/download). - Install TensorRT 11 following [these instructions](https://developer.nvidia.com/tensorrt#section-get-started-with-tensorrt). - Get a single-node, multi-GPU machine. - Install [OpenMPI](https://docs.open-mpi.org/en/v5.0.x/installing-open-mpi/quickstart.html) in your chosen development environment (bare metal or in a container) 1. **Create a network for multi-device inference** At the network level, multi-device inference is enabled through `IDistCollectiveLayer` for cross-GPU communication. Collective operations can be added directly to a TensorRT network using `INetworkDefinition::addDistCollective`: ```cpp usingnamespacenvinfer1; // create empty network autonetwork = std::unique_ptr(builder->createNetworkV2( 1U << static_cast(kSTRONGLY_TYPED))); auto* input = network->addInput("input", DataType::kFLOAT, Dims2{3, 4}); ITensor& inputTensor = *network->getInput(0); auto* collectiveLayer = network->addDistCollective( inputTensor, CollectiveOperation::kALL_REDUCE, ReduceOperation::kSUM, -1, // root: -1 for collectives without a root rank nullptr, // groups: nullptr means all ranks participate 0 // groupSize ); // set the world size aka total number of GPUs collectiveLayer->setNbRanks(8); ``` For reduction collectives such as `ALL_REDUCE`, `REDUCE`, and `REDUCE_SCATTER`, specify a valid `ReduceOperation`, such as `kSUM`. For non-reduction collectives such as `ALL_GATHER`, `BROADCAST`, `ALL_TO_ALL`, `GATHER`, and `SCATTER`, use `ReduceOperation::kNONE`. Root-based operations, including `BROADCAST`, `REDUCE`, `GATHER`, and `SCATTER`, require a valid root rank. 2. **Build an engine** ```cpp // create builder config autobuilderConfig = std::unique_ptr(builder->createBuilderConfig()); // build engine autoserializedEngine = std::unique_ptr(builder->buildSerializedNetwork (*network, *builderConfig)); ``` 3. **Create execution context** ```cpp autoruntime = std::unique_ptr(createInferRuntime( sample::gLogger.getTRTLogger())); ``` 4. **Bind IO tensors** ```cpp charconst* inputName = engine->getIOTensorName(0); charconst* outputName = engine->getIOTensorName(1); std::vector const& inputChunk = (rank == 0) ? config.rank0Input : config.rank1Input; std::vector outputChunk(config.outputElementCount, 0.0F); size_tconstinputBytes = inputChunk.size() * sizeof(float); size_tconstoutputBytes = outputChunk.size() * sizeof(float); void* dInput = nullptr; void* dOutput = nullptr; CHECK_CUDA(cudaMalloc(&dInput, inputBytes)); CHECK_CUDA(cudaMalloc(&dOutput, outputBytes)); // Copy input data to GPU asynchronously CHECK_CUDA(cudaMemcpyAsync(dInput, inputChunk.data(), inputBytes, cudaMemcpyHostToDevice, stream)); // Set input/output tensor addresses in the execution context context->setInputTensorAddress(inputName, dInput); context->setTensorAddress(outputName, dOutput); context->setInputShape(inputName, Dims2{kINPUT_ROWS, kINPUT_COLS}); ``` 5. **Set communicator and enqueue inference** ```cpp context->setCommunicator(comm); context->enqueueV3(stream); ``` Note: the NCCL communicator must also remain valid for the lifetime of the execution context that uses it. 6. **Kick off inference** Run the application with OpenMPI on 8 GPUs. Each rank selects its local CUDA device, initializes NCCL, creates its own TensorRT engine, creates its own execution context, and attaches the NCCL communicator. ```cpp mpirun -np 8 bash -lc 'export TRT_MY_RANK=$OMPI_COMM_WORLD_RANK; \ export TRT_WORLD_SIZE=$OMPI_COMM_WORLD_SIZE; \ export TRT_NCCL_ID_FILE=/tmp/nccl_id.txt; \ ./sample_dist_collective --op all_reduce' ``` ## Learn more If you want to learn more about the topics introduced in this article, we included some useful links for further reading. **NCCL:** [NVIDIA Collective Communications Library (NCCL)](https://developer.nvidia.com/nccl) **Parallelism:** - [Context Parallelism for Scalable Million-Token Inference](https://arxiv.org/abs/2411.01783) - [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://arxiv.org/pdf/2310.01889) - [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://arxiv.org/pdf/2309.14509) **NVIDIA TensorRT:** - [NVIDIA TensorRT Product Page](https://developer.nvidia.com/tensorrt) - [Download NVIDIA TensorRT](https://developer.nvidia.com/tensorrt/download) - [NVIDIA TensorRT GitHub Repository](https://github.com/NVIDIA/TensorRT) **NVIDIA Torch-TensorRT:** [Torch-TensorRT Documentation](https://docs.pytorch.org/TensorRT/) ![An image showing the side-by-side comparison of column-wise and row-wise parallel projections.](./assets/img-fa7bfd11.webp)_Figure 1. Column-wise and row-wise parallel projections_ ![An image showing that context parallelism partitions a token sequence across GPUs along the sequence dimension. Each rank processes a slice of Q locally. Before attention, ranks run AllGather on K and V so every slice can attend over the full sequence.](./assets/img-d1636e90.webp)_Figure 2. AllGather KV strategy for context parallelism_ ![An image showing an alternative implementation of context parallelism, using Ring Attention. The image shows that only partial-sized K and V are present on any given GPU, resulting in peak memory reduction.](./assets/img-bcce06d1.webp)_Figure 3. Ring Attention strategy for context parallelism_ ![An image showing an alternative implementation of context parallelism, using DeepSpeed Ulysses. The image illustrates the dual all-to-all communication steps that swap the parallelism dimension from the sequence length to the attention heads right before the attention block, and then back to the sequence length immediately afterward.](./assets/img-08219572.webp)_Figure 4. DeepSpeed Ulysses strategy for context parallelism_ ![A column chart showing the latency of different context parallelism strategies on 1,2,4 and 8 GPUs. The image shows that the latency scales well using both the Ulysses and the AllGather KV strategy.](./assets/img-b6ac731b.webp)_Figure 5. NVIDIA Cosmos 3 E2E latencies in milliseconds on N GPUs with different CP strategies_ ![A column chart showing the scaling of different context parallelism strategies on 1,2,4 and 8 GPUs.](./assets/img-1b19adaf.webp)_Figure 6. NVIDIA Cosmos 3 backbone speedup on GPUs with different context parallelism strategies_ ![A column chart showing the latency of different context parallelism strategies on 1,2,4 and 8 GPUs. The image shows that the lowest latency is achieved by applying the Ulysses strategy.](./assets/img-d9b00bbc.webp)_Figure 8. Flux E2E latencies in milliseconds on N GPUs with different CP strategies_ ![A column chart showing the scaling of different context parallelism strategies on 1,2,4 ,and 8 GPUs.](./assets/img-239ce909.webp)_Figure 9. Flux backbone speedup on GPUs with different CP strategies_ ![A side-by-side comparison of the outputs of AllGather KV, Ring Attention ,and Ulysses CP strategies. The output images look very similar to each other. All 3 output images show Mt. Fuji with a cherry tree branch in the foreground.](./assets/img-dff275fe.webp)_Figure 10. Sample outputs of the Black Forest Lab Flux.1 model on 8 GPUs with different CP strategies_ - ![图片](./assets/img-bab70b95.webp) - ![图片](./assets/img-9d3a9443.webp)