---
格式版本: 2
标题: "Networking — NVIDIA DGX GB Rack Scale Systems User Guide"
原文链接: "https://docs.nvidia.com/dgx/dgxgb200-user-guide/networking.html"
发布日期: "2026-05-30"
发现时间: "2026-05-30T10:27:36+08:00"
入库时间: "2026-05-30T02:35:27.883Z"
来源平台: "Tavily"
搜索渠道: "tavily_web"
搜索词: "Switch tray"
匹配关键词:
  - "Switch tray"
  - "Compute tray"
  - "NVIDIA"
  - "GPU"
  - "Scale-up"
相关厂家:
  - "NVIDIA"
相关专家:
  []
内容类型: "网页"
抓取工具: "XCrawl Scrape"
清洗工具: "XCrawl Markdown + LLM 正文裁剪"
原始附件:
  []
AI优质: "是"
AI打分: 91
AI分档: "高置信优质"
AI质检状态: "通过"
AI打分理由: "官方NVIDIA DGX GB200/GB300机柜级系统网络架构文档，直接涵盖Compute/Switch Tray、NVLink/IB/以太网多网络拓扑、端口规格与PCIe/RDMA映射，技术细节极深，来源权威，完整度高，属核心工程参…"
AI质检模型: "qwen3.6-plus"
AI质检时间: "2026-06-02T23:59:33+08:00"
AI主题相关性: 20
AI来源权威性: 15
AI新颖性: 18
AI技术细节: 20
AI商业部署信号: 8
AI完整性: 10
采集批次: "2026年5月30日10点27分27秒"
采集批次ID: "20260530-102727-775"
去重键: "https://docs.nvidia.com/dgx/dgxgb200-user-guide/networking.html"
---

# Networking

DGX Grace Blackwell rack scale systems leverage a sophisticated hybrid networking approach to enable seamless, high-performance communication both within the rack and across multiple racks. This model combines the unparalleled speed and coherence of NVLink with the traditional, highly scalable InfiniBand and Ethernet technologies.
* NVLink forms the ultra-fast, memory-coherent scale-up fabric within each rack, making 72 GPUs act as one.
* InfiniBand provides the high-bandwidth, low-latency scale-out compute fabric between racks, enabling massive multi-node AI training clusters.
* Ethernet handles storage, management (in-band and out-of-band), and external connectivity, ensuring all components are integrated and manageable.

## NVLink Networking

NVIDIA NVLink is a high-speed, low-latency interconnect technology designed to overcome PCIe bandwidth limitations in multi-GPU and GPU-to-CPU communication, crucial for HPC and AI platforms. Its key features include ultra-high bandwidth (e.g., 1.8 TB/s bidirectional per GPU with NVLink 5.0) and extremely low latency for direct GPU-to-GPU data transfer. NVLink enables unified memory access across GPUs, simplifying programming, and scales efficiently through NVSwitch chips, which act as non-blocking switches between servers, creating high-bandwidth fabrics. The NVLink fabric interconnects are provided by the connectors at the back of the trays, and connect to the cable cartridges that communicate all compute and NVLink switch trays up and down the back of the system. This fabric is managed by NVIDIA Fabric Manager (or NVLSM), integrated with GPU drivers and the CUDA software stack (like NCCL), which configures the NVSwitch topology and monitors performance. Additionally, NVSwitch incorporates in-network computing (SHARP) to accelerate collective operations, further optimizing distributed AI training by performing computations within the network. For further and detailed information on NVIDIA NMX, please refer to [NVIDIA NMX Documentation](https://docs.nvidia.com/networking/software/nvlink-management-software/index.html)

## Network Overview

A DGX GB rack installation uses several discrete networks:

DGX GB200
1. `externalnet` \- This network provides external communication for the headnode to the enterprise network or to the Internet if allowed.
2. `internalnet` \- This network provides provisioning capabilities as well as system management to the control plane nodes.
3. `dgxnet` \- This network is used to provision and manage the compute trays.
4. `ipminet` \- This network provides Base Command Manager and the headnode access to the out-of-band interfaces of all the components in the rack, including the compute trays, NVLink switch trays, the power shelves, the top-of-rack switches, and the control plane nodes.
5. `computenet` \- This network is used for node communication across racks (east/west traffic).
6. `storagenet` \- This network provides each node access to storage.
7. `failovernet` \- This network is only used by the headnodes when configured in redundant failover configuration.

DGX GB300
1. `externalnet` \- This network provides external communication for the headnode to the enterprise network or to the Internet if allowed.
2. `internalnet` \- This network provides provisioning capabilities as well as system management to the control plane nodes.
3. `dgxnet` \- This network is used to provision and manage the compute trays.
4. `ipminet` \- This network provides Base Command Manager and the headnode access to the out-of-band/BMC interfaces of all the components in the rack, including the compute trays, NVLink switch trays, the power shelves, the top-of-rack switches, and the control plane nodes.
5. `computenet` \- This network is used for RDMA communication across racks (east/west traffic).
6. `storagenet` \- This network provides each node access to storage.
7. `failovernet` \- This network is only used by the headnodes when configured in redundant failover configuration.

The following images identify the network ports used to communicate to these networks on the compute and switch trays. Note that the number in the image corresponds to the number of the network in the numbered list above.

DGX GB200 ![Image showing port locations for each of the networks](https://docs.nvidia.com/_images/network-numbered-ports.png)
DGX GB200 Compute and Switch Trays - Identification of network port locations#

DGX GB300 ![Image showing port locations for each of the networks](https://docs.nvidia.com/_images/network-numbered-ports-gb300.png)
DGX GB300 Compute and Switch Trays - Identification of network port locations#

The following image describes the port types available in the compute tray, and how DGX OS identifies each of the ports. Note that the port names in black are the default configurations. Since the BlueField-3 card ports can be switched to InfiniBand mode, those network port names are called out in light gray.

DGX GB200 ![Image showing network port type for each port](https://docs.nvidia.com/_images/network-port-types.png)
DGX GB200 Compute and Switch Trays - Identification of network port types#

DGX GB300 ![Image showing network port type for each port](https://docs.nvidia.com/_images/network-port-types-gb300.png)
DGX GB300 Compute and Switch Trays - Identification of network port types#

## Compute Tray Network Ports

The following table describes the function and type of each of the compute tray network ports.

Compute Tray Network Ports#
| Port | Port Type | Network / switch |
|---|---|---|
| ConnectX-7 port | OSFP | computenet (InfiniBand) |
| BlueField-3 left port | QSFP | storagenet (Ethernet) |
| BlueField-3 right port | QSFP | dgxnet (Ethernet) |
| Bluefield-3 RJ-45 port | RJ45 | ipminet (Ethernet) |
| 1 GbEthernet port LAN | RJ45 | Not Connected |
| 1 GbEthernet Port BMC | RJ45 | ipminet (Ethernet) |

## Network Interface Names

The following table describes the network interface names for each of the ports on the compute tray.

DGX GB200 Network Interface Names#
| Port | PCIe Bus | Interface name | RDMA |
|---|---|---|---|
| Left Bay OSFP P3 | 00:03:00.0 | ibp3s0 | mlx5_0 |
| Left Bay OSFP P4 | 02:03:00.0 | ibp2p3s0 | mlx5_1 |
| Right Bay OSFP P3 | 10:03:00.0 | ibp16p3s0 | mlx5_4 |
| Right Bay OSFP P4 | 12:03:00.0 | ibp11p3s0 | mlx5_5 |
| Left Bay BF3 P1 | 06:03:00.0 | enP6p3s0f0np0 | mlx5_2 |
| Left Bay BF3 P2 | 06:03:00.1 | enP6p3s0f1np1 | mlx5_3 |
| Right Bay BF3 P1 | 16:03:00.0 | enP22p3s0f0np0 | mlx5_6 |
| Right Bay BF3 P2 | 16:03:00.1 | enP22p3s0f1np1 | mlx5_7 |
| LAN | | enP5p9s0 | |

DGX GB300 Network Interface Names#
| Port | PCIe Bus | Interface name | RDMA |
|---|---|---|---|
| Left Bay OSFP P3 | 00:03:00.0 | ibp3s0 | mlx5_0 |
| Left Bay OSFP P4 | 02:03:00.0 | ibP2p3s0 | mlx5_1 |
| Right Bay OSFP P3 | 10:03:00.0 | ibP16p3s0 | mlx5_2 |
| Right Bay OSFP P4 | 12:03:00.0 | ibP18p3s0 | mlx5_3 |
| Right Bay BF3 P1 | 16:03:00.0 | enP22p3s0f0np0 | mlx5_4 |
| Right Bay BF3 P2 | 16:03:00.1 | enP22p3s0f1np1 | mlx5_5 |
| LAN | 05:09:00.0 | enP5p9s0 | |

## NVLink Switch Network Ports

The following table describes the function and type of each of the NVLink switch network ports.

NVLink Switch Network Ports#
| Port (from left to right) | Port Type | Network / switch |
|---|---|---|
| USB port | USB | Not applicable |
| RS232 to CPU UART or for BMC | RJ45 | Not connected |
| 1GbEthernet port BMC | RJ45 | ipminet (Ethernet) |
| 1GbEthernet for switch management | RJ45 | ipminet (Ethernet) |
| 1GbEthernet for switch management | RJ45 | ipminet (Ethernet) |
| Leakage connector | RJ45 | Not connected |