AI cluster networking is the network that joins the thousands of GPUs in an AI cluster, so that they can share their results quickly enough to work as one machine. It is built either with InfiniBand, a specialist network technology whose main supplier today is NVIDIA, or with Ethernet, the ordinary network technology of data centres, run at much higher speeds. It is also called the back-end network or the GPU fabric.
Why does an AI cluster need its own network? When a model is trained on many GPUs, each GPU works on a different slice of the data. After every step, all the GPUs must combine their results, so that every GPU has the same updated model before the next step. This exchange is called all-reduce. The step cannot finish until the slowest GPU has received its results. So the network must carry very large amounts of data with very little delay, and it must not lose packets, because a lost packet has to be sent again while every GPU waits. The page on bandwidth and latency explains these two measures.
Say a GPU cloud provider in Pune runs a cluster of 1,024 GPUs, in 128 servers of 8 GPUs each. Each GPU has its own network card with a port of 400 gigabits per second, so each server has eight such ports. The switches are arranged so that every GPU can send at full speed to any other GPU at the same time. A customer training a model finds that each step takes 2 seconds, and that 0.4 seconds of it is the network exchange. On a slower network, that share would grow, and GPUs that cost lakhs of rupees each would spend more of their time waiting.
The GPUs use a method called RDMA, remote direct memory access. With RDMA, one server's network card writes data straight into another server's memory, without asking either server's CPU to handle it. InfiniBand was built for this. In 2023, more than 80 per cent of AI cluster networks used InfiniBand. By 2025, Ethernet made up more than two-thirds of switch sales for AI clusters. Ethernet now does the same job with a method called RoCE, which stands for RDMA over Converged Ethernet. Ethernet switch makers, the networking chip makers and NVIDIA itself all now sell Ethernet for AI. Some of these switches are white-box switches running SONiC.
Inside each server, the GPUs are joined by a still faster link, described on the page on chip interconnect. Thousands of optical cables and cable assemblies connect the servers to the switches. Clusters in different buildings are joined by data centre interconnect fibre. The AI cluster is the whole machine: servers, network, storage and cooling.