A parallel file system is storage software that spreads each file across many storage servers, so that thousands of computers can read different parts of the same file at the same time. It is used to train AI models, and to run supercomputers for science and engineering. This kind of computing is called high-performance computing, or HPC.
Why is it needed? AI models are trained on GPUs and other AI chips, which run thousands of calculations at once. Thousands of GPUs work together in an AI cluster. Each GPU must receive new training data many times a second. GPUs are very expensive. So every second that they spend waiting for data is money wasted.
An ordinary storage array answers every request through one or two controllers. A controller is the computer inside the array that reads and writes the drives. When thousands of GPUs ask for data at once, their requests queue at these controllers, and the GPUs wait.
A parallel file system removes that queue. It cuts each file into pieces, called stripes, and stores the stripes on many storage servers. A separate metadata server keeps a list of which server holds which stripe. A GPU server first asks the metadata server where the stripes are. Then it reads them directly from all the storage servers at once. With 100 storage servers, the total reading speed is roughly 100 times that of one server.
Imagine a GPU cloud company in Navi Mumbai that rents 1,024 GPUs to an AI start-up. The start-up trains a speech model for Indian languages on 40 TB of recorded audio. Its storage is a parallel file system on 32 flash servers, which together deliver about 200 gigabytes a second. That is enough for the GPUs to read all 40 TB in under four minutes. Every two hours, the training job also saves a checkpoint. A checkpoint is a full copy of the model so far, so that training can restart from it if a server fails. For a large model, one checkpoint can be several terabytes. The file system must write it within a few minutes, because the GPUs stop and wait while it is written.
Lustre is the best-known open-source parallel file system, and many of the world's supercomputers use it. Newer systems are built for flash drives and for GPU clouds. The training itself is split across machines by a distributed computing framework. A GPU-as-a-service platform decides whose job runs on which GPU.
A parallel file system differs from object storage. Object storage keeps each file whole, and it is cheap and nearly unlimited, but it is too slow to feed GPUs directly. Many AI teams keep their full datasets in object storage, and copy the part they are training on into the parallel file system.