Tech, ExplainedCloud platforms and developer tools
Part

Version control for large files

Game art and chip designs that break Git.

INSIDE: CODE REPOSITORYHOW IT WORKS · REPAINTING A 1.8 GB CAR TEXTURE IN HYDERABADThe studio's4 TB projectsits on acentral serverAn artist checksout the 1.8 GBtexture, whichlocks itAnother artistsees "Locked byanother user"Two days later,the file issubmitted asversion 47All 46 earlierversions stay onthe serverThe nightlybuild collectsthe latest filesALSO TALKS TOElectronic design automation (EDA)↗ SEMICONDUCTORSVFX pipeline software↗ FILM PRODUCTION AND VFXINSIDE: CODE REPOSITORYHOW IT WORKS · REPAINTING A1.8 GB CAR TEXTURE IN HYDERABADThe studio's 4 TB projectsits on a central serverAn artist checks out the1.8 GB texture, whichlocks itAnother artist sees"Locked by another user"Two days later, thefile is submittedas version 47All 46 earlier versionsstay on the serverThe nightly buildcollects the latest filesALSO TALKS TOElectronic design automation (EDA)↗ SEMICONDUCTORSVFX pipeline software↗ FILM PRODUCTION AND VFXINSIDE: CODE REPOSITORY · REPAINTING A 1.8 GBCAR TEXTURE IN HYDERABADThe studio's 4 TB project sits on acentral server↓An artist checks out the 1.8 GB texture,which locks it↓Another artist sees"Locked by another user"↓Two days later, the file is submitted asversion 47↓All 46 earlier versionsstay on the server↓The nightly buildcollects the latest filesALSO TALKS TOElectronic design automation (EDA)↗ SEMICONDUCTORSVFX pipeline software↗ FILM PRODUCTION AND VFX
A 1.8 GB texture is locked to one artist, submitted to the central server as version 47 with 46 earlier versions kept, and collected by the nightly build.
opens a page inside this one a page in the same scene a page in another scene a concept page a label only, with no page

Version control for large files is software that keeps every version of very large files, such as a game's 3D models or a chip's design data, which ordinary Git handles badly. It is used mainly by game studios and chip designers, and India has many of both, in cities such as Hyderabad, Pune and Bengaluru.

Why does Git struggle? Git was made for text files, such as source code. When a developer copies a Git repository to a laptop, Git copies every version of every file. For code, that is small. But a game's files are images, sounds, 3D models and level maps. One file can be 2 GB, and it may have hundreds of versions. Copying the full history of such files to every laptop is not practical. Git also cannot merge two people's edits to a binary file, a file that is not plain text. If two artists edit the same 3D model at once, there are two versions, and one artist's work is lost.

Version control for large files solves both problems. The history stays on a central server. Each person downloads only the latest version of the files they need. And a file can be locked. When an artist opens a file for editing, the server marks it as locked in her name. Nobody else can submit changes to it until she checks it back in.

Consider a game studio in Hyderabad with 150 artists and programmers, working on a racing game. The project holds 4 TB of files. An artist needs to repaint a car's body. She checks out the 1.8 GB texture file, which locks it. Another artist tries to open it and sees "Locked by another user". The first artist finishes in two days and submits the file with a note. The server stores it as version 47, and keeps all 46 earlier versions. That night, the studio's build machine collects the latest files and builds the game in its game engine, ready for testing the next morning.

Chip designers have the same problem. A chip is described, simulated and laid out in electronic design automation (EDA) software, and the layout files can be very large. A design team must know exactly which version of each block went into the chip that was sent for manufacture. Film studios keep every version of each shot in their VFX pipeline for a similar reason.

How is this different from an ordinary code repository? Git gives every developer the full history, and lets two people edit the same file and merge their changes later. Version control for large files keeps the history on one server, and lets only one person edit a binary file at a time. Some teams use both: Git for the code, and large-file version control for the art.

Tech, Explained is a part of ingrid.fyi. Every page explains one piece of technology in plain words.