Foundation models

Foundation models are very large AI models trained on a very wide range of data, which other companies then use as a base to build their own products. A language model such as the one behind a chatbot is the best-known example. The word "foundation" is used because many different applications are built on top of one such model.

Here is how it works in general. A company first trains the model on a huge collection of text, images, audio or other data. This takes thousands of special chips and weeks or months, and costs a very large sum. The result is a model with a general ability, for example to read, write or describe images. Other companies then use it in two ways. They can send questions to it through an API, or they can adjust it with their own data so that it does one task better. They add their own software and data around it to make a product.

A model is made of numbers, called parameters, which are learned during training. The large models have billions of them. A company can use a model in two ways. In prompting, it gives instructions and examples in the question. In fine-tuning, it trains the model a little further on its own data. Each answer also costs computing power, so using a model at large scale is a continuing cost, not a one-time one. A model knows only what was in its training data, and it can state wrong things with confidence. So companies often add their own documents at the moment of the question, so that the model answers from those.

Take, for example, a bank that builds a customer chat service on top of a language model. The model understands the question, and the bank's own software looks up the account. Consider a hospital that adjusts an image model to read X-rays; see radiology AI. Say a robot company uses a model trained on video of tasks to guide a robot arm; see robot foundation models.

Because training is so costly, only a few companies build these models, and many others build on them. Countries also build their own, as in sovereign AI. Useful pages are language models, voice models, building on a model, training and serving and hallucination, which is a risk of such models.

How is it different from a model made for one task? A task model is trained on specific data to do one job, such as spotting a defect. A foundation model is trained broadly, then used for many jobs.