A foundation model is a large-scale AI model trained on a vast quantity of broad, unlabeled data that can be adapted to a wide range of downstream tasks. These models, such as GPT-4, Llama, or Stable Diffusion, are characterized by their emergent capabilities and the fact that they serve as a base for many specialized applications through fine-tuning.
The term 'foundation model' was coined and popularized by the Stanford Institute for Human-Centered AI (HAI) in their 2021 report, 'On the Opportunities and Risks of Foundation Models.' The report aimed to describe the paradigm shift in AI where a single, massive model could serve as the foundation for numerous applications, highlighting both the potential and the risks of this homogenization.
The concept of foundation models has become central to the modern AI landscape. They power a vast array of applications, from search engines and chatbots to code generation and scientific discovery. The development and deployment of these models are concentrated in a few large tech companies and research labs due to the immense computational resources required for their training. The widespread use of foundation models has also sparked important discussions around their societal impact, including issues of bias, safety, and economic disruption.