Pre-training is the initial phase of training a large model on a massive, general dataset. This phase is computationally expensive and is where the model learns general features and representations of the data.
A core concept in transfer learning.
The foundation of modern deep learning models.