A parameter-efficient fine-tuning (PEFT) method that enhances LoRA (Low-Rank Adaptation). DoRA, or Weight-Decomposed Low-Rank Adaptation, works by decomposing a pre-trained weight matrix into two components: magnitude and direction. It applies LoRA specifically to the direction component, while making the magnitude component trainable separately. This decoupling allows for more effective and stable fine-tuning by addressing the issue that LoRA updates both magnitude and direction simultaneously, which can lead to suboptimal results. DoRA often achieves the performance of full fine-tuning with significantly fewer trainable parameters.
Introduced in the 2024 paper 'DoRA: Weight-Decomposed Low-Rank Adaptation' by researchers from NVIDIA and others.
Following its publication, DoRA was quickly recognized for its performance gains over LoRA and has been integrated into major machine learning libraries, including Hugging Face's PEFT (Parameter-Efficient Fine-Tuning) package. It is increasingly being adopted by practitioners for fine-tuning large language models (LLMs) and vision models where maximizing performance while minimizing computational cost is critical.