AdaLoRA paper review

https://medium.com/@aseer-ansari/the-lora-family-rosa-dora-adalora-delta-lora-lora-vera-lorafa-lora-drop-79989803ace9

The Lora Family : RoSA, DoRA, Adalora, Delta LoRA, LoRA+ , VeRA, LoraFA, Lora-drop

 

The Lora Family

: RoSA, DoRA, Adalora, Delta LoRA, LoRA+ , VeRA, LoraFA, Lora-drop

LoRA: Low-Rank Adaptation of Large Language Models by EJ Hu · 2021 · Cited by 12522 

AdaLoRA: Adaptive Budget Allocation for Parameter- ... by Q Zhang · 2023 · Cited by 556

DoRA: Weight-Decomposed Low-Rank Adaptation by SY Liu · 2024 · Cited by 358

VeRA: Vector-based Random Matrix Adaptation by DJ Kopiczko · 2023 · Cited by 172

LoRA+: Efficient Low Rank Adaptation of Large Models by S Hayou · 2024 · Cited by 146 

LoRA-FA: Memory-efficient Low-rank Adaptation for Large ... by L Zhang · 2023 · Cited by 107 

Delta-LoRA: Fine-Tuning High-Rank Parameters with the ... by B Zi · 2023 · Cited by 49 

RoSA: Accurate Parameter-Efficient Fine-Tuning via ... by M Nikdan · 2024 · Cited by 30

LoRA-drop: Efficient LoRA Parameter Pruning based on ... by H Zhou · 2024 · Cited by 15 

 

• LoRA: Low-Rank Adaptation of Large Language Models (2021)​
  • 링크: https://arxiv.org/abs/2106.09685
• ADALORA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023년 3월)​
  • 링크: https://arxiv.org/abs/2303.10512
• LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning (2023년 8월)​
  • 링크: https://arxiv.org/abs/2308.03303
• Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices (2023년 9월)​
  • 링크: https://arxiv.org/abs/2309.02411
• VeRA: Vector-based Random Matrix Adaptation (2023년 10월)​
  • 링크: https://arxiv.org/abs/2310.11454
• RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation (2024년 1월)​
  • 링크: https://arxiv.org/abs/2401.04679
• DoRA: Weight-Decomposed Low-Rank Adaptation (2024년 2월)​
  • 링크: https://arxiv.org/abs/2402.09353
• LoRA+: Efficient Low Rank Adaptation of Large Models (2024년 2월)​
  • 링크: https://arxiv.org/abs/2402.12354
• LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation (2024년 2월)​ResearchGate+3허깅페이스+3aXi+3
  • 링크: https://arxiv.org/abs/2402.07721

 

 

--> LORA family 중에 가장 인용수가 높음.  읽어봐야겠다고 선정!

 

관련 블로그

https://jiogenes.github.io/%EB%85%BC%EB%AC%B8%EB%A6%AC%EB%B7%B0/2024/03/07/adalora.html

https://velog.io/@khs0415p/Paper-DoRA

https://jiogenes.github.io/%EB%85%BC%EB%AC%B8%EB%A6%AC%EB%B7%B0/2024/03/07/adalora.html

https://towardsdatascience.com/an-overview-of-the-lora-family-515d81134725/