Semanlink - sl:creationTime : 2023-07-27T01:54:57Z

Search Tag:

Search Doc:

Preferences...

Tag Cloud

LoRA -

sl:creationTime : 2023-07-27T01:54:57Z

Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AI - Sebastian Raschka

Tags:

LoRA

> how to tune an LLM with Low-Rank Adaptation (LoRA) in a computationally efficient manner

[tweet](https://twitter.com/rasbt/status/1651226178353614854) [Karpathy](https://twitter.com/karpathy/status/1651288867247640578)

> the paper LoRA: Low-Rank Adaptation of Large Language Models proposes to decompose the weight changes, ΔW, into a lower-rank representation. (To be technically correct, LoRA does not decompose the matrices directly, but it learns the decomposed matrices via backpropagation).
>
> suppose ΔW is the weight update for an A × B weight matrix. Then, we can decompose the weight update matrix into two smaller matrices: ΔW = WA WB, where WA is an an A × r-dimensional matrix, and WB is an an r × B-dimensional matrix.

LORA and LLama :

> Lit-LLaMA repository a simple, readable reimplementation of Meta’s popular LLaMA model. Besides code for training and running LLaMA itself (with the original Meta LLaMA weights), it also contains code for finetuning LLaMA using LLaMA-Adapter and LoRA.

2023-07-27 About