Semanlink - sl:creationDate : 2023-07-27

Search Tag:

Search Doc:

Preferences...

Tag Cloud

C'est la Terre qui se révolte - Introduction - LoRA - Orque - Vengeance -

sl:creationDate : 2023-07-27

Dans le détroit de Gibraltar, la vendetta de l’orque Gladis

Tags:

2023-07-27 About

Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AI - Sebastian Raschka

Tags:

LoRA

> how to tune an LLM with Low-Rank Adaptation (LoRA) in a computationally efficient manner

[tweet](https://twitter.com/rasbt/status/1651226178353614854) [Karpathy](https://twitter.com/karpathy/status/1651288867247640578)

> the paper LoRA: Low-Rank Adaptation of Large Language Models proposes to decompose the weight changes, ΔW, into a lower-rank representation. (To be technically correct, LoRA does not decompose the matrices directly, but it learns the decomposed matrices via backpropagation).
>
> suppose ΔW is the weight update for an A × B weight matrix. Then, we can decompose the weight update matrix into two smaller matrices: ΔW = WA WB, where WA is an an A × r-dimensional matrix, and WB is an an r × B-dimensional matrix.

LORA and LLama :

> Lit-LLaMA repository a simple, readable reimplementation of Meta’s popular LLaMA model. Besides code for training and running LLaMA itself (with the original Meta LLaMA weights), it also contains code for finetuning LLaMA using LLaMA-Adapter and LoRA.

2023-07-27 About

What is low-rank adaptation (LoRA)? - TechTalks

Tags:

2023-07-27 About