Semanlink - Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AI

Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AI - Sebastian Raschka

Tags:

LoRA

> how to tune an LLM with Low-Rank Adaptation (LoRA) in a computationally efficient manner

[tweet](https://twitter.com/rasbt/status/1651226178353614854) [Karpathy](https://twitter.com/karpathy/status/1651288867247640578)

> the paper LoRA: Low-Rank Adaptation of Large Language Models proposes to decompose the weight changes, ΔW, into a lower-rank representation. (To be technically correct, LoRA does not decompose the matrices directly, but it learns the decomposed matrices via backpropagation).
>
> suppose ΔW is the weight update for an A × B weight matrix. Then, we can decompose the weight update matrix into two smaller matrices: ΔW = WA WB, where WA is an an A × r-dimensional matrix, and WB is an an r × B-dimensional matrix.

LORA and LLama :

> Lit-LLaMA repository a simple, readable reimplementation of Meta’s popular LLaMA model. Besides code for training and running LLaMA itself (with the original Meta LLaMA weights), it also contains code for finetuning LLaMA using LLaMA-Adapter and LoRA.

About This Document

sl:bookmarkOf : https://lightning.ai/pages/community/article/lora-llm/
sl:creationDate : 2023-07-27
sl:creationTime : 2023-07-27T01:54:57Z

File info

Bookmark of: https://lightning.ai/pages/community/article/lora-llm/

Documents with similar tags (experimental)

LlamaIndex 🦙 sur X : "Fine-tuning Embedding Models for RAG with LoRA'

Tags:

2024-04-23 About

Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)

Tags:

2023-11-20 About

Batched Low-Rank Adaptation of Foundation Models

Tags:

LoRA

2023-11-12 About

[2309.12307] LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

Tags:

2023-09-26 About

peft/examples/token_classification/peft_lora_token_cls.ipynb at main · huggingface/peft

Tags:

2023-08-27 About

[2307.13269] LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

Tags:

2023-08-08 About

What is low-rank adaptation (LoRA)? - TechTalks

Tags:

2023-07-27 About

LoRA for semantic similarity tasks

Tags:

2023-07-21 About

How to use Alpaca-LoRA to fine-tune a model like ChatGPT

Tags:

2023-05-26 About

Google "We Have No Moat, And Neither Does OpenAI"

Tags:

> low-cost public involvement was enabled
by a vastly cheaper mechanism for fine tuning called low
rank adaptation ()[LoRA](tag:lora)

> **Part of what makes LoRA so effective is that ... it’s stackable.**
>
> By contrast, training giant models from scratch not only
throws away the pretraining, but also any iterative
improvements that have been made on top.

> LoRA updates are very cheap to produce (~$100) for the
most popular model sizes.

> Many of these projects are saving time by training on
small, highly curated datasets...
> These
datasets are built using synthetic methods (e.g. filtering
the best responses from an existing model) and
scavenging from other projects

> Directly Competing With Open Source
Is a Losing Proposition

> Paradoxically, the one clear winner in all of this is Meta.
Because the leaked model was theirs ([LLaMA](tag:llama)), they have
effectively garnered an entire planet's worth of free labor.
Since most open source innovation is happening on top of
their architecture, there is nothing stopping them from
directly incorporating it into their products.

2023-05-04 About

[2106.09685] LoRA: Low-Rank Adaptation of Large Language Models

Tags:

2023-03-21 About