Semanlink - Favoris ; Technical girls and guys ; Language Model AND LM Fine-tuning

Printer friendly

Search Tag:

Search Doc:

Preferences...

Favoris ; Technical girls and guys ; Language Model AND LM Fine-tuning

Common descendants

21 Documents

Jeremy Howard sur X : "FSDP/QDoRA with Llama3 : I believe that this combination is likely to create better task-specific models than anything else at any cost."

Tags:

2024-04-23 About

[2404.03592] ReFT: Representation Finetuning for Language Models

Tags:

2024-04-08 About

Answer.AI - You can now train a 70b language model at home

Tags:

2024-03-09 About

[2311.11077] Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning

Tags:

2023-11-25 About

How many Labelled Examples do you need for a BERT-sized Model to Beat GPT-4 on Predictive Tasks? - Speaker Deck

Tags:

2023-10-27 About

fast.ai - Can LLMs learn from a single example?

Tags:

2023-10-21 About

The End of Finetuning — with Jeremy Howard of Fast.ai

Tags:

2023-10-20 About

[2302.06600] Task-Specific Skill Localization in Fine-tuned Language Models

Tags:

2023-08-25 About

Easily Finetune Llama 2 for Your Text-to-SQL Applications | by Jerry Liu | Aug, 2023

Tags:

2023-08-18 About

Modular and Parameter-Efficient Fine-Tuning for NLP Models

Tags:

2023-08-08 About

EMNLP 2022 Tutorial — “Modular and Parameter-Efficient Fine-Tuning for NLP Models” | by Oscar Shih | Jun, 2023 | Medium

Tags:

2023-08-08 About

Sanjeev Arora sur Twitter : "new `skills' induced by LLM fine-tuning can be localized in tiny fraction of the model."

Tags:

2023-07-07 About

Sanjeev Arora sur Twitter : "Fine-tuning language models using just forward pass!...r

Tags:

2023-06-09 About

Andrej Karpathy sur Twitter : "There's a chance that LoRA finetunes work so well that it dramatically alters the finetuning vs. retrieval + few-shot prompting power dynamic..."

Tags:

2023-04-21 About

(((ل()(ل() 'yoav))))👾 sur Twitter : "there is this genre of papers that show you can train/tune only some subset of a network's weights, freezing the rest, and things still work as well as (or better than) full training/tuning..."

Tags:

2023-02-15 About

Sanjeev Arora sur Twitter : "A priori, fine-tuning a huge LM on a few datapoints could lead to catastrophic overfitting. So why doesn’t it? Our theory + experiments..."

Tags:

2022-10-14 About

Recent Advances in Language Model Fine-tuning (Feb 2021)

Tags:

2022-09-02 About

[2106.10199] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Tags:

2022-09-01 About

Domain adaptation of word embeddings through the exploitation of in-domain corpora and knowledge bases (PhD Thesis 2021)

Tags:

Thèse de Hicham El Boukkouri, univ Paris-Saclay

[Github](https://github.com/helboukkouri/phd-code)

### Goal

Given a target specialized domain, improve the quality of general-domain
word representations using in-domain corpora and/or knowledge bases

### Contributions

#### a method for specializing general-domain embeddings in a [Low-Resource](tag:nlp_low_resource_scenarios) context.

> - train static representations on the task corpus, 
> - resume the
pre-training of general-domain contextual embeddings on the same task corpus,
> - finally, combine both static and contextual representations into
one final model

#### we tackle the issue of using a general-domain vocabulary in a specialized domain

#### Evaluation of  re-training vs training from scratch on specialized corpora using a specialized vocabulary

training from scratch better, but not that much: re-training from a general model
is still appropriate as it is less expensive and leads to comparable, although
slightly lower, performance

#### Regarding subword-based tokenization systems
> we argue that they are inconvenient in practice -> CharacterBERT, a variant
of BERT that uses ELMo’s character-based system instead of WordPieces. More convenient ti use, superior robustness to misspellings

#### Ways to specialize general-domain representations using knowledge bases

a strong baseline using a simple
method relying on graph embeddings and concatenation, using only is_a relation

> both static and contextual embeddings may effectively be
specialized using this simple approach

#### Knowledge Injection Modules (KIM) that
inject the knowledge representations directly within the BERT-like models' architecture

### Notes

> our experiments focused on a single
setting (i.e. the medical domain and the English language)

> meta-embeddings, an approach that consists in combining
different sets of representations for achieving improved performance

2022-03-23 About

[2106.04647] Compacter: Efficient Low-Rank Hypercomplex Adapter Layers

Tags:

2021-09-29 About

Recent Advances in Language Model Fine-tuning

Tags:

2021-02-24 About