Semanlink - Fine-tuning AND NLP@Google

UKP Lab sur X : "a lightweight solution for few-shot domain-specific sentence classification: AdaSent!..."

Tags:

2023-12-09 About

Modular and Parameter-Efficient Fine-Tuning for NLP Models

Tags:

2023-08-08 About

Peter J. Liu sur Twitter : "RLHF-alternative without RL"

Tags:

2023-05-18 About

Google "We Have No Moat, And Neither Does OpenAI"

Tags:

> low-cost public involvement was enabled
by a vastly cheaper mechanism for fine tuning called low
rank adaptation ()[LoRA](tag:lora)

> **Part of what makes LoRA so effective is that ... it’s stackable.**
>
> By contrast, training giant models from scratch not only
throws away the pretraining, but also any iterative
improvements that have been made on top.

> LoRA updates are very cheap to produce (~$100) for the
most popular model sizes.

> Many of these projects are saving time by training on
small, highly curated datasets...
> These
datasets are built using synthetic methods (e.g. filtering
the best responses from an existing model) and
scavenging from other projects

> Directly Competing With Open Source
Is a Losing Proposition

> Paradoxically, the one clear winner in all of this is Meta.
Because the leaked model was theirs ([LLaMA](tag:llama)), they have
effectively garnered an entire planet's worth of free labor.
Since most open source innovation is happening on top of
their architecture, there is nothing stopping them from
directly incorporating it into their products.

2023-05-04 About

Domain Adaptation with Generative Pseudo-Labeling (GPL) | Pinecone

Tags:

2023-04-09 About

Daniel Vila Suero sur Twitter : "Data quality is key for LLMs, but we're building Open Source LLMs with data of "unknown" quality... Introducing Alpaca GarbageCollector..."

Tags:

2023-04-05 About

The Flan Collection: Advancing open source methods for instruction tuning – Google AI Blog

Tags:

2023-02-02 About

Shayne Longpre sur Twitter : "What’s the best completely public competitor to #ChatGPT? Flan-T5 beats all public models we tested..."

Tags:

2023-02-01 About

ValueError "invalid literal for int() with base 10" in trainer.evaluate (dataset created from pandas) · Issue #228 · huggingface/setfit

Tags:

2022-12-13 About

Lewis Tunstall sur Twitter : "The SetFit library for few-shot learning with Sentence Transformers now supports *multi-label text classification*..."

Tags:

2022-10-14 About

huggingface/setfit: Efficient few-shot learning with Sentence Transformers

Tags:

2022-10-12 About

[2205.11498] Domain Adaptation for Memory-Efficient Dense Retrieval

Tags:

2022-09-26 About

[2209.11055] Efficient Few-Shot Learning Without Prompts

Tags:

2022-09-23 About

[2106.10199] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Tags:

2022-09-01 About

On Stability of Few-Sample Transformer Fine-Tuning | Kaggle

Tags:

2022-08-29 About

Unsupervised Learning — Sentence-Transformers documentation

Tags:

2022-08-20 About

Domain transfer with GGPL: German Generative Pseudo Labeling 🥨 | by Matthias Richter | Jun, 2022 | ML6team

Tags:

2022-06-02 About

Nils Reimers sur Twitter : "GPL goes multi-lingual..."

Tags:

2022-06-01 About

Ramsri Goutham Golla sur Twitter : "Hi @Nils_Reimers For GPL you used "msmarco-distilbert-base-tas-b" model and ..."

Tags:

2022-04-27 About

[2008.11228] A simple method for domain adaptation of sentence embeddings

Tags:

2022-04-01 About

[2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Tags:

2022-03-31 About

Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)

Tags:

2022-03-31 About

Nils Reimers sur Twitter : "Great post on SetFit"

Tags:

2022-03-31 About

Sentence Embedding Fine-tuning for the French Language | by La Javaness R&D | Feb, 2022 | Medium

Tags:

2022-03-31 About

Domain Adaptation — Sentence-Transformers documentation

Tags:

2022-03-31 About

[2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning

Tags:

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 About

[2006.05987] Revisiting Few-sample BERT Fine-tuning

Tags:

2022-03-21 About

NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"

Tags:

2022-03-09 About

[2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Tags:

An unsupervised domain adaptation technique for dense retrieval models

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 About

Unsupervised Training for Sentence Transformers | Pinecone

Tags:

2021-11-24 About

On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (2021)

Tags:

2021-10-30 About

[2106.04647] Compacter: Efficient Low-Rank Hypercomplex Adapter Layers

Tags:

2021-09-29 About

[1902.00751] Parameter-Efficient Transfer Learning for NLP

Tags:

2021-04-11 About