Semanlink - Thomas Müller sur Twitter : "paper & code of a novel light-weight few-shot model based on sentence embeddings..."

Thomas Müller sur Twitter : "paper & code of a novel light-weight few-shot model based on sentence embeddings..."

Tags:

Au sujet de ce document

sl:bookmarkOf : https://twitter.com/muelletm/status/1508717548019331072
sl:creationDate : 2022-03-30
sl:creationTime : 2022-03-30T15:48:13Z
sl:relatedDoc : http://www.semanlink.net/doc/2022/03/2203_14655_few_shot_learning_

Infos sur le fichier

Bookmark of: https://twitter.com/muelletm/status/1508717548019331072

Linked From

[2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning

Tags:

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 A propos

Documents with similar tags (experimental)

Tu Vu sur Twitter : "Enormous LMs like GPT-3 exhibit impressive few-shot performance, but w/ self-training a BERT base sized model can achieve much better results!

Tags:

2022-04-13 A propos

Nils Reimers sur Twitter : "How to train state-of-the-art sentence embeddings?"

Tags:

2021-06-22 A propos

Nandan Thakur sur Twitter : "how to create sentence-embeddings when little or zero in-domain training data is available"

Tags:

2021-06-03 A propos