Semanlink - [2008.11228] A simple method for domain adaptation of sentence embeddings

[2008.11228] A simple method for domain adaptation of sentence embeddings

Tags:

Au sujet de ce document

sl:arxiv_author : Anna Kruspe
sl:arxiv_firstAuthor : Anna Kruspe
sl:arxiv_num : 2008.11228
sl:arxiv_published : 2020-08-25T18:31:08Z
sl:arxiv_summary : Pre-trained sentence embeddings have been shown to be very useful for a variety of NLP tasks. Due to the fact that training such embeddings requires a large amount of data, they are commonly trained on a variety of text data. An adaptation to specific domains could improve results in many cases, but such a finetuning is usually problem-dependent and poses the risk of over-adapting to the data used for adaptation. In this paper, we present a simple universal method for finetuning Google's Universal Sentence Encoder (USE) using a Siamese architecture. We demonstrate how to use this approach for a variety of data sets and present results on different data sets representing similar problems. The approach is also compared to traditional finetuning on these data sets. As a further advantage, the approach can be used for combining data sets with different annotations. We also present an embedding finetuned on all data sets in parallel.@en
sl:arxiv_title : A simple method for domain adaptation of sentence embeddings@en
sl:arxiv_updated : 2020-08-25T18:31:08Z
sl:bookmarkOf : https://arxiv.org/abs/2008.11228
sl:creationDate : 2022-04-01
sl:creationTime : 2022-04-01T14:07:28Z

Infos sur le fichier

Bookmark of: https://arxiv.org/abs/2008.11228

Documents with similar tags (experimental)

[2302.08091] Do We Still Need Clinical Language Models?

Tags:

2023-02-17 A propos

[2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Tags:

2022-03-31 A propos

Domain Adaptation — Sentence-Transformers documentation

Tags:

2022-03-31 A propos

[2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning

Tags:

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 A propos

[1911.02655] Towards Domain Adaptation from Limited Data for Question Answering Using Deep Neural Networks

Tags:

2021-11-19 A propos

[2108.13854] Contrastive Domain Adaptation for Question Answering using Limited Text Corpora

Tags:

2021-11-19 A propos

[1706.03610] Neural Domain Adaptation for Biomedical Question Answering

Tags:

2021-11-19 A propos

[2106.13474] Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Tags:

2021-10-21 A propos

[1908.11860] Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification

Tags:

2021-10-21 A propos

[2104.06979] TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

Tags:

> The most
successful previous approaches like InferSent (Conneau
et al., 2017), Universial Sentence Encoder
(USE) (Cer et al., 2018) and SBERT (Reimers and
Gurevych, 2019) heavily relied on labeled data to
train sentence embedding models.
>
> TSDAE can
achieve up to 93.1% of the performance of indomain
supervised approaches. Further, we
show that TSDAE is **a strong domain adaptation
and pre-training method for sentence
embeddings**, significantly outperforming other
approaches like Masked Language Model.

> During training, TSDAE
encodes corrupted sentences into fixed-sized
vectors and requires the decoder to reconstruct the
original sentences from this sentence embedding.

- <https://www.sbert.net/examples/unsupervised_learning/TSDAE/README.html>
- [github](https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning/TSDAE)
- [UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet](doc:2020/07/ukplab_sentence_transformers_s)
- [twitter](https://twitter.com/KexinWang2049/status/1433361957579538432):

> **TSDAE can learn domain-specific sentence embeddings with unlabeled sentences**
>
> Most importantly, instead of STS (Semantic Textual Similarity), **we suggest evaluating unsupervised sentence embeddings on the domain-specific tasks&datasets, which is the real use case for them**. Actually, STS scores do not correlate with performance on specific tasks.

2021-09-01 A propos

[2004.10964] Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

Tags:

2020-12-01 A propos

[1803.11175] Universal Sentence Encoder

Tags:

2018-05-29 A propos