Semanlink - Unsupervised Training for Sentence Transformers

Tags:

Au sujet de ce document

sl:bookmarkOf : https://www.pinecone.io/learn/unsupervised-training-sentence-transformers/
sl:creationDate : 2021-11-24
sl:creationTime : 2021-11-24T21:03:44Z
sl:relatedDoc : http://www.semanlink.net/doc/2021/09/2104_06979_tsdae_using_trans

Infos sur le fichier

Bookmark of: https://www.pinecone.io/learn/unsupervised-training-sentence-transformers/

Documents with similar tags (experimental)

Unsupervised Learning — Sentence-Transformers documentation

Tags:

2022-08-20 A propos

Domain Adaptation — Sentence-Transformers documentation

Tags:

2022-03-31 A propos

[2104.06979] TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

Tags:

> The most
successful previous approaches like InferSent (Conneau
et al., 2017), Universial Sentence Encoder
(USE) (Cer et al., 2018) and SBERT (Reimers and
Gurevych, 2019) heavily relied on labeled data to
train sentence embedding models.
>
> TSDAE can
achieve up to 93.1% of the performance of indomain
supervised approaches. Further, we
show that TSDAE is **a strong domain adaptation
and pre-training method for sentence
embeddings**, significantly outperforming other
approaches like Masked Language Model.

> During training, TSDAE
encodes corrupted sentences into fixed-sized
vectors and requires the decoder to reconstruct the
original sentences from this sentence embedding.

- <https://www.sbert.net/examples/unsupervised_learning/TSDAE/README.html>
- [github](https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning/TSDAE)
- [UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet](doc:2020/07/ukplab_sentence_transformers_s)
- [twitter](https://twitter.com/KexinWang2049/status/1433361957579538432):

> **TSDAE can learn domain-specific sentence embeddings with unlabeled sentences**
>
> Most importantly, instead of STS (Semantic Textual Similarity), **we suggest evaluating unsupervised sentence embeddings on the domain-specific tasks&datasets, which is the real use case for them**. Actually, STS scores do not correlate with performance on specific tasks.

2021-09-01 A propos