Au sujet de ce document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Tianyu Gao
- sl:arxiv_num : 2104.08821
- sl:arxiv_published : 2021-04-18T11:27:08Z
- sl:arxiv_summary : This paper presents SimCSE, a simple contrastive learning framework that
greatly advances state-of-the-art sentence embeddings. We first describe an
unsupervised approach, which takes an input sentence and predicts itself in a
contrastive objective, with only standard dropout used as noise. This simple
method works surprisingly well, performing on par with previous supervised
counterparts. We find that dropout acts as minimal data augmentation, and
removing it leads to a representation collapse. Then, we propose a supervised
approach, which incorporates annotated pairs from natural language inference
datasets into our contrastive learning framework by using \"entailment\" pairs as
positives and \"contradiction\" pairs as hard negatives. We evaluate SimCSE on
standard semantic textual similarity (STS) tasks, and our unsupervised and
supervised models using BERT base achieve an average of 76.3% and 81.6%
Spearman's correlation respectively, a 4.2% and 2.2% improvement compared to
the previous best results. We also show -- both theoretically and empirically
-- that the contrastive learning objective regularizes pre-trained embeddings'
anisotropic space to be more uniform, and it better aligns positive pairs when
supervised signals are available.@en
- sl:arxiv_title : SimCSE: Simple Contrastive Learning of Sentence Embeddings@en
- sl:arxiv_updated : 2022-05-18T12:29:49Z
- sl:bookmarkOf : https://arxiv.org/abs/2104.08821
- sl:creationDate : 2022-10-17
- sl:creationTime : 2022-10-17T16:59:03Z
- sl:relatedDoc : http://www.semanlink.net/doc/2022/10/2210_07316_mteb_massive_text
Documents with similar tags (experimental)
Préférences...