Au sujet de ce document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Jianmo Ni
- sl:arxiv_num : 2108.08877
- sl:arxiv_published : 2021-08-19T18:58:02Z
- sl:arxiv_summary : We provide the first exploration of sentence embeddings from text-to-text
transformers (T5). Sentence embeddings are broadly useful for language
processing tasks. While T5 achieves impressive performance on language tasks
cast as sequence-to-sequence mapping problems, it is unclear how to produce
sentence embeddings from encoder-decoder models. We investigate three methods
for extracting T5 sentence embeddings: two utilize only the T5 encoder and one
uses the full T5 encoder-decoder model. To support our investigation, we
establish a new sentence representation transfer benchmark, SentGLUE, which
extends the SentEval toolkit to nine tasks from the GLUE benchmark. Our
encoder-only models outperforms Sentence-BERT and SimCSE sentence embeddings on
both SentEval and SentGLUE transfer tasks, including semantic textual
similarity (STS). Scaling up T5 from millions to billions of parameters is
found to produce consistent further improvements. Finally, our encoder-decoder
method achieves a new state-of-the-art on STS when using sentence embeddings.
Our models are released at https://tfhub.dev/google/collections/sentence-t5/1.@en
- sl:arxiv_title : Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models@en
- sl:arxiv_updated : 2021-12-14T06:19:33Z
- sl:bookmarkOf : https://arxiv.org/abs/2108.08877
- sl:creationDate : 2023-02-17
- sl:creationTime : 2023-02-17T18:20:47Z
Documents with similar tags (experimental)
Préférences...