Au sujet de ce document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Mike Lewis
- sl:arxiv_num : 2006.15020
- sl:arxiv_published : 2020-06-26T14:43:43Z
- sl:arxiv_summary : We introduce MARGE, a pre-trained sequence-to-sequence model learned with an
unsupervised multi-lingual multi-document paraphrasing objective. MARGE
provides an alternative to the dominant masked language modeling paradigm,
where we self-supervise the reconstruction of target text by retrieving a set
of related texts (in many languages) and conditioning on them to maximize the
likelihood of generating the original. We show it is possible to jointly learn
to do retrieval and reconstruction, given only a random initialization. The
objective noisily captures aspects of paraphrase, translation, multi-document
summarization, and information retrieval, allowing for strong zero-shot
performance on several tasks. For example, with no additional task-specific
training we achieve BLEU scores of up to 35.8 for document translation. We
further show that fine-tuning gives strong performance on a range of
discriminative and generative tasks in many languages, making MARGE the most
generally applicable pre-training method to date.@en
- sl:arxiv_title : Pre-training via Paraphrasing@en
- sl:arxiv_updated : 2020-06-26T14:43:43Z
- sl:bookmarkOf : https://arxiv.org/abs/2006.15020
- sl:creationDate : 2020-06-30
- sl:creationTime : 2020-06-30T11:32:08Z
Documents with similar tags (experimental)
Préférences...