Au sujet de ce document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Yue Yu
- sl:arxiv_num : 2010.07835
- sl:arxiv_published : 2020-10-15T15:55:08Z
- sl:arxiv_summary : Fine-tuned pre-trained language models (LMs) have achieved enormous success
in many natural language processing (NLP) tasks, but they still require
excessive labeled data in the fine-tuning stage. We study the problem of
fine-tuning pre-trained LMs using only weak supervision, without any labeled
data. This problem is challenging because the high capacity of LMs makes them
prone to overfitting the noisy labels generated by weak supervision. To address
this problem, we develop a contrastive self-training framework, COSINE, to
enable fine-tuning LMs with weak supervision. Underpinned by contrastive
regularization and confidence-based reweighting, this contrastive self-training
framework can gradually improve model fitting while effectively suppressing
error propagation. Experiments on sequence, token, and sentence pair
classification tasks show that our model outperforms the strongest baseline by
large margins on 7 benchmarks in 6 tasks, and achieves competitive performance
with fully-supervised fine-tuning methods.@en
- sl:arxiv_title : Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach@en
- sl:arxiv_updated : 2021-03-31T02:25:55Z
- sl:bookmarkOf : https://arxiv.org/abs/2010.07835
- sl:creationDate : 2022-09-02
- sl:creationTime : 2022-09-02T11:02:48Z
Documents with similar tags (experimental)
Préférences...