Au sujet de ce document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Xiaochuang Han
- sl:arxiv_num : 1904.02817
- sl:arxiv_published : 2019-04-04T23:05:45Z
- sl:arxiv_summary : Contextualized word embeddings such as ELMo and BERT provide a foundation for
strong performance across a wide range of natural language processing tasks by
pretraining on large corpora of unlabeled text. However, the applicability of
this approach is unknown when the target domain varies substantially from the
pretraining corpus. We are specifically interested in the scenario in which
labeled data is available in only a canonical source domain such as newstext,
and the target domain is distinct from both the labeled and pretraining texts.
To address this scenario, we propose domain-adaptive fine-tuning, in which the
contextualized embeddings are adapted by masked language modeling on text from
the target domain. We test this approach on sequence labeling in two
challenging domains: Early Modern English and Twitter. Both domains differ
substantially from existing pretraining corpora, and domain-adaptive
fine-tuning yields substantial improvements over strong BERT baselines, with
particularly impressive results on out-of-vocabulary words. We conclude that
domain-adaptive fine-tuning offers a simple and effective approach for the
unsupervised adaptation of sequence labeling to difficult new domains.@en
- sl:arxiv_title : Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling@en
- sl:arxiv_updated : 2019-09-05T00:18:25Z
- sl:bookmarkOf : https://arxiv.org/abs/1904.02817
- sl:creationDate : 2023-01-12
- sl:creationTime : 2023-01-12T16:29:04Z
Documents with similar tags (experimental)
Préférences...