Au sujet de ce document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Hongjin Su
- sl:arxiv_num : 2209.01975
- sl:arxiv_published : 2022-09-05T14:01:15Z
- sl:arxiv_summary : Many recent approaches to natural language tasks are built on the remarkable
abilities of large language models. Large language models can perform
in-context learning, where they learn a new task from a few task
demonstrations, without any parameter updates. This work examines the
implications of in-context learning for the creation of datasets for new
natural language tasks. Departing from recent in-context learning methods, we
formulate an annotation-efficient, two-step framework: selective annotation
that chooses a pool of examples to annotate from unlabeled data in advance,
followed by prompt retrieval that retrieves task examples from the annotated
pool at test time. Based on this framework, we propose an unsupervised,
graph-based selective annotation method, voke-k, to select diverse,
representative examples to annotate. Extensive experiments on 10 datasets
(covering classification, commonsense reasoning, dialogue, and text/code
generation) demonstrate that our selective annotation method improves the task
performance by a large margin. On average, vote-k achieves a 12.9%/11.4%
relative gain under an annotation budget of 18/100, as compared to randomly
selecting examples to annotate. Compared to state-of-the-art supervised
finetuning approaches, it yields similar performance with 10-100x less
annotation cost across 10 tasks. We further analyze the effectiveness of our
framework in various scenarios: language models with varying sizes, alternative
selective annotation methods, and cases where there is a test data domain
shift. We hope that our studies will serve as a basis for data annotations as
large language models are increasingly applied to new tasks. Our code is
available at https://github.com/HKUNLP/icl-selective-annotation.@en
- sl:arxiv_title : Selective Annotation Makes Language Models Better Few-Shot Learners@en
- sl:arxiv_updated : 2022-09-05T14:01:15Z
- sl:bookmarkOf : https://arxiv.org/abs/2209.01975
- sl:creationDate : 2022-09-07
- sl:creationTime : 2022-09-07T13:20:58Z
Documents with similar tags (experimental)
Préférences...