Semanlink - Favoris ; Semantic technology AND Knowledge distillation

Printer friendly

Search Tag:

Search Doc:

Preferences...

Favoris ; Semantic technology AND Knowledge distillation

Common descendants

DistilBERT

17 Documents

[2002.06275] TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval

Tags:

2023-08-27 About

[2207.06300] Re2G: Retrieve, Rerank, Generate

Tags:

2022-07-14 About

[2004.09813] Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation

Tags:

2022-03-18 About

[2010.02666] Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation

Tags:

2021-12-16 About

Multilingual Sentence Transformers | Pinecone

Tags:

2021-11-04 About

[2106.13474] Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Tags:

2021-10-21 About

[2010.02194] Self-training Improves Pre-training for Natural Language Understanding

Tags:

2021-03-12 About

Distill our @huggingface zero-shot classifier with your specified class names

Tags:

2021-02-23 About

Zero-shot classifier distillation at master · huggingface/transformers

Tags:

2021-02-23 About

[2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering

Tags:

> a method to train an information retrieval module for downstream tasks, **without using pairs of queries and documents as annotations**.

Uses two models (standard pipeline for open-domain QA):

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 About

[1907.04829] BAM! Born-Again Multi-Task Networks for Natural Language Understanding

Tags:

2020-05-12 About

NLP Newsletter: The Annotated GPT-2, Understanding self-distillation, Haiku, GANILLA, Sparkwiki, Ethics in NLP, Torchmeta,…

Tags:

2020-02-24 About

Distilling BERT models with spaCy - Towards Data Science (2019)

Tags:

2020-02-15 About

Hugging Face sur Twitter : DistilBERT-cased for Question Answering w/ just 3 lines of javascript

Tags:

2020-02-14 About

[2002.02925] BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

Tags:

2020-02-10 About

Canwen Xu sur Twitter : "WTF? We brutally dismember BERT and replace all his organs?"

Tags:

2020-02-10 About

Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT

Tags:

2019-08-28 About