Semanlink - Domain-Specific BERT Models

Tags:

Domain-Specific BERT

Au sujet de ce document

sl:bookmarkOf : https://mccormickml.com/2020/06/22/domain-specific-bert-tutorial/
sl:creationDate : 2020-12-01
sl:creationTime : 2020-12-01T15:08:22Z

Infos sur le fichier

Bookmark of: https://mccormickml.com/2020/06/22/domain-specific-bert-tutorial/

Documents with similar tags (experimental)

NLP | How to add a domain-specific vocabulary (new tokens) to a subword tokenizer already trained like BERT WordPiece | by Pierre Guillou | Medium

Tags:

2022-03-18 A propos

[2007.15779] Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Tags:

2021-04-11 A propos

exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources - ACL Anthology

Tags:

**Focus on the Embedding of Domain-specific Vocabulary.**

> exBERT
adds a new domain-specific vocabulary and the corresponding
embedding layer, as well as a small
extension module to the original unmodified model

> a pretraining
method allowing **low-cost embedding of
domain-specific vocabulary in the context of an
existing large pre-trained model such as BERT**

> exBERT... explicitly incorporates
the new domain’s vocabulary, while being able to
**reuse the original pre-trained model’s weights as is**
to reduce required computation and training data. Specifically, exBERT extends BERT by augmenting
its embeddings for the original vocabulary with
new embeddings for the domain-specific vocabulary
via **a learned small “extension” module**. **The
output of the original and extension modules are
combined via a trainable weighted sum operation**

In a way similar to concept developed in

> [[1902.00751] Parameter-Efficient Transfer Learning for NLP](doc:2021/04/1902_00751_parameter_efficien), but not in the fine-tuning paradigm.

[Github](https://github.com/cgmhaicenter/exBERT)

2021-04-11 A propos

[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce

Tags:

2020-12-14 A propos