Semanlink - Domain adaptation AND Language Model: Adaptation

LlamaIndex 🦙 sur X : "Fine-tuning Embedding Models for RAG with LoRA'

Tags:

2024-04-23 A propos

UKP Lab sur X : "a lightweight solution for few-shot domain-specific sentence classification: AdaSent!..."

Tags:

2023-12-09 A propos

[2203.10581] Cluster & Tune: Boost Cold Start Performance in Text Classification

Tags:

[Leshem Choshen sur Twitter : "Labelled data is scarce, what can we do?..."](doc:2022/04/leshem_choshen_sur_twitter_l)

> **One-sentence Summary**: we suggest adding an unsupervised intermediate classification step, before finetunning and after pretraining BERT, and show it improves performance for data-constrained cases.

> for text classification cold start (when labeled
data is scarce), **add an intermediate unsupervised
classification task**, between the pretraining
and fine-tuning phases:
> perform clustering and
train the pre-trained model on predicting the
cluster labels.

> this additional
classification phase can significantly improve
performance, mainly for **topical classification**
tasks

> we use an efficient clustering technique,
that relies on simple Bag Of Words (BOW)
representations, to partition the unlabeled training
data into relatively homogeneous clusters of text
instances.
>
> Next, we treat these clusters as labeled
data for an intermediate text classification task, and
train the pre-trained model – with or without additional
MLM pretraining – with respect to this
multi-class problem, prior to the final fine-tuning
over the actual target-task labels

> The underlying
intuition is that inter-training the model
over a related text classification task would be more
beneficial compared to MLM inter-training, which
focuses on different textual entities, namely predicting
the identity of a single token.

2022-04-06 A propos

Leshem Choshen sur Twitter : "Labelled data is scarce, what can we do?..."

Tags:

2022-04-06 A propos

Domain adaptation of word embeddings through the exploitation of in-domain corpora and knowledge bases (PhD Thesis 2021)

Tags:

Thèse de Hicham El Boukkouri, univ Paris-Saclay

[Github](https://github.com/helboukkouri/phd-code)

### Goal

Given a target specialized domain, improve the quality of general-domain
word representations using in-domain corpora and/or knowledge bases

### Contributions

#### a method for specializing general-domain embeddings in a [Low-Resource](tag:nlp_low_resource_scenarios) context.

> - train static representations on the task corpus, 
> - resume the
pre-training of general-domain contextual embeddings on the same task corpus,
> - finally, combine both static and contextual representations into
one final model

#### we tackle the issue of using a general-domain vocabulary in a specialized domain

#### Evaluation of  re-training vs training from scratch on specialized corpora using a specialized vocabulary

training from scratch better, but not that much: re-training from a general model
is still appropriate as it is less expensive and leads to comparable, although
slightly lower, performance

#### Regarding subword-based tokenization systems
> we argue that they are inconvenient in practice -> CharacterBERT, a variant
of BERT that uses ELMo’s character-based system instead of WordPieces. More convenient ti use, superior robustness to misspellings

#### Ways to specialize general-domain representations using knowledge bases

a strong baseline using a simple
method relying on graph embeddings and concatenation, using only is_a relation

> both static and contextual embeddings may effectively be
specialized using this simple approach

#### Knowledge Injection Modules (KIM) that
inject the knowledge representations directly within the BERT-like models' architecture

### Notes

> our experiments focused on a single
setting (i.e. the medical domain and the English language)

> meta-embeddings, an approach that consists in combining
different sets of representations for achieving improved performance

2022-03-23 A propos

GuideToTransformersDomainAdaptation.ipynb - Colaboratory

Tags:

2022-03-18 A propos

NLP | How to add a domain-specific vocabulary (new tokens) to a subword tokenizer already trained like BERT WordPiece | by Pierre Guillou | Medium

Tags:

2022-03-18 A propos

Retraining roberta-base using the RoBERTa MLM Procedure | Medium

Tags:

2022-03-18 A propos

[2106.13474] Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Tags:

2021-10-21 A propos

[1908.11860] Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification

Tags:

2021-10-21 A propos

[2004.10964] Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

Tags:

2020-12-01 A propos