Semanlink - Text Classification AND Attention mechanism

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 A propos

New pipeline for zero-shot text classification - 🤗Transformers - Hugging Face Forums

Tags:

2021-03-15 A propos

Zero-shot classifier distillation at master · huggingface/transformers

Tags:

2021-02-23 A propos

Zero-Shot Learning in Modern NLP | Joe Davison Blog (2020-05)

Tags:

2021-02-23 A propos

X-BERT: eXtreme Multi-label Text Classification using Bidirectional Encoder Representations from Transformers

Tags:

2021-01-10 A propos

raphaelsty/ckb: Contextual knowledge bases

Tags:

2020-11-09 A propos

Sylvain Gugger sur Twitter : "Training a transformer model for text classification..."

Tags:

2020-10-19 A propos

[1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification

Tags:

A model that performs **text classification using entities in a knowledge base**.

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 A propos

[2003.11644] MAGNET: Multi-Label Text Classification using Attention-based Graph Neural Network

Tags:

2020-08-14 A propos

Hugging Face sur Twitter : "No labeled data? No problem. The 🤗 Transformers master branch now includes a built-in pipeline for zero-shot text classification...

Tags:

2020-08-12 A propos

FastHugs | ntentional

Tags:

2020-02-19 A propos

[1805.04174] Joint Embedding of Words and Labels for Text Classification (ACL Anthology 2018)

Tags:

2020-02-18 A propos

[1904.08398] DocBERT: BERT for Document Classification

Tags:

2019-04-18 A propos