Semanlink - Favoris ; Machine learning AND Classification

> **One-sentence Summary**: we suggest adding an unsupervised intermediate classification step, before finetunning and after pretraining BERT, and show it improves performance for data-constrained cases.

> for text classification cold start (when labeled
data is scarce), **add an intermediate unsupervised
classification task**, between the pretraining
and fine-tuning phases:
> perform clustering and
train the pre-trained model on predicting the
cluster labels.

> this additional
classification phase can significantly improve
performance, mainly for **topical classification**
tasks

> we use an efficient clustering technique,
that relies on simple Bag Of Words (BOW)
representations, to partition the unlabeled training
data into relatively homogeneous clusters of text
instances.
>
> Next, we treat these clusters as labeled
data for an intermediate text classification task, and
train the pre-trained model – with or without additional
MLM pretraining – with respect to this
multi-class problem, prior to the final fine-tuning
over the actual target-task labels

> The underlying
intuition is that inter-training the model
over a related text classification task would be more
beneficial compared to MLM inter-training, which
focuses on different textual entities, namely predicting
the identity of a single token.

2022-04-06 A propos

Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)

Tags:

2022-03-31 A propos

Nils Reimers sur Twitter : "Great post on SetFit"

Tags:

2022-03-31 A propos

[2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning

Tags:

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 A propos

Domain Adaptation with BERT-based Domain Classification and Data Selection - ACL Anthology (2019)

Tags:

2022-03-16 A propos

Label-Specific Document Representation for Multi-Label Text Classification - ACL Anthology

Tags:

2021-12-20 A propos

[1712.05972] Train Once, Test Anywhere: Zero-Shot Learning for Text Classification

Tags:

2021-10-16 A propos

[2010.07245] Text Classification Using Label Names Only: A Language Model Self-Training Approach

Tags:

2021-10-16 A propos

Selective Classification Can Magnify Disparities Across Groups | SAIL Blog

Tags:

2021-10-16 A propos

Building a sentence embedding index with fastText and BM25 | by David Mezzetti | Towards Data Science

Tags:

2021-09-30 A propos

Term Based Semantic Clusters for Very Short Text Classification (2019)

Tags:

2021-05-26 A propos

Adventures in Zero-Shot Text Classification

Tags:

2021-05-25 A propos

New pipeline for zero-shot text classification - 🤗Transformers - Hugging Face Forums

Tags:

2021-03-15 A propos

Distill our @huggingface zero-shot classifier with your specified class names

Tags:

2021-02-23 A propos

Zero-shot classifier distillation at master · huggingface/transformers

Tags:

2021-02-23 A propos

Zero-Shot Learning in Modern NLP | Joe Davison Blog (2020-05)

Tags:

2021-02-23 A propos

X-BERT: eXtreme Multi-label Text Classification using Bidirectional Encoder Representations from Transformers

Tags:

2021-01-10 A propos

raphaelsty/ckb: Contextual knowledge bases

Tags:

2020-11-09 A propos

Sylvain Gugger sur Twitter : "Training a transformer model for text classification..."

Tags:

2020-10-19 A propos

[1904.09078] EmbraceNet: A robust deep learning architecture for multimodal classification

Tags:

2020-10-14 A propos

[1911.11506] Word-Class Embeddings for Multiclass Text Classification

Tags:

> In supervised tasks such as multiclass
text classification (the focus of this article) it seems appealing to enhance word representations
with ad-hoc embeddings that encode task-specific information. We propose (supervised) word-class
embeddings (WCEs), and show that, when concatenated to (unsupervised) pre-trained word embeddings,
they substantially facilitate the training of deep-learning models in multiclass classification by
topic.
>
> A differentiating aspect of our method is that it keeps the modelling of word-class interactions separate from the
original word embedding. Word-class correlations are confined in a dedicated vector space, whose vectors enhance
(by concatenation) the unsupervised representations. The net effect is an embedding matrix that is better suited to
classification, and imposes no restriction to the network architecture using it.

[github](https://github.com/AlexMoreo/word-class-embeddings). Refers to [LEAM](doc:2020/02/joint_embedding_of_words_and_la) :

> [in LEAM] Once words and labels are embedded in a common vector space, word-label
compatibility is measured via cosine similarity. Our method instead models these compatibilities directly, without
generating intermediate embeddings for words or labels.

2020-10-11 A propos

[2004.03705] Deep Learning Based Text Classification: A Comprehensive Review

Tags:

2020-10-11 A propos

Top 6 Open Source Pretrained Models for Text Classification you should use

Tags:

Text Classification

2020-10-11 A propos

Towards Unsupervised Text Classification Leveraging Experts and Word Embeddings - (ACL 2019)

Tags:

2020-10-05 A propos

Unsupervised text classification with word embeddings - Max Halford

Tags:

2020-10-05 A propos

[1609.02521] DiSMEC - Distributed Sparse Machines for Extreme Multi-label Classification

Tags:

2020-09-06 A propos

[1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification

Tags:

A model that performs **text classification using entities in a knowledge base**.

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 A propos

Initializing neural networks for hierarchical multi-label text classification (2017)

Tags:

2020-09-01 A propos

[1306.6802] Evaluation Measures for Hierarchical Classification: a unified view and novel approaches

Tags:

2020-09-01 A propos

Hierarchical Multi-label Classification of Text with Capsule Networks (2019)

Tags:

2020-08-15 A propos

A Study of multilabel text classification and the effect of label hierarchy (2015)

Tags:

2020-08-15 A propos

[2003.11644] MAGNET: Multi-Label Text Classification using Attention-based Graph Neural Network

Tags:

2020-08-14 A propos

[1812.02956] LNEMLC: Label Network Embeddings for Multi-Label Classification

Tags:

2020-08-12 A propos

Hugging Face sur Twitter : "No labeled data? No problem. The 🤗 Transformers master branch now includes a built-in pipeline for zero-shot text classification...

Tags:

2020-08-12 A propos

The Extreme Classification Repository

Tags:

2020-08-12 A propos

Everything you always wanted to know about extreme classification (but were afraid to ask) - Microsoft Research - 2019

Tags:

2020-08-12 A propos

SLEEC: Sparse Local Embeddings for Extreme Multi-label Classification (2015)

Tags:

2020-08-11 A propos

FastHugs | ntentional

Tags:

2020-02-19 A propos

[1805.04174] Joint Embedding of Words and Labels for Text Classification (ACL Anthology 2018)

Tags:

2020-02-18 A propos

[1503.08677] Label-Embedding for Image Classification

Tags:

2020-02-18 A propos

Interpretable Named entity recognition with keras and LIME – Depends on the definition

Tags:

2020-01-09 A propos

[1912.03263] Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One

Tags:

2019-12-09 A propos

Baselines and Bigrams: Simple, Good Sentiment and Topic Classification. Sida Wang and Christopher D. Manning

Tags:

2019-09-03 A propos

Open-sourcing hyperparameter autotuning for fastText

Tags:

FastText

2019-08-27 A propos

20% Accuracy Bump in Text Classification with ME-ULMFiT

Tags:

2019-06-23 A propos

[1905.10070] Label-aware Document Representation via Hybrid Attention for Extreme Multi-Label Text Classification

Tags:

2019-06-22 A propos

"I made a bet that a Naive Bayes classifier would work as well on humor recognition as a neural net with fine-tuned Bert embeddings. I won"

Tags:

2019-06-06 A propos

Knowledge-Based Short Text Categorization Using Entity and Category Embedding | Springer for Research & Development (2019)

Tags:

2019-05-30 A propos

A Technique for Building NLP Classifiers Efficiently with Transfer Learning and Weak Supervision

Tags:

2019-05-22 A propos

Combining Knowledge with Deep Convolutional Neural Networks for Short Text Classification (2017)

Tags:

2019-05-20 A propos

[1904.08398] DocBERT: BERT for Document Classification

Tags:

2019-04-18 A propos

[1902.05196] Categorical Metadata Representation for Customized Text Classification

Tags:

2019-02-18 A propos

Enhancing Binary Classification by Modeling Uncertain Boundary in Three-Way Decisions - IEEE Journals & Magazine

Tags:

Three-way decisions

2019-02-02 A propos

[1711.09677] Binary classification models with "Uncertain" predictions

Tags:

2019-02-02 A propos

Cheatsheet · fastText

Tags:

2019-01-29 A propos

Using FastText models (not vectors) for robust embeddings | Kaggle

Tags:

2019-01-29 A propos

Tutorial on Text Classification (NLP) using ULMFiT and fastai Library in Python - Analytics Vidhya

Tags:

2018-12-01 A propos

Practical Text Classification With Python and Keras – Real Python

Tags:

2018-10-25 A propos

[Learning Note] StarSpace For Multi-label Text Classification

Tags:

2018-09-11 A propos

GitHub - marcotcr/lime: Lime: Explaining the predictions of any machine learning classifier

Tags:

2018-09-09 A propos

[1602.04938] "Why Should I Trust You?": Explaining the Predictions of Any Classifier

Tags:

2018-09-09 A propos

Supplementary : Extreme Multi-label Learning with Label Features for Warm-start Tagging, Ranking & Recommendation

Tags:

2018-08-07 A propos

Practical guide to text classification | Google Developers

Tags:

2018-07-23 A propos

Training Classifiers with Natural Language Explanations

Tags:

2018-06-23 A propos

Deep Unordered Composition Rivals Syntactic Methods for Text Classification (2015)

Tags:

2018-05-29 A propos

[1607.07956] Joint Embedding of Hierarchical Categories and Entities for Concept Categorization and Dataless Classification (COLING 2016)

Tags:

2018-05-12 A propos

Under the hood: Multilingual embeddings | Engineering Blog | Facebook Code

Tags:

2018-05-11 A propos

GitHub - Babylonpartners/fastText_multilingual: Multilingual word vectors

Tags:

2018-05-11 A propos

Text Classification with TensorFlow Estimators

Tags:

2018-04-17 A propos

ML-knn: A lazy learning approach to multi-label learning (2007)

Tags:

a lazy learning approach derived from the traditional [k-nearest neighbors algorithm](tag:k_nearest_neighbors_algorithm)

> for each unseen instance, its K nearest neighbors in the training set are firstly identified. After that, based on statistical information gained from the label sets of these neighboring instances, i.e. the number of neighboring instances belonging to each possible class, [Maximum a posteriori (MAP)](tag:maximum_a_posteriori_estimation) principle is utilized to determine the label set for the unseen instance.

Implemented in [scikit-multilearn](http://scikit.ml/api/skmultilearn.adapt.mlknn.html), in [java](https://github.com/lefman/mulan-extended/blob/master/mulan/src/mulan/classifier/lazy/MLkNN.java)

> the first lazy approach proposed specifically for multi-label classification. This is also a binary relevance approach which considers each label independently as a binary classification problem. Instead of a standard k-NN method, however, MLkNN uses the maximum a-posteriori (MAP) (Kelleher et al., 2015) approach combined with k-NN. [src](https://pdfs.semanticscholar.org/af9b/33da37d290c063cd826ab5923d96892a9767.pdf)

2018-03-18 A propos

[1707.00418] Learning Deep Latent Spaces for Multi-Label Classification

Tags:

2018-03-16 A propos

Examination of machine learning methods for multi-label classification of intellectual property documents (2017)

Tags:

2018-03-05 A propos

Extreme Multi-label Learning with Label Features for Warm-start Tagging, Ranking & Recommendation (2018)

Tags:

2018-03-04 A propos

[1307.5101] Large-scale Multi-label Learning with Missing Labels

Tags:

2018-03-04 A propos

Intro to text classification with Keras: automatically tagging Stack Overflow posts | Google Cloud Big Data and Machine Learning Blog

Tags:

2018-03-04 A propos

[1604.06737] Entity Embeddings of Categorical Variables

Tags:

2018-03-03 A propos

A Benchmark of Text Classification in PyTorch

Tags:

2018-02-28 A propos

RESEARCH TRACK: Web Content Analysis, Semantics and Knowledge

Tags:

[CFP](https://www2018.thewebconf.org/call-for-papers/research-tracks-cfp/web-content-analysis/)

> In previous years, ‘content analysis’ and ‘semantic and knowledge’ were in separate track. This year, we combined these tracks to emphasize the close relationship between these topics; **the use of content to curate knowledge and the use of knowledge to guide content analysis and intelligent usage**.

Some of the accepted papers:
### [Large-Scale Hierarchical Text Classification with Recursively Regularized Deep Graph-CNN](https://doi.org/10.1145/3178876.3186005)

[Hierarchical Text Classification](/tag/nlp_hierarchical_text_classification): Text classification to a hierarchical taxonomy of topics, using graph representation of text, and CNN over this graph

Renvoie à ce qui a été vu dans le tutorial "Graph-based Text Representations"

from the abstract:

> a graph-CNN based deep learning model to first convert texts to graph-of-words, and then use graph convolution operations to convolve the word graph. Graph-of-words representation of texts has the advantage of capturing non-consecutive and long-distance semantics. CNN models have the advantage of learning different level of semantics. To further leverage the hierarchy of labels, we regularize the deep architecture with the dependency among labels

Conversion of text to graph: potentially given a single document

### [Weakly-supervised Relation Extraction by Pattern-enhanced Embedding Learning](https://doi.org/10.1145/3178876.3186024 )

Extraction de relations de corpus de textes de façon semi-supervisée, dans un contexte où on a peu de données labellisées décrivant les relations.

Par exemple, des données labellisées indique que le texte "Beijing, capital of China" correspond à la relation entre entités : ("Beijing", "Capital Of", "China), et on voudrait pouvoir extraire les entités et relations pertinentes à partir de texte tel que "Paris, France's capital,..."

Le papier décrit une méthode qui combine deux modules, l'un basé sur l'extraction automatique de patterns (par ex "[Head], Capital Of [Tail]") et l'autre sur la "sémantique distributionnelle" (du type "word embeddings"). Ces deux modules collaborent, le premier permettant de créer des instances de relations augmentant la base de connaissance sur lequel entrainer le second, et le second aidant le premier à déterminer des patterns informatifs ("co-entrainement")

### [Scalable Instance Reconstruction in Knowledge Bases via Relatedness Affiliated Embedding](https://doi.org/10.1145/3178876.3186017)

Knowledge base completion problem: usually, it is formulated as a link prediction problem, but not here. A novel knowledge embedding model ("Joint Modelling and Learning of Relatedness and Embedding")

### [Improving Word Embedding Compositionality using Lexicographic Definitions](https://doi.org/10.1145/3178876.3186007)

comment obtenir les meilleures représentations de texte à partir de représentations de mots (word embeddings) ? L'auteur utilise des ressources lexicographiques (wordnet) pour ses tests : l'embedding obtenu pour la définition d'un mot est-il proche de celui du mot ?

Le papier s'appuie sur une [thèse du même auteur](https://esc.fnwi.uva.nl/thesis/centraal/files/f1554608041.pdf), claire et bien écrite.

### [CESI: Canonicalizing Open Knowledge Bases using Embeddings and Side Information](https://doi.org/10.1145/3178876.3186030)

Amélioration de l'extraction de triplets (nom phrase, property, nom phrase) à partir de texte en calculant des embeddings pour les "nom phrases" (~entités)

### [Short-Text Topic Modeling via Non-negative Matrix Factorization Enriched with Local Word-Context Correlations](https://doi.org/10.1145/3178876.3186009)

Topic modeling for short texts, leveraging the word-context semantic correlations in the training

### [Towards Annotating Relational Data on the Web with Language Models](https://doi.org/10.1145/3178876.3186029)

### A paper by [David Blei](/tag/david_blei): (Dynamic Embeddings for Language Evolution)

2018-01-27 A propos

[1801.06146] Universal Language Model Fine-tuning for Text Classification

Tags: