Semanlink - Supervised machine learning AND Neural networks

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 A propos

Building a sentence embedding index with fastText and BM25 | by David Mezzetti | Towards Data Science

Tags:

2021-09-30 A propos

Google AI Blog: From Vision to Language: Semi-supervised Learning in Action…at Scale

Tags:

2021-07-14 A propos

Yann LeCun sur Twitter : "Barlow Twins: a new super-simple self-supervised method to train joint-embedding architectures (aka Siamese nets) non contrastively. "

Tags:

2021-05-09 A propos

[1901.04085] Passage Re-ranking with BERT

Tags:

2021-03-26 A propos

New pipeline for zero-shot text classification - 🤗Transformers - Hugging Face Forums

Tags:

2021-03-15 A propos

Rodrigo Nogueira sur Twitter : "Slides of our WSDM 2021 tutorial "Pretrained Transformers for Text Ranking: BERT and Beyond"

Tags:

2021-03-09 A propos

Distill our @huggingface zero-shot classifier with your specified class names

Tags:

2021-02-23 A propos

Zero-shot classifier distillation at master · huggingface/transformers

Tags:

2021-02-23 A propos

Zero-Shot Learning in Modern NLP | Joe Davison Blog (2020-05)

Tags:

2021-02-23 A propos

X-BERT: eXtreme Multi-label Text Classification using Bidirectional Encoder Representations from Transformers

Tags:

2021-01-10 A propos

raphaelsty/ckb: Contextual knowledge bases

Tags:

2020-11-09 A propos

Sylvain Gugger sur Twitter : "Training a transformer model for text classification..."

Tags:

2020-10-19 A propos

Towards Unsupervised Text Classification Leveraging Experts and Word Embeddings - (ACL 2019)

Tags:

2020-10-05 A propos

Unsupervised text classification with word embeddings - Max Halford

Tags:

2020-10-05 A propos

[1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification

Tags:

A model that performs **text classification using entities in a knowledge base**.

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 A propos

Hierarchical Multi-label Classification of Text with Capsule Networks (2019)

Tags:

2020-08-15 A propos

[2003.11644] MAGNET: Multi-Label Text Classification using Attention-based Graph Neural Network

Tags:

2020-08-14 A propos

Hugging Face sur Twitter : "No labeled data? No problem. The 🤗 Transformers master branch now includes a built-in pipeline for zero-shot text classification...

Tags:

2020-08-12 A propos

FastHugs | ntentional

Tags:

2020-02-19 A propos

[1805.04174] Joint Embedding of Words and Labels for Text Classification (ACL Anthology 2018)

Tags:

2020-02-18 A propos

Siamese CNN for job–candidate matching (slides)

Tags:

2020-02-10 A propos

Matching Resumes to Jobs via Deep Siamese Network | Companion Proceedings of the The Web Conference 2018

Tags:

2020-02-10 A propos

[1703.07464] No Fuss Distance Metric Learning using Proxies

Tags:

> We address the problem of distance metric learning (DML), defined as learning a distance consistent with a notion of semantic similarity...
> Traditionnaly, supervision is expressed in the form of sets of points that follow
an ordinal relationship – an anchor point x is similar to
a set of positive points Y , and dissimilar to a set of negative
points Z, and a loss defined over these distances is minimized.
> Triplet-Based methods are challenging to optimize (a main issue is the need for finding informative triplets).
>
> We propose to **optimize the triplet loss on a different space of triplets, consisting of an anchor data point and similar and dissimilar proxy points which are learned as well**. These proxies approximate the original data points, so that a triplet loss over the proxies is a tight upper bound of the original loss.

Mentioned in this [blog post](/doc/2020/01/training_a_speaker_embedding_fr):

> "**Proxy based triplet learning**": instead of generating triplets, we learn an embedding for each class and use the learnt embedding as a proxy for triplets as part of the training. In other words, we can train end to end without the computationally expensive step of resampling triplets after each network update.

Near the conclusion:

> Our formulation of Proxy-NCA loss produces a loss very
similar to the standard cross-entropy loss used in classification.
However, we arrive at our formulation from a different
direction: we are not interested in the actual classifier and
indeed discard the proxies once the model has been trained.
Instead, the proxies are auxiliary variables, enabling more
effective optimization of the embedding model parameters.
**As such, our formulation not only enables us to surpass the
state of the art in zero-shot learning, but also offers an explanation
to the effectiveness of the standard trick of training
a classifier, and using its penultimate layer’s output as the
embedding.**

2020-02-09 A propos

[1503.03832] FaceNet: A Unified Embedding for Face Recognition and Clustering

Tags:

2020-01-25 A propos

Paris NLP Season 4 Meetup #3 – Paris NLP (2020)

Tags:

2020-01-23 A propos

Semantic Text Matching for Long-Form Documents (2019)

Tags:

2020-01-23 A propos

Training a Speaker Embedding from Scratch with Triplet Learning (2018)

Tags:

2020-01-23 A propos

Siamese Network for Image and Text similarity using Keras

Tags:

2020-01-22 A propos

Interpretable Named entity recognition with keras and LIME – Depends on the definition

Tags:

2020-01-09 A propos

One Shot learning, Siamese networks and Triplet Loss with Keras

Tags:

2019-10-13 A propos

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 A propos

Open-sourcing hyperparameter autotuning for fastText

Tags:

FastText

2019-08-27 A propos

Learning Text Similarity with Siamese Recurrent Networks (2016)

Tags:

2019-08-07 A propos

Siamese Neural Networks for One-shot Image Recognition (2015)

Tags:

2019-08-06 A propos

Classifying duplicate questions from Quora with Keras | R-bloggers

Tags:

2019-07-03 A propos

Tags:

2019-07-03 A propos

One Shot Learning with Siamese Networks using Keras

Tags:

2019-06-28 A propos

20% Accuracy Bump in Text Classification with ME-ULMFiT

Tags:

2019-06-23 A propos

"I made a bet that a Naive Bayes classifier would work as well on humor recognition as a neural net with fine-tuned Bert embeddings. I won"

Tags:

2019-06-06 A propos

A Technique for Building NLP Classifiers Efficiently with Transfer Learning and Weak Supervision

Tags:

2019-05-22 A propos

[1904.08398] DocBERT: BERT for Document Classification

Tags:

2019-04-18 A propos

How to do Deep Learning on Graphs with Graph Convolutional Networks Part 2

Tags:

2019-01-30 A propos

Cheatsheet · fastText

Tags:

2019-01-29 A propos

Using FastText models (not vectors) for robust embeddings | Kaggle

Tags:

2019-01-29 A propos

Tutorial on Text Classification (NLP) using ULMFiT and fastai Library in Python - Analytics Vidhya

Tags:

2018-12-01 A propos

Practical Text Classification With Python and Keras – Real Python

Tags:

2018-10-25 A propos

GitHub - marcotcr/lime: Lime: Explaining the predictions of any machine learning classifier

Tags:

2018-09-09 A propos

[1602.04938] "Why Should I Trust You?": Explaining the Predictions of Any Classifier

Tags:

2018-09-09 A propos

Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification | Scientific Reports

Tags:

2018-08-28 A propos

Deep Unordered Composition Rivals Syntactic Methods for Text Classification (2015)

Tags:

2018-05-29 A propos

Under the hood: Multilingual embeddings | Engineering Blog | Facebook Code

Tags:

2018-05-11 A propos

GitHub - Babylonpartners/fastText_multilingual: Multilingual word vectors

Tags:

2018-05-11 A propos

Text Classification with TensorFlow Estimators

Tags:

2018-04-17 A propos

Intro to text classification with Keras: automatically tagging Stack Overflow posts | Google Cloud Big Data and Machine Learning Blog

Tags:

2018-03-04 A propos

Codes of Interest: Using Bottleneck Features for Multi-Class Classification in Keras and TensorFlow

Tags:

2018-03-04 A propos

A Benchmark of Text Classification in PyTorch

Tags:

2018-02-28 A propos

RESEARCH TRACK: Web Content Analysis, Semantics and Knowledge

Tags:

[CFP](https://www2018.thewebconf.org/call-for-papers/research-tracks-cfp/web-content-analysis/)

> In previous years, ‘content analysis’ and ‘semantic and knowledge’ were in separate track. This year, we combined these tracks to emphasize the close relationship between these topics; **the use of content to curate knowledge and the use of knowledge to guide content analysis and intelligent usage**.

Some of the accepted papers:
### [Large-Scale Hierarchical Text Classification with Recursively Regularized Deep Graph-CNN](https://doi.org/10.1145/3178876.3186005)

[Hierarchical Text Classification](/tag/nlp_hierarchical_text_classification): Text classification to a hierarchical taxonomy of topics, using graph representation of text, and CNN over this graph

Renvoie à ce qui a été vu dans le tutorial "Graph-based Text Representations"

from the abstract:

> a graph-CNN based deep learning model to first convert texts to graph-of-words, and then use graph convolution operations to convolve the word graph. Graph-of-words representation of texts has the advantage of capturing non-consecutive and long-distance semantics. CNN models have the advantage of learning different level of semantics. To further leverage the hierarchy of labels, we regularize the deep architecture with the dependency among labels

Conversion of text to graph: potentially given a single document

### [Weakly-supervised Relation Extraction by Pattern-enhanced Embedding Learning](https://doi.org/10.1145/3178876.3186024 )

Extraction de relations de corpus de textes de façon semi-supervisée, dans un contexte où on a peu de données labellisées décrivant les relations.

Par exemple, des données labellisées indique que le texte "Beijing, capital of China" correspond à la relation entre entités : ("Beijing", "Capital Of", "China), et on voudrait pouvoir extraire les entités et relations pertinentes à partir de texte tel que "Paris, France's capital,..."

Le papier décrit une méthode qui combine deux modules, l'un basé sur l'extraction automatique de patterns (par ex "[Head], Capital Of [Tail]") et l'autre sur la "sémantique distributionnelle" (du type "word embeddings"). Ces deux modules collaborent, le premier permettant de créer des instances de relations augmentant la base de connaissance sur lequel entrainer le second, et le second aidant le premier à déterminer des patterns informatifs ("co-entrainement")

### [Scalable Instance Reconstruction in Knowledge Bases via Relatedness Affiliated Embedding](https://doi.org/10.1145/3178876.3186017)

Knowledge base completion problem: usually, it is formulated as a link prediction problem, but not here. A novel knowledge embedding model ("Joint Modelling and Learning of Relatedness and Embedding")

### [Improving Word Embedding Compositionality using Lexicographic Definitions](https://doi.org/10.1145/3178876.3186007)

comment obtenir les meilleures représentations de texte à partir de représentations de mots (word embeddings) ? L'auteur utilise des ressources lexicographiques (wordnet) pour ses tests : l'embedding obtenu pour la définition d'un mot est-il proche de celui du mot ?

Le papier s'appuie sur une [thèse du même auteur](https://esc.fnwi.uva.nl/thesis/centraal/files/f1554608041.pdf), claire et bien écrite.

### [CESI: Canonicalizing Open Knowledge Bases using Embeddings and Side Information](https://doi.org/10.1145/3178876.3186030)

Amélioration de l'extraction de triplets (nom phrase, property, nom phrase) à partir de texte en calculant des embeddings pour les "nom phrases" (~entités)

### [Short-Text Topic Modeling via Non-negative Matrix Factorization Enriched with Local Word-Context Correlations](https://doi.org/10.1145/3178876.3186009)

Topic modeling for short texts, leveraging the word-context semantic correlations in the training

### [Towards Annotating Relational Data on the Web with Language Models](https://doi.org/10.1145/3178876.3186029)

### A paper by [David Blei](/tag/david_blei): (Dynamic Embeddings for Language Evolution)

2018-01-27 A propos

[1801.06146] Universal Language Model Fine-tuning for Text Classification

Tags: