Semanlink - Favoris ; Machine learning: problems AND Unsupervised machine learning

a language
model with access to information available in a
KB, no assumptions
about the availability of additional components
(such as Named Entity Taggers) or annotations

> While classes of
named entities (e.g., person or location) occur frequently,
each individual name (e.g, Atherton or
Zhouzhuang) may be observed infrequently even
in a very large corpus of text. As a result language
models learn to represent accurately only the most
popular named entities

> knowing that Alice is a name
used to refer to a person should give ample information
about the context in which the word may
occur (e.g., Bob visited Alice).

> ---

> extends a traditional **RNN LM**

> we enhance a traditional LM with a
gating mechanism that controls whether a particular
word is modeled as a general word or as a reference
to an entity
>
> We train the model end-to-end
with only the traditional predictive language modeling
perplexity objective
>
> KALM is trained end-to-end using
a predictive objective on large corpus of text.

> To the best of our knowledge, KALM is the first
unsupervised neural NER approach.

> KALM extends a traditional, RNN-based neural
LM.

2022-08-31 About

Unsupervised Learning — Sentence-Transformers documentation

Tags:

2022-08-20 About

What Makes a Good Classification Example?

Tags:

2022-08-16 About

Train and Fine-Tune Sentence Transformers Models

Tags:

Sentence-BERT

2022-08-13 About

[2207.06300] Re2G: Retrieve, Rerank, Generate

Tags:

2022-07-14 About

[2205.00820] Entity-aware Transformers for Entity Search

Tags:

2022-07-12 About

sentence bert model in onnx format · Issue #46 · UKPLab/sentence-transformers

Tags:

2022-06-13 About

[2205.15952] Knowledge Graph -- Deep Learning: A Case Study in Question Answering in Aviation Safety Domain

Tags:

2022-06-11 About

Domain transfer with GGPL: German Generative Pseudo Labeling 🥨 | by Matthias Richter | Jun, 2022 | ML6team

Tags:

2022-06-02 About

Nils Reimers sur Twitter : "GPL goes multi-lingual..."

Tags:

2022-06-01 About

[2205.04260] EASE: Entity-Aware Contrastive Learning of Sentence Embedding

Tags:

2022-05-11 About

Ramsri Goutham Golla sur Twitter : "Hi @Nils_Reimers For GPL you used "msmarco-distilbert-base-tas-b" model and ..."

Tags:

2022-04-27 About

[2110.08151] mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

Tags:

[Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_)

> Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**.
>
> the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features.

[Github](https://github.com/studio-ousia/luke)

> Entity representations are known to enhance
language models in mono-lingual settings
(Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019:  [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al.,
2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context))
presumably by introducing real-world knowledge.
We show that using entity representations facilitates
cross-lingual transfer by providing languageindependent
features.
>
> Multilingual extension of LUKE. The model is trained with the multilingual
masked language modeling (MLM) task as well
as the masked entity prediction (MEP) task with
Wikipedia entity embeddings

> We investigate two ways of using the entity representations
in cross-lingual transfer tasks:
> 1. perform
entity linking for the input text, and append
the detected entity tokens to the input sequence.
The entity tokens are expected to provide language independent
features to the model
> 2. use the entity
[MASK] token from the MEP task as a languageindependent
feature extractor.

2022-04-17 About

Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."

Tags:

2022-04-13 About

EASE: Entity-Aware Contrastive Learning of Sentence Embedding | Papers With Code

Tags:

2022-04-08 About

[2203.10581] Cluster & Tune: Boost Cold Start Performance in Text Classification

Tags:

[Leshem Choshen sur Twitter : "Labelled data is scarce, what can we do?..."](doc:2022/04/leshem_choshen_sur_twitter_l)

> **One-sentence Summary**: we suggest adding an unsupervised intermediate classification step, before finetunning and after pretraining BERT, and show it improves performance for data-constrained cases.

> for text classification cold start (when labeled
data is scarce), **add an intermediate unsupervised
classification task**, between the pretraining
and fine-tuning phases:
> perform clustering and
train the pre-trained model on predicting the
cluster labels.

> this additional
classification phase can significantly improve
performance, mainly for **topical classification**
tasks

> we use an efficient clustering technique,
that relies on simple Bag Of Words (BOW)
representations, to partition the unlabeled training
data into relatively homogeneous clusters of text
instances.
>
> Next, we treat these clusters as labeled
data for an intermediate text classification task, and
train the pre-trained model – with or without additional
MLM pretraining – with respect to this
multi-class problem, prior to the final fine-tuning
over the actual target-task labels

> The underlying
intuition is that inter-training the model
over a related text classification task would be more
beneficial compared to MLM inter-training, which
focuses on different textual entities, namely predicting
the identity of a single token.

2022-04-06 About

Leshem Choshen sur Twitter : "Labelled data is scarce, what can we do?..."

Tags:

2022-04-06 About

[2008.11228] A simple method for domain adaptation of sentence embeddings

Tags:

2022-04-01 About

[1910.06294] Training Compact Models for Low Resource Entity Tagging using Pre-trained Language Models

Tags:

2022-03-31 About

[2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Tags:

2022-03-31 About

Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)

Tags:

2022-03-31 About

Nils Reimers sur Twitter : "Great post on SetFit"

Tags:

2022-03-31 About

Sentence Embedding Fine-tuning for the French Language | by La Javaness R&D | Feb, 2022 | Medium

Tags:

2022-03-31 About

Domain Adaptation — Sentence-Transformers documentation

Tags:

2022-03-31 About

[2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning

Tags:

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 About

[2006.00632] Neural Unsupervised Domain Adaptation in NLP---A Survey

Tags:

2022-03-30 About

Domain adaptation of word embeddings through the exploitation of in-domain corpora and knowledge bases (PhD Thesis 2021)

Tags:

Thèse de Hicham El Boukkouri, univ Paris-Saclay

[Github](https://github.com/helboukkouri/phd-code)

### Goal

Given a target specialized domain, improve the quality of general-domain
word representations using in-domain corpora and/or knowledge bases

### Contributions

#### a method for specializing general-domain embeddings in a [Low-Resource](tag:nlp_low_resource_scenarios) context.

> - train static representations on the task corpus, 
> - resume the
pre-training of general-domain contextual embeddings on the same task corpus,
> - finally, combine both static and contextual representations into
one final model

#### we tackle the issue of using a general-domain vocabulary in a specialized domain

#### Evaluation of  re-training vs training from scratch on specialized corpora using a specialized vocabulary

training from scratch better, but not that much: re-training from a general model
is still appropriate as it is less expensive and leads to comparable, although
slightly lower, performance

#### Regarding subword-based tokenization systems
> we argue that they are inconvenient in practice -> CharacterBERT, a variant
of BERT that uses ELMo’s character-based system instead of WordPieces. More convenient ti use, superior robustness to misspellings

#### Ways to specialize general-domain representations using knowledge bases

a strong baseline using a simple
method relying on graph embeddings and concatenation, using only is_a relation

> both static and contextual embeddings may effectively be
specialized using this simple approach

#### Knowledge Injection Modules (KIM) that
inject the knowledge representations directly within the BERT-like models' architecture

### Notes

> our experiments focused on a single
setting (i.e. the medical domain and the English language)

> meta-embeddings, an approach that consists in combining
different sets of representations for achieving improved performance

2022-03-23 About

GuideToTransformersDomainAdaptation.ipynb - Colaboratory

Tags:

2022-03-18 About

NLP | How to add a domain-specific vocabulary (new tokens) to a subword tokenizer already trained like BERT WordPiece | by Pierre Guillou | Medium

Tags:

2022-03-18 About

Retraining roberta-base using the RoBERTa MLM Procedure | Medium

Tags:

2022-03-18 About

Studio Ousia sur Twitter : "Now using LUKE is easier than ever!" / Twitter

Tags:

2022-03-15 About

Document Matching for Job Descriptions | Semantic Scholar (2021)

Tags:

2022-03-09 About

Unsupervised Training of Retrievers Using GenQ (The Art of Asking Questions with GenQ) | Pinecone

Tags:

2022-03-09 About

NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"

Tags:

2022-03-09 About

[2109.06304] Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration

Tags:

2022-02-25 About

Nils Reimers sur Twitter : "Creating intent classes for chatbots is challenging This tutorial shows how to use sentence-transformers to find potentially overlapping intent classes and how to improve your data annotation work." / Twitter

Tags:

2022-02-19 About

Nils Reimers sur Twitter : "how to use the fast clustering algorithm from sentence-transformers..."

Tags:

2022-02-19 About

Semantic Keyword Clustering For 10,000+ Keywords [With Script]

Tags:

2022-02-18 About

sentence-transformers/fast_clustering.py at master · UKPLab/sentence-transformers

Tags:

2022-02-18 About

gsarti/scibert-nli · Hugging Face

Tags:

2022-01-29 About

Semantic Search — Sentence-Transformers documentation

Tags:

2022-01-29 About

[2108.13934] Robust Retrieval Augmented Generation for Zero-shot Slot Filling

Tags:

2022-01-19 About

Domain Transfer with BERT | Pinecone

Tags:

2022-01-04 About

[2112.09118] Towards Unsupervised Dense Information Retrieval with Contrastive Learning

Tags:

2021-12-21 About

Using pretrained SBERT model in cross-encoder · Issue #726 · UKPLab/sentence-transformers

Tags:

2021-12-17 About

Advance BERT model via transferring knowledge from Cross-Encoders to Bi-Encoders | by Chien Vu | Towards Data Science

Tags:

2021-12-17 About

[2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Tags:

An unsupervised domain adaptation technique for dense retrieval models

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 About

Nils Reimers sur Twitter : "Do dense retrieval models work out-of-the-box for your specific domain? Often the answer was No😢..."

Tags:

2021-12-15 About

Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine

Tags:

2021-12-05 About

Unsupervised_Extractive_Summarization - a Hugging Face Space by Hellisotherpeople

Tags:

2021-12-03 About

Unsupervised Training for Sentence Transformers | Pinecone

Tags:

2021-11-24 About

How to Fine-Tune Sentence-BERT for Question Answering | Capital One

Tags:

2021-11-21 About

Multilingual Sentence Transformers | Pinecone

Tags:

2021-11-04 About

Next-Gen Sentence Embeddings with Multiple Negatives Ranking Loss | Pinecone

Tags:

2021-10-27 About

Sentence Embeddings and Transformers | Pinecone

Tags:

2021-10-23 About

[2106.13474] Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Tags:

2021-10-21 About

[1908.11860] Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification

Tags:

2021-10-21 About

Application of Self-Organizing Maps in Text Clustering: A Review | IntechOpen (2012)

Tags:

2021-10-17 About

[1712.05972] Train Once, Test Anywhere: Zero-Shot Learning for Text Classification

Tags:

2021-10-16 About

[2010.07245] Text Classification Using Label Names Only: A Language Model Self-Training Approach

Tags:

2021-10-16 About

Sahajtomar/french_semantic · Hugging Face

Tags:

2021-10-14 About

[2010.12566] DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries

Tags:

2021-09-06 About

[2104.06979] TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

Tags:

> The most
successful previous approaches like InferSent (Conneau
et al., 2017), Universial Sentence Encoder
(USE) (Cer et al., 2018) and SBERT (Reimers and
Gurevych, 2019) heavily relied on labeled data to
train sentence embedding models.
>
> TSDAE can
achieve up to 93.1% of the performance of indomain
supervised approaches. Further, we
show that TSDAE is **a strong domain adaptation
and pre-training method for sentence
embeddings**, significantly outperforming other
approaches like Masked Language Model.

> During training, TSDAE
encodes corrupted sentences into fixed-sized
vectors and requires the decoder to reconstruct the
original sentences from this sentence embedding.

- <https://www.sbert.net/examples/unsupervised_learning/TSDAE/README.html>
- [github](https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning/TSDAE)
- [UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet](doc:2020/07/ukplab_sentence_transformers_s)
- [twitter](https://twitter.com/KexinWang2049/status/1433361957579538432):

> **TSDAE can learn domain-specific sentence embeddings with unlabeled sentences**
>
> Most importantly, instead of STS (Semantic Textual Similarity), **we suggest evaluating unsupervised sentence embeddings on the domain-specific tasks&datasets, which is the real use case for them**. Actually, STS scores do not correlate with performance on specific tasks.

2021-09-01 About

[2107.00676] A Primer on Pretrained Multilingual Language Models

Tags:

2021-07-13 About

Semantic Search with S-BERT is all you need

Tags:

2021-06-05 About

Improving Quality of Search Results Clustering with Approximate Matrix Factorisations (2006)

Tags:

Lingo

2021-06-02 About

Simple Unsupervised Keyphrase Extraction using Sentence Embeddings - ACL Anthology (2018)

Tags:

2021-05-31 About

Carrot2 search results clustering engine (online)

Tags:

Carrot2

2021-05-27 About

Clustering of semantically enriched short texts (2018)

Tags:

2021-05-26 About

Integrating Document Clustering and Multidocument Summarization

Tags:

2021-05-25 About

[2001.11631] Enhancement of Short Text Clustering by Iterative Classification

Tags:

2021-05-20 About

[2103.12953] Supporting Clustering with Contrastive Learning

Tags:

2021-05-20 About

A Self-Training Approach for Short Text Clustering - (Hadifar 2019)

Tags:

2021-05-20 About

Nils Reimers sur Twitter : "SBERT Release v1.1.0"

Tags:

2021-04-22 About

Simultaneous Categorization of Text Documents And Identification of Cluster-dependent Keywords (2003)

Tags:

Clustering of text documents

2021-04-20 About

A Survey of Text Clustering Algorithms - C. C. Aggarwal (2012)

Tags:

2021-04-20 About

[2011.05864] On the Sentence Embeddings from Pre-trained Language Models

Tags:

2021-04-19 About

SimCSE: Simple Contrastive Learning of Sentence Embeddings

Tags:

2021-04-18 About

Nils Reimers sur Twitter : "New models for Neural Information Retrieval..."

Tags:

2021-04-17 About

[2007.12603] IR-BERT: Leveraging BERT for Semantic Search in Background Linking for News Articles

Tags:

2021-04-12 About

[2007.15779] Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Tags:

2021-04-11 About

exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources - ACL Anthology

Tags:

**Focus on the Embedding of Domain-specific Vocabulary.**

> exBERT
adds a new domain-specific vocabulary and the corresponding
embedding layer, as well as a small
extension module to the original unmodified model

> a pretraining
method allowing **low-cost embedding of
domain-specific vocabulary in the context of an
existing large pre-trained model such as BERT**

> exBERT... explicitly incorporates
the new domain’s vocabulary, while being able to
**reuse the original pre-trained model’s weights as is**
to reduce required computation and training data. Specifically, exBERT extends BERT by augmenting
its embeddings for the original vocabulary with
new embeddings for the domain-specific vocabulary
via **a learned small “extension” module**. **The
output of the original and extension modules are
combined via a trainable weighted sum operation**

In a way similar to concept developed in

> [[1902.00751] Parameter-Efficient Transfer Learning for NLP](doc:2021/04/1902_00751_parameter_efficien), but not in the fine-tuning paradigm.

[Github](https://github.com/cgmhaicenter/exBERT)

2021-04-11 About

[1901.04085] Passage Re-ranking with BERT

Tags:

2021-03-26 About

SentenceTransformers Documentation

Tags:

2021-03-25 About

Rodrigo Nogueira sur Twitter : "Slides of our WSDM 2021 tutorial "Pretrained Transformers for Text Ranking: BERT and Beyond"

Tags:

2021-03-09 About

Distill our @huggingface zero-shot classifier with your specified class names

Tags:

2021-02-23 About

Zero-shot classifier distillation at master · huggingface/transformers

Tags:

2021-02-23 About

Zero-Shot Learning in Modern NLP | Joe Davison Blog (2020-05)

Tags:

2021-02-23 About

kamalkraj/BERT-NER: Pytorch-Named-Entity-Recognition-with-BERT

Tags:

2021-02-07 About

[1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Tags:

> way of **injecting factual knowledge about entities into the pretrained BERT model**.

(Feeding entity vectors
into BERT as if they
were wordpiece vectors without additional encoder
pretraining)

>
> **We align [Wikipedia2Vec](tag:wikipedia2vec) entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors**. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to [ERNIE](tag:ernie) (Zhang et al., 2019) and [KnowBert](tag:knowbert) (Peters et al., 2019), but it **requires no expensive further pretraining of the BERT encoder**.
>
> Our vector space alignment strategy is inspired by
cross-lingual word vector alignment

Related work on Entity-enhanced BERT:

> ([ERNIE](doc:2019/08/_1905_07129_ernie_enhanced_la) and [Knowbert](doc:2020/05/1909_04164_knowledge_enhanced)) are based on the design principle
that BERT be adapted to entity vectors. They introduce
new encoder layers to feed pretrained entity
vectors into the Transformer, and they require additional
pretraining to integrate the new parameters.
In contrast, E-BERT’s design principle is that entity
vectors be adapted to BERT.
>
> Two other knowledge-enhanced MLMs are [KEPLER](doc:2020/11/1911_06136_kepler_a_unified_)
(Wang et al., 2019c) and K-Adapter (Wang
et al., 2020)... Their factual knowledge
does not stem from entity vectors – instead, they
are trained in a multi-task setting on relation classification
and knowledge base completion.

Not to be cofounded with [[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](doc:2020/12/2009_02835_e_bert_a_phrase_a)

2021-01-12 About

[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce

Tags:

2020-12-14 About

Supporting content decision makers with machine learning | Dec, 2020 | Netflix TechBlog

Tags:

2020-12-11 About

[2004.10964] Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

Tags:

2020-12-01 About

Domain-Specific BERT Models · Chris McCormick

Tags:

Domain-Specific BERT

2020-12-01 About

[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

Tags:

2020-11-26 About

raphaelsty/ckb: Contextual knowledge bases

Tags:

2020-11-09 About

[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

Tags:

2020-11-03 About

[2010.03496] Inductive Entity Representations from Text via Link Prediction

Tags:

BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a
link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training)

> a method for **learning representations
of entities**, that uses a **pre-trained Transformer** based
architecture as an entity encoder, and
**link prediction training on a knowledge graph
with textual entity descriptions**.

> using entity descriptions,
an entity encoder is trained for link prediction in
a knowledge graph. The encoder can then be used
without fine-tuning to obtain features for entity classification
and information retrieval

Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their
objective targeted exclusively for link prediction (and not an objective that combines language modeling
and link prediction as Kepler)
performs better than Kepler's more complex one.

2020-11-03 About

Towards Unsupervised Text Classification Leveraging Experts and Word Embeddings - (ACL 2019)

Tags:

2020-10-05 About

Unsupervised text classification with word embeddings - Max Halford

Tags:

2020-10-05 About

[2010.00402] From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering

Tags:

2020-10-03 About

UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet

Tags:

2020-07-14 About

How to use BERT for finding similar sentences or similar news? · Issue #876 · huggingface/transformers

Tags:

2020-07-12 About

BERT Word Embeddings Tutorial · Chris McCormick

Tags:

2020-07-06 About

Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge (ACL Anthology 2020)

Tags:

Aim to leverage both contextual representation of input text (deep LMs)  and  knowledge derived
from curated KBs ([Wordnet](tag:wordnet)) to improve [slot tagging](tag:slot_tagging) in the presence of [out-of-vocab](tag:oov) words ([few-shot scenario](tag:few_shot_learning))

Method:

1. retrieve potentially relevant KB entities and
encode them into distributed representations that
describe global graph-structured information
2. BERT encoder
layer to capture context-aware representations of
the sequence and attend to the KB embeddings
using multi-level graph attention
3. integrate
BERT embeddings and the KB embeddings
to predict the slot type

Contributions:

1. feasibility of applying lexical ontology
to facilitate recognizing OOV words. First to consider the large-scale background
knowledge for enhancing context-aware
slot tagging models.
2. a knowledge integration mechanism that uses multi-level graph
attention to model explicit lexical relations.
3.experiments on two benchmark datasets

> our method makes a notable difference in a
scenario where samples are linguistically diverse,
and large vocab exists.

(Better improvements when using RNN than BERT, because BERT already contains a lot of background knowledge)

2020-07-04 About

[1909.04164] Knowledge Enhanced Contextual Word Representations

Tags:

2020-05-13 About

[2004.14958] A Call for More Rigor in Unsupervised Cross-lingual Learning

Tags:

2020-05-02 About

Unsupervised NER using BERT - Hands-on NLP model review - Quora

Tags:

2020-03-06 About

[2002.11402] Detecting Potential Topics In News Using BERT, CRF and Wikipedia

Tags:

2020-02-27 About

Paris NLP Season 4 Meetup #3 – Paris NLP (2020)

Tags:

2020-01-23 About

Named Entity Recognition with Bert – Depends on the definition

Tags:

2020-01-09 About

[1902.10909] BERT for Joint Intent Classification and Slot Filling

Tags:

2020-01-09 About

Richer Sentence Embeddings using Sentence-BERT — Part I

Tags:

Sentence-BERT

2020-01-06 About

Lecture 14 – Contextual Vectors | Stanford CS224U: Natural Language Understanding | Spring 2019

Tags:

2020-01-05 About

[1707.00306] Variable Selection Methods for Model-based Clustering

Tags:

2019-12-11 About

CONCEPTUAL GROUNDING FOR TEXT REPRESENTATION LEARNING

Tags:

2019-12-03 About

Clustering of semantically enriched short texts (2019)

Tags:

2019-11-15 About

Efficient multi-lingual language model fine-tuning · fast.ai NLP

Tags:

2019-10-22 About

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 About

[1905.07129] ERNIE: Enhanced Language Representation with Informative Entities

Tags:

2019-08-05 About

20% Accuracy Bump in Text Classification with ME-ULMFiT

Tags:

2019-06-23 About

[1906.02715] Visualizing and Measuring the Geometry of BERT

Tags:

2019-06-07 About

"I made a bet that a Naive Bayes classifier would work as well on humor recognition as a neural net with fine-tuned Bert embeddings. I won"

Tags:

2019-06-06 About

Knowledge-Based Short Text Categorization Using Entity and Category Embedding | Springer for Research & Development (2019)

Tags:

2019-05-30 About

A Technique for Building NLP Classifiers Efficiently with Transfer Learning and Weak Supervision

Tags:

2019-05-22 About

[1904.08398] DocBERT: BERT for Document Classification

Tags:

2019-04-18 About

kheyer/Genomic-ULMFiT: ULMFiT for Genomic Sequence Data

Tags:

2019-04-02 About

[1902.11269] Efficient Contextual Representation Learning Without Softmax Layer

Tags:

**how to accelerate contextual representation learning**.

> Contextual representation models are difficult to train due to the large parameter sizes and high computational complexity

> We find that the softmax layer (the output layer) causes significant inefficiency due to the large vocabulary size.
Therefore, we redesign the learning objectiv.
> Specifically, the proposed approach bypasses the softmax layer by performing language modeling with dimension reduction, and allows the models to leverage pre-trained word embeddings.
Our framework reduces the time spent on the output layer to a negligible level, eliminates almost all the trainable parameters of the softmax layer and performs language modeling without truncating the vocabulary.
When applied to ELMo, our method achieves a 4 times speedup and eliminates 80% trainable parameters while achieving competitive performance on downstream tasks.

**decouples learning contexts and words**

> Instead of using
a softmax layer to predict the distribution of the
missing word, we utilize and extend the SEMFIT
layer (Kumar and Tsvetkov, 2018) to **predict the
embedding of the missing word**.

2019-03-02 About

[Jaeyoung2018] Patent Document Clustering with Deep Embeddings

Tags:

2019-02-23 About

[1511.06335] Unsupervised Deep Embedding for Clustering Analysis

Tags:

2019-02-19 About

[1812.04616] Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs

Tags:

2018-12-14 About

The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar

Tags:

2018-12-03 About

Tutorial on Text Classification (NLP) using ULMFiT and fastai Library in Python - Analytics Vidhya

Tags:

2018-12-01 About

[1811.05370] Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents

Tags:

2018-11-20 About

Associative Multichannel Autoencoder for Multimodal Word Representation (2018)

Tags:

2018-11-15 About

iliaschalkidis/ELMo-keras: Re-implementation of ELMo on Keras

Tags:

2018-11-14 About

[1810.04805] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Tags:

2018-10-12 About

Can Global Semantic Context Improve Neural Language Models? - Apple (2018)

Tags:

2018-09-27 About

Linear algebraic structure of word meanings – Off the convex path

Tags:

2018-09-20 About

AI Can Recognize Images, But Text Has Been Tricky—Until Now | WIRED

Tags:

2018-09-08 About

[1601.03764] Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Tags:

> Here it is shown that multiple word senses reside
in linear superposition within the word
embedding and simple sparse coding can recover
vectors that approximately capture the
senses

> Each extracted word sense is accompanied by one of about  2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense.

> The success of the approach is mathematically explained using a variant of
the random walk on discourses model

("random walk": a generative model for language). Under the assumptions of this model,  there
exists a linear relationship between the vector of a
word w and the vectors of the words in its contexts (It is not the average of the words in w's context, but in a given corpus the matrix of the linear relationship does not depend on w. It can be estimated, and so we can compute the embedding of a word from the contexts it belongs to)

[Related blog post](/doc/?uri=https%3A%2F%2Fwww.offconvex.org%2F2016%2F07%2F10%2Fembeddingspolysemy%2F)

2018-08-28 About

[1807.03748] Representation Learning with Contrastive Predictive Coding

Tags:

2018-07-21 About

[1806.06259] Evaluation of sentence embeddings in downstream and linguistic probing tasks

Tags:

2018-06-19 About

ELMo: Deep contextualized word representations (2018)

Tags:

ELMo

2018-02-16 About

[1801.06146] Universal Language Model Fine-tuning for Text Classification

Tags:

2018-01-19 About

Modifications for the Cluster Content Discovery and the Cluster Label Induction Phases of the Lingo Algorithm (2014)

Tags:

Lingo

2017-11-11 About

[1602.06797] Semi-supervised Clustering for Short Text via Deep Representation Learning

Tags:

2017-11-07 About

Semantic Enriched Short Text Clustering | SpringerLink

Tags:

Short Text Clustering

2017-11-07 About

[1701.00185] Self-Taught Convolutional Neural Networks for Short Text Clustering

Tags:

2017-11-04 About

[1710.04087] Word Translation Without Parallel Data

Tags:

2017-10-14 About

Lingo: Search Results Clustering Algorithm Based on Singular Value Decomposition (slides)

Tags:

2017-07-11 About

Lingo: Search Results Clustering Algorithm Based on Singular Value Decomposition (2004) (paper)

Tags:

2017-07-11 About

Hierarchical clustering in Python and beyond

Tags:

2017-07-11 About

Document Clustering with Python

Tags:

2017-06-28 About

Document Representation and Dimension Reduction for Text Clustering

Tags:

2017-06-24 About

Lingo3G or Carrot2? | Carrot Search

Tags:

Carrot2

2017-06-04 About

Carrot2 manual

Tags:

Carrot2

2017-05-23 About

Carrot2: Text Clustering Algorithms and Applications

Tags:

2017-05-23 About

Result Clustering - Apache Solr Reference Guide - Apache Software Foundation

Tags:

2017-05-23 About

Lingo3G: real-time text clustering engine | Carrot Search

Tags:

2017-05-22 About

What are the best open source tools for unsupervised clustering of text documents? - Quora

Tags:

Clustering of text documents

2017-05-22 About

Quick review on Text Clustering and Text Similarity Approaches

Tags:

Author: Maali Mnasri (PhD @ CEA)

First transform text units to vectors? not always (eg. sentence similarity task using lexical word alignment). But vectors are efficient to process, and benefit from existing clustering algorithms such as k-means.

Sentence level or document level? Sentence clustering to summarise large documents.

Thematic clustering vs Semantic clustering: depends on the similarity measure.

Text similarity measures:

- Cosine similarity of tf-idf (suitable to produce thematic clusters)
- Knowledge-based Measures (wordNet) (quantify semantic relatedness of words),
- Word embedings

Examples, sample code:

- using wordnet with NLTK, and the formula to compute sentence similarities from word similarities.
- computing similarities between docs using gensim/word2vec

Which clustering algorithm?

- when we have an approximation of the clusters number, and when the similarity measure is not expensive in terms of computation time, clustering algo are suitable and fast. Sample code of k-means clustering using tf-idf vectors with scikit-learn
- Hierarchical clustering algorithms
    - don't need to give the number of clusters
    - but time consuming (calculate a similarity matrix for the sentences) 
- for voluminous data, use an incremental clustering algorithm: sentences are processed one at a time ; each new sentence is compared to each of the already formed clusters.

2017-05-18 About

Sapping Attention: When you have a MALLET, everything looks like a nail

Tags:

2014-04-25 About

Provable Algorithms for Machine Learning Problems by Rong Ge.

Tags:

2014-04-23 About