Semanlink - Favoris ; Data mining AND Embeddings

[Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_)

> Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**.
>
> the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features.

[Github](https://github.com/studio-ousia/luke)

> Entity representations are known to enhance
language models in mono-lingual settings
(Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019:  [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al.,
2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context))
presumably by introducing real-world knowledge.
We show that using entity representations facilitates
cross-lingual transfer by providing languageindependent
features.
>
> Multilingual extension of LUKE. The model is trained with the multilingual
masked language modeling (MLM) task as well
as the masked entity prediction (MEP) task with
Wikipedia entity embeddings

> We investigate two ways of using the entity representations
in cross-lingual transfer tasks:
> 1. perform
entity linking for the input text, and append
the detected entity tokens to the input sequence.
The entity tokens are expected to provide language independent
features to the model
> 2. use the entity
[MASK] token from the MEP task as a languageindependent
feature extractor.

2022-04-17 A propos

I.A.B sur Twitter : "When we "know the meaning" of a word, what is it that we know? For example, what does knowing the words "dolphin" and "tiger" entail?..."

Tags:

2022-04-15 A propos

Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."

Tags:

2022-04-13 A propos

[2008.11228] A simple method for domain adaptation of sentence embeddings

Tags:

2022-04-01 A propos

[2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Tags:

2022-03-31 A propos

Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)

Tags:

2022-03-31 A propos

Nils Reimers sur Twitter : "Great post on SetFit"

Tags:

2022-03-31 A propos

Sentence Embedding Fine-tuning for the French Language | by La Javaness R&D | Feb, 2022 | Medium

Tags:

2022-03-31 A propos

Domain Adaptation — Sentence-Transformers documentation

Tags:

2022-03-31 A propos

[2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning

Tags:

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 A propos

[2004.09813] Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation

Tags:

2022-03-18 A propos

Studio Ousia sur Twitter : "Now using LUKE is easier than ever!" / Twitter

Tags:

2022-03-15 A propos

[2110.10778] Contrastive Document Representation Learning with Graph Attention Networks

Tags:

2022-03-10 A propos

Document Matching for Job Descriptions | Semantic Scholar (2021)

Tags:

2022-03-09 A propos

NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"

Tags:

2022-03-09 A propos

[2109.06304] Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration

Tags:

2022-02-25 A propos

Nils Reimers sur Twitter : "Creating intent classes for chatbots is challenging This tutorial shows how to use sentence-transformers to find potentially overlapping intent classes and how to improve your data annotation work." / Twitter

Tags:

2022-02-19 A propos

Nils Reimers sur Twitter : "how to use the fast clustering algorithm from sentence-transformers..."

Tags:

2022-02-19 A propos

sentence-transformers/fast_clustering.py at master · UKPLab/sentence-transformers

Tags:

2022-02-18 A propos

gsarti/scibert-nli · Hugging Face

Tags:

2022-01-29 A propos

Semantic Search — Sentence-Transformers documentation

Tags:

2022-01-29 A propos

[2004.07180] SPECTER: Document-level Representation Learning using Citation-informed Transformers

Tags:

2022-01-29 A propos

Using pretrained SBERT model in cross-encoder · Issue #726 · UKPLab/sentence-transformers

Tags:

2021-12-17 A propos

Advance BERT model via transferring knowledge from Cross-Encoders to Bi-Encoders | by Chien Vu | Towards Data Science

Tags:

2021-12-17 A propos

[2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Tags:

An unsupervised domain adaptation technique for dense retrieval models

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 A propos

Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine

Tags:

2021-12-05 A propos

Unsupervised_Extractive_Summarization - a Hugging Face Space by Hellisotherpeople

Tags:

2021-12-03 A propos

Unsupervised Training for Sentence Transformers | Pinecone

Tags:

2021-11-24 A propos

How to Fine-Tune Sentence-BERT for Question Answering | Capital One

Tags:

2021-11-21 A propos

Multilingual Sentence Transformers | Pinecone

Tags:

2021-11-04 A propos

Next-Gen Sentence Embeddings with Multiple Negatives Ranking Loss | Pinecone

Tags:

2021-10-27 A propos

Sentence Embeddings and Transformers | Pinecone

Tags:

2021-10-23 A propos

Sahajtomar/french_semantic · Hugging Face

Tags:

2021-10-14 A propos

[2109.08133] Phrase Retrieval Learns Passage Retrieval, Too

Tags:

2021-09-30 A propos

Building a sentence embedding index with fastText and BM25 | by David Mezzetti | Towards Data Science

Tags:

2021-09-30 A propos

[2104.06979] TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

Tags:

> The most
successful previous approaches like InferSent (Conneau
et al., 2017), Universial Sentence Encoder
(USE) (Cer et al., 2018) and SBERT (Reimers and
Gurevych, 2019) heavily relied on labeled data to
train sentence embedding models.
>
> TSDAE can
achieve up to 93.1% of the performance of indomain
supervised approaches. Further, we
show that TSDAE is **a strong domain adaptation
and pre-training method for sentence
embeddings**, significantly outperforming other
approaches like Masked Language Model.

> During training, TSDAE
encodes corrupted sentences into fixed-sized
vectors and requires the decoder to reconstruct the
original sentences from this sentence embedding.

- <https://www.sbert.net/examples/unsupervised_learning/TSDAE/README.html>
- [github](https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning/TSDAE)
- [UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet](doc:2020/07/ukplab_sentence_transformers_s)
- [twitter](https://twitter.com/KexinWang2049/status/1433361957579538432):

> **TSDAE can learn domain-specific sentence embeddings with unlabeled sentences**
>
> Most importantly, instead of STS (Semantic Textual Similarity), **we suggest evaluating unsupervised sentence embeddings on the domain-specific tasks&datasets, which is the real use case for them**. Actually, STS scores do not correlate with performance on specific tasks.

2021-09-01 A propos

Semantic Search with S-BERT is all you need

Tags:

2021-06-05 A propos

A Self-Training Approach for Short Text Clustering - (Hadifar 2019)

Tags:

2021-05-20 A propos

Nils Reimers sur Twitter : "SBERT Release v1.1.0"

Tags:

2021-04-22 A propos

[2011.05864] On the Sentence Embeddings from Pre-trained Language Models

Tags:

2021-04-19 A propos

SimCSE: Simple Contrastive Learning of Sentence Embeddings

Tags:

2021-04-18 A propos

Nils Reimers sur Twitter : "New models for Neural Information Retrieval..."

Tags:

2021-04-17 A propos

[2007.12603] IR-BERT: Leveraging BERT for Semantic Search in Background Linking for News Articles

Tags:

2021-04-12 A propos

SentenceTransformers Documentation

Tags:

2021-03-25 A propos

Zero-Shot Learning in Modern NLP | Joe Davison Blog (2020-05)

Tags:

2021-02-23 A propos

[1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Tags:

> way of **injecting factual knowledge about entities into the pretrained BERT model**.

(Feeding entity vectors
into BERT as if they
were wordpiece vectors without additional encoder
pretraining)

>
> **We align [Wikipedia2Vec](tag:wikipedia2vec) entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors**. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to [ERNIE](tag:ernie) (Zhang et al., 2019) and [KnowBert](tag:knowbert) (Peters et al., 2019), but it **requires no expensive further pretraining of the BERT encoder**.
>
> Our vector space alignment strategy is inspired by
cross-lingual word vector alignment

Related work on Entity-enhanced BERT:

> ([ERNIE](doc:2019/08/_1905_07129_ernie_enhanced_la) and [Knowbert](doc:2020/05/1909_04164_knowledge_enhanced)) are based on the design principle
that BERT be adapted to entity vectors. They introduce
new encoder layers to feed pretrained entity
vectors into the Transformer, and they require additional
pretraining to integrate the new parameters.
In contrast, E-BERT’s design principle is that entity
vectors be adapted to BERT.
>
> Two other knowledge-enhanced MLMs are [KEPLER](doc:2020/11/1911_06136_kepler_a_unified_)
(Wang et al., 2019c) and K-Adapter (Wang
et al., 2020)... Their factual knowledge
does not stem from entity vectors – instead, they
are trained in a multi-task setting on relation classification
and knowledge base completion.

Not to be cofounded with [[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](doc:2020/12/2009_02835_e_bert_a_phrase_a)

2021-01-12 A propos

Pablo Castro sur Twitter : "Random finding of the day for word embeddings: vec("apple")-vec("apples") yields a vector close to ipad, ipod, etc. (apples removes the "fruitness" from apple)

Tags:

2020-12-18 A propos

google/tapas-base-finetuned-wtq · Hugging Face

Tags:

2020-12-17 A propos

Supporting content decision makers with machine learning | Dec, 2020 | Netflix TechBlog

Tags:

2020-12-11 A propos

[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

Tags:

2020-11-26 A propos

raphaelsty/ckb: Contextual knowledge bases

Tags:

2020-11-09 A propos

[2010.03496] Inductive Entity Representations from Text via Link Prediction

Tags:

BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a
link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training)

> a method for **learning representations
of entities**, that uses a **pre-trained Transformer** based
architecture as an entity encoder, and
**link prediction training on a knowledge graph
with textual entity descriptions**.

> using entity descriptions,
an entity encoder is trained for link prediction in
a knowledge graph. The encoder can then be used
without fine-tuning to obtain features for entity classification
and information retrieval

Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their
objective targeted exclusively for link prediction (and not an objective that combines language modeling
and link prediction as Kepler)
performs better than Kepler's more complex one.

2020-11-03 A propos

Towards Unsupervised Text Classification Leveraging Experts and Word Embeddings - (ACL 2019)

Tags:

2020-10-05 A propos

Unsupervised text classification with word embeddings - Max Halford

Tags:

2020-10-05 A propos

[1802.05930] Learning beyond datasets: Knowledge Graph Augmented Neural Networks for Natural language Processing

Tags:

2020-10-02 A propos

[1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification

Tags:

A model that performs **text classification using entities in a knowledge base**.

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 A propos

[1812.06280] Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia

Tags:

2020-09-02 A propos

UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet

Tags:

2020-07-14 A propos

How to use BERT for finding similar sentences or similar news? · Issue #876 · huggingface/transformers

Tags:

2020-07-12 A propos

BERT Word Embeddings Tutorial · Chris McCormick

Tags:

2020-07-06 A propos

[1910.00163] Specializing Word Embeddings (for Parsing) by Information Bottleneck

Tags:

2020-06-29 A propos

On word embeddings

Tags:

2020-06-05 A propos

Approximating the Softmax for Learning Word Embeddings

Tags:

2020-06-04 A propos

[1709.03933] Hash Embeddings for Efficient Word Representations

Tags:

2020-05-19 A propos

[1909.04164] Knowledge Enhanced Contextual Word Representations

Tags:

2020-05-13 A propos

[1807.08447] LinkNBed: Multi-Graph Representation Learning with Entity Linkage

Tags:

2020-05-11 A propos

[1906.01195] Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs

Tags:

2020-04-30 A propos

[2001.09522] TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network

Tags:

how to add a set of new concepts to an existing taxonomy.

[Tweet](https://twitter.com/mickeyjs6/status/1253772146142216194?s=20) [GitHub](https://github.com/mickeystroller/TaxoExpan)

> we study the taxonomy expansion task: given an
existing taxonomy and a set of new emerging concepts, we aim
to automatically expand the taxonomy to incorporate these new
concepts (without changing the existing relations in the given taxonomy).

> To the best of our knowledge, this is the first study on **how to
expand an existing directed acyclic graph (as we model a taxonomy
as a DAG) using self-supervised learning**.

Self-supervised framework, the existing taxonomy being used as training data: it learns a model to predict whether a query concept is the direct hyponym of an anchor concept.

> 2 techniques:
>
> 1. a **position-enhanced graph neural network that encodes the local structure of an anchor concept** in the existing taxonomy,
> 2. a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data.

Regarding 1: uses [GNN](/tag/graph_neural_networks.html) to model the "ego network" of concepts (potential “siblings”
and “grand parents” of the query concept).

> Regular
GNNs fail to distinguish nodes with different relative positions to
the query (i.e., some nodes are grand parents of the query while
the others are siblings of the query). To address this limitation, we
present a simple but effective enhancement to inject such position
information into GNNs using position embedding. We show that
such embedding can be easily integrated with existing GNN architectures
(e.g., [GCN](/tag/graph_convolutional_networks) and GAT) and significantly boosts the
prediction performance

Regarding point 2: uses InfoNCE loss, cf. [Contrastive Predictive Coding](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1807.03748)

> Instead of predicting
whether each individual ⟨query concept, anchor concept⟩ pair
is positive or not, we first group all pairs sharing the same query
concept into a single training instance and learn a model to select
the positive pair among other negative ones from the group.

(Hum, ça me rappelle quelque chose)

> assume each concept (in existing taxonomy + set of new concepts) has an initial embedding
vector learned from some text associated with this concept.

To keep things tractable, only attempts to find a single parent node of each new concept.

2020-04-25 A propos

[1911.02168] CoKE: Contextualized Knowledge Graph Embedding

Tags:

2020-03-22 A propos

[1902.10197] RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space

Tags:

2020-03-03 A propos

[1910.04126] Scalable Nearest Neighbor Search for Optimal Transport

Tags:

2020-02-20 A propos

[1805.04174] Joint Embedding of Words and Labels for Text Classification (ACL Anthology 2018)

Tags:

2020-02-18 A propos

davidsbatista/BREDS: "Bootstrapping Relationship Extractors with Distributional Semantics" (Batista et al., 2015) - code for EMNLP'15 paper

Tags:

2020-01-11 A propos

Tags:

2020-01-10 A propos

Richer Sentence Embeddings using Sentence-BERT — Part I

Tags:

Sentence-BERT

2020-01-06 A propos

Lecture 14 – Contextual Vectors | Stanford CS224U: Natural Language Understanding | Spring 2019

Tags:

2020-01-05 A propos

Efficient multi-lingual language model fine-tuning · fast.ai NLP

Tags:

2019-10-22 A propos

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 A propos

Open-sourcing hyperparameter autotuning for fastText

Tags:

FastText

2019-08-27 A propos

[1905.07129] ERNIE: Enhanced Language Representation with Informative Entities

Tags:

2019-08-05 A propos

Reasoning With Neural Tensor Networks for Knowledge Base Completion (2013)

Tags:

**Predicting the likely truth of additional facts based on existing facts in the knowledge base.**

> we introduce an expressive neural
tensor network suitable for reasoning over relationships between two entities.

Most similar work: [Bordes et al.](http://127.0.0.1:8080/semanlink/doc/2019/08/learning_structured_embeddings_) (2011)

Contributions:

1. new neural tensor
network (**NTN**) suitable for reasoning over relationships between two entities. Generalizes several previous neural network models and provides a more
powerful way to model relational information than a standard neural network layer.
2. a new way to represent entities in knowledge bases, as the
average of  their constituting word vectorss, allowing the sharing of statistical strength between the words describing
each entity (e.g., Bank of China and China).
3. incorporation of word vectors which are trained on large unlabeled text

> We **learn to modify word representations
via grounding in world knowledge**. This essentially allows us to analyze word embeddings and
query them for specific relations. Furthermore, the resulting vectors could be used in other tasks
such as named entity recognition or relation classification in natural language

**Makes use of entity name**: NTN
first learns word vectors from an auxiliary news corpus, and
then initializes the representation of each entity by averaging
the vectors of words contained in its name. For example,
**the embedding of AlfredHitchcock is initialized by the
average word vectors of “alfred” and “hitchcock”**... This kind of methods model textual information
separately from KG facts, and hence fail to leverage
interactions between them.[src](doc:2019/05/knowledge_graph_embedding_a_su)

2019-08-03 A propos

[1602.01137] A Dual Embedding Space Model for Document Ranking

Tags:

2019-07-17 A propos

[1810.04882] Towards Understanding Linear Word Analogies

Tags:

2019-06-24 A propos

Kawin Ethayarajh sur Twitter : "When and why does king - man + woman = queen?"

Tags:

Word embeddings

2019-06-24 A propos

[1906.02715] Visualizing and Measuring the Geometry of BERT

Tags:

2019-06-07 A propos

Word Embeddings: 6 Years Later

Tags:

Word embeddings

2019-06-03 A propos

[1703.02507] Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features

Tags:

2019-03-25 A propos

Contrastive Unsupervised Learning of Semantic Representations: A Theoretical Framework – Off the convex path (2019-03)

Tags:

2019-03-20 A propos

Word Embeddings: Explaining their properties – Off the convex path (2016)

Tags:

2019-03-20 A propos

[1902.11269] Efficient Contextual Representation Learning Without Softmax Layer

Tags:

**how to accelerate contextual representation learning**.

> Contextual representation models are difficult to train due to the large parameter sizes and high computational complexity

> We find that the softmax layer (the output layer) causes significant inefficiency due to the large vocabulary size.
Therefore, we redesign the learning objectiv.
> Specifically, the proposed approach bypasses the softmax layer by performing language modeling with dimension reduction, and allows the models to leverage pre-trained word embeddings.
Our framework reduces the time spent on the output layer to a negligible level, eliminates almost all the trainable parameters of the softmax layer and performs language modeling without truncating the vocabulary.
When applied to ELMo, our method achieves a 4 times speedup and eliminates 80% trainable parameters while achieving competitive performance on downstream tasks.

**decouples learning contexts and words**

> Instead of using
a softmax layer to predict the distribution of the
missing word, we utilize and extend the SEMFIT
layer (Kumar and Tsvetkov, 2018) to **predict the
embedding of the missing word**.

2019-03-02 A propos

[1902.10618] Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition

Tags:

2019-02-28 A propos

Keywords2vec

Tags:

2019-02-09 A propos

nlp-notebooks/Simple Sentence Similarity.ipynb at master · nlptown/nlp-notebooks

Tags:

2019-02-07 A propos

BPEmb: Subword Embeddings

Tags:

2019-01-31 A propos

Cheatsheet · fastText

Tags:

2019-01-29 A propos

Using FastText models (not vectors) for robust embeddings | Kaggle

Tags:

2019-01-29 A propos

[1601.01343] Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation

Tags:

2019-01-27 A propos

Wikipedia2Vec

Tags:

2018-12-23 A propos

The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar

Tags:

2018-12-03 A propos

[1811.05370] Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents

Tags:

2018-11-20 A propos

iliaschalkidis/ELMo-keras: Re-implementation of ELMo on Keras

Tags:

2018-11-14 A propos

Word Mover's Embedding: From Word2Vec to Document Embedding (2018)

Tags:

2018-11-10 A propos

Cross-Lingual Word Representations: Induction and Evaluation (Tutorial EMNLP 2017)

Tags:

2018-11-05 A propos

[1710.06632] Towards a Seamless Integration of Word Senses into Downstream NLP Applications

Tags:

2018-10-09 A propos

Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline (Ethayarajh 2018)

Tags:

2018-10-08 A propos

Can Global Semantic Context Improve Neural Language Models? - Apple (2018)

Tags:

2018-09-27 A propos

Linear algebraic structure of word meanings – Off the convex path

Tags:

2018-09-20 A propos

A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (2018)

Tags:

A La Carte Embedding

2018-09-18 A propos

Simple and efficient semantic embeddings for rare words, n-grams, and language features – Off the convex path

Tags:

2018-09-18 A propos

AI Can Recognize Images, But Text Has Been Tricky—Until Now | WIRED

Tags:

2018-09-08 A propos

A Latent Variable Model Approach to PMI-based Word Embeddings (2016)

Tags:

2018-08-28 A propos

[1601.03764] Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Tags:

> Here it is shown that multiple word senses reside
in linear superposition within the word
embedding and simple sparse coding can recover
vectors that approximately capture the
senses

> Each extracted word sense is accompanied by one of about  2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense.

> The success of the approach is mathematically explained using a variant of
the random walk on discourses model

("random walk": a generative model for language). Under the assumptions of this model,  there
exists a linear relationship between the vector of a
word w and the vectors of the words in its contexts (It is not the average of the words in w's context, but in a given corpus the matrix of the linear relationship does not depend on w. It can be estimated, and so we can compute the embedding of a word from the contexts it belongs to)

[Related blog post](/doc/?uri=https%3A%2F%2Fwww.offconvex.org%2F2016%2F07%2F10%2Fembeddingspolysemy%2F)

2018-08-28 A propos

zalandoresearch/flair: A very simple framework for state-of-the-art NLP

Tags:

2018-08-24 A propos

Contextual String Embeddings for Sequence Labeling (2018)

Tags:

Word embeddings

2018-08-24 A propos

Learning to Understand Phrases by Embedding the Dictionary (2016)

Tags:

2018-08-23 A propos

what are the pros and cons of the various unsupervised word and sentence/ document embedding models? - Quora

Tags:

2018-08-19 A propos

[1608.05426] A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments

Tags:

2018-07-23 A propos

SIGIR 2018 Tutorial - Knowledge Extraction and Inference from Text: Shallow, Deep, and Everything in Between

Tags:

2018-07-09 A propos

How sampling works in Word2vec? Can someone please make me understand NCE and negative sampling? - Cross Validated

Tags:

2018-07-07 A propos

Deep-learning-free Text and Sentence Embedding, Part 2 – Off the convex path

Tags:

2018-06-25 A propos

Deep-learning-free Text and Sentence Embedding, Part 1 – Off the convex path

Tags:

2018-06-25 A propos

[1806.06259] Evaluation of sentence embeddings in downstream and linguistic probing tasks

Tags:

2018-06-19 A propos

Sanjeev Arora on "A theoretical approach to semantic representations" - YouTube (2016)

Tags:

2018-06-10 A propos

A Tri-Partite Neural Document Language Model for Semantic Information Retrieval (2018 - ESWC conference)

Tags:

2018-06-08 A propos

A Word Embedding Approach to Predicting the Compositionality of Multiword Expressions (2015)

Tags:

2018-06-08 A propos

The Current Best of Universal Word Embeddings and Sentence Embeddings (2018)

Tags:

2018-05-30 A propos

Deep Unordered Composition Rivals Syntactic Methods for Text Classification (2015)

Tags:

2018-05-29 A propos

A Theoretical Approach to Semantic Coding and Hashing | Simons Institute for the Theory of Computing (2016)

Tags:

2018-05-26 A propos

A simple spell checker built from word vectors – Ed Rushton – Medium

Tags:

2018-05-25 A propos

Module google/universal-sentence-encoder | TensorFlow

Tags:

2018-05-23 A propos

[1404.5367] Lexicon Infused Phrase Embeddings for Named Entity Resolution

Tags:

2018-05-22 A propos

[1706.04902] A Survey Of Cross-lingual Word Embedding Models

Tags:

2018-05-20 A propos

NLP: Any libraries/dictionaries out there for fixing common spelling errors? - Part 2 & Alumni - Deep Learning Course Forums

Tags:

2018-05-18 A propos

An Exploration of Embeddings for Generalized Phrases (2014)

Tags:

2018-05-12 A propos

Under the hood: Multilingual embeddings | Engineering Blog | Facebook Code

Tags:

2018-05-11 A propos

GitHub - Babylonpartners/fastText_multilingual: Multilingual word vectors

Tags:

2018-05-11 A propos

NLP using Word Vectors with Spacy - CLDSPN | Kaggle

Tags:

2018-05-11 A propos

Dict2vec : Learning Word Embeddings using Lexical Dictionaries

Tags:

2018-05-11 A propos

A Simple but Tough-to-Beat Baseline for Sentence Embeddings (2017)

Tags:

2018-05-10 A propos

Improving Word Embedding Compositionality using Lexicographic Definitions

Tags:

2018-05-10 A propos

That Makes Sense: Joint Sense Retrofitting from Contextual and Ontological Information

Tags:

post-processing method for generating low-dimensional sense embedding. Emploies the ontological and contextual information simultaneously.

(Poster at the Web Conf) [Github](https://github.com/y95847frank/Joint-Retrofitting)

Calcule des "sense embeddings", en partant de word embeddings pré-calculés (par ex avec word2vec), et de données de type lexicographiques (ex wordnet), en contraignant, pour un sens, la distance entre sense et word embedding.

Abstract:

> While recent word embedding models demonstrate their abilities to capture syntactic and semantic information, the demand for sense level embedding is getting higher. In this study, we propose a novel joint sense embedding learning model that retrofits the word representation into sense representation from contextual and ontological information. The experiments show the effectiveness and robustness of our model that outperforms previous approaches in four public available benchmark datasets.

> Given a trained word embedding and a lexical ontology that contains sense level relationships (e.g., synonym, hypernym, etc.), our model generates new sense vectors via constraining the distance between the sense vector and its word form vector, its sense neighbors and its contextual neighbors

[Influenced by](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1411.4166) (which post-processes and modifies word vectors to incorporate knowledge from semantic lexicons, while this creates new sense vectors)

2018-05-10 A propos

[1804.01486] Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data

Tags:

2018-04-14 A propos

GitHub - ijkilchenko/Fuzbal: Chrome extension: Gives Ctrl+F like find results which include non-exact (fuzzy) matches using string edit-distance and GloVe/Word2Vec. Also searches by regular expressions.

Tags:

2018-03-29 A propos

[1803.05651] Word2Bits - Quantized Word Vectors

Tags:

2018-03-20 A propos

GitHub - anvaka/word2vec-graph: Exploring word2vec embeddings as a graph of nearest neighbors

Tags:

2018-03-12 A propos

Deep learning with word embeddings improves biomedical named entity recognition | Bioinformatics | Oxford Academic (2017)

Tags:

2018-03-05 A propos

[1411.4166] Retrofitting Word Vectors to Semantic Lexicons

Tags:

2018-02-25 A propos

ELMo: Deep contextualized word representations (2018)

Tags:

ELMo

2018-02-16 A propos

A gentle introduction to Doc2Vec – ScaleAbout – Medium

Tags:

Doc2Vec

2018-02-14 A propos

Explanation for Doc2Vec - Quora

Tags:

Doc2Vec

2018-02-14 A propos

[1710.04099] Wembedder: Wikidata entity embedding web service

Tags:

2018-02-13 A propos

Improving the Compositionality of Word Embeddings (2017)

Tags:

(MS thesis, a [paper at TheWebConf 2018](/doc/?uri=https%3A%2F%2Fdoi.org%2F10.1145%2F3178876.3186007))

> This thesis explores a method to find better encodings of meaning a computer can work with. We specifically want to combine encodings of word meanings in such a way that a good encoding of their joint meaning is created. The act of combining multiple representations of meaning into a new representation of meaning is called semantic composition.

Analysis of four word embeddings (Word2Vec, GloVe, fastText and Paragram)  in terms of their semantic compositionality. A method to tune these embeddings towards better compositionality, using a simple neural network architecture with definitions and lemmas from WordNet.

> Since dictionary definitions are semantically similar to their associated lemmas, they are the ideal candidate for our tuning method, as well as evaluating for compositionality. Our architecture allows for the embeddings to be composed using simple arithmetic operations, which makes these embeddings specifically suitable for production applications such as web search and data mining. We also explore more elaborate and involved compositional models, such as recurrent composition and convolutional composition.

2018-02-13 A propos

Modèle Neuronal de Recherche d'Information Augmenté par une Ressource Sémantique (2017)

Tags:

2018-02-12 A propos

[1412.6623] Word Representations via Gaussian Embedding

Tags:

2018-01-28 A propos

Evaluating the Impact of Word Embeddings on Similarity Scoring in Practical Information Retrieval (2017)

Tags:

> Transferring the success of word embeddings to Information Retrieval (IR) task is currently an active research topic. While embedding-based retrieval models could tackle the vocabulary mismatch problem by making use of the embedding’s inherent similarity between distinct words, most of them struggle to compete with the prevalent strong baselines such as TF-IDF and BM25.

Considering a practical ad-hoc IR task composed of two steps, matching and scoring, compares the performance of several techniques that leverage word embeddings in the retrieval models to compute the similarity between the query and the documents (namely word centroid similarity, paragraph vectors, Word Mover’s distance, as well as a novel inverse document frequency (IDF) re-weighted word centroid similarity).

> We confirm that word embeddings can be successfully employed in a practical information retrieval setting. The proposed cosine similarity of IDF re-weighted, aggregated word vectors is competitive to the TF-IDF baseline.

2018-01-28 A propos

RESEARCH TRACK: Web Content Analysis, Semantics and Knowledge

Tags:

[CFP](https://www2018.thewebconf.org/call-for-papers/research-tracks-cfp/web-content-analysis/)

> In previous years, ‘content analysis’ and ‘semantic and knowledge’ were in separate track. This year, we combined these tracks to emphasize the close relationship between these topics; **the use of content to curate knowledge and the use of knowledge to guide content analysis and intelligent usage**.

Some of the accepted papers:
### [Large-Scale Hierarchical Text Classification with Recursively Regularized Deep Graph-CNN](https://doi.org/10.1145/3178876.3186005)

[Hierarchical Text Classification](/tag/nlp_hierarchical_text_classification): Text classification to a hierarchical taxonomy of topics, using graph representation of text, and CNN over this graph

Renvoie à ce qui a été vu dans le tutorial "Graph-based Text Representations"

from the abstract:

> a graph-CNN based deep learning model to first convert texts to graph-of-words, and then use graph convolution operations to convolve the word graph. Graph-of-words representation of texts has the advantage of capturing non-consecutive and long-distance semantics. CNN models have the advantage of learning different level of semantics. To further leverage the hierarchy of labels, we regularize the deep architecture with the dependency among labels

Conversion of text to graph: potentially given a single document

### [Weakly-supervised Relation Extraction by Pattern-enhanced Embedding Learning](https://doi.org/10.1145/3178876.3186024 )

Extraction de relations de corpus de textes de façon semi-supervisée, dans un contexte où on a peu de données labellisées décrivant les relations.

Par exemple, des données labellisées indique que le texte "Beijing, capital of China" correspond à la relation entre entités : ("Beijing", "Capital Of", "China), et on voudrait pouvoir extraire les entités et relations pertinentes à partir de texte tel que "Paris, France's capital,..."

Le papier décrit une méthode qui combine deux modules, l'un basé sur l'extraction automatique de patterns (par ex "[Head], Capital Of [Tail]") et l'autre sur la "sémantique distributionnelle" (du type "word embeddings"). Ces deux modules collaborent, le premier permettant de créer des instances de relations augmentant la base de connaissance sur lequel entrainer le second, et le second aidant le premier à déterminer des patterns informatifs ("co-entrainement")

### [Scalable Instance Reconstruction in Knowledge Bases via Relatedness Affiliated Embedding](https://doi.org/10.1145/3178876.3186017)

Knowledge base completion problem: usually, it is formulated as a link prediction problem, but not here. A novel knowledge embedding model ("Joint Modelling and Learning of Relatedness and Embedding")

### [Improving Word Embedding Compositionality using Lexicographic Definitions](https://doi.org/10.1145/3178876.3186007)

comment obtenir les meilleures représentations de texte à partir de représentations de mots (word embeddings) ? L'auteur utilise des ressources lexicographiques (wordnet) pour ses tests : l'embedding obtenu pour la définition d'un mot est-il proche de celui du mot ?

Le papier s'appuie sur une [thèse du même auteur](https://esc.fnwi.uva.nl/thesis/centraal/files/f1554608041.pdf), claire et bien écrite.

### [CESI: Canonicalizing Open Knowledge Bases using Embeddings and Side Information](https://doi.org/10.1145/3178876.3186030)

Amélioration de l'extraction de triplets (nom phrase, property, nom phrase) à partir de texte en calculant des embeddings pour les "nom phrases" (~entités)

### [Short-Text Topic Modeling via Non-negative Matrix Factorization Enriched with Local Word-Context Correlations](https://doi.org/10.1145/3178876.3186009)

Topic modeling for short texts, leveraging the word-context semantic correlations in the training

### [Towards Annotating Relational Data on the Web with Language Models](https://doi.org/10.1145/3178876.3186029)

### A paper by [David Blei](/tag/david_blei): (Dynamic Embeddings for Language Evolution)

2018-01-27 A propos

Knowledge Graph and Text Jointly Embedding (2014)

Tags:

2018-01-05 A propos

Learning Concept-Driven Document Embeddings for Medical Information Search (2017)

Tags:

2018-01-03 A propos

Learning Deep Structured Semantic Models for Web Search using Clickthrough Data - Microsoft Research (2013)

Tags:

2017-12-30 A propos

DSSM ("Deep Semantic Similarity Model") - Microsoft Research

Tags:

DSSM (Deep Semantic Similarity Model)

2017-12-30 A propos

Combining word and entity embeddings for entity linking (ESWC 2017)

Tags:

2017-12-30 A propos

[1712.09405] Advances in Pre-Training Distributed Word Representations

Tags:

2017-12-29 A propos

gensim/WMD_tutorial.ipynb

Tags:

2017-12-23 A propos

facebookresearch/MUSE: A library for Multilingual Unsupervised or Supervised word Embeddings

Tags:

2017-12-22 A propos

LDA2vec: Word Embeddings in Topic Models – Towards Data Science

Tags:

LDA2vec

2017-12-11 A propos

A Latent Variable Model Approach to PMI-basedWord Embeddings

Tags:

2017-12-07 A propos

(fastText) Euclidean distance instead of cosine-similarity?

Tags:

2017-12-07 A propos

machine learning - Text categorization: combining different kind of features - Data Science Stack Exchange

Tags:

2017-12-06 A propos

How can we get the vector of a paragraph? · Issue #26 · facebookresearch/fastText

Tags:

2017-11-25 A propos

How to use pre-trained word vectors from Facebook’s fastText

Tags:

2017-11-25 A propos

DS Toolbox - Topic Models - DS lore

Tags:

2017-11-21 A propos

Semantic Word Embeddings – Off the convex path

Tags:

2017-11-21 A propos

Gaussian LDA for Topic Models with Word Embeddings (2015)

Tags:

2017-11-21 A propos

Calculates Word Mover's Distance Insanely Fast

Tags:

2017-11-12 A propos

Word Mover’s Distance in Python

Tags:

Word Mover’s Distance

2017-11-12 A propos

Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings (2015)

Tags:

2017-11-12 A propos

A Comparison of Vector-based Representations for Semantic Composition (Blacoe and Lapata - 2012)

Tags:

2017-11-12 A propos

Improving Distributional Similarity with Lessons Learned from Word Embeddings (O Levy - 2015)

Tags:

2017-11-11 A propos

Semantics with Dense Vectors

Tags:

2017-11-11 A propos

Dependency-Based Word Embeddings | Omer Levy

Tags:

2017-11-08 A propos

Word embeddings in 2017: Trends and future directions

Tags:

2017-11-06 A propos

Spell Checker using Word2vec | Kaggle

Tags:

2017-11-03 A propos

LSTM with word2vec embeddings | Kaggle

Tags:

2017-10-25 A propos

[1511.07972] Learning with Memory Embeddings

Tags:

2017-10-24 A propos

Using Gensim Word2Vec Embeddings in Keras | Ben Bolte's Blog

Tags:

2017-10-23 A propos

A Word2Vec Keras tutorial

Tags:

2017-10-23 A propos

Using pre-trained word embeddings in a Keras model

Tags:

2017-10-23 A propos

Towards a Seamless Integration of Word Senses into Downstream NLP Applications (2017)

Tags:

2017-10-21 A propos

[1710.04087] Word Translation Without Parallel Data

Tags:

2017-10-14 A propos

Enriching Word Embeddings Using Knowledge Graph for Semantic Tagging in Conversational Dialog Systems - Microsoft Research (2015)

Tags:

> new simple, yet effective approaches to
learn domain specific word embeddings.

## Intro

> Adapting word embeddings, such as jointly capturing
syntactic and semantic information, can further enrich semantic
word representations for several tasks, e.g., sentiment
analysis (Tang et al. 2014), named entity recognition
(Lebret, Legrand, and Collobert 2013), entity-relation extraction
(Weston et al. 2013), etc. (Yu and Dredze 2014)
has introduced a lightly supervised word embedding learning
extending word2vec. They incorporate prior information to the objective
function as a regularization term considering synonymy relations
between words from Wordnet (Fellbaum 1999).

> In this work, we go one step further and investigate if
enriching the word2vec word embeddings trained on unstructured/
unlabeled text with domain specific semantic relations
obtained from knowledge sources (e.g., knowledge
graphs, search query logs, etc.) can help to discover relation
aware word embeddings. Unlike earlier work, **we encode the
information about the relations between phrases, thereby,
entities and relation mentions are all embedded into a low dimensional
vector space**.

## Related work (Learning Word Embeddings with Priors)

- word2vec
- Relational Constrained Model (RTM) (Yu and Dredze 2014)
While CBOW learns lexical word embeddings from provided text, the RTM learns embeddings of words based on their similarity to other words provided by a knowledge resource (eg. wordnet)
- Joint model  (Yu and Dredze 2014)
combines CBOW and RTM through linear combination

2017-10-02 A propos

Named Entity Recognition using Word Embedding as a Feature (2016)

Tags: