Semanlink - Favoris ; AI teams AND Word embeddings

> We find that the softmax layer (the output layer) causes significant inefficiency due to the large vocabulary size.
Therefore, we redesign the learning objectiv.
> Specifically, the proposed approach bypasses the softmax layer by performing language modeling with dimension reduction, and allows the models to leverage pre-trained word embeddings.
Our framework reduces the time spent on the output layer to a negligible level, eliminates almost all the trainable parameters of the softmax layer and performs language modeling without truncating the vocabulary.
When applied to ELMo, our method achieves a 4 times speedup and eliminates 80% trainable parameters while achieving competitive performance on downstream tasks.

**decouples learning contexts and words**

> Instead of using
a softmax layer to predict the distribution of the
missing word, we utilize and extend the SEMFIT
layer (Kumar and Tsvetkov, 2018) to **predict the
embedding of the missing word**.

2019-03-02 About

Keywords2vec

Tags:

2019-02-09 About

Cheatsheet · fastText

Tags:

2019-01-29 About

Using FastText models (not vectors) for robust embeddings | Kaggle

Tags:

2019-01-29 About

The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar

Tags:

2018-12-03 About

[1811.05370] Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents

Tags:

2018-11-20 About

iliaschalkidis/ELMo-keras: Re-implementation of ELMo on Keras

Tags:

2018-11-14 About

Word Mover's Embedding: From Word2Vec to Document Embedding (2018)

Tags:

2018-11-10 About

AI Can Recognize Images, But Text Has Been Tricky—Until Now | WIRED

Tags:

2018-09-08 About

How sampling works in Word2vec? Can someone please make me understand NCE and negative sampling? - Cross Validated

Tags:

2018-07-07 About

[1806.06259] Evaluation of sentence embeddings in downstream and linguistic probing tasks

Tags:

2018-06-19 About

Under the hood: Multilingual embeddings | Engineering Blog | Facebook Code

Tags:

2018-05-11 About

GitHub - Babylonpartners/fastText_multilingual: Multilingual word vectors

Tags:

2018-05-11 About

[1803.05651] Word2Bits - Quantized Word Vectors

Tags:

2018-03-20 About

GitHub - anvaka/word2vec-graph: Exploring word2vec embeddings as a graph of nearest neighbors

Tags:

2018-03-12 About

ELMo: Deep contextualized word representations (2018)

Tags:

ELMo

2018-02-16 About

A gentle introduction to Doc2Vec – ScaleAbout – Medium

Tags:

Doc2Vec

2018-02-14 About

Explanation for Doc2Vec - Quora

Tags:

Doc2Vec

2018-02-14 About

[1710.04099] Wembedder: Wikidata entity embedding web service

Tags:

2018-02-13 About

Knowledge Graph and Text Jointly Embedding (2014)

Tags:

2018-01-05 About

[1712.09405] Advances in Pre-Training Distributed Word Representations

Tags:

2017-12-29 About

gensim/WMD_tutorial.ipynb

Tags:

2017-12-23 About

facebookresearch/MUSE: A library for Multilingual Unsupervised or Supervised word Embeddings

Tags:

2017-12-22 About

(fastText) Euclidean distance instead of cosine-similarity?

Tags:

2017-12-07 About

How can we get the vector of a paragraph? · Issue #26 · facebookresearch/fastText

Tags:

2017-11-25 About

How to use pre-trained word vectors from Facebook’s fastText

Tags:

2017-11-25 About

Spell Checker using Word2vec | Kaggle

Tags:

2017-11-03 About

LSTM with word2vec embeddings | Kaggle

Tags:

2017-10-25 About

Using Gensim Word2Vec Embeddings in Keras | Ben Bolte's Blog

Tags:

2017-10-23 About

A Word2Vec Keras tutorial

Tags:

2017-10-23 About

[1710.04087] Word Translation Without Parallel Data

Tags:

2017-10-14 About

Enriching Word Embeddings Using Knowledge Graph for Semantic Tagging in Conversational Dialog Systems - Microsoft Research (2015)

Tags:

> new simple, yet effective approaches to
learn domain specific word embeddings.

## Intro

> Adapting word embeddings, such as jointly capturing
syntactic and semantic information, can further enrich semantic
word representations for several tasks, e.g., sentiment
analysis (Tang et al. 2014), named entity recognition
(Lebret, Legrand, and Collobert 2013), entity-relation extraction
(Weston et al. 2013), etc. (Yu and Dredze 2014)
has introduced a lightly supervised word embedding learning
extending word2vec. They incorporate prior information to the objective
function as a regularization term considering synonymy relations
between words from Wordnet (Fellbaum 1999).

> In this work, we go one step further and investigate if
enriching the word2vec word embeddings trained on unstructured/
unlabeled text with domain specific semantic relations
obtained from knowledge sources (e.g., knowledge
graphs, search query logs, etc.) can help to discover relation
aware word embeddings. Unlike earlier work, **we encode the
information about the relations between phrases, thereby,
entities and relation mentions are all embedded into a low dimensional
vector space**.

## Related work (Learning Word Embeddings with Priors)

- word2vec
- Relational Constrained Model (RTM) (Yu and Dredze 2014)
While CBOW learns lexical word embeddings from provided text, the RTM learns embeddings of words based on their similarity to other words provided by a knowledge resource (eg. wordnet)
- Joint model  (Yu and Dredze 2014)
combines CBOW and RTM through linear combination

2017-10-02 About

Distributed Word Representations for Information Retrieval

Tags: