Semanlink - Embeddings in NLP

Parents:

Embeddings

Embeddings in NLP

Related Tags:

Meaning in NLP

Descendants

Document embeddings
- Doc2Vec
- DSSM (Deep Semantic Similarity Model)
- Representation of text + metadata
Multilingual embeddings
- Multilingual Language Models
Phrase embeddings
Sense embeddings
- Sense2vec
- Word embeddings with lexical resources
Sentence Embeddings
- A La Carte Embedding
- Doc2Vec
- DSSM (Deep Semantic Similarity Model)
- E5
- Sent2Vec
- Sentence embeddings: Domain adaptation
  - Sentence-BERT: Domain adaptation
    - GPL (Generative Pseudo Labeling)
    - sBERT fine-tuning
      - SetFit (SBERT fine-tuning)
        
        UKP Lab sur X : "a lightweight solution for few-shot domain-specific sentence classification: AdaSent!..."
        
        Tags:
        
        Adapter modules (LM finetuning)
        EMNLP 2023
        Few-shot text classification
        Sentence-BERT: Domain adaptation
        Sentence embeddings: Domain adaptation
        SetFit (SBERT fine-tuning)
        Tweet
        
        AdaSent is an approach to creating domain-specialized sentence encoders for few-shot sentence classification > Reusable general sentence adapter across domains > AdaSent decouples DAPT (Domain Adaptative Pre-Training) & SEPT (Sentence Embedding Pre-Training) **by storing the sentence encoding abilities into an adapter**, which is trained only once in the general domain and plugged into various DAPT-ed PLMs [Github](https://github.com/UKPLab/AdaSent)
        
        2023-12-09 About
        
        Daniel Vila Suero sur Twitter : "Data quality is key for LLMs, but we're building Open Source LLMs with data of "unknown" quality... Introducing Alpaca GarbageCollector..."
        
        Tags:
        
        Alpaca
        Dataset quality
        SetFit (SBERT fine-tuning)
        Tweet
        
        > a cross-lingual SetFit model to identify potential bad instructions in Alpaca-like datasets
        
        2023-04-05 About
        
        ValueError "invalid literal for int() with base 10" in trainer.evaluate (dataset created from pandas) · Issue #228 · huggingface/setfit
        
        Tags:
        
        fps' post
        SetFit (SBERT fine-tuning)
        
        see <https://github.com/huggingface/setfit/blob/main/notebooks/zero-shot-classification.ipynb> > Note: some datasets on the Hugging Face Hub don't have a ClassLabel feature for the label column. In these cases, you should compute the candidate labels manually by first computing the id2label mapping as follows:
        
        2022-12-13 About
        
        Lewis Tunstall sur Twitter : "The SetFit library for few-shot learning with Sentence Transformers now supports *multi-label text classification*..."
        
        Tags:
        
        Multi-label Text classification
        SetFit (SBERT fine-tuning)
        Tweet
        
        Multilabel support [github issue](https://github.com/huggingface/setfit/issues/65)
        
        2022-10-14 About
        
        huggingface/setfit: Efficient few-shot learning with Sentence Transformers
        
        Tags:
        
        GitHub project
        SetFit (SBERT fine-tuning)
        
        2022-10-12 About
        
        [2209.11055] Efficient Few-Shot Learning Without Prompts
        
        Tags:
        
        Arxiv Doc
        Few-shot learning
        Moshe Wasserblat
        Nils Reimers
        SetFit (SBERT fine-tuning)
        
        [tweet](https://twitter.com/_akhaliq/status/1573109469646561280?s=20&t=RTpK9dh90az0zT1Xg2ohpQ): > So if I have 4 classes and say 2 labels per class, I would first fine tune an ST on these 4 pairs and then vectorize the 8 total examples for fine-tuning the classifier
        
        2022-09-23 About
        
        Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)
        
        Tags:
        
        Few-shot text classification
        Moshe Wasserblat
        sBERT fine-tuning
        SetFit (SBERT fine-tuning)
        
        Finetuning d'un SBERT sur une tâche de classification (in fine, produit un SBERT) > **Few-shot text classification based on fine-tuning a Sentence Transformer with task-specific data** that can easily be implemented with the sentence-transformers library > Surprisingly, we did not find any work that performed an end-to-end ST fine-tuning for text classification in a Siamese manner. [COLAB](https://colab.research.google.com/github/MosheWasserb/SetFit/blob/main/SetFit_SST_2.ipynb) [Nils Reimers sur Twitter](doc:2022/03/nils_reimers_sur_twitter_gre)
        
        2022-03-31 About
        
        Nils Reimers sur Twitter : "Great post on SetFit"
        
        Tags:
        
        Nils Reimers
        SetFit (SBERT fine-tuning)
        Tweet
        
        About [Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat](doc:2022/03/sentence_transformer_fine_tunin) > - Outperforms GPT-3 in few-shot text-classification (50 labeled examples, secret test set) > - 1600 times smaller > - Can be run on your CPU > - No limitation on the number of training examples > - Just few lines of code needed
        
        2022-03-31 About
- Sentence Transformers
  - BERT + Sentence Embeddings
    - Sentence-BERT
      - sBERT fine-tuning
      - Sentence-BERT: Domain adaptation
      - SentenceBERT and Words
        
        Few-Shot Text Classification (Cloudera 2020)
        
        Tags:
        
        Few-shot text classification
        SentenceBERT and Words
        
        > Sentence-BERT has been optimized… well, for sentences! It’s reasonable to suspect that SBERT’s representations of single words or short phrases like “Business” or “Science & Technology” won’t be as semantically relevant as representations derived from a word-level method, like word2vec or GloVe
        
        2022-11-24 About
- SIF embeddings
- Universal Sentence Encoder
- Unsupervised Sentence Embedding Learning
  - GPL (Generative Pseudo Labeling)
  - TSDAE
Text Embeddings
- E5
- Geometry of language embeddings
- MTEB
- OpenAI Embeddings
- Representation of text + metadata
- Sentence Embeddings
  Cohere int8 & binary Embeddings - Scale Your Vector Database to Large Datasets
  Tags:
  [tweet](https://x.com/Nils_Reimers/status/1769809006762037368?s=20) > Instead of reducing the number of dimensions, a better method is to train the model specifically to use fewer bytes per dimension. By using 1 byte per dimension, we reduce the memory 4x (954 GB → 238 GB) while keeping 99.99% of the original search quality. We can go even further...
  
  2024-03-19 About
  Jo Kristian Bergum sur X : "...helping people understand the shortcomings of text embedding models for their data. The most powerful has been demonstrating how the embedding models' tokenizers work."
  Tags:
  2024-03-16 About
  Nils Reimers sur X : "Embeddings can store only 1 aspect/topic per embedding well."
  Tags:
  > On Wikipedia, one paragraph typically focuses on one topic. So this gives you a good chunking for Wikipeda
  
  2024-03-13 About
  FlagOpen/FlagEmbedding: Dense Retrieval and Retrieval-augmented LLMs
  Tags:
  - Sentence Embeddings
  - Text Embeddings
  "BGE-M3 - The Mother of all embedding models", [Vespa](tag:vespa) says > I believe that M3 is a welcome direction in training embedding models. A single forward pass through the model obtains several representations. Highest possible flexibility to express the model in retrieval and ranking pipelines! [src](https://x.com/jobergum/status/1753359534150590508?s=20) ([@jobergum](tag:jo_kristian_bergum))
  
  2024-02-03 About
  Voyage AI (@Voyage_AI_) / X
  Tags:
  - Domain adaptation
  - Text Embeddings
  Building embedding/vectorization models, customized for your domain and company, for better retrieval quality [Blog post](https://blog.voyageai.com/2023/10/29/voyage-embeddings/)
  
  2023-10-31 About
  intfloat/e5-large-v2 · Hugging Face: Text Embeddings by Weakly-Supervised Contrastive Pre-training
  Tags:
  2023-10-13 About
  [2305.12517] Retrieving Texts based on Abstract Descriptions
  Tags:
  [@yoavgo sur Twitter](doc:2023/05/ل_ل_yoav_👾_sur_twit) > In this work, we aim to connect two research areas: instruction models and retrieval-based models. > **While instruction-tuned Large Language Models (LLMs) excel at extracting information from text, they are not suitable for semantic retrieval**. > Similarity search over embedding vectors allows to index and query vectors, but the **similarity reflected in the embedding is sub-optimal** retrieving sentences based on abstract descriptions of their content. Eg. >sentences related to “an important battle fought during World War II” or “a significant scientific discovery in the field of physics". fine-tune a sentence transformer. > To obtain the training data, we observe that the reverse direction of the process, going from a text to its description, is a task that can quite easily be performed ... by large language models such as GPT-3
  
  2023-06-15 About
  @yoavgo sur Twitter : "searching by description can be very useful, but current embedding models will give you texts that are *similar* to the description, rather than texts that *adhere to it*..."
  Tags:
  > GPT3... can easily do the opposite direction. Given a text, return an abstract description of it. So we collected many of these, and trained a dual-embedding model > a key takeaway for me is that **we can use the power of LLMs to perform tasks that LLMs cannot do on their own**. Like cool retrieval setups [Paper](doc:2023/06/2305_12517_retrieving_texts_b)
  
  2023-05-24 About
  hkunlp/instructor-xl · Hugging Face
  Tags:
  - Hugging Face
  - Text Embeddings
  > Instructor, an **instruction-finetuned text embedding model that can generate text embeddings tailored to any task** (e.g., classification, retrieval, clustering, text evaluation, etc.) and domains (e.g., science, finance, etc.) **by simply providing the task instruction, without any finetuning.** > > easy to use with our **customized sentence-transformer library**
  
  2023-02-17 About
  [2212.09741] One Embedder, Any Task: Instruction-Finetuned Text Embeddings
  Tags:
  > INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. > every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). [Documentation](https://instructor-embedding.github.io) ; [At Hugging Face](doc:2023/02/hkunlp_instructor_xl_·_hugging_) ex of use [here](https://postgresml.org/blog/generating-llm-embeddings-with-open-source-models-in-postgresml)
  
  2023-02-17 About
  [2210.07316] MTEB: Massive Text Embedding Benchmark
  Tags:
  > It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce MTEB [Leaderbord](https://huggingface.co/spaces/mteb/leaderboard)
  
  2022-10-17 About
  Nils Reimers sur Twitter : "MTEB - Massive Text Embedding Benchmark ..."
  Tags:
  2022-10-17 About
  Google Developers Blog: Text Embedding Models Contain Bias. Here's Why That Matters.
  Tags:
  - Bias
  - Text Embeddings
  2018-04-14 About
Topic embeddings
Word embeddings
- A La Carte Embedding
- Cross-lingual Word Embeddings
- ELMo
- FastText
- GloVe
- LDA2vec
- Subword embeddings
- Topic Models + Word embedding
- Using word embeddings
  - Word2vec: howto
  - Word Mover’s Distance
- Word + Entity embeddings
  - Wikipedia2Vec
- Word2vec
  - Doc2Vec
  - Word2vec: howto
    On word embeddings
    Tags:
    
    Sebastian Ruder
    Word2vec
    Word embeddings
    History of word embeddings in the context of language modelling. [Next post in serie](doc:2020/06/approximating_the_softmax_for_l)
    
    2020-06-05 About
    Contrastive Unsupervised Learning of Semantic Representations: A Theoretical Framework – Off the convex path (2019-03)
    Tags:
    
    Contrastive Self-Supervised Learning
    Embeddings
    Sanjeev Arora
    Word2vec
    [paper](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1902.09229). Why do objectives similar the one used by word2vec succeed in such diverse settings? ("Contrastive Unsupervised Representation Learning" (CURL): **methods that leverage similar pairs of data points**) > In contrastive learning the objective used at test time is very different from the training objective: generalization error is not the right way to think about this. -> a framework that formalizes the notion of semantic similarity that is implicitly used by these algorithms > **if the unsupervised loss happens to be small at the end of contrastive learning then the resulting representations perform well on downstream classification**
    
    2019-03-20 About
    Keywords2vec
    Tags:
    
    GitHub project
    Keyword/keyphrase extraction
    Medical IR, ML, IA
    Phrase mining
    RAKE
    Simple idea
    Word2vec
    To generate a word2vec model, but using keywords instead of one word. Tokenize on stopwords + non word characters (This remembers me author of [FlashText algorithm](tag:flashtext_algorithm.html) saying he had developed it to create word2vec models)
    
    2019-02-09 About
    How sampling works in Word2vec? Can someone please make me understand NCE and negative sampling? - Cross Validated
    Tags:
    
    Noise contrastive estimation
    Word2vec
    > In order to deal with the issue of the expensive computation of the softmax, Word2Vec uses a technique called noise-contrastive estimation... **The basic idea is to convert a multinomial classification problem (as it is the problem of predicting the next word) to a binary classification problem.**
    
    2018-07-07 About
    [1803.05651] Word2Bits - Quantized Word Vectors
    Tags:
    
    Arxiv Doc
    Word2vec
    Word embeddings
    We show that high quality quantized word vectors using 1-2 bits per parameter can be learned by introducing a quantization function into Word2Vec. We furthermore show that training with the quantization function acts as a regularizer
    
    2018-03-20 About
    GitHub - anvaka/word2vec-graph: Exploring word2vec embeddings as a graph of nearest neighbors
    Tags:
    
    GitHub project
    Graph visualization
    Word2vec
    2018-03-12 About
    [1710.04099] Wembedder: Wikidata entity embedding web service
    Tags:
    
    Arxiv Doc
    gensim
    Knowledge Graphs
    Word2vec
    web service for querying an embedding of entities in the Wikidata knowledge graph. The embedding is trained on the Wikidata dump using Gensim's Word2Vec implementation and a simple graph walk
    
    2018-02-13 About
    [1712.09405] Advances in Pre-Training Distributed Word Representations
    Tags:
    
    Arxiv Doc
    FastText
    NLP@Facebook
    Tomas Mikolov
    Word2vec
    Word embeddings
    > we show how to train high-quality word vector representations by using a combination of known tricks that are however rarely used together. The main result of our work is the new set of publicly available pre-trained models that outperform the current state of the art by a large margin on a number of tasks
    
    2017-12-29 About
    gensim/WMD_tutorial.ipynb
    Tags:
    
    gensim
    Word2vec
    Word Mover’s Distance
    Finding similar documents with Word2Vec and WMD (Word Mover’s Distance)
    
    2017-12-23 About
    Spell Checker using Word2vec | Kaggle
    Tags:
    
    Spellchecker
    Word2vec
    2017-11-03 About
    LSTM with word2vec embeddings | Kaggle
    Tags:
    
    Keras embedding layer
    LSTM
    NLP sample code
    Using word embeddings
    Word2vec
    2017-10-25 About
    Using Gensim Word2Vec Embeddings in Keras | Ben Bolte's Blog
    Tags:
    
    gensim
    Keras
    NLP sample code
    Word2vec
    2017-10-23 About
    A Word2Vec Keras tutorial
    Tags:
    
    Keras
    Tutorial
    Word2vec
    2017-10-23 About
    Distributed Word Representations for Information Retrieval
    Tags:
    
    Chris Manning
    Embeddings in Information Retrieval
    Word2vec
    Word embeddings
    includes description of word2vec
    
    2017-10-01 About
    Word2Vec Resources · Chris McCormick
    Tags:
    
    Links
    Word2vec
    2017-09-12 About
    Word2Vec Tutorial Part 2 - Negative Sampling · Chris McCormick
    Tags:
    
    Negative Sampling
    Tutorial
    Word2vec
    the tweaks to make training feasible
    
    2017-09-10 About
    Word2Vec Tutorial - The Skip-Gram Model · Chris McCormick
    Tags:
    
    Good
    Tutorial
    Word2vec
    skip-gram
    
    2017-09-10 About
    How does word2vec work? Can someone walk through a specific example? - Quora
    Tags:
    
    Word2vec
    2017-08-28 About
    Vector Representations of Words | TensorFlow
    Tags:
    
    TensorFlow
    Tutorial
    Word2vec
    Word embeddings
    2017-08-28 About
    An overview of word embeddings and their connection to distributional semantic models - AYLIEN (2016)
    Tags:
    
    Distributional semantics
    GloVe
    Good
    Sebastian Ruder
    Word2vec
    Word embedding: evaluation
    Word embeddings
    > While on the surface DSMs and word embedding models use varying algorithms to learn word representations – the former count, the latter predict – both types of model fundamentally act on the same underlying statistics of the data, i.e. the co-occurrence counts between words... > These results are in contrast to the general consensus that word embeddings are superior to traditional methods. Rather, they indicate that it typically makes no difference whatsoever whether word embeddings or distributional methods are used. What really matters is that your hyperparameters are tuned and that you utilize the appropriate pre-processing and post-processing steps.
    
    2017-07-20 About
    Some pre-trained word2vec models for French
    Tags:
    
    NLP: French
    Word2vec
    2017-07-20 About
    word2vec-api
    Tags:
    
    François Scharffe
    gensim
    GitHub project
    Word2vec
    Simple web service providing a word embedding API. The methods are based on Gensim Word2Vec implementation.<br/> List of word2vec datasets
    
    2017-06-09 About
    gensim: models.word2vec – Deep learning with word2vec
    Tags:
    
    gensim
    Word2vec
    2017-06-01 About
    Word2vec in gensim Tutorial | RaRe Technologies
    Tags:
    
    gensim
    Tutorial
    Word2vec
    2017-06-01 About
    alternatives to word2vec? - Quora
    Tags:
    
    Word2vec
    Word embeddings
    2017-05-23 About
    Improving Topic Models with Latent Feature Word Representations | Nguyen | Transactions of the Association for Computational Linguistics
    Tags:
    
    Topic Models + Word embedding
    Word2vec
    2017-05-20 About
    Using Word2Vec for topic modeling - Stack Overflow
    Tags:
    
    Stack Overflow
    Topic Modeling
    Word2vec
    2017-05-19 About
    Text Classification With Word2Vec - DS lore (2016)
    Tags:
    
    NLP sample code
    Python sample code
    Text Classification
    Word2vec
    > Overall, we won’t be throwing away our SVMs any time soon in favor of word2vec but it has it’s place in text classification. > > 1. SVM’s are pretty great at text classification tasks > 2. Models based on simple averaging of word-vectors can be surprisingly good too (given how much information is lost in taking the average) > 3. but they only seem to have a clear advantage when there is ridiculously little labeled training data > > Update 2017: actually, the best way to utilise the pretrained embeddings would probably be this [using keras](https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html) Sample code to benchmark a few text categorization models to test whehter word embeddings like word2vec can improve text classification accuracy. Sample code (based on scikit-learn) includes an embedding vectorizer that is given embedding dataset and vectorizes texts by taking the mean of all the vectors corresponding to individual words.
    
    2017-05-18 About
    Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors (2014)
    Tags:
    
    GloVe
    Word2vec
    Word embedding: evaluation
    Word embeddings
    (good presentation in the intro of context-counting vs. context-predicting vectors)
    
    2017-05-18 About
    How is GloVe different from word2vec? - Quora
    Tags:
    
    GloVe
    Word2vec
    Word embeddings
    Both learn geometrical encodings (vectors) of words from their co-occurrence information. Word2vec is a "predictive" model, whereas GloVe is a "count-based" model.
    
    2017-05-18 About
    Using Cognonto to Generate Domain Specific word2vec Models | Frederick Giasson
    Tags:
    
    Frederick Giasson
    Word2vec
    creating domain-specific training corpuses to use with word2vec can have a dramatic impact on the results and how results can be much more meaningful within the scope of that domain. Another advantage of the domain-specific training corpuses is that they create much smaller models.
    
    2016-09-29 About
    2Vec or Not 2Vec?
    Tags:
    
    Word2vec
    Word embeddings
    2016-03-05 About
    Word2vec: Neural Word Embeddings in Java - Deeplearning4j: Open-source, distributed deep learning for the JVM
    Tags:
    
    Java
    Word2vec
    2016-02-26 About
    [1301.3781] Efficient Estimation of Word Representations in Vector Space
    Tags:
    
    Arxiv Doc
    Tomas Mikolov
    Word2vec
    Word embeddings
    We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.
    
    2016-01-13 About
- Word embedding: evaluation
- Word Embedding Compositionality
  - SIF embeddings
    [1902.10618] Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition
    Tags:
    
    Arxiv Doc
    Contextualized word representations
    Word Embedding Compositionality
    How well do contextualized word embeddings address lexical composition? They are good in recognizing meaning shift ("give in" is different from "give") but much worse with revealing implicit meaning ("hot tea" is about temperature, "hot debate" isn't).
    
    2019-02-28 About
    A Word Embedding Approach to Predicting the Compositionality of Multiword Expressions (2015)
    Tags:
    
    Multiword Expressions
    Word Embedding Compositionality
    Word embeddings
    2018-06-08 About
    Deep Unordered Composition Rivals Syntactic Methods for Text Classification (2015)
    Tags:
    
    DANs (Deep Averaging Neural Networks)
    Neural Bag of Words
    Text Classification
    Word Embedding Compositionality
    > Many existing deep learning models for natural language processing tasks focus on learning the compositionality of their inputs, which requires many expensive computations. We present a simple deep neural network that competes with and, in some cases, outperforms such models on sentiment analysis and factoid question answering tasks while taking only a fraction of the training time. While our model is syntactically-ignorant, we show significant improvements over previous bag-of-words models by deepening our network and applying a novel variant of dropout
    
    2018-05-29 About
    Improving Word Embedding Compositionality using Lexicographic Definitions
    Tags:
    
    Embedding evaluation
    TheWebConf 2018
    Word Embedding Compositionality
    Word embeddings
    Word embeddings with lexical resources
    comment obtenir les meilleures représentations de texte à partir de représentations de mots (word embeddings) ? L'auteur utilise des ressources lexicographiques (wordnet) pour ses tests : l'embedding obtenu pour la définition d'un mot est-il proche de celui du mot ? Le papier s'appuie sur une [thèse du même auteur](/doc/?uri=https%3A%2F%2Fesc.fnwi.uva.nl%2Fthesis%2Fcentraal%2Ffiles%2Ff1554608041.pdf), claire et bien écrite.
    
    2018-05-10 About
    Improving the Compositionality of Word Embeddings (2017)
    Tags:
    
    Embedding evaluation
    TheWebConf 2018
    Word Embedding Compositionality
    Word embeddings
    (MS thesis, a [paper at TheWebConf 2018](/doc/?uri=https%3A%2F%2Fdoi.org%2F10.1145%2F3178876.3186007)) > This thesis explores a method to find better encodings of meaning a computer can work with. We specifically want to combine encodings of word meanings in such a way that a good encoding of their joint meaning is created. The act of combining multiple representations of meaning into a new representation of meaning is called semantic composition. Analysis of four word embeddings (Word2Vec, GloVe, fastText and Paragram) in terms of their semantic compositionality. A method to tune these embeddings towards better compositionality, using a simple neural network architecture with definitions and lemmas from WordNet. > Since dictionary definitions are semantically similar to their associated lemmas, they are the ideal candidate for our tuning method, as well as evaluating for compositionality. Our architecture allows for the embeddings to be composed using simple arithmetic operations, which makes these embeddings specifically suitable for production applications such as web search and data mining. We also explore more elaborate and involved compositional models, such as recurrent composition and convolutional composition.
    
    2018-02-13 About
- Word embeddings: introduction
- Word embeddings with lexical resources
  I.A.B sur Twitter : "When we "know the meaning" of a word, what is it that we know? For example, what does knowing the words "dolphin" and "tiger" entail?..."
  Tags:
  2022-04-15 About
  Pablo Castro sur Twitter : "Random finding of the day for word embeddings: vec("apple")-vec("apples") yields a vector close to ipad, ipod, etc. (apples removes the "fruitness" from apple)
  Tags:
  2020-12-18 About
  BERT Word Embeddings Tutorial · Chris McCormick
  Tags:
  2020-07-06 About
  [1910.00163] Specializing Word Embeddings (for Parsing) by Information Bottleneck
  Tags:
  EMNLP best paper award. [Related blog post](doc:2020/06/information_bottleneck_for_nlp_)
  
  2020-06-29 About
  On word embeddings
  Tags:
  History of word embeddings in the context of language modelling. [Next post in serie](doc:2020/06/approximating_the_softmax_for_l)
  
  2020-06-05 About
  Approximating the Softmax for Learning Word Embeddings
  Tags:
  - Softmax-based Approaches - Hierarchical Softmax - Differentiated Softmax - CNN-Softmax - Sampling-based Approaches - Importance Sampling - ... - [Noise contrastive estimation](tag:noise_contrastive_estimation) - [Negative Sampling](tag:negative_sampling) - ... [prev post in serie](doc:2020/06/on_word_embeddings)
  
  2020-06-04 About
  [1709.03933] Hash Embeddings for Efficient Word Representations
  Tags:
  > A hash embedding may be seen as an interpolation between a standard word embedding and a word embedding created using a random hash function (the hashing trick). recommandé par [Raphaël Sourty](tag:raphaelsty)
  
  2020-05-19 About
  [1810.04882] Towards Understanding Linear Word Analogies
  Tags:
  - Arxiv Doc
  - Word embeddings
  2019-06-24 About
  Kawin Ethayarajh sur Twitter : "When and why does king - man + woman = queen?"
  Tags:
  - Word embeddings
  [paper](doc/2019/06/_1810_04882_towards_understand) ; [blog post](/doc/2019/06/when_and_why_does_king_man_)
  
  2019-06-24 About
  Word Embeddings: 6 Years Later
  Tags:
  - Word embeddings
  2019-06-03 About
  Word Embeddings: Explaining their properties – Off the convex path (2016)
  Tags:
  - Sanjeev Arora
  - Word embeddings
  second part for [this post](/doc/?uri=http%3A%2F%2Fwww.offconvex.org%2F2015%2F12%2F12%2Fword-embeddings-1%2F) >- What properties of natural languages cause these low-dimensional embeddings to exist? >- Why do low-dimensional embeddings work better at analogy solving than high dimensional embeddings? >- Why do Semantic Relations correspond to Directions?
  
  2019-03-20 About
  Word Mover's Embedding: From Word2Vec to Document Embedding (2018)
  Tags:
  unsupervised embeddings of sentences of variable length from pre-trained word embeddings (better on short length text). (Builds on the word mover's distance, but using ideas borrowed from kernel methods approximation, gets a representation of sentences, instead of just a distance between them)
  
  2018-11-10 About
  Can Global Semantic Context Improve Neural Language Models? - Apple (2018)
  Tags:
  2018-09-27 About
  Linear algebraic structure of word meanings – Off the convex path
  Tags:
  Extracting word senses from embeddings. [About this paper](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1601.03764)
  
  2018-09-20 About
  Simple and efficient semantic embeddings for rare words, n-grams, and language features – Off the convex path
  Tags:
  A La Carte embeddings > Distributional methods for capturing meaning, such as word embeddings, often require observing many examples of words in context. But most humans can infer a reasonable meaning from very few or even a single occurrence... > > “Porgies live in shallow temperate marine waters” > > Inducing word embedding from their contexts: a surprising linear relationship
  
  2018-09-18 About
  A Latent Variable Model Approach to PMI-based Word Embeddings (2016)
  Tags:
  - Sanjeev Arora
  - Word embeddings
  [Related YouTube video](/doc/?uri=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKR46z_V0BVw) Based on a generative model (random walk on words involving a latent discourse vector), a rigorous justification for models such as word2vec and GloVe, including the hyperparameter choices for the latter, and a mathematical explanation for why these word embeddings allow analogies to be solved using linear algebra.
  
  2018-08-28 About
  [1601.03764] Linear Algebraic Structure of Word Senses, with Applications to Polysemy
  Tags:
  > Here it is shown that multiple word senses reside in linear superposition within the word embedding and simple sparse coding can recover vectors that approximately capture the senses > Each extracted word sense is accompanied by one of about 2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense. > The success of the approach is mathematically explained using a variant of the random walk on discourses model ("random walk": a generative model for language). Under the assumptions of this model, there exists a linear relationship between the vector of a word w and the vectors of the words in its contexts (It is not the average of the words in w's context, but in a given corpus the matrix of the linear relationship does not depend on w. It can be estimated, and so we can compute the embedding of a word from the contexts it belongs to) [Related blog post](/doc/?uri=https%3A%2F%2Fwww.offconvex.org%2F2016%2F07%2F10%2Fembeddingspolysemy%2F)
  
  2018-08-28 About
  zalandoresearch/flair: A very simple framework for state-of-the-art NLP
  Tags:
  > A very simple framework for state-of-the-art NLP. Developed by Zalando Research. paper: ["Contextual String Embeddings for Sequence Labeling (2018)"](/doc/?uri=http%3A%2F%2Faclweb.org%2Fanthology%2FC18-1139)
  
  2018-08-24 About
  Contextual String Embeddings for Sequence Labeling (2018)
  Tags:
  - Word embeddings
  > we propose to leverage the internal states of a trained character language model to produce a novel type of word embedding which we refer to as contextual string embeddings. Our proposed embeddings have the distinct properties that they (a) are trained without any explicit notion of words and thus fundamentally model words as sequences of characters, and (b) are contextualized by their surrounding text, meaning that the same word will have different embeddings depending on its contextual use. [Github](https://github.com/zalandoresearch/flair)
  
  2018-08-24 About
  what are the pros and cons of the various unsupervised word and sentence/ document embedding models? - Quora
  Tags:
  2018-08-19 About
  Sanjeev Arora on "A theoretical approach to semantic representations" - YouTube (2016)
  Tags:
  Why do low-dimensional word vectors exist? > a text corpus is imagined as being generated by a random walk in a latent variable space, and the word production is via a loglinear distribution. This model is shown to imply several empirically discovered past methods for word embedding like word2vec, GloVe, PMI etc [Related paper](/doc/?uri=http%3A%2F%2Fwww.aclweb.org%2Fanthology%2FQ16-1028)
  
  2018-06-10 About
  A Word Embedding Approach to Predicting the Compositionality of Multiword Expressions (2015)
  Tags:
  2018-06-08 About
  The Current Best of Universal Word Embeddings and Sentence Embeddings (2018)
  Tags:
  Word embeddings SOTA: [ELMo](/tag/elmo) Sentence embeddings: While unsupervised representation learning of sentences had been the norm for quite some time, with simple baselines like averaging word embeddings, a few novel unsupervised and supervised approaches, as well as multi-task learning schemes, have emerged in late 2017-early 2018.
  
  2018-05-30 About
  A Theoretical Approach to Semantic Coding and Hashing | Simons Institute for the Theory of Computing (2016)
  Tags:
  2018-05-26 About
  A simple spell checker built from word vectors – Ed Rushton – Medium
  Tags:
  - Spellchecker
  - Word embeddings
  2018-05-25 About
  NLP: Any libraries/dictionaries out there for fixing common spelling errors? - Part 2 & Alumni - Deep Learning Course Forums
  Tags:
  - Spellchecker
  - Word embeddings
  2018-05-18 About
  NLP using Word Vectors with Spacy - CLDSPN | Kaggle
  Tags:
  including an implementation of a "what to read next?" recommender system
  
  2018-05-11 About
  Dict2vec : Learning Word Embeddings using Lexical Dictionaries
  Tags:
  - Word embeddings
  - Word embeddings with lexical resources
  2018-05-11 About
  Improving Word Embedding Compositionality using Lexicographic Definitions
  Tags:
  comment obtenir les meilleures représentations de texte à partir de représentations de mots (word embeddings) ? L'auteur utilise des ressources lexicographiques (wordnet) pour ses tests : l'embedding obtenu pour la définition d'un mot est-il proche de celui du mot ? Le papier s'appuie sur une [thèse du même auteur](/doc/?uri=https%3A%2F%2Fesc.fnwi.uva.nl%2Fthesis%2Fcentraal%2Ffiles%2Ff1554608041.pdf), claire et bien écrite.
  
  2018-05-10 About
  [1803.05651] Word2Bits - Quantized Word Vectors
  Tags:
  We show that high quality quantized word vectors using 1-2 bits per parameter can be learned by introducing a quantization function into Word2Vec. We furthermore show that training with the quantization function acts as a regularizer
  
  2018-03-20 About
  Deep learning with word embeddings improves biomedical named entity recognition | Bioinformatics | Oxford Academic (2017)
  Tags:
  2018-03-05 About
  Improving the Compositionality of Word Embeddings (2017)
  Tags:
  (MS thesis, a [paper at TheWebConf 2018](/doc/?uri=https%3A%2F%2Fdoi.org%2F10.1145%2F3178876.3186007)) > This thesis explores a method to find better encodings of meaning a computer can work with. We specifically want to combine encodings of word meanings in such a way that a good encoding of their joint meaning is created. The act of combining multiple representations of meaning into a new representation of meaning is called semantic composition. Analysis of four word embeddings (Word2Vec, GloVe, fastText and Paragram) in terms of their semantic compositionality. A method to tune these embeddings towards better compositionality, using a simple neural network architecture with definitions and lemmas from WordNet. > Since dictionary definitions are semantically similar to their associated lemmas, they are the ideal candidate for our tuning method, as well as evaluating for compositionality. Our architecture allows for the embeddings to be composed using simple arithmetic operations, which makes these embeddings specifically suitable for production applications such as web search and data mining. We also explore more elaborate and involved compositional models, such as recurrent composition and convolutional composition.
  
  2018-02-13 About
  [1412.6623] Word Representations via Gaussian Embedding
  Tags:
  > Current work in lexical distributed representations maps each word to a point vector in low-dimensional space. Mapping instead to a density provides many interesting advantages > Novel word embedding algorithms that embed words directly as Gaussian distributional potential functions in an infinite dimensional function space. This allows us to map word types not only to vectors but to soft regions in space, modeling uncertainty, inclusion, and entailment, as well as providing a rich geometry of the latent space.
  
  2018-01-28 About
  RESEARCH TRACK: Web Content Analysis, Semantics and Knowledge
  Tags:
  [CFP](https://www2018.thewebconf.org/call-for-papers/research-tracks-cfp/web-content-analysis/) > In previous years, ‘content analysis’ and ‘semantic and knowledge’ were in separate track. This year, we combined these tracks to emphasize the close relationship between these topics; **the use of content to curate knowledge and the use of knowledge to guide content analysis and intelligent usage**. Some of the accepted papers: ### [Large-Scale Hierarchical Text Classification with Recursively Regularized Deep Graph-CNN](https://doi.org/10.1145/3178876.3186005) [Hierarchical Text Classification](/tag/nlp_hierarchical_text_classification): Text classification to a hierarchical taxonomy of topics, using graph representation of text, and CNN over this graph Renvoie à ce qui a été vu dans le tutorial "Graph-based Text Representations" from the abstract: > a graph-CNN based deep learning model to first convert texts to graph-of-words, and then use graph convolution operations to convolve the word graph. Graph-of-words representation of texts has the advantage of capturing non-consecutive and long-distance semantics. CNN models have the advantage of learning different level of semantics. To further leverage the hierarchy of labels, we regularize the deep architecture with the dependency among labels Conversion of text to graph: potentially given a single document ### [Weakly-supervised Relation Extraction by Pattern-enhanced Embedding Learning](https://doi.org/10.1145/3178876.3186024 ) Extraction de relations de corpus de textes de façon semi-supervisée, dans un contexte où on a peu de données labellisées décrivant les relations. Par exemple, des données labellisées indique que le texte "Beijing, capital of China" correspond à la relation entre entités : ("Beijing", "Capital Of", "China), et on voudrait pouvoir extraire les entités et relations pertinentes à partir de texte tel que "Paris, France's capital,..." Le papier décrit une méthode qui combine deux modules, l'un basé sur l'extraction automatique de patterns (par ex "[Head], Capital Of [Tail]") et l'autre sur la "sémantique distributionnelle" (du type "word embeddings"). Ces deux modules collaborent, le premier permettant de créer des instances de relations augmentant la base de connaissance sur lequel entrainer le second, et le second aidant le premier à déterminer des patterns informatifs ("co-entrainement") ### [Scalable Instance Reconstruction in Knowledge Bases via Relatedness Affiliated Embedding](https://doi.org/10.1145/3178876.3186017) Knowledge base completion problem: usually, it is formulated as a link prediction problem, but not here. A novel knowledge embedding model ("Joint Modelling and Learning of Relatedness and Embedding") ### [Improving Word Embedding Compositionality using Lexicographic Definitions](https://doi.org/10.1145/3178876.3186007) comment obtenir les meilleures représentations de texte à partir de représentations de mots (word embeddings) ? L'auteur utilise des ressources lexicographiques (wordnet) pour ses tests : l'embedding obtenu pour la définition d'un mot est-il proche de celui du mot ? Le papier s'appuie sur une [thèse du même auteur](https://esc.fnwi.uva.nl/thesis/centraal/files/f1554608041.pdf), claire et bien écrite. ### [CESI: Canonicalizing Open Knowledge Bases using Embeddings and Side Information](https://doi.org/10.1145/3178876.3186030) Amélioration de l'extraction de triplets (nom phrase, property, nom phrase) à partir de texte en calculant des embeddings pour les "nom phrases" (~entités) ### [Short-Text Topic Modeling via Non-negative Matrix Factorization Enriched with Local Word-Context Correlations](https://doi.org/10.1145/3178876.3186009) Topic modeling for short texts, leveraging the word-context semantic correlations in the training ### [Towards Annotating Relational Data on the Web with Language Models](https://doi.org/10.1145/3178876.3186029) ### A paper by [David Blei](/tag/david_blei): (Dynamic Embeddings for Language Evolution)
  
  2018-01-27 About
  [1712.09405] Advances in Pre-Training Distributed Word Representations
  Tags:
  > we show how to train high-quality word vector representations by using a combination of known tricks that are however rarely used together. The main result of our work is the new set of publicly available pre-trained models that outperform the current state of the art by a large margin on a number of tasks
  
  2017-12-29 About
  facebookresearch/MUSE: A library for Multilingual Unsupervised or Supervised word Embeddings
  Tags:
  2017-12-22 About
  A Latent Variable Model Approach to PMI-basedWord Embeddings
  Tags:
  (improves on [Yoav Goldberg](/tag/yoav_goldberg)'s findings)
  
  2017-12-07 About
  DS Toolbox - Topic Models - DS lore
  Tags:
  Usefulness of topic models and word embeddings for non-NLP tasks
  
  2017-11-21 About
  Semantic Word Embeddings – Off the convex path
  Tags:
  ([Part 2](/doc/?uri=http%3A%2F%2Fwww.offconvex.org%2F2016%2F02%2F14%2Fword-embeddings-2%2F))
  
  2017-11-21 About
  Gaussian LDA for Topic Models with Word Embeddings (2015)
  Tags:
  - Latent Dirichlet allocation
  - Word embeddings
  2017-11-21 About
  Improving Distributional Similarity with Lessons Learned from Word Embeddings (O Levy - 2015)
  Tags:
  > We reveal that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves. Furthermore, we show that these modifications can be transferred to traditional distributional models, yielding similar gains. In contrast to prior reports, we observe mostly local or insignificant performance differences between the methods, with no global advantage to any single approach over the others.
  
  2017-11-11 About
  Semantics with Dense Vectors
  Tags:
  > We will introduce three methods of generating very dense, short vectors: > 1. using dimensionality reduction methods like SVD, > 2. using neural nets like the popular skip-gram or CBOW approaches. > 3. a quite different approach based on neighboring words called Brown clustering.
  
  2017-11-11 About
  Dependency-Based Word Embeddings | Omer Levy
  Tags:
  - Word embeddings
  - Yoav Goldberg
  > While continuous word embeddings are gaining popularity, current models are based solely on linear contexts. In this work, we generalize the skip-gram model with negative sampling introduced by Mikolov et al. to include arbitrary contexts. > Experiments with dependency-based contexts show that they produce markedly different kinds of similarities. > In particular, the bag-of-words nature of the contexts in the “original” SKIPGRAM model yield broad topical similarities, while the dependency-based contexts yield more functional similarities of a cohyponym nature.
  
  2017-11-08 About
  Word embeddings in 2017: Trends and future directions
  Tags:
  - Sebastian Ruder
  - Word embeddings
  - Subword-level embeddings: several methods: > Word embeddings have been augmented with subword-level information for many applications such as named entity recognition, POS, ..., Language Modeling. > Most of these models employ a CNN or a BiLSTM that takes as input the characters of a word and outputs a character-based word representation. > For incorporating character information into pre-trained embeddings, however, **character n-grams features** have been shown to be more powerful. [#FastText] > Subword units based on **byte-pair encoding** have been found to be particularly useful for machine translation where they have replaced words as the standard input units - Out-of-vocabulary (OOV) words - Polysemy. Multi-sense embeddings - [Towards a Seamless Integration of Word Senses into Downstream NLP Applications](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1710.06632) - ...
  
  2017-11-06 About
  [1710.04087] Word Translation Without Parallel Data
  Tags:
  > we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way
  
  2017-10-14 About
  Named Entity Recognition using Word Embedding as a Feature (2016)
  Tags:
  Uses word embeddings as features for named entity recognition (NER) training, and CRF as learning algorithm
  
  2017-10-01 About
  Distributed Word Representations for Information Retrieval
  Tags:
  includes description of word2vec
  
  2017-10-01 About
  ConceptNet
  Tags:
  - Knowledge Graphs
  - Word embeddings
  An open, multilingual knowledge graph
  
  2017-09-18 About
  Learned in translation: contextualized word vectors (Salesforce Research)
  Tags:
  Models that use pretrained word vectors must learn how to use them. Our work picks up where word vectors left off by looking to improve over randomly initialized methods for contextualizing word vectors through training on an intermediate task -> We teach a neural network how to understand words in context by first teaching it how to translate English to German
  
  2017-09-18 About
  Vector Representations of Words | TensorFlow
  Tags:
  2017-08-28 About
  [1703.00993] A Comparative Study of Word Embeddings for Reading Comprehension
  Tags:
  abstract: The focus of past machine learning research for Reading Comprehension tasks has been primarily on the design of novel deep learning architectures. Here we show that seemingly minor choices made on 1. the use of pre-trained word embeddings, and 2. the representation of out-of-vocabulary tokens at test time, can turn out to have a larger impact than architectural choices on the final performance
  
  2017-08-28 About
  An overview of word embeddings and their connection to distributional semantic models - AYLIEN (2016)
  Tags:
  > While on the surface DSMs and word embedding models use varying algorithms to learn word representations – the former count, the latter predict – both types of model fundamentally act on the same underlying statistics of the data, i.e. the co-occurrence counts between words... > These results are in contrast to the general consensus that word embeddings are superior to traditional methods. Rather, they indicate that it typically makes no difference whatsoever whether word embeddings or distributional methods are used. What really matters is that your hyperparameters are tuned and that you utilize the appropriate pre-processing and post-processing steps.
  
  2017-07-20 About
  Embed, encode, attend, predict: The new deep learning formula for state-of-the-art NLP models | Blog | Explosion AI
  Tags:
  > A four-step strategy for deep learning with text > Word embeddings let you treat individual words as related units of meaning, rather than entirely distinct IDs. However, most NLP problems require understanding of longer spans of text, not just individual words. There's now a simple and flexible solution that is achieving excellent performance on a wide range of problems. After embedding the text into a sequence of vectors, bidirectional RNNs are used to encode the vectors into a sentence matrix. The rows of this matrix can be understood as token vectors — they are sensitive to the sentential context of the token. The final piece of the puzzle is called an attention mechanism. This lets you reduce the sentence matrix down to a sentence vector, ready for prediction.
  
  2017-07-20 About
  Distributed Semantics & Embeddings
  Tags:
  Yejin Choi - University of Washington [Slides adapted from Dan Jurafsky]
  
  2017-07-10 About
  Recurrent Neural Networks with Word Embeddings — DeepLearning 0.1 documentation
  Tags:
  2017-06-16 About
  A Simple Introduction to Word Embeddings
  Tags:
  topical vs typical relatedness
  
  2017-06-16 About
  An Intuitive Understanding of Word Embeddings: From Count Vectors to Word2Vec
  Tags:
  Types of word embeddings: - Frequency based Embedding - Count Vector - TF-IDF Vector - Co-Occurrence Vector - Co_occurence matrix (with a fixed context window), size V*V or V * N (Vocab size * subset of V size) matrix. - PCA or SVD: keeping the k most important eigenvalues - Prediction based Embedding - CBOW (Continuous Bag Of Words). 1 hidden layer, one output layer. Predict the probability of a word given a context - Skip-gram. Predict the proba of the context given a word Sample code using gensim
  
  2017-06-09 About
  [1607.00570] Representation learning for very short texts using weighted word embedding aggregation
  Tags:
  A method based on word embeddings and frequency information to arrive at low-dimensional representations for short texts designed to capture semantic similarity. <a href="https://github.com/cedricdeboom/RepresentationLearning">Github</a> (hmm...) (python code)
  
  2017-06-09 About
  [1512.00765] Learning Semantic Similarity for Very Short Texts
  Tags:
  In order to pair short text fragments—as a concatenation of separate words—an adequate distributed sentence representation is needed. Main contribution: a first step towards a hybrid method that combines the strength of dense distributed representations— as opposed to sparse term matching—with the strength of tf-idf based methods. The combination of word embeddings and tf-idf information might lead to a better model for semantic content within very short text fragments.
  
  2017-06-09 About
  When is using word embeddings harmful? - Quora
  Tags:
  - Word embeddings
  2017-06-05 About
  alternatives to word2vec? - Quora
  Tags:
  - Word2vec
  - Word embeddings
  2017-05-23 About
  Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors (2014)
  Tags:
  (good presentation in the intro of context-counting vs. context-predicting vectors)
  
  2017-05-18 About
  How is GloVe different from word2vec? - Quora
  Tags:
  Both learn geometrical encodings (vectors) of words from their co-occurrence information. Word2vec is a "predictive" model, whereas GloVe is a "count-based" model.
  
  2017-05-18 About
  Short Text Similarity with Word Embeddings
  Tags:
  We investigate whether determining short text similarity is possible using only semantic features. A novel feature of our approach is that an arbitrary number of word embedding sets can be incorporated.
  
  2017-05-18 About
  2Vec or Not 2Vec?
  Tags:
  - Word2vec
  - Word embeddings
  2016-03-05 About
  [1301.3781] Efficient Estimation of Word Representations in Vector Space
  Tags:
  We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.
  
  2016-01-13 About
  Deep Learning, NLP, and Representations - colah's blog
  Tags:
  2015-10-16 About

0 Document (Long List)

Properties

sl:creationDate : 2020-03-20
sl:creationTime : 2020-03-20T00:34:10Z
rdf:type : sl:Tag
skos:prefLabel : Embeddings in NLP