Semanlink - Favoris AND Information retrieval: techniques

> **approach for training dense retrieval models that does not require any labeled training data**. Dense retrieval is a central challenge for open-domain tasks, such as Open QA, where state-of-the-art methods typically require large supervised datasets with custom hard-negative mining and denoising of positive examples.
>
> ART, in contrast, only requires access to unpaired inputs and outputs (e.g. questions and potential answer documents).
>
> It uses a new document-retrieval autoencoding scheme, where
> 1. an input question is used to retrieve a set of evidence documents, and
> 2. the documents are then used to compute the probability of reconstructing the original question.
>
> Training for retrieval based on question reconstruction enables effective unsupervised learning of both document and question encoders, which can be later incorporated into complete Open QA systems without any further finetuning.

[Tweet](doc:2022/07/devendra_singh_sachan_sur_twitt)

> Given an
input question, ART first retrieves a small set
of possible evidences documents. It then recon
structs
the original question by attending to these
documents
>
> The
key idea in ART is to consider the retrieved documents
as a noisy representation of the original
question and question reconstruction probability
as a way of denoising that provides soft-labels for
how likely each document is to have been the correct
result

Refers to [[IZACARD 2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering](doc:2020/12/2012_04584_distilling_knowled)

2022-07-06 A propos

Devendra Singh Sachan sur Twitter : "ART (Autoencoding-based Retriever Training), an unsupervised method to train a dense retriever that only uses questions and a collection of unpaired documents as the training data."

Tags:

2022-07-06 A propos

[2203.08913] Memorizing Transformers

Tags:

2022-05-07 A propos

Devendra Singh Sachan sur Twitter : "...Unsupervised Passage Re-ranker (UPR), an approach to re-rank retrieved passages for information retrieval tasks."

Tags:

2022-04-18 A propos

Jeremy Howard sur Twitter : "NLP competition at Kaggle about patent concept similarity...."

Tags:

2022-04-15 A propos

How to Build a Semantic Search Engine With Transformers and Faiss | by Kostas Stathoulopoulos | Towards Data Science

Tags:

2022-01-29 A propos

[2108.13934] Robust Retrieval Augmented Generation for Zero-shot Slot Filling

Tags:

2022-01-19 A propos

[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Tags:

2022-01-19 A propos

semi-technologies/weaviate: Weaviate is a cloud-native, modular, real-time vector search engine

Tags:

2021-12-05 A propos

Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine

Tags:

2021-12-05 A propos

Zexuan Zhong sur Twitter : ...Does this really mean dense models are better? No Our #EMNLP2021 paper shows dense retrievers even fail to answer simple entity-centric questions

Tags:

2021-10-07 A propos

Building a sentence embedding index with fastText and BM25 | by David Mezzetti | Towards Data Science

Tags:

2021-09-30 A propos

[2104.08663] BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models

Tags:

2021-07-09 A propos

[2007.12603] IR-BERT: Leveraging BERT for Semantic Search in Background Linking for News Articles

Tags:

2021-04-12 A propos

[1901.04085] Passage Re-ranking with BERT

Tags:

2021-03-26 A propos

Rodrigo Nogueira sur Twitter : "Slides of our WSDM 2021 tutorial "Pretrained Transformers for Text Ranking: BERT and Beyond"

Tags:

2021-03-09 A propos

Hugging Face sur Twitter : "Transformers release of the Retrieval-Augmented Generation model in collaboration with @facebookai!"

Tags:

2021-02-23 A propos

Retrieval Augmented Generation with Huggingface Transformers and Ray | Distributed Computing with Ray

Tags:

2021-02-11 A propos

Linking Entities with Knowledge Graphs | by Sigurd Berglann | Strise | Medium

Tags:

2021-02-05 A propos

[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training

Tags:

**Augment language model pre-training with a retriever module**, which
is trained using the masked language modeling objective.

> To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. **For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner**, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents

Hum, #TODO: parallel to be drawn with techniques in [KG-augmented Language Models](tag:knowledge_graph_augmented_language_models) which focus "on the problem of capturing declarative knowledge in the learned parameters of a language model."

[Google AI Blog Post](doc:2020/08/google_ai_blog_realm_integrat)

[Summary](https://joeddav.github.io/blog/2020/03/03/REALM.html) for the [Hugging Face awesome-papers reading group](doc:2021/03/huggingface_awesome_papers_pap)

2020-12-12 A propos

[2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering

Tags:

> a method to train an information retrieval module for downstream tasks, **without using pairs of queries and documents as annotations**.

Uses two models (standard pipeline for open-domain QA):

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 A propos

Searching with Shingles | Elastic Blog

Tags:

2020-09-03 A propos

SLEEC: Sparse Local Embeddings for Extreme Multi-label Classification (2015)

Tags:

2020-08-11 A propos

Tags:

2020-07-10 A propos

Ranked Entities in Search Results at Google

Tags:

2020-07-02 A propos

Tags:

2020-07-02 A propos

[1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Tags:

2020-05-02 A propos

Sebastian Riedel sur Twitter : Happy to introduce BLINK, the @facebookai open-source entity linker!...

Tags:

2020-05-02 A propos

[1910.04126] Scalable Nearest Neighbor Search for Optimal Transport

Tags:

2020-02-20 A propos

Information Retrieval for HR (2018)

Tags:

2020-02-14 A propos

Building a Search Engine with BERT and TensorFlow - Towards Data Science

Tags:

2020-01-12 A propos

Building a real-time embeddings similarity matching system | Solutions | Google Cloud

Tags:

2020-01-11 A propos

Tags:

2020-01-10 A propos

Thomas Wolf sur Twitter : "I liked the LSH attention in the reformer..."

Tags:

2020-01-05 A propos

[1911.00172] Generalization through Memorization: Nearest Neighbor Language Models

Tags:

2019-12-20 A propos

[1910.03524] Beyond Vector Spaces: Compact Data Representation as Differentiable Weighted Graphs

Tags:

2019-10-09 A propos

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 A propos

[1602.01137] A Dual Embedding Space Model for Document Ranking

Tags:

2019-07-17 A propos

A Short Introduction to Learning to Rank (2010)

Tags:

2019-04-18 A propos

Ranking Measures and Loss Functions in Learning to Rank (2009)

Tags:

2019-04-18 A propos

Sent2Vec: An unsupervised approach towards learning sentence embeddings | RARE Technologies

Tags:

2019-03-25 A propos

[1704.08803] Neural Ranking Models with Weak Supervision

Tags:

2019-01-27 A propos

[1703.03129] Learning to Remember Rare Events

Tags:

2018-10-23 A propos

[1810.07150] Subword Semantic Hashing for Intent Classification on Small Datasets

Tags:

2018-10-22 A propos

Paris NLP Meetup #6

Tags:

2018-07-26 A propos

SIGIR 2018 Tutorial - Knowledge Extraction and Inference from Text: Shallow, Deep, and Everything in Between

Tags:

2018-07-09 A propos

A Tri-Partite Neural Document Language Model for Semantic Information Retrieval (2018 - ESWC conference)

Tags:

2018-06-08 A propos

A Theoretical Approach to Semantic Coding and Hashing | Simons Institute for the Theory of Computing (2016)

Tags:

2018-05-26 A propos

Tutorial on Poincaré Embeddings (Jupyter Notebook )

Tags:

2018-05-20 A propos

Implementing Poincaré Embeddings | RARE Technologies

Tags:

2018-05-20 A propos

Semantic hashing using tags and topic modeling (2013)

Tags:

2018-03-22 A propos

ML-knn: A lazy learning approach to multi-label learning (2007)

Tags:

a lazy learning approach derived from the traditional [k-nearest neighbors algorithm](tag:k_nearest_neighbors_algorithm)

> for each unseen instance, its K nearest neighbors in the training set are firstly identified. After that, based on statistical information gained from the label sets of these neighboring instances, i.e. the number of neighboring instances belonging to each possible class, [Maximum a posteriori (MAP)](tag:maximum_a_posteriori_estimation) principle is utilized to determine the label set for the unseen instance.

Implemented in [scikit-multilearn](http://scikit.ml/api/skmultilearn.adapt.mlknn.html), in [java](https://github.com/lefman/mulan-extended/blob/master/mulan/src/mulan/classifier/lazy/MLkNN.java)

> the first lazy approach proposed specifically for multi-label classification. This is also a binary relevance approach which considers each label independently as a binary classification problem. Instead of a standard k-NN method, however, MLkNN uses the maximum a-posteriori (MAP) (Kelleher et al., 2015) approach combined with k-NN. [src](https://pdfs.semanticscholar.org/af9b/33da37d290c063cd826ab5923d96892a9767.pdf)

2018-03-18 A propos

[1710.04099] Wembedder: Wikidata entity embedding web service

Tags:

2018-02-13 A propos

Modèle Neuronal de Recherche d'Information Augmenté par une Ressource Sémantique (2017)

Tags:

2018-02-12 A propos

Evaluating the Impact of Word Embeddings on Similarity Scoring in Practical Information Retrieval (2017)

Tags:

> Transferring the success of word embeddings to Information Retrieval (IR) task is currently an active research topic. While embedding-based retrieval models could tackle the vocabulary mismatch problem by making use of the embedding’s inherent similarity between distinct words, most of them struggle to compete with the prevalent strong baselines such as TF-IDF and BM25.

Considering a practical ad-hoc IR task composed of two steps, matching and scoring, compares the performance of several techniques that leverage word embeddings in the retrieval models to compute the similarity between the query and the documents (namely word centroid similarity, paragraph vectors, Word Mover’s distance, as well as a novel inverse document frequency (IDF) re-weighted word centroid similarity).

> We confirm that word embeddings can be successfully employed in a practical information retrieval setting. The proposed cosine similarity of IDF re-weighted, aggregated word vectors is competitive to the TF-IDF baseline.

2018-01-28 A propos

Learning Deep Structured Semantic Models for Web Search using Clickthrough Data - Microsoft Research (2013)

Tags:

2017-12-30 A propos

DSSM ("Deep Semantic Similarity Model") - Microsoft Research

Tags:

DSSM (Deep Semantic Similarity Model)

2017-12-30 A propos

gensim/WMD_tutorial.ipynb

Tags:

2017-12-23 A propos

[1712.01208] The Case for Learned Index Structures

Tags:

2017-12-11 A propos

(fastText) Euclidean distance instead of cosine-similarity?

Tags:

2017-12-07 A propos

How to use pre-trained word vectors from Facebook’s fastText

Tags:

2017-11-25 A propos

[1706.00957] Semantic Vector Encoding and Similarity Search Using Fulltext Search Engines

Tags:

2017-11-11 A propos

Semantics with Dense Vectors

Tags:

2017-11-11 A propos

Semantic Hashing [9 mins] - Université de Toronto | Coursera

Tags:

2017-11-07 A propos

Semantic hashing (2008) - Ruslan Salakhutdinov, Geoffrey Hinton

Tags:

> We show how to learn a deep graphical model of the word-count vectors obtained from a
large set of documents. The values of the latent variables in the deepest layer are easy to
infer and give a much better representation of each document than Latent Semantic Analysis.
When the deepest layer is forced to use a small number of binary variables (e.g. 32),
the graphical model performs ‘‘semantic hashing”: Documents are mapped to memory
addresses in such a way that semantically similar documents are located at nearby
addresses. Documents similar to a query document can then be found by simply accessing
all the addresses that differ by only a few bits from the address of the query document. This
way of extending the efficiency of hash-coding to approximate matching is much faster
than locality sensitive hashing, which is the fastest current method. By using semantic
hashing to filter the documents given to TF-IDF, we achieve higher accuracy than applying
TF-IDF to the entire document set.

Indexing is implemented in the following manner: a document is mapped to a word-count vector and then this vector is passed through a [#Restricted Boltzmann Machine](/tag/restricted_boltzmann_machine) autoencoder and encoded to 32-bit address.

2017-11-07 A propos

[1004.5370] Self-Taught Hashing for Fast Similarity Search

Tags:

2017-11-07 A propos

Using Gensim Word2Vec Embeddings in Keras | Ben Bolte's Blog

Tags:

2017-10-23 A propos

Efficient unsupervised keywords extraction using graphs

Tags:

2017-10-04 A propos

Distributed Word Representations for Information Retrieval

Tags:

2017-10-01 A propos

Vectorland: Brief Notes from Using Text Embeddings for Search

Tags:

2017-09-18 A propos

Using Text Embeddings for Information Retrieval

Tags:

2017-09-18 A propos

Concept Search on Wikipedia · Chris McCormick

Tags:

2017-09-10 A propos

Finding Similar Items

Tags:

**Jaccard similarity**: similarity of sets, based on the relative size of their intersection -> **finding textually similar documents in a large corpus, near duplicates**. [Collaborative Filtering](/tag/collaborative_filtering) as a Similar-Sets Problem (cf. online purchases, movie ratings)

**Shingling** turns the problem of textual similarity of documents into a pb of similarity of sets

k-shingle: substring of length k found within a document. k: 5 for emails. Hashing shingles. Shingles built from words (stop word + 2 following words)

Similarity-Preserving Summaries of Sets: shingles sets are large -> compress large sets into small representations (“signatures”) that preserve similarity: **[Minhashing](/tag/minhash)** - related to Jaccard similarity (good explanation in [wikipedia](https://en.wikipedia.org/wiki/MinHash))

It still may be impossible to find the pairs of docs with greatest similarity efficiently -> **[Locality-Sensitive Hashing](/tag/locality_sensitive_hashing)** for Documents

Distance measures

Theory of Locality-Sensitive Functions

LSH famiies for other distance measures

Applications of Locality-Sensitive Hashing:

- entity resolution
- matching fingerprints
- matching newpapers articles

Methods for High Degrees of Similarity: LSH-based methods most effective when the degree of similarity we
accept is relatively low. When we want to find sets that are almost identical, other methods can be faster.

2017-07-26 A propos

How to spot first stories on Twitter using Storm | Michael Vogiatzis

Tags:

2017-07-26 A propos

Effective measures for inter-document similarity

Tags:

2017-07-21 A propos

Gensim tutorial: Similarity Queries

Tags:

2017-07-19 A propos

Latent semantic indexing ("Introduction to Information Retrieval" Manning 2008)

Tags:

VSM : problem with synonymy and polysemy (eg. synonyms are accorded separate dimensions)

Could we use the co-occurrences of terms to capture the latent semantic associations of terms and alleviate these problems?

Concluding remarks:

- computational cost of the SVD is significant
    - biggest obstacle to the widespread adoption to LSI.
    - One approach to this obstacle: build the LSI representation on a randomly sampled subset of the documents, following which the remaining documents are ``folded in'' (cf Gensim tutorial "[Random Projection (used as an option to speed up LSI)](https://radimrehurek.com/gensim/models/rpmodel.html)")
- As we reduce k, recall tends to increase, as expected.
- **Most surprisingly**, a value of k in the low hundreds can actually increase precision. **This appears to suggest that for a suitable value of *k*, LSI addresses some of the challenges of synonymy**.
- LSI works best in applications where there is little overlap between queries and documents. (--??)

The experiments also documented some modes where LSI failed to match the effectiveness of more traditional indexes and score computations.

LSI shares two basic drawbacks of vector space retrieval:
    
- no good way of expressing negations
- no way of enforcing Boolean conditions.

LSI can be viewed as soft clustering by interpreting each dimension of the reduced space as a cluster and the value that a document has on that dimension as its fractional membership in that cluster.

2017-07-19 A propos

Indexing by Latent Semantic Analysis - Deerwester et al. (1990)

Tags:

2017-07-18 A propos

What is a simple but detailed explanation of Textrank? - Quora

Tags: