Semanlink - NLP tasks / problems AND Information retrieval: techniques

> **approach for training dense retrieval models that does not require any labeled training data**. Dense retrieval is a central challenge for open-domain tasks, such as Open QA, where state-of-the-art methods typically require large supervised datasets with custom hard-negative mining and denoising of positive examples.
>
> ART, in contrast, only requires access to unpaired inputs and outputs (e.g. questions and potential answer documents).
>
> It uses a new document-retrieval autoencoding scheme, where
> 1. an input question is used to retrieve a set of evidence documents, and
> 2. the documents are then used to compute the probability of reconstructing the original question.
>
> Training for retrieval based on question reconstruction enables effective unsupervised learning of both document and question encoders, which can be later incorporated into complete Open QA systems without any further finetuning.

[Tweet](doc:2022/07/devendra_singh_sachan_sur_twitt)

> Given an
input question, ART first retrieves a small set
of possible evidences documents. It then recon
structs
the original question by attending to these
documents
>
> The
key idea in ART is to consider the retrieved documents
as a noisy representation of the original
question and question reconstruction probability
as a way of denoising that provides soft-labels for
how likely each document is to have been the correct
result

Refers to [[IZACARD 2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering](doc:2020/12/2012_04584_distilling_knowled)

2022-07-06 About

Devendra Singh Sachan sur Twitter : "ART (Autoencoding-based Retriever Training), an unsupervised method to train a dense retriever that only uses questions and a collection of unpaired documents as the training data."

Tags:

2022-07-06 About

[2108.13934] Robust Retrieval Augmented Generation for Zero-shot Slot Filling

Tags:

2022-01-19 About

[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Tags:

2022-01-19 About

Zexuan Zhong sur Twitter : ...Does this really mean dense models are better? No Our #EMNLP2021 paper shows dense retrievers even fail to answer simple entity-centric questions

Tags:

2021-10-07 About

Building a sentence embedding index with fastText and BM25 | by David Mezzetti | Towards Data Science

Tags:

2021-09-30 About

Linking Entities with Knowledge Graphs | by Sigurd Berglann | Strise | Medium

Tags:

2021-02-05 About

[2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering

Tags:

> a method to train an information retrieval module for downstream tasks, **without using pairs of queries and documents as annotations**.

Uses two models (standard pipeline for open-domain QA):

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 About

Tags:

2020-07-10 About

[1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Tags:

2020-05-02 About

Sebastian Riedel sur Twitter : Happy to introduce BLINK, the @facebookai open-source entity linker!...

Tags:

2020-05-02 About

[1910.04126] Scalable Nearest Neighbor Search for Optimal Transport

Tags:

2020-02-20 About

Building a real-time embeddings similarity matching system | Solutions | Google Cloud

Tags:

2020-01-11 About

Tags:

2020-01-10 About

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 About

Sent2Vec: An unsupervised approach towards learning sentence embeddings | RARE Technologies

Tags:

2019-03-25 About

A Tri-Partite Neural Document Language Model for Semantic Information Retrieval (2018 - ESWC conference)

Tags:

2018-06-08 About

Tutorial on Poincaré Embeddings (Jupyter Notebook )

Tags:

2018-05-20 About

Implementing Poincaré Embeddings | RARE Technologies

Tags:

2018-05-20 About

Semantic hashing using tags and topic modeling (2013)

Tags:

2018-03-22 About

[1710.04099] Wembedder: Wikidata entity embedding web service

Tags:

2018-02-13 About

Evaluating the Impact of Word Embeddings on Similarity Scoring in Practical Information Retrieval (2017)

Tags:

> Transferring the success of word embeddings to Information Retrieval (IR) task is currently an active research topic. While embedding-based retrieval models could tackle the vocabulary mismatch problem by making use of the embedding’s inherent similarity between distinct words, most of them struggle to compete with the prevalent strong baselines such as TF-IDF and BM25.

Considering a practical ad-hoc IR task composed of two steps, matching and scoring, compares the performance of several techniques that leverage word embeddings in the retrieval models to compute the similarity between the query and the documents (namely word centroid similarity, paragraph vectors, Word Mover’s distance, as well as a novel inverse document frequency (IDF) re-weighted word centroid similarity).

> We confirm that word embeddings can be successfully employed in a practical information retrieval setting. The proposed cosine similarity of IDF re-weighted, aggregated word vectors is competitive to the TF-IDF baseline.

2018-01-28 About

gensim/WMD_tutorial.ipynb

Tags:

2017-12-23 About

(fastText) Euclidean distance instead of cosine-similarity?

Tags:

2017-12-07 About

How to use pre-trained word vectors from Facebook’s fastText

Tags:

2017-11-25 About

Using Gensim Word2Vec Embeddings in Keras | Ben Bolte's Blog

Tags:

2017-10-23 About

Efficient unsupervised keywords extraction using graphs

Tags:

2017-10-04 About

Concept Search on Wikipedia · Chris McCormick

Tags:

2017-09-10 About

Finding Similar Items

Tags:

**Jaccard similarity**: similarity of sets, based on the relative size of their intersection -> **finding textually similar documents in a large corpus, near duplicates**. [Collaborative Filtering](/tag/collaborative_filtering) as a Similar-Sets Problem (cf. online purchases, movie ratings)

**Shingling** turns the problem of textual similarity of documents into a pb of similarity of sets

k-shingle: substring of length k found within a document. k: 5 for emails. Hashing shingles. Shingles built from words (stop word + 2 following words)

Similarity-Preserving Summaries of Sets: shingles sets are large -> compress large sets into small representations (“signatures”) that preserve similarity: **[Minhashing](/tag/minhash)** - related to Jaccard similarity (good explanation in [wikipedia](https://en.wikipedia.org/wiki/MinHash))

It still may be impossible to find the pairs of docs with greatest similarity efficiently -> **[Locality-Sensitive Hashing](/tag/locality_sensitive_hashing)** for Documents

Distance measures

Theory of Locality-Sensitive Functions

LSH famiies for other distance measures

Applications of Locality-Sensitive Hashing:

- entity resolution
- matching fingerprints
- matching newpapers articles

Methods for High Degrees of Similarity: LSH-based methods most effective when the degree of similarity we
accept is relatively low. When we want to find sets that are almost identical, other methods can be faster.

2017-07-26 About

How to spot first stories on Twitter using Storm | Michael Vogiatzis

Tags:

2017-07-26 About

Effective measures for inter-document similarity

Tags:

2017-07-21 About

Gensim tutorial: Similarity Queries

Tags:

2017-07-19 About

Latent semantic indexing ("Introduction to Information Retrieval" Manning 2008)

Tags:

VSM : problem with synonymy and polysemy (eg. synonyms are accorded separate dimensions)

Could we use the co-occurrences of terms to capture the latent semantic associations of terms and alleviate these problems?

Concluding remarks:

- computational cost of the SVD is significant
    - biggest obstacle to the widespread adoption to LSI.
    - One approach to this obstacle: build the LSI representation on a randomly sampled subset of the documents, following which the remaining documents are ``folded in'' (cf Gensim tutorial "[Random Projection (used as an option to speed up LSI)](https://radimrehurek.com/gensim/models/rpmodel.html)")
- As we reduce k, recall tends to increase, as expected.
- **Most surprisingly**, a value of k in the low hundreds can actually increase precision. **This appears to suggest that for a suitable value of *k*, LSI addresses some of the challenges of synonymy**.
- LSI works best in applications where there is little overlap between queries and documents. (--??)

The experiments also documented some modes where LSI failed to match the effectiveness of more traditional indexes and score computations.

LSI shares two basic drawbacks of vector space retrieval:
    
- no good way of expressing negations
- no way of enforcing Boolean conditions.

LSI can be viewed as soft clustering by interpreting each dimension of the reduced space as a cluster and the value that a document has on that dimension as its fractional membership in that cluster.

2017-07-19 About

What is a simple but detailed explanation of Textrank? - Quora

Tags: