Semanlink - NLP girls and guys AND Information retrieval: techniques

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 About

SLEEC: Sparse Local Embeddings for Extreme Multi-label Classification (2015)

Tags:

2020-08-11 About

Thomas Wolf sur Twitter : "I liked the LSH attention in the reformer..."

Tags:

2020-01-05 About

[1911.00172] Generalization through Memorization: Nearest Neighbor Language Models

Tags:

2019-12-20 About

[1602.01137] A Dual Embedding Space Model for Document Ranking

Tags:

2019-07-17 About

[1704.08803] Neural Ranking Models with Weak Supervision

Tags:

2019-01-27 About

A Tri-Partite Neural Document Language Model for Semantic Information Retrieval (2018 - ESWC conference)

Tags:

2018-06-08 About

A Theoretical Approach to Semantic Coding and Hashing | Simons Institute for the Theory of Computing (2016)

Tags:

2018-05-26 About

Modèle Neuronal de Recherche d'Information Augmenté par une Ressource Sémantique (2017)

Tags:

2018-02-12 About

Semantics with Dense Vectors

Tags:

2017-11-11 About

Distributed Word Representations for Information Retrieval

Tags:

2017-10-01 About

Vectorland: Brief Notes from Using Text Embeddings for Search

Tags:

2017-09-18 About

Using Text Embeddings for Information Retrieval

Tags:

2017-09-18 About

Latent semantic indexing ("Introduction to Information Retrieval" Manning 2008)

Tags:

VSM : problem with synonymy and polysemy (eg. synonyms are accorded separate dimensions)

Could we use the co-occurrences of terms to capture the latent semantic associations of terms and alleviate these problems?

Concluding remarks:

- computational cost of the SVD is significant
    - biggest obstacle to the widespread adoption to LSI.
    - One approach to this obstacle: build the LSI representation on a randomly sampled subset of the documents, following which the remaining documents are ``folded in'' (cf Gensim tutorial "[Random Projection (used as an option to speed up LSI)](https://radimrehurek.com/gensim/models/rpmodel.html)")
- As we reduce k, recall tends to increase, as expected.
- **Most surprisingly**, a value of k in the low hundreds can actually increase precision. **This appears to suggest that for a suitable value of *k*, LSI addresses some of the challenges of synonymy**.
- LSI works best in applications where there is little overlap between queries and documents. (--??)

The experiments also documented some modes where LSI failed to match the effectiveness of more traditional indexes and score computations.

LSI shares two basic drawbacks of vector space retrieval:
    
- no good way of expressing negations
- no way of enforcing Boolean conditions.

LSI can be viewed as soft clustering by interpreting each dimension of the reduced space as a cluster and the value that a document has on that dimension as its fractional membership in that cluster.

2017-07-19 About

TextRank: Bringing Order into Texts (2004)

Tags:

2017-06-14 About

Introduction to Information Retrieval, Cambridge University Press (2008) Manning, Raghavan, and Schütze

Tags:

2012-04-01 About