Semanlink - Information retrieval AND NLP tasks / problems

> **approach for training dense retrieval models that does not require any labeled training data**. Dense retrieval is a central challenge for open-domain tasks, such as Open QA, where state-of-the-art methods typically require large supervised datasets with custom hard-negative mining and denoising of positive examples.
>
> ART, in contrast, only requires access to unpaired inputs and outputs (e.g. questions and potential answer documents).
>
> It uses a new document-retrieval autoencoding scheme, where
> 1. an input question is used to retrieve a set of evidence documents, and
> 2. the documents are then used to compute the probability of reconstructing the original question.
>
> Training for retrieval based on question reconstruction enables effective unsupervised learning of both document and question encoders, which can be later incorporated into complete Open QA systems without any further finetuning.

[Tweet](doc:2022/07/devendra_singh_sachan_sur_twitt)

> Given an
input question, ART first retrieves a small set
of possible evidences documents. It then recon
structs
the original question by attending to these
documents
>
> The
key idea in ART is to consider the retrieved documents
as a noisy representation of the original
question and question reconstruction probability
as a way of denoising that provides soft-labels for
how likely each document is to have been the correct
result

Refers to [[IZACARD 2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering](doc:2020/12/2012_04584_distilling_knowled)

2022-07-06 A propos

Devendra Singh Sachan sur Twitter : "ART (Autoencoding-based Retriever Training), an unsupervised method to train a dense retriever that only uses questions and a collection of unpaired documents as the training data."

Tags:

2022-07-06 A propos

Understanding Semantic Search and Question Answering | deepset

Tags:

2022-06-03 A propos

BERTopic: The Future of Topic Modeling | Pinecone

Tags:

2022-05-12 A propos

Unsupervised Training of Retrievers Using GenQ (The Art of Asking Questions with GenQ) | Pinecone

Tags:

2022-03-09 A propos

NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"

Tags:

2022-03-09 A propos

Semantic Keyword Clustering For 10,000+ Keywords [With Script]

Tags:

2022-02-18 A propos

Modern Question Answering Systems Explained

Tags:

Haystack

2022-01-27 A propos

Haystack Annotation Tool

Tags:

Haystack Annotation Tool

2022-01-27 A propos

[2108.13934] Robust Retrieval Augmented Generation for Zero-shot Slot Filling

Tags:

2022-01-19 A propos

[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Tags:

2022-01-19 A propos

Integrate ORQA and REALM for Open Domain Question Answering · Issue #312 · deepset-ai/haystack

Tags:

2022-01-12 A propos

How to build a chatbot that reads all your data to find the right answer - Xatkit

Tags:

2022-01-11 A propos

[1904.08375] Document Expansion by Query Prediction

Tags:

2022-01-05 A propos

Domain Transfer with BERT | Pinecone

Tags:

2022-01-04 A propos

[2112.09118] Towards Unsupervised Dense Information Retrieval with Contrastive Learning

Tags:

2021-12-21 A propos

Making the Most of Data: Augmentation with BERT | Pinecone

Tags:

2021-12-18 A propos

[2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Tags:

An unsupervised domain adaptation technique for dense retrieval models

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 A propos

Nils Reimers sur Twitter : "Do dense retrieval models work out-of-the-box for your specific domain? Often the answer was No😢..."

Tags:

2021-12-15 A propos

[2112.01488] ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction

Tags:

2021-12-05 A propos

Unsupervised Training for Sentence Transformers | Pinecone

Tags:

2021-11-24 A propos

Multilingual Sentence Transformers | Pinecone

Tags:

2021-11-04 A propos

Nils Reimers sur Twitter : "Neural Search for Low Resource Scenarios..."

Tags:

2021-10-27 A propos

Zexuan Zhong sur Twitter : ...Does this really mean dense models are better? No Our #EMNLP2021 paper shows dense retrievers even fail to answer simple entity-centric questions

Tags:

2021-10-07 A propos

[2109.08133] Phrase Retrieval Learns Passage Retrieval, Too

Tags:

2021-09-30 A propos

Building a sentence embedding index with fastText and BM25 | by David Mezzetti | Towards Data Science

Tags:

2021-09-30 A propos

Haystack (deepset)

Tags:

2021-09-20 A propos

Build NLP features into your product | deepset

Tags:

2021-09-20 A propos

Agostina Calabrese sur Twitter : Structured Sentiment Analysis as Dependency Graph Parsing

Tags:

2021-08-02 A propos

[2010.06467] Pretrained Transformers for Text Ranking: BERT and Beyond

Tags:

2021-07-09 A propos

[2106.00882] Efficient Passage Retrieval with Hashing for Open-domain Question Answering

Tags:

2021-06-03 A propos

[2004.04906] Dense Passage Retrieval for Open-Domain Question Answering

Tags:

2021-06-03 A propos

Linking Entities with Knowledge Graphs | by Sigurd Berglann | Strise | Medium

Tags:

2021-02-05 A propos

[2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering

Tags:

> a method to train an information retrieval module for downstream tasks, **without using pairs of queries and documents as annotations**.

Uses two models (standard pipeline for open-domain QA):

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 A propos

Recherche multilingue grâce à la détection de la langue dans Elasticsearch | Elastic Blog

Tags:

2020-11-26 A propos

Tags:

2020-07-10 A propos

[1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Tags:

2020-05-02 A propos

Sebastian Riedel sur Twitter : Happy to introduce BLINK, the @facebookai open-source entity linker!...

Tags:

2020-05-02 A propos

[1910.04126] Scalable Nearest Neighbor Search for Optimal Transport

Tags:

2020-02-20 A propos

Building a real-time embeddings similarity matching system | Solutions | Google Cloud

Tags:

2020-01-11 A propos

Tags:

2020-01-10 A propos

Elasticsearch meets BERT: Building Search Engine with Elasticsearch and BERT

Tags:

2020-01-10 A propos

Joint Intent Classification and Slot Filling with Transformers (Jupyter Notebook Viewer)

Tags:

2020-01-09 A propos

[1902.10909] BERT for Joint Intent Classification and Slot Filling

Tags:

2020-01-09 A propos

Natural Language Understanding with Sequence to Sequence Models

Tags:

2020-01-09 A propos

[1912.08904] Macaw: An Extensible Conversational Information Seeking Platform

Tags:

2020-01-01 A propos

Détection d'intention: application industrielle d'un projet de recherche

Tags:

2019-12-14 A propos

Meetup Paris #40 : Beyond plain text: elasticsearch’s annotated text field type (en anglais) - YouTube

Tags:

2019-11-24 A propos

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 A propos

[1806.04411] Named Entity Recognition with Extremely Limited Data

Tags:

2019-04-11 A propos

Sent2Vec: An unsupervised approach towards learning sentence embeddings | RARE Technologies

Tags:

2019-03-25 A propos

A Tri-Partite Neural Document Language Model for Semantic Information Retrieval (2018 - ESWC conference)

Tags:

2018-06-08 A propos

Tutorial on Poincaré Embeddings (Jupyter Notebook )

Tags:

2018-05-20 A propos

Implementing Poincaré Embeddings | RARE Technologies

Tags:

2018-05-20 A propos

Semantic hashing using tags and topic modeling (2013)

Tags:

2018-03-22 A propos

[1710.04099] Wembedder: Wikidata entity embedding web service

Tags:

2018-02-13 A propos

Evaluating the Impact of Word Embeddings on Similarity Scoring in Practical Information Retrieval (2017)

Tags:

> Transferring the success of word embeddings to Information Retrieval (IR) task is currently an active research topic. While embedding-based retrieval models could tackle the vocabulary mismatch problem by making use of the embedding’s inherent similarity between distinct words, most of them struggle to compete with the prevalent strong baselines such as TF-IDF and BM25.

Considering a practical ad-hoc IR task composed of two steps, matching and scoring, compares the performance of several techniques that leverage word embeddings in the retrieval models to compute the similarity between the query and the documents (namely word centroid similarity, paragraph vectors, Word Mover’s distance, as well as a novel inverse document frequency (IDF) re-weighted word centroid similarity).

> We confirm that word embeddings can be successfully employed in a practical information retrieval setting. The proposed cosine similarity of IDF re-weighted, aggregated word vectors is competitive to the TF-IDF baseline.

2018-01-28 A propos

gensim/WMD_tutorial.ipynb

Tags:

2017-12-23 A propos

(fastText) Euclidean distance instead of cosine-similarity?

Tags:

2017-12-07 A propos

How to use pre-trained word vectors from Facebook’s fastText

Tags:

2017-11-25 A propos

Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings (2015)

Tags:

2017-11-12 A propos

Using Gensim Word2Vec Embeddings in Keras | Ben Bolte's Blog

Tags:

2017-10-23 A propos

Efficient unsupervised keywords extraction using graphs

Tags:

2017-10-04 A propos

Concept Search on Wikipedia · Chris McCormick

Tags:

2017-09-10 A propos

Finding Similar Items

Tags:

**Jaccard similarity**: similarity of sets, based on the relative size of their intersection -> **finding textually similar documents in a large corpus, near duplicates**. [Collaborative Filtering](/tag/collaborative_filtering) as a Similar-Sets Problem (cf. online purchases, movie ratings)

**Shingling** turns the problem of textual similarity of documents into a pb of similarity of sets

k-shingle: substring of length k found within a document. k: 5 for emails. Hashing shingles. Shingles built from words (stop word + 2 following words)

Similarity-Preserving Summaries of Sets: shingles sets are large -> compress large sets into small representations (“signatures”) that preserve similarity: **[Minhashing](/tag/minhash)** - related to Jaccard similarity (good explanation in [wikipedia](https://en.wikipedia.org/wiki/MinHash))

It still may be impossible to find the pairs of docs with greatest similarity efficiently -> **[Locality-Sensitive Hashing](/tag/locality_sensitive_hashing)** for Documents

Distance measures

Theory of Locality-Sensitive Functions

LSH famiies for other distance measures

Applications of Locality-Sensitive Hashing:

- entity resolution
- matching fingerprints
- matching newpapers articles

Methods for High Degrees of Similarity: LSH-based methods most effective when the degree of similarity we
accept is relatively low. When we want to find sets that are almost identical, other methods can be faster.

2017-07-26 A propos

How to spot first stories on Twitter using Storm | Michael Vogiatzis

Tags:

2017-07-26 A propos

Effective measures for inter-document similarity

Tags:

2017-07-21 A propos

Gensim tutorial: Similarity Queries

Tags:

2017-07-19 A propos

Tags:

2017-07-19 A propos

Tags:

2017-07-19 A propos

Latent semantic indexing ("Introduction to Information Retrieval" Manning 2008)

Tags:

VSM : problem with synonymy and polysemy (eg. synonyms are accorded separate dimensions)

Could we use the co-occurrences of terms to capture the latent semantic associations of terms and alleviate these problems?

Concluding remarks:

- computational cost of the SVD is significant
    - biggest obstacle to the widespread adoption to LSI.
    - One approach to this obstacle: build the LSI representation on a randomly sampled subset of the documents, following which the remaining documents are ``folded in'' (cf Gensim tutorial "[Random Projection (used as an option to speed up LSI)](https://radimrehurek.com/gensim/models/rpmodel.html)")
- As we reduce k, recall tends to increase, as expected.
- **Most surprisingly**, a value of k in the low hundreds can actually increase precision. **This appears to suggest that for a suitable value of *k*, LSI addresses some of the challenges of synonymy**.
- LSI works best in applications where there is little overlap between queries and documents. (--??)

The experiments also documented some modes where LSI failed to match the effectiveness of more traditional indexes and score computations.

LSI shares two basic drawbacks of vector space retrieval:
    
- no good way of expressing negations
- no way of enforcing Boolean conditions.

LSI can be viewed as soft clustering by interpreting each dimension of the reduced space as a cluster and the value that a document has on that dimension as its fractional membership in that cluster.

2017-07-19 A propos

What is a simple but detailed explanation of Textrank? - Quora

Tags: