Semanlink - Favoris AND Search

> **approach for training dense retrieval models that does not require any labeled training data**. Dense retrieval is a central challenge for open-domain tasks, such as Open QA, where state-of-the-art methods typically require large supervised datasets with custom hard-negative mining and denoising of positive examples.
>
> ART, in contrast, only requires access to unpaired inputs and outputs (e.g. questions and potential answer documents).
>
> It uses a new document-retrieval autoencoding scheme, where
> 1. an input question is used to retrieve a set of evidence documents, and
> 2. the documents are then used to compute the probability of reconstructing the original question.
>
> Training for retrieval based on question reconstruction enables effective unsupervised learning of both document and question encoders, which can be later incorporated into complete Open QA systems without any further finetuning.

[Tweet](doc:2022/07/devendra_singh_sachan_sur_twitt)

> Given an
input question, ART first retrieves a small set
of possible evidences documents. It then recon
structs
the original question by attending to these
documents
>
> The
key idea in ART is to consider the retrieved documents
as a noisy representation of the original
question and question reconstruction probability
as a way of denoising that provides soft-labels for
how likely each document is to have been the correct
result

Refers to [[IZACARD 2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering](doc:2020/12/2012_04584_distilling_knowled)

2022-07-06 A propos

Devendra Singh Sachan sur Twitter : "ART (Autoencoding-based Retriever Training), an unsupervised method to train a dense retriever that only uses questions and a collection of unpaired documents as the training data."

Tags:

2022-07-06 A propos

Evaluation Measures in Information Retrieval | Pinecone

Tags:

2022-06-29 A propos

Understanding Semantic Search and Question Answering | deepset

Tags:

2022-06-03 A propos

Max Irwin sur Twitter : "Instant Neural Search for your website! ..."

Tags:

2022-05-19 A propos

BERTopic: The Future of Topic Modeling | Pinecone

Tags:

2022-05-12 A propos

[2203.08913] Memorizing Transformers

Tags:

2022-05-07 A propos

[2008.09470] Top2Vec: Distributed Representations of Topics

Tags:

2022-04-28 A propos

Nils Reimers sur Twitter : "A nice thread on generalization performance for Dense Retrieval models..."

Tags:

2022-04-27 A propos

Devendra Singh Sachan sur Twitter : "...Unsupervised Passage Re-ranker (UPR), an approach to re-rank retrieved passages for information retrieval tasks."

Tags:

2022-04-18 A propos

Jeremy Howard sur Twitter : "NLP competition at Kaggle about patent concept similarity...."

Tags:

2022-04-15 A propos

[2203.13088] Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

Tags:

2022-03-30 A propos

[2203.06169] LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval

Tags:

2022-03-29 A propos

ddangelov/Top2Vec: Top2Vec learns jointly embedded topic, document and word vectors.

Tags:

> Top2Vec is an algorithm for topic modeling and semantic search. It automatically detects topics present in text and generates jointly embedded topic, document and word vectors.
>
> "Update: Pre-trained Universal Sentence Encoders and BERT Sentence Transformer now available for embedding."

> **The assumption the algorithm makes is that many semantically similar documents are indicative of an underlying topic**. The first step is to create a joint embedding of document and word vectors. Once documents and words are embedded in a vector space the goal of the algorithm is to find dense clusters of documents, then identify which words attracted those documents together. Each dense area is a topic and the words that attracted the documents to the dense area are the topic words.

> Once you train the Top2Vec model you can:
> - ...
> - Get **hierarchical topics**.
> - Search topics by keywords.
> - Search documents by topic, by keywords.
> - Find similar words, similar documents.

Refered by [BERTopic](doc:2022/03/maartengr_bertopic_leveraging_)

2022-03-10 A propos

Unsupervised Training of Retrievers Using GenQ (The Art of Asking Questions with GenQ) | Pinecone

Tags:

2022-03-09 A propos

NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"

Tags:

2022-03-09 A propos

SPIKE: Extractive Search from Allen Institute for AI

Tags:

2022-02-22 A propos

SPIKE for Knowledge Base Construction

Tags:

2022-02-22 A propos

Yosi Shamay sur Twitter : "a new platform for rapid ad-hoc knowledgebase construction using extractive search...."

Tags:

2022-02-22 A propos

Semantic Keyword Clustering For 10,000+ Keywords [With Script]

Tags:

2022-02-18 A propos

How to Build a Semantic Search Engine With Transformers and Faiss | by Kostas Stathoulopoulos | Towards Data Science

Tags:

2022-01-29 A propos

Semantic Search — Sentence-Transformers documentation

Tags:

2022-01-29 A propos

Modern Question Answering Systems Explained

Tags:

Haystack

2022-01-27 A propos

Haystack Annotation Tool

Tags:

Haystack Annotation Tool

2022-01-27 A propos

REALM: Retrieval-Augmented Language Model Pre-Training (Paper Explained) - YouTube

Tags:

2022-01-23 A propos

[2108.13934] Robust Retrieval Augmented Generation for Zero-shot Slot Filling

Tags:

2022-01-19 A propos

[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Tags:

2022-01-19 A propos

Integrate ORQA and REALM for Open Domain Question Answering · Issue #312 · deepset-ai/haystack

Tags:

2022-01-12 A propos

[2004.12832] ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

Tags:

2022-01-12 A propos

How to build a chatbot that reads all your data to find the right answer - Xatkit

Tags:

2022-01-11 A propos

raphaelsty/cherche: Neural search

Tags:

2022-01-11 A propos

[2007.00814] Relevance-guided Supervision for OpenQA with ColBERT

Tags:

2022-01-07 A propos

[1904.08375] Document Expansion by Query Prediction

Tags:

2022-01-05 A propos

Domain Transfer with BERT | Pinecone

Tags:

2022-01-04 A propos

[2112.09118] Towards Unsupervised Dense Information Retrieval with Contrastive Learning

Tags:

2021-12-21 A propos

Making the Most of Data: Augmentation with BERT | Pinecone

Tags:

2021-12-18 A propos

Vespa Blog

Tags:

2021-12-17 A propos

Vespa - the big data serving engine

Tags:

Vespa

2021-12-17 A propos

[2010.02666] Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation

Tags:

2021-12-16 A propos

[2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Tags:

An unsupervised domain adaptation technique for dense retrieval models

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 A propos

Nils Reimers sur Twitter : "Do dense retrieval models work out-of-the-box for your specific domain? Often the answer was No😢..."

Tags:

2021-12-15 A propos

semi-technologies/weaviate: Weaviate is a cloud-native, modular, real-time vector search engine

Tags:

2021-12-05 A propos

Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine

Tags:

2021-12-05 A propos

[2112.01488] ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction

Tags:

2021-12-05 A propos

Unsupervised Training for Sentence Transformers | Pinecone

Tags:

2021-11-24 A propos

Multilingual Sentence Transformers | Pinecone

Tags:

2021-11-04 A propos

Nils Reimers sur Twitter : "Neural Search for Low Resource Scenarios..."

Tags:

2021-10-27 A propos

Next-Gen Sentence Embeddings with Multiple Negatives Ranking Loss | Pinecone

Tags:

2021-10-27 A propos

Isotropy in the Contextual Embedding Space: Clusters and Manifolds | OpenReview

Tags:

2021-10-26 A propos

Sentence Embeddings and Transformers | Pinecone

Tags:

2021-10-23 A propos

Building Scalable, Explainable, and Adaptive NLP Models with Retrieval | SAIL Blog

Tags:

2021-10-07 A propos

Zexuan Zhong sur Twitter : ...Does this really mean dense models are better? No Our #EMNLP2021 paper shows dense retrievers even fail to answer simple entity-centric questions

Tags:

2021-10-07 A propos

raphaelsty/RetrieverReader: Fast API QA

Tags:

2021-10-04 A propos

princeton-nlp/DensePhrases

Tags:

2021-09-30 A propos

[2109.08133] Phrase Retrieval Learns Passage Retrieval, Too

Tags:

2021-09-30 A propos

Building a sentence embedding index with fastText and BM25 | by David Mezzetti | Towards Data Science

Tags:

2021-09-30 A propos

neuml/txtai: Build AI-powered semantic search applications

Tags:

txtai

2021-09-30 A propos

Haystack (deepset)

Tags:

2021-09-20 A propos

Build NLP features into your product | deepset

Tags:

2021-09-20 A propos

Nils Reimers sur Twitter : "Introduction - Neural Search"

Tags:

2021-09-20 A propos

Agostina Calabrese sur Twitter : Structured Sentiment Analysis as Dependency Graph Parsing

Tags:

2021-08-02 A propos

A Moderate Proposal for Radically Better AI-powered Web Search

Tags:

2021-07-10 A propos

[2010.06467] Pretrained Transformers for Text Ranking: BERT and Beyond

Tags:

2021-07-09 A propos

[2104.08663] BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models

Tags:

2021-07-09 A propos

Nandan Thakur sur Twitter : "@ikuyamada @Nils_Reimers Thanks @ikuyamad..."

Tags:

2021-07-09 A propos

UKPLab/beir: A Heterogeneous Benchmark for Information Retrieval.

Tags:

2021-07-09 A propos

How Dense Passage Retrievers (DPR) Work | Towards Data Science

Tags:

Dense Passage Retrieval

2021-06-23 A propos

[2106.04612] Neural Extractive Search

Tags:

how to extend a
search paradigm we call “**extractive search**” with
neural similarity techniques.

> some information needs require extracting
and aggregating sub-sentence information
(words, phrases, or entities) from multiple documents
(e.g. a list of all the risk factors for a specific
disease and their number of mentions, or a comprehensive
table of startups and CEOs).

> extractive search combines
document selection with information extraction. **The query is extended with capture slots**:
these are **search terms that act as variables, whose
values should be extracted**.
> The user
is then presented with the matched documents, each
annotated with the corresponding captured spans,
as well as aggregate information over the captured
spans

Conclusion :

> We presented a system for neural extractive search.
While we found our system to be useful for scientific
search, it also has clear limitations and areas
for improvement, both in terms of accuracy (only
72.2% of the returned results are relevant, both the
alignment and similarity models generalize well to
some relations but not to others), and in terms of
scale

[Video of demo](https://www.youtube.com/watch?v=TtqWi2GgB5A&t=1832s)

2021-06-23 A propos

librAIry

Tags:

2021-06-22 A propos

Efficient open-domain question-answering on Vespa.ai | Vespa Blog

Tags:

Dense Passage Retrieval

2021-06-06 A propos

Semantic Search with S-BERT is all you need

Tags:

2021-06-05 A propos

[2106.00882] Efficient Passage Retrieval with Hashing for Open-domain Question Answering

Tags:

2021-06-03 A propos

[2004.04906] Dense Passage Retrieval for Open-Domain Question Answering

Tags:

2021-06-03 A propos

Nils Reimers sur Twitter : "New models for Neural Information Retrieval..."

Tags:

2021-04-17 A propos

[2007.12603] IR-BERT: Leveraging BERT for Semantic Search in Background Linking for News Articles

Tags:

2021-04-12 A propos

[1901.04085] Passage Re-ranking with BERT

Tags:

2021-03-26 A propos

Rodrigo Nogueira sur Twitter : "Slides of our WSDM 2021 tutorial "Pretrained Transformers for Text Ranking: BERT and Beyond"

Tags:

2021-03-09 A propos

Hugging Face sur Twitter : "Transformers release of the Retrieval-Augmented Generation model in collaboration with @facebookai!"

Tags:

2021-02-23 A propos

Retrieval Augmented Generation with Huggingface Transformers and Ray | Distributed Computing with Ray

Tags:

2021-02-11 A propos

Linking Entities with Knowledge Graphs | by Sigurd Berglann | Strise | Medium

Tags:

2021-02-05 A propos

NLP Based Information Retrieval System | by Nikhil Sharma | Towards Data Science

Tags:

2021-01-27 A propos

Weaviate

Tags:

2021-01-18 A propos

[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training

Tags:

**Augment language model pre-training with a retriever module**, which
is trained using the masked language modeling objective.

> To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. **For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner**, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents

Hum, #TODO: parallel to be drawn with techniques in [KG-augmented Language Models](tag:knowledge_graph_augmented_language_models) which focus "on the problem of capturing declarative knowledge in the learned parameters of a language model."

[Google AI Blog Post](doc:2020/08/google_ai_blog_realm_integrat)

[Summary](https://joeddav.github.io/blog/2020/03/03/REALM.html) for the [Hugging Face awesome-papers reading group](doc:2021/03/huggingface_awesome_papers_pap)

2020-12-12 A propos

[2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering

Tags:

> a method to train an information retrieval module for downstream tasks, **without using pairs of queries and documents as annotations**.

Uses two models (standard pipeline for open-domain QA):

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 A propos

Recherche multilingue grâce à la détection de la langue dans Elasticsearch | Elastic Blog

Tags:

2020-11-26 A propos

Semantic Scholar | AI-Powered Research Tool

Tags:

2020-11-24 A propos

Building a Faster and Accurate Search Engine on Custom Dataset with Transformers 🤗 | by Shivanand Roy | Analytics Vidhya | Sep, 2020 | Medium

Tags:

2020-10-22 A propos

Elastic Transformers. Making BERT stretchy — Scalable… | by Mihail Dungarov | Sep, 2020 | Medium

Tags:

2020-09-08 A propos

Searching with Shingles | Elastic Blog

Tags:

2020-09-03 A propos

Google AI Blog: REALM: Integrating Retrieval into Language Representation Models

Tags:

2020-08-13 A propos

SLEEC: Sparse Local Embeddings for Extreme Multi-label Classification (2015)

Tags:

2020-08-11 A propos

Tags:

2020-07-10 A propos

Ranked Entities in Search Results at Google

Tags:

2020-07-02 A propos

Tags:

2020-07-02 A propos

[1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Tags:

2020-05-02 A propos

Sebastian Riedel sur Twitter : Happy to introduce BLINK, the @facebookai open-source entity linker!...

Tags:

2020-05-02 A propos

[2004.06842] Layered Graph Embedding for Entity Recommendation using Wikipedia in the Yahoo! Knowledge Graph

Tags:

2020-04-17 A propos

[1911.02168] CoKE: Contextualized Knowledge Graph Embedding

Tags:

2020-03-22 A propos

[1910.04126] Scalable Nearest Neighbor Search for Optimal Transport

Tags:

2020-02-20 A propos

Information Retrieval for HR (2018)

Tags:

2020-02-14 A propos

Building a Search Engine with BERT and TensorFlow - Towards Data Science

Tags:

2020-01-12 A propos

Building a real-time embeddings similarity matching system | Solutions | Google Cloud

Tags:

2020-01-11 A propos

Tags:

2020-01-10 A propos

Elasticsearch meets BERT: Building Search Engine with Elasticsearch and BERT

Tags:

2020-01-10 A propos

Joint Intent Classification and Slot Filling with Transformers (Jupyter Notebook Viewer)

Tags:

2020-01-09 A propos

[1902.10909] BERT for Joint Intent Classification and Slot Filling

Tags:

2020-01-09 A propos

Natural Language Understanding with Sequence to Sequence Models

Tags:

2020-01-09 A propos

Thomas Wolf sur Twitter : "I liked the LSH attention in the reformer..."

Tags:

2020-01-05 A propos

[1912.08904] Macaw: An Extensible Conversational Information Seeking Platform

Tags:

2020-01-01 A propos

[1911.00172] Generalization through Memorization: Nearest Neighbor Language Models

Tags:

2019-12-20 A propos

Détection d'intention: application industrielle d'un projet de recherche

Tags:

2019-12-14 A propos

Elasticsearch RSS feed indexer with Spacy entity extraction

Tags:

2019-11-24 A propos

Meetup Paris #40 : Beyond plain text: elasticsearch’s annotated text field type (en anglais) - YouTube

Tags:

2019-11-24 A propos

TEXTUAL REPRESENTATION LEARNING DRIVEN BY KNOWLEDGE RESOURCES: APPLICATION TO INFORMATION RETRIEVAL

Tags:

2019-10-21 A propos

[1910.03524] Beyond Vector Spaces: Compact Data Representation as Differentiable Weighted Graphs

Tags:

2019-10-09 A propos

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 A propos

Neural Models for Information Retrieval (2017)

Tags:

2019-08-18 A propos

[1602.01137] A Dual Embedding Space Model for Document Ranking

Tags:

2019-07-17 A propos

A Short Introduction to Learning to Rank (2010)

Tags:

2019-04-18 A propos

Ranking Measures and Loss Functions in Learning to Rank (2009)

Tags:

2019-04-18 A propos

[1806.04411] Named Entity Recognition with Extremely Limited Data

Tags:

2019-04-11 A propos

Sent2Vec: An unsupervised approach towards learning sentence embeddings | RARE Technologies

Tags:

2019-03-25 A propos

[1704.08803] Neural Ranking Models with Weak Supervision

Tags:

2019-01-27 A propos

[1703.03129] Learning to Remember Rare Events

Tags:

2018-10-23 A propos

[1810.07150] Subword Semantic Hashing for Intent Classification on Small Datasets

Tags:

2018-10-22 A propos

Paris NLP Meetup #6

Tags:

2018-07-26 A propos

SIGIR 2018 Tutorial - Knowledge Extraction and Inference from Text: Shallow, Deep, and Everything in Between

Tags:

2018-07-09 A propos

A Tri-Partite Neural Document Language Model for Semantic Information Retrieval (2018 - ESWC conference)

Tags:

2018-06-08 A propos

A Theoretical Approach to Semantic Coding and Hashing | Simons Institute for the Theory of Computing (2016)

Tags:

2018-05-26 A propos

Tutorial on Poincaré Embeddings (Jupyter Notebook )

Tags:

2018-05-20 A propos

Implementing Poincaré Embeddings | RARE Technologies

Tags:

2018-05-20 A propos

Semantic hashing using tags and topic modeling (2013)

Tags:

2018-03-22 A propos

ML-knn: A lazy learning approach to multi-label learning (2007)

Tags:

a lazy learning approach derived from the traditional [k-nearest neighbors algorithm](tag:k_nearest_neighbors_algorithm)

> for each unseen instance, its K nearest neighbors in the training set are firstly identified. After that, based on statistical information gained from the label sets of these neighboring instances, i.e. the number of neighboring instances belonging to each possible class, [Maximum a posteriori (MAP)](tag:maximum_a_posteriori_estimation) principle is utilized to determine the label set for the unseen instance.

Implemented in [scikit-multilearn](http://scikit.ml/api/skmultilearn.adapt.mlknn.html), in [java](https://github.com/lefman/mulan-extended/blob/master/mulan/src/mulan/classifier/lazy/MLkNN.java)

> the first lazy approach proposed specifically for multi-label classification. This is also a binary relevance approach which considers each label independently as a binary classification problem. Instead of a standard k-NN method, however, MLkNN uses the maximum a-posteriori (MAP) (Kelleher et al., 2015) approach combined with k-NN. [src](https://pdfs.semanticscholar.org/af9b/33da37d290c063cd826ab5923d96892a9767.pdf)

2018-03-18 A propos

[1710.04099] Wembedder: Wikidata entity embedding web service

Tags:

2018-02-13 A propos

Modèle Neuronal de Recherche d'Information Augmenté par une Ressource Sémantique (2017)

Tags:

2018-02-12 A propos

Evaluating the Impact of Word Embeddings on Similarity Scoring in Practical Information Retrieval (2017)

Tags:

> Transferring the success of word embeddings to Information Retrieval (IR) task is currently an active research topic. While embedding-based retrieval models could tackle the vocabulary mismatch problem by making use of the embedding’s inherent similarity between distinct words, most of them struggle to compete with the prevalent strong baselines such as TF-IDF and BM25.

Considering a practical ad-hoc IR task composed of two steps, matching and scoring, compares the performance of several techniques that leverage word embeddings in the retrieval models to compute the similarity between the query and the documents (namely word centroid similarity, paragraph vectors, Word Mover’s distance, as well as a novel inverse document frequency (IDF) re-weighted word centroid similarity).

> We confirm that word embeddings can be successfully employed in a practical information retrieval setting. The proposed cosine similarity of IDF re-weighted, aggregated word vectors is competitive to the TF-IDF baseline.

2018-01-28 A propos

Learning Deep Structured Semantic Models for Web Search using Clickthrough Data - Microsoft Research (2013)

Tags:

2017-12-30 A propos

DSSM ("Deep Semantic Similarity Model") - Microsoft Research

Tags:

DSSM (Deep Semantic Similarity Model)

2017-12-30 A propos

gensim/WMD_tutorial.ipynb

Tags:

2017-12-23 A propos

[1712.01208] The Case for Learned Index Structures

Tags:

2017-12-11 A propos

(fastText) Euclidean distance instead of cosine-similarity?

Tags:

2017-12-07 A propos

How to use pre-trained word vectors from Facebook’s fastText

Tags:

2017-11-25 A propos

Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings (2015)

Tags:

2017-11-12 A propos

[1706.00957] Semantic Vector Encoding and Similarity Search Using Fulltext Search Engines

Tags:

2017-11-11 A propos

Semantics with Dense Vectors

Tags:

2017-11-11 A propos

Semantic Hashing [9 mins] - Université de Toronto | Coursera

Tags:

2017-11-07 A propos

Semantic hashing (2008) - Ruslan Salakhutdinov, Geoffrey Hinton

Tags:

> We show how to learn a deep graphical model of the word-count vectors obtained from a
large set of documents. The values of the latent variables in the deepest layer are easy to
infer and give a much better representation of each document than Latent Semantic Analysis.
When the deepest layer is forced to use a small number of binary variables (e.g. 32),
the graphical model performs ‘‘semantic hashing”: Documents are mapped to memory
addresses in such a way that semantically similar documents are located at nearby
addresses. Documents similar to a query document can then be found by simply accessing
all the addresses that differ by only a few bits from the address of the query document. This
way of extending the efficiency of hash-coding to approximate matching is much faster
than locality sensitive hashing, which is the fastest current method. By using semantic
hashing to filter the documents given to TF-IDF, we achieve higher accuracy than applying
TF-IDF to the entire document set.

Indexing is implemented in the following manner: a document is mapped to a word-count vector and then this vector is passed through a [#Restricted Boltzmann Machine](/tag/restricted_boltzmann_machine) autoencoder and encoded to 32-bit address.

2017-11-07 A propos

[1004.5370] Self-Taught Hashing for Fast Similarity Search

Tags:

2017-11-07 A propos

Using Gensim Word2Vec Embeddings in Keras | Ben Bolte's Blog

Tags:

2017-10-23 A propos

Efficient unsupervised keywords extraction using graphs

Tags:

2017-10-04 A propos

Distributed Word Representations for Information Retrieval

Tags:

2017-10-01 A propos

Vectorland: Brief Notes from Using Text Embeddings for Search

Tags:

2017-09-18 A propos

Using Text Embeddings for Information Retrieval

Tags:

2017-09-18 A propos

Concept Search on Wikipedia · Chris McCormick

Tags:

2017-09-10 A propos

Finding Similar Items

Tags:

**Jaccard similarity**: similarity of sets, based on the relative size of their intersection -> **finding textually similar documents in a large corpus, near duplicates**. [Collaborative Filtering](/tag/collaborative_filtering) as a Similar-Sets Problem (cf. online purchases, movie ratings)

**Shingling** turns the problem of textual similarity of documents into a pb of similarity of sets

k-shingle: substring of length k found within a document. k: 5 for emails. Hashing shingles. Shingles built from words (stop word + 2 following words)

Similarity-Preserving Summaries of Sets: shingles sets are large -> compress large sets into small representations (“signatures”) that preserve similarity: **[Minhashing](/tag/minhash)** - related to Jaccard similarity (good explanation in [wikipedia](https://en.wikipedia.org/wiki/MinHash))

It still may be impossible to find the pairs of docs with greatest similarity efficiently -> **[Locality-Sensitive Hashing](/tag/locality_sensitive_hashing)** for Documents

Distance measures

Theory of Locality-Sensitive Functions

LSH famiies for other distance measures

Applications of Locality-Sensitive Hashing:

- entity resolution
- matching fingerprints
- matching newpapers articles

Methods for High Degrees of Similarity: LSH-based methods most effective when the degree of similarity we
accept is relatively low. When we want to find sets that are almost identical, other methods can be faster.

2017-07-26 A propos

How to spot first stories on Twitter using Storm | Michael Vogiatzis

Tags:

2017-07-26 A propos

Effective measures for inter-document similarity

Tags:

2017-07-21 A propos

Gensim tutorial: Similarity Queries

Tags:

2017-07-19 A propos

Tags:

2017-07-19 A propos

Tags:

2017-07-19 A propos

Latent semantic indexing ("Introduction to Information Retrieval" Manning 2008)

Tags:

VSM : problem with synonymy and polysemy (eg. synonyms are accorded separate dimensions)

Could we use the co-occurrences of terms to capture the latent semantic associations of terms and alleviate these problems?

Concluding remarks:

- computational cost of the SVD is significant
    - biggest obstacle to the widespread adoption to LSI.
    - One approach to this obstacle: build the LSI representation on a randomly sampled subset of the documents, following which the remaining documents are ``folded in'' (cf Gensim tutorial "[Random Projection (used as an option to speed up LSI)](https://radimrehurek.com/gensim/models/rpmodel.html)")
- As we reduce k, recall tends to increase, as expected.
- **Most surprisingly**, a value of k in the low hundreds can actually increase precision. **This appears to suggest that for a suitable value of *k*, LSI addresses some of the challenges of synonymy**.
- LSI works best in applications where there is little overlap between queries and documents. (--??)

The experiments also documented some modes where LSI failed to match the effectiveness of more traditional indexes and score computations.

LSI shares two basic drawbacks of vector space retrieval:
    
- no good way of expressing negations
- no way of enforcing Boolean conditions.

LSI can be viewed as soft clustering by interpreting each dimension of the reduced space as a cluster and the value that a document has on that dimension as its fractional membership in that cluster.

2017-07-19 A propos

Indexing by Latent Semantic Analysis - Deerwester et al. (1990)

Tags:

2017-07-18 A propos

Dealing with Human Language | Elasticsearch: The Definitive Guide [master]

Tags:

2017-07-18 A propos

What is a simple but detailed explanation of Textrank? - Quora

Tags: