Semanlink - NLP tasks / problems AND Text: similarity, embeddings...

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 About

NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"

Tags:

2022-03-09 About

Nils Reimers sur Twitter : "Creating intent classes for chatbots is challenging This tutorial shows how to use sentence-transformers to find potentially overlapping intent classes and how to improve your data annotation work." / Twitter

Tags:

2022-02-19 About

Nils Reimers sur Twitter : "how to use the fast clustering algorithm from sentence-transformers..."

Tags:

2022-02-19 About

gsarti/scibert-nli · Hugging Face

Tags:

2022-01-29 About

Making the Most of Data: Augmentation with BERT | Pinecone

Tags:

2021-12-18 About

Using pretrained SBERT model in cross-encoder · Issue #726 · UKPLab/sentence-transformers

Tags:

2021-12-17 About

Advance BERT model via transferring knowledge from Cross-Encoders to Bi-Encoders | by Chien Vu | Towards Data Science

Tags:

2021-12-17 About

[2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Tags:

An unsupervised domain adaptation technique for dense retrieval models

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 About

Unsupervised_Extractive_Summarization - a Hugging Face Space by Hellisotherpeople

Tags:

2021-12-03 About

Unsupervised Training for Sentence Transformers | Pinecone

Tags:

2021-11-24 About

How to Fine-Tune Sentence-BERT for Question Answering | Capital One

Tags:

2021-11-21 About

Multilingual Sentence Transformers | Pinecone

Tags:

2021-11-04 About

Detecting Duplicate Questions (2019)

Tags:

2021-10-14 About

Building a sentence embedding index with fastText and BM25 | by David Mezzetti | Towards Data Science

Tags:

2021-09-30 About

[2104.06979] TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

Tags:

> The most
successful previous approaches like InferSent (Conneau
et al., 2017), Universial Sentence Encoder
(USE) (Cer et al., 2018) and SBERT (Reimers and
Gurevych, 2019) heavily relied on labeled data to
train sentence embedding models.
>
> TSDAE can
achieve up to 93.1% of the performance of indomain
supervised approaches. Further, we
show that TSDAE is **a strong domain adaptation
and pre-training method for sentence
embeddings**, significantly outperforming other
approaches like Masked Language Model.

> During training, TSDAE
encodes corrupted sentences into fixed-sized
vectors and requires the decoder to reconstruct the
original sentences from this sentence embedding.

- <https://www.sbert.net/examples/unsupervised_learning/TSDAE/README.html>
- [github](https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning/TSDAE)
- [UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet](doc:2020/07/ukplab_sentence_transformers_s)
- [twitter](https://twitter.com/KexinWang2049/status/1433361957579538432):

> **TSDAE can learn domain-specific sentence embeddings with unlabeled sentences**
>
> Most importantly, instead of STS (Semantic Textual Similarity), **we suggest evaluating unsupervised sentence embeddings on the domain-specific tasks&datasets, which is the real use case for them**. Actually, STS scores do not correlate with performance on specific tasks.

2021-09-01 About

Simple Unsupervised Keyphrase Extraction using Sentence Embeddings - ACL Anthology (2018)

Tags:

2021-05-31 About

A Self-Training Approach for Short Text Clustering - (Hadifar 2019)

Tags:

2021-05-20 About

Zero-Shot Learning in Modern NLP | Joe Davison Blog (2020-05)

Tags:

2021-02-23 About

Tags:

2020-07-10 About

[1910.04126] Scalable Nearest Neighbor Search for Optimal Transport

Tags:

2020-02-20 About

Semantic Text Matching for Long-Form Documents (2019)

Tags:

2020-01-23 About

Siamese Network for Image and Text similarity using Keras

Tags:

2020-01-22 About

Building a real-time embeddings similarity matching system | Solutions | Google Cloud

Tags:

2020-01-11 About

Tags:

2020-01-10 About

Elasticsearch meets BERT: Building Search Engine with Elasticsearch and BERT

Tags:

2020-01-10 About

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 About

Learning Text Similarity with Siamese Recurrent Networks (2016)

Tags:

2019-08-07 About

Is That a Duplicate Quora Question? | LinkedIn

Tags:

Quora Question Pairs

2019-07-03 About

Classifying duplicate questions from Quora with Keras | R-bloggers

Tags:

2019-07-03 About

Tags:

2019-07-02 About

Tags:

2019-07-02 About

Semantic textual similarity | NLP-progress

Tags:

2019-07-02 About

Quora Question Pairs | Kaggle

Tags:

Quora Question Pairs

2019-07-02 About

Sent2Vec: An unsupervised approach towards learning sentence embeddings | RARE Technologies

Tags:

2019-03-25 About

nlp-notebooks/Simple Sentence Similarity.ipynb at master · nlptown/nlp-notebooks

Tags:

2019-02-07 About

Tags:

2018-11-25 About

Word Mover's Embedding: From Word2Vec to Document Embedding (2018)

Tags:

2018-11-10 About

Unsupervised Text Summarization using Sentence Embeddings

Tags:

2018-10-09 About

[1704.05358] Representing Sentences as Low-Rank Subspaces

Tags:

2018-10-06 About

A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (2018)

Tags:

A La Carte Embedding

2018-09-18 About

Simple and efficient semantic embeddings for rare words, n-grams, and language features – Off the convex path

Tags:

2018-09-18 About

[1806.06259] Evaluation of sentence embeddings in downstream and linguistic probing tasks

Tags:

2018-06-19 About

Comparing Sentence Similarity Methods

Tags:

2018-05-25 About

Evaluating the Impact of Word Embeddings on Similarity Scoring in Practical Information Retrieval (2017)

Tags:

> Transferring the success of word embeddings to Information Retrieval (IR) task is currently an active research topic. While embedding-based retrieval models could tackle the vocabulary mismatch problem by making use of the embedding’s inherent similarity between distinct words, most of them struggle to compete with the prevalent strong baselines such as TF-IDF and BM25.

Considering a practical ad-hoc IR task composed of two steps, matching and scoring, compares the performance of several techniques that leverage word embeddings in the retrieval models to compute the similarity between the query and the documents (namely word centroid similarity, paragraph vectors, Word Mover’s distance, as well as a novel inverse document frequency (IDF) re-weighted word centroid similarity).

> We confirm that word embeddings can be successfully employed in a practical information retrieval setting. The proposed cosine similarity of IDF re-weighted, aggregated word vectors is competitive to the TF-IDF baseline.

2018-01-28 About

gensim/WMD_tutorial.ipynb

Tags:

2017-12-23 About

Calculates Word Mover's Distance Insanely Fast

Tags:

2017-11-12 About

Word Mover’s Distance in Python

Tags:

Word Mover’s Distance

2017-11-12 About

A Comparison of Vector-based Representations for Semantic Composition (Blacoe and Lapata - 2012)

Tags:

2017-11-12 About

[1607.01759] Bag of Tricks for Efficient Text Classification

Tags:

2017-09-10 About

Finding Similar Items

Tags:

**Jaccard similarity**: similarity of sets, based on the relative size of their intersection -> **finding textually similar documents in a large corpus, near duplicates**. [Collaborative Filtering](/tag/collaborative_filtering) as a Similar-Sets Problem (cf. online purchases, movie ratings)

**Shingling** turns the problem of textual similarity of documents into a pb of similarity of sets

k-shingle: substring of length k found within a document. k: 5 for emails. Hashing shingles. Shingles built from words (stop word + 2 following words)

Similarity-Preserving Summaries of Sets: shingles sets are large -> compress large sets into small representations (“signatures”) that preserve similarity: **[Minhashing](/tag/minhash)** - related to Jaccard similarity (good explanation in [wikipedia](https://en.wikipedia.org/wiki/MinHash))

It still may be impossible to find the pairs of docs with greatest similarity efficiently -> **[Locality-Sensitive Hashing](/tag/locality_sensitive_hashing)** for Documents

Distance measures

Theory of Locality-Sensitive Functions

LSH famiies for other distance measures

Applications of Locality-Sensitive Hashing:

- entity resolution
- matching fingerprints
- matching newpapers articles

Methods for High Degrees of Similarity: LSH-based methods most effective when the degree of similarity we
accept is relatively low. When we want to find sets that are almost identical, other methods can be faster.

2017-07-26 About

A brief overview of query/sentence similarity functions | searchivarius.org

Tags:

Text Similarity

2017-07-21 About

Effective measures for inter-document similarity

Tags:

2017-07-21 About

Word Meaning and Similarity - Stanford University

Tags:

2017-07-20 About

Gensim tutorial: Similarity Queries

Tags:

2017-07-19 About

Tags:

2017-07-19 About

Tags:

2017-07-19 About

From Word Embeddings To Document Distances (Kusner 2015)

Tags:

Word Mover’s Distance

2017-06-09 About

How to find semantic similarity between two documents? (researchgate)

Tags:

Text Similarity

2017-05-18 About

Tags:

2017-05-18 About

Quick review on Text Clustering and Text Similarity Approaches

Tags:

Author: Maali Mnasri (PhD @ CEA)

First transform text units to vectors? not always (eg. sentence similarity task using lexical word alignment). But vectors are efficient to process, and benefit from existing clustering algorithms such as k-means.

Sentence level or document level? Sentence clustering to summarise large documents.

Thematic clustering vs Semantic clustering: depends on the similarity measure.

Text similarity measures:

- Cosine similarity of tf-idf (suitable to produce thematic clusters)
- Knowledge-based Measures (wordNet) (quantify semantic relatedness of words),
- Word embedings

Examples, sample code:

- using wordnet with NLTK, and the formula to compute sentence similarities from word similarities.
- computing similarities between docs using gensim/word2vec

Which clustering algorithm?

- when we have an approximation of the clusters number, and when the similarity measure is not expensive in terms of computation time, clustering algo are suitable and fast. Sample code of k-means clustering using tf-idf vectors with scikit-learn
- Hierarchical clustering algorithms
    - don't need to give the number of clusters
    - but time consuming (calculate a similarity matrix for the sentences) 
- for voluminous data, use an incremental clustering algorithm: sentences are processed one at a time ; each new sentence is compared to each of the already formed clusters.

2017-05-18 About

Fuzzy-Fingerprints for Text-Based Information Retrieval

Tags:

2013-05-31 About