Semanlink - Favoris ; Embeddings ; NLP@Google AND NLP@Google

[Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_)

> Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**.
>
> the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features.

[Github](https://github.com/studio-ousia/luke)

> Entity representations are known to enhance
language models in mono-lingual settings
(Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019:  [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al.,
2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context))
presumably by introducing real-world knowledge.
We show that using entity representations facilitates
cross-lingual transfer by providing languageindependent
features.
>
> Multilingual extension of LUKE. The model is trained with the multilingual
masked language modeling (MLM) task as well
as the masked entity prediction (MEP) task with
Wikipedia entity embeddings

> We investigate two ways of using the entity representations
in cross-lingual transfer tasks:
> 1. perform
entity linking for the input text, and append
the detected entity tokens to the input sequence.
The entity tokens are expected to provide language independent
features to the model
> 2. use the entity
[MASK] token from the MEP task as a languageindependent
feature extractor.

2022-04-17 A propos

Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."

Tags:

2022-04-13 A propos

[2008.11228] A simple method for domain adaptation of sentence embeddings

Tags:

2022-04-01 A propos

[2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Tags:

2022-03-31 A propos

Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)

Tags:

2022-03-31 A propos

Nils Reimers sur Twitter : "Great post on SetFit"

Tags:

2022-03-31 A propos

Sentence Embedding Fine-tuning for the French Language | by La Javaness R&D | Feb, 2022 | Medium

Tags:

2022-03-31 A propos

Domain Adaptation — Sentence-Transformers documentation

Tags:

2022-03-31 A propos

[2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning

Tags:

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 A propos

Studio Ousia sur Twitter : "Now using LUKE is easier than ever!" / Twitter

Tags:

2022-03-15 A propos

Document Matching for Job Descriptions | Semantic Scholar (2021)

Tags:

2022-03-09 A propos

NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"

Tags:

2022-03-09 A propos

[2109.06304] Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration

Tags:

2022-02-25 A propos

Nils Reimers sur Twitter : "Creating intent classes for chatbots is challenging This tutorial shows how to use sentence-transformers to find potentially overlapping intent classes and how to improve your data annotation work." / Twitter

Tags:

2022-02-19 A propos

Nils Reimers sur Twitter : "how to use the fast clustering algorithm from sentence-transformers..."

Tags:

2022-02-19 A propos

sentence-transformers/fast_clustering.py at master · UKPLab/sentence-transformers

Tags:

2022-02-18 A propos

gsarti/scibert-nli · Hugging Face

Tags:

2022-01-29 A propos

Semantic Search — Sentence-Transformers documentation

Tags:

2022-01-29 A propos

Using pretrained SBERT model in cross-encoder · Issue #726 · UKPLab/sentence-transformers

Tags:

2021-12-17 A propos

Advance BERT model via transferring knowledge from Cross-Encoders to Bi-Encoders | by Chien Vu | Towards Data Science

Tags:

2021-12-17 A propos

[2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Tags:

An unsupervised domain adaptation technique for dense retrieval models

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 A propos

Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine

Tags:

2021-12-05 A propos

Unsupervised_Extractive_Summarization - a Hugging Face Space by Hellisotherpeople

Tags:

2021-12-03 A propos

Unsupervised Training for Sentence Transformers | Pinecone

Tags:

2021-11-24 A propos

How to Fine-Tune Sentence-BERT for Question Answering | Capital One

Tags:

2021-11-21 A propos

Multilingual Sentence Transformers | Pinecone

Tags:

2021-11-04 A propos

Next-Gen Sentence Embeddings with Multiple Negatives Ranking Loss | Pinecone

Tags:

2021-10-27 A propos

Sentence Embeddings and Transformers | Pinecone

Tags:

2021-10-23 A propos

Sahajtomar/french_semantic · Hugging Face

Tags:

2021-10-14 A propos

[2010.12566] DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries

Tags:

2021-09-06 A propos

Semantic Search with S-BERT is all you need

Tags:

2021-06-05 A propos

Nils Reimers sur Twitter : "SBERT Release v1.1.0"

Tags:

2021-04-22 A propos

[2011.05864] On the Sentence Embeddings from Pre-trained Language Models

Tags:

2021-04-19 A propos

SimCSE: Simple Contrastive Learning of Sentence Embeddings

Tags:

2021-04-18 A propos

Nils Reimers sur Twitter : "New models for Neural Information Retrieval..."

Tags:

2021-04-17 A propos

[2007.12603] IR-BERT: Leveraging BERT for Semantic Search in Background Linking for News Articles

Tags:

2021-04-12 A propos

SentenceTransformers Documentation

Tags:

2021-03-25 A propos

Zero-Shot Learning in Modern NLP | Joe Davison Blog (2020-05)

Tags:

2021-02-23 A propos

[1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Tags:

> way of **injecting factual knowledge about entities into the pretrained BERT model**.

(Feeding entity vectors
into BERT as if they
were wordpiece vectors without additional encoder
pretraining)

>
> **We align [Wikipedia2Vec](tag:wikipedia2vec) entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors**. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to [ERNIE](tag:ernie) (Zhang et al., 2019) and [KnowBert](tag:knowbert) (Peters et al., 2019), but it **requires no expensive further pretraining of the BERT encoder**.
>
> Our vector space alignment strategy is inspired by
cross-lingual word vector alignment

Related work on Entity-enhanced BERT:

> ([ERNIE](doc:2019/08/_1905_07129_ernie_enhanced_la) and [Knowbert](doc:2020/05/1909_04164_knowledge_enhanced)) are based on the design principle
that BERT be adapted to entity vectors. They introduce
new encoder layers to feed pretrained entity
vectors into the Transformer, and they require additional
pretraining to integrate the new parameters.
In contrast, E-BERT’s design principle is that entity
vectors be adapted to BERT.
>
> Two other knowledge-enhanced MLMs are [KEPLER](doc:2020/11/1911_06136_kepler_a_unified_)
(Wang et al., 2019c) and K-Adapter (Wang
et al., 2020)... Their factual knowledge
does not stem from entity vectors – instead, they
are trained in a multi-task setting on relation classification
and knowledge base completion.

Not to be cofounded with [[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](doc:2020/12/2009_02835_e_bert_a_phrase_a)

2021-01-12 A propos

Supporting content decision makers with machine learning | Dec, 2020 | Netflix TechBlog

Tags:

2020-12-11 A propos

[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

Tags:

2020-11-26 A propos

raphaelsty/ckb: Contextual knowledge bases

Tags:

2020-11-09 A propos

[2010.03496] Inductive Entity Representations from Text via Link Prediction

Tags:

BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a
link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training)

> a method for **learning representations
of entities**, that uses a **pre-trained Transformer** based
architecture as an entity encoder, and
**link prediction training on a knowledge graph
with textual entity descriptions**.

> using entity descriptions,
an entity encoder is trained for link prediction in
a knowledge graph. The encoder can then be used
without fine-tuning to obtain features for entity classification
and information retrieval

Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their
objective targeted exclusively for link prediction (and not an objective that combines language modeling
and link prediction as Kepler)
performs better than Kepler's more complex one.

2020-11-03 A propos

UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet

Tags: