Semanlink - Internet AND NLP tasks / problems

a language
model with access to information available in a
KB, no assumptions
about the availability of additional components
(such as Named Entity Taggers) or annotations

> While classes of
named entities (e.g., person or location) occur frequently,
each individual name (e.g, Atherton or
Zhouzhuang) may be observed infrequently even
in a very large corpus of text. As a result language
models learn to represent accurately only the most
popular named entities

> knowing that Alice is a name
used to refer to a person should give ample information
about the context in which the word may
occur (e.g., Bob visited Alice).

> ---

> extends a traditional **RNN LM**

> we enhance a traditional LM with a
gating mechanism that controls whether a particular
word is modeled as a general word or as a reference
to an entity
>
> We train the model end-to-end
with only the traditional predictive language modeling
perplexity objective
>
> KALM is trained end-to-end using
a predictive objective on large corpus of text.

> To the best of our knowledge, KALM is the first
unsupervised neural NER approach.

> KALM extends a traditional, RNN-based neural
LM.

2022-08-31 About

AllenNLP sur Twitter : "Dataset: training data for @MetaAI 's No Language Left Behind NLLB-200 models!..."

Tags:

2022-08-25 About

Timo Schick sur Twitter : "PEER, a language model trained to incrementally write texts & collaborate w/ humans ..."

Tags:

2022-08-25 About

Unsupervised Learning — Sentence-Transformers documentation

Tags:

2022-08-20 About

BlenderBot 3: A 175B parameter, publicly available chatbot that improves its skills and safety over time

Tags:

BlenderBot

2022-08-09 About

[2012.15156] A Memory Efficient Baseline for Open Domain Question Answering

Tags:

2022-08-08 About

[2205.00820] Entity-aware Transformers for Entity Search

Tags:

2022-07-12 About

[2206.10658] Questions Are All You Need to Train a Dense Passage Retriever

Tags:

> **approach for training dense retrieval models that does not require any labeled training data**. Dense retrieval is a central challenge for open-domain tasks, such as Open QA, where state-of-the-art methods typically require large supervised datasets with custom hard-negative mining and denoising of positive examples.
>
> ART, in contrast, only requires access to unpaired inputs and outputs (e.g. questions and potential answer documents).
>
> It uses a new document-retrieval autoencoding scheme, where
> 1. an input question is used to retrieve a set of evidence documents, and
> 2. the documents are then used to compute the probability of reconstructing the original question.
>
> Training for retrieval based on question reconstruction enables effective unsupervised learning of both document and question encoders, which can be later incorporated into complete Open QA systems without any further finetuning.

[Tweet](doc:2022/07/devendra_singh_sachan_sur_twitt)

> Given an
input question, ART first retrieves a small set
of possible evidences documents. It then recon
structs
the original question by attending to these
documents
>
> The
key idea in ART is to consider the retrieved documents
as a noisy representation of the original
question and question reconstruction probability
as a way of denoising that provides soft-labels for
how likely each document is to have been the correct
result

Refers to [[IZACARD 2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering](doc:2020/12/2012_04584_distilling_knowled)

2022-07-06 About

No Language Left Behind

Tags:

2022-07-06 About

Using BERT For Classifying Documents with Long Texts | by Armand Olivares | Medium

Tags:

2022-06-29 About

[2205.15952] Knowledge Graph -- Deep Learning: A Case Study in Question Answering in Aviation Safety Domain

Tags:

2022-06-11 About

Understanding Semantic Search and Question Answering | deepset

Tags:

2022-06-03 About

Domain transfer with GGPL: German Generative Pseudo Labeling 🥨 | by Matthias Richter | Jun, 2022 | ML6team

Tags:

2022-06-02 About

Nils Reimers sur Twitter : "GPL goes multi-lingual..."

Tags:

2022-06-01 About

BERTopic: The Future of Topic Modeling | Pinecone

Tags:

2022-05-12 About

[2008.09470] Top2Vec: Distributed Representations of Topics

Tags:

2022-04-28 About

Ramsri Goutham Golla sur Twitter : "Hi @Nils_Reimers For GPL you used "msmarco-distilbert-base-tas-b" model and ..."

Tags:

2022-04-27 About

[1909.00426] Global Entity Disambiguation with BERT

Tags:

2022-04-18 About

[2110.08151] mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

Tags:

[Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_)

> Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**.
>
> the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features.

[Github](https://github.com/studio-ousia/luke)

> Entity representations are known to enhance
language models in mono-lingual settings
(Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019:  [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al.,
2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context))
presumably by introducing real-world knowledge.
We show that using entity representations facilitates
cross-lingual transfer by providing languageindependent
features.
>
> Multilingual extension of LUKE. The model is trained with the multilingual
masked language modeling (MLM) task as well
as the masked entity prediction (MEP) task with
Wikipedia entity embeddings

> We investigate two ways of using the entity representations
in cross-lingual transfer tasks:
> 1. perform
entity linking for the input text, and append
the detected entity tokens to the input sequence.
The entity tokens are expected to provide language independent
features to the model
> 2. use the entity
[MASK] token from the MEP task as a languageindependent
feature extractor.

2022-04-17 About

Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."

Tags:

2022-04-13 About

Tu Vu sur Twitter : "Enormous LMs like GPT-3 exhibit impressive few-shot performance, but w/ self-training a BERT base sized model can achieve much better results!

Tags:

2022-04-13 About

[2008.11228] A simple method for domain adaptation of sentence embeddings

Tags:

2022-04-01 About

[2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Tags:

2022-03-31 About

Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)

Tags:

2022-03-31 About

Nils Reimers sur Twitter : "Great post on SetFit"

Tags:

2022-03-31 About

Sentence Embedding Fine-tuning for the French Language | by La Javaness R&D | Feb, 2022 | Medium

Tags:

2022-03-31 About

Domain Adaptation — Sentence-Transformers documentation

Tags:

2022-03-31 About

[2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning

Tags:

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 About

NLP | How to add a domain-specific vocabulary (new tokens) to a subword tokenizer already trained like BERT WordPiece | by Pierre Guillou | Medium

Tags:

2022-03-18 About

Studio Ousia sur Twitter : "Now using LUKE is easier than ever!" / Twitter

Tags:

2022-03-15 About

ddangelov/Top2Vec: Top2Vec learns jointly embedded topic, document and word vectors.

Tags:

> Top2Vec is an algorithm for topic modeling and semantic search. It automatically detects topics present in text and generates jointly embedded topic, document and word vectors.
>
> "Update: Pre-trained Universal Sentence Encoders and BERT Sentence Transformer now available for embedding."

> **The assumption the algorithm makes is that many semantically similar documents are indicative of an underlying topic**. The first step is to create a joint embedding of document and word vectors. Once documents and words are embedded in a vector space the goal of the algorithm is to find dense clusters of documents, then identify which words attracted those documents together. Each dense area is a topic and the words that attracted the documents to the dense area are the topic words.

> Once you train the Top2Vec model you can:
> - ...
> - Get **hierarchical topics**.
> - Search topics by keywords.
> - Search documents by topic, by keywords.
> - Find similar words, similar documents.

Refered by [BERTopic](doc:2022/03/maartengr_bertopic_leveraging_)

2022-03-10 About

MaartenGr/BERTopic: Leveraging BERT and c-TF-IDF to create easily interpretable topics.

Tags:

2022-03-10 About

NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"

Tags:

2022-03-09 About

[2109.06304] Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration

Tags:

2022-02-25 About

Nils Reimers sur Twitter : "Creating intent classes for chatbots is challenging This tutorial shows how to use sentence-transformers to find potentially overlapping intent classes and how to improve your data annotation work." / Twitter

Tags:

2022-02-19 About

Nils Reimers sur Twitter : "how to use the fast clustering algorithm from sentence-transformers..."

Tags:

2022-02-19 About

Semantic Keyword Clustering For 10,000+ Keywords [With Script]

Tags:

2022-02-18 About

gsarti/scibert-nli · Hugging Face

Tags:

2022-01-29 About

[2009.02252] KILT: a Benchmark for Knowledge Intensive Language Tasks

Tags:

2022-01-23 About

[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Tags:

2022-01-19 About

[1906.00300] Latent Retrieval for Weakly Supervised Open Domain Question Answering

Tags:

2022-01-11 About

Domain Transfer with BERT | Pinecone

Tags:

2022-01-04 About

[2112.09118] Towards Unsupervised Dense Information Retrieval with Contrastive Learning

Tags:

2021-12-21 About

Making the Most of Data: Augmentation with BERT | Pinecone

Tags:

2021-12-18 About

Using pretrained SBERT model in cross-encoder · Issue #726 · UKPLab/sentence-transformers

Tags:

2021-12-17 About

Advance BERT model via transferring knowledge from Cross-Encoders to Bi-Encoders | by Chien Vu | Towards Data Science

Tags:

2021-12-17 About

[2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Tags:

An unsupervised domain adaptation technique for dense retrieval models

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 About

facebookresearch/DrQA: Reading Wikipedia to Answer Open-Domain Questions

Tags:

2021-12-08 About

[1906.04980] Unsupervised Question Answering by Cloze Translation

Tags:

2021-12-08 About

facebookresearch/UnsupervisedQA: Unsupervised Question answering via Cloze Translation

Tags:

2021-12-07 About

Unsupervised_Extractive_Summarization - a Hugging Face Space by Hellisotherpeople

Tags:

2021-12-03 About

Sea - Internet-Augmented Dialogue Generation

Tags:

2021-11-25 About

Unsupervised Training for Sentence Transformers | Pinecone

Tags:

2021-11-24 About

[1705.06476] ParlAI: A Dialog Research Software Platform

Tags:

2021-11-21 About

How to Fine-Tune Sentence-BERT for Question Answering | Capital One

Tags:

2021-11-21 About

Tutorial: Training AI bots to chat using MTurk and Facebook’s ParlAI | by Amazon Mechanical Turk | Happenings at MTurk

Tags:

2021-11-19 About

ParlAI

Tags:

ParlAI

2021-11-19 About

Blenderbot2

Tags:

2021-11-19 About

Multilingual Sentence Transformers | Pinecone

Tags:

2021-11-04 About

Train embeddings by using the Two-Tower built-in algorithm | Vertex AI

Tags:

> The Two-Tower model pairs similar types of objects, such as user profiles, search queries, web documents, answer passages, or images, in the same vector space, so that related items are close to each other. **The Two-Tower model consists of two encoder towers: the query tower and the candidate tower**. These towers embed independent items into a shared embedding space, which lets Matching Engine retrieve similarly matched items.
>
> To train a Two-Tower model, Google uses **pairs of relevant items**. Each pair consists of a query document and a candidate document. Documents contain arbitrary customer-defined features including text, numeric, and categorical features. After training, the Two-Tower built-in algorithm exports two TensorFlow SavedModels—a query encoder and a candidate encoder... Given a query item, Matching Engine uses the query encoder to generate a query embedding, and uses the index to find similar candidate embeddings. Matching Engine uses the candidate encoder to index all the items and serve them by using an approximate nearest neighbor solution.

2021-11-04 About

Google AI Blog: Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer (2020)

Tags:

2021-10-13 About

raphaelsty/RetrieverReader: Fast API QA

Tags:

2021-10-04 About

Building a sentence embedding index with fastText and BM25 | by David Mezzetti | Towards Data Science

Tags:

2021-09-30 About

[2010.12566] DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries

Tags:

2021-09-06 About

Agostina Calabrese sur Twitter : Structured Sentiment Analysis as Dependency Graph Parsing

Tags:

2021-08-02 About

[1911.02116] Unsupervised Cross-lingual Representation Learning at Scale

Tags:

2021-07-29 About

[2102.07043] Reasoning Over Virtual Knowledge Bases With Open Predicate Relations

Tags:

> a method for constructing **a virtual KB (VKB) trained entirely from text**

Open Predicate Query Language (OPQL): constructing a virtual knowledge base (VKB) that supports KB reasoning & open-domain QA, tackling the incompleteness of knowledge bases by constructing a virtual KB only from text

> OPQL constructs
a VKB by **encoding and indexing a set of
relation mentions** in a way that naturally enables
reasoning and can be trained without any structured
supervision.

> can be used
as an **external memory integrated into a language
model**

cf. this earlier paper [[2002.10640] Differentiable Reasoning over a Virtual Knowledge Base](doc:2020/07/2002_10640_differentiable_rea). But does not require an initial structured KB for distant
supervision.

> The key idea in constructing the OPQL VKB is to use a
dual-encoder pre-training process, similar to 
[[1906.03158] Matching the Blanks: Distributional Similarity for Relation Learning](doc:2021/05/1906_03158_matching_the_blank)

Related work section refers to [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced). Also refers to [[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge](doc:2020/07/2007_00849_facts_as_experts_) (some authors in common)

2021-06-20 About

[2004.04906] Dense Passage Retrieval for Open-Domain Question Answering

Tags:

2021-06-03 About

[1906.03158] Matching the Blanks: Distributional Similarity for Relation Learning

Tags:

2021-05-13 About

[1909.10506] Learning Dense Representations for Entity Retrieval

Tags:

> We show that it is feasible to perform **entity
linking by training a dual encoder (two-tower)
model that encodes mentions and entities in
the same dense vector space**, where candidate
entities are retrieved by approximate nearest
neighbor search. Unlike prior work, **this setup
does not rely on an alias table followed by a
re-ranker, and is thus the first fully learned entity
retrieval model**.

Contributions:

> -  a dual encoder architecture for
learning entity and mention encodings suitable for
retrieval. A key feature of the architecture is that it
employs a modular **hierarchy of sub-encoders that
capture different aspects of mentions and entities**
> - a simple, fully unsupervised **hard negative
mining** strategy that produces massive gains
in retrieval performance, compared to using only
random negatives
> - high
quality candidate entities very efficiently using approximate nearest neighbor search
> - outperforms discrete retrieval
baselines like an alias table or BM25

> strong retrieval
performance across all 5.7 million Wikipedia entities in
around 3ms per mention

> since we are using a two-tower or dual
encoder architecture, **our model cannot use any kind of attention over
both mentions and entities at once**, nor feature-wise
comparisons as done by Francis-Landau et al. (2016).
This is a fairly severe constraint – for example, **we cannot
directly compare the mention span to the entity title**
– but it permits retrieval with nearest neighbor search
for the entire context against a single, all encompassing
representation for each entity

2021-05-01 About

[2007.15779] Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Tags:

2021-04-11 About

[1902.00751] Parameter-Efficient Transfer Learning for NLP

Tags:

2021-04-11 About

exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources - ACL Anthology

Tags:

**Focus on the Embedding of Domain-specific Vocabulary.**

> exBERT
adds a new domain-specific vocabulary and the corresponding
embedding layer, as well as a small
extension module to the original unmodified model

> a pretraining
method allowing **low-cost embedding of
domain-specific vocabulary in the context of an
existing large pre-trained model such as BERT**

> exBERT... explicitly incorporates
the new domain’s vocabulary, while being able to
**reuse the original pre-trained model’s weights as is**
to reduce required computation and training data. Specifically, exBERT extends BERT by augmenting
its embeddings for the original vocabulary with
new embeddings for the domain-specific vocabulary
via **a learned small “extension” module**. **The
output of the original and extension modules are
combined via a trainable weighted sum operation**

In a way similar to concept developed in

> [[1902.00751] Parameter-Efficient Transfer Learning for NLP](doc:2021/04/1902_00751_parameter_efficien), but not in the fine-tuning paradigm.

[Github](https://github.com/cgmhaicenter/exBERT)

2021-04-11 About

Zero-Shot Learning in Modern NLP | Joe Davison Blog (2020-05)

Tags:

2021-02-23 About

kamalkraj/BERT-NER: Pytorch-Named-Entity-Recognition-with-BERT

Tags:

2021-02-07 About

Linking Entities with Knowledge Graphs | by Sigurd Berglann | Strise | Medium

Tags:

2021-02-05 About

QAnswer · Accessing your Knowledge via Natural Language

Tags:

2021-01-26 About

[2010.00904] Autoregressive Entity Retrieval

Tags:

2021-01-14 About

[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce

Tags:

2020-12-14 About

[2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering

Tags:

> a method to train an information retrieval module for downstream tasks, **without using pairs of queries and documents as annotations**.

Uses two models (standard pipeline for open-domain QA):

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 About

Keyword Extraction with BERT | Towards Data Science

Tags:

2020-12-06 About

Salmon Run: Word Sense Disambiguation using BERT as a Language Model

Tags:

2020-12-01 About

Domain-Specific BERT Models · Chris McCormick

Tags:

Domain-Specific BERT

2020-12-01 About

Recherche multilingue grâce à la détection de la langue dans Elasticsearch | Elastic Blog

Tags:

2020-11-26 About

raphaelsty/ckb: Contextual knowledge bases

Tags:

2020-11-09 About

TAGME: on-the-fly annotation of short text fragments!

Tags:

2020-10-11 About

Which flavor of BERT should you use for your QA task? | by Olesya Bondarenko | Towards Data Science

Tags:

2020-10-04 About

[1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification

Tags:

A model that performs **text classification using entities in a knowledge base**.

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 About

[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision

Tags:

2020-07-11 About

Tags:

2020-07-10 About

Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge (ACL Anthology 2020)

Tags:

Aim to leverage both contextual representation of input text (deep LMs)  and  knowledge derived
from curated KBs ([Wordnet](tag:wordnet)) to improve [slot tagging](tag:slot_tagging) in the presence of [out-of-vocab](tag:oov) words ([few-shot scenario](tag:few_shot_learning))

Method:

1. retrieve potentially relevant KB entities and
encode them into distributed representations that
describe global graph-structured information
2. BERT encoder
layer to capture context-aware representations of
the sequence and attend to the KB embeddings
using multi-level graph attention
3. integrate
BERT embeddings and the KB embeddings
to predict the slot type

Contributions:

1. feasibility of applying lexical ontology
to facilitate recognizing OOV words. First to consider the large-scale background
knowledge for enhancing context-aware
slot tagging models.
2. a knowledge integration mechanism that uses multi-level graph
attention to model explicit lexical relations.
3.experiments on two benchmark datasets

> our method makes a notable difference in a
scenario where samples are linguistically diverse,
and large vocab exists.

(Better improvements when using RNN than BERT, because BERT already contains a lot of background knowledge)

2020-07-04 About

dicksontsai/stanford-nlp-local-extension: Chrome extension for sending content to localhost server running Stanford NLP tools.

Tags:

2020-07-03 About

Representation Learning for Information Extraction from Form-like Documents – Google Research

Tags:

2020-06-15 About

Label unstructured data using Enterprise Knowledge Graphs 2

Tags:

2020-05-22 About

[1909.04164] Knowledge Enhanced Contextual Word Representations

Tags:

2020-05-13 About

Disambiguating KBpedia Knowledge Graph Concepts

Tags:

2020-05-03 About

[1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Tags:

2020-05-02 About

Sebastian Riedel sur Twitter : Happy to introduce BLINK, the @facebookai open-source entity linker!...

Tags:

2020-05-02 About

Martynas Jusevicius sur Twitter : "Is there a solution for entity recognition that would use a local #KnowledgeGraph to look for matches? Ideally any SPARQL datasource..."

Tags:

2020-03-13 About

Unsupervised NER using BERT - Hands-on NLP model review - Quora

Tags:

2020-03-06 About

[2002.11402] Detecting Potential Topics In News Using BERT, CRF and Wikipedia

Tags:

2020-02-27 About

Distilling BERT models with spaCy - Towards Data Science (2019)

Tags:

2020-02-15 About

Hugging Face sur Twitter : DistilBERT-cased for Question Answering w/ just 3 lines of javascript

Tags:

2020-02-14 About

How Much Knowledge Can You Pack Into the Parameters of a Language Model?

Tags:

> It has recently been observed that neural language
models trained on unstructured text can
implicitly store and retrieve knowledge using
natural language queries.

indeed, cf. Facebook's paper [Language Models as Knowledge Bases?](/doc/2019/09/_1909_01066_language_models_as)

> In this short paper,
we measure the practical utility of this
approach by fine-tuning pre-trained models to
answer questions without access to any external
context or knowledge.

> we show that a large language
model pre-trained on unstructured text can
attain competitive results on open-domain question
answering benchmarks without any access
to external knowledge

BUT:

>1. state-of-the-art results only with the largest model
which had 11 billion parameters.
>1. “open-book” models
typically provide some indication of what information
they accessed when answering a question
that provides a useful form of interpretability.
In contrast, our model distributes knowledge
in its parameters in an inexplicable way, which
precludes this form of interpretability.
>1. **the maximum-likelihood objective provides no guarantees as to whether
a model will learn a fact or not.**

So, what's the point? To be compared with this [IBM's paper](/doc/2019/09/_1909_04120_span_selection_pre): "a new pre-training task inspired by reading comprehension and an effort to avoid encoding general knowledge in the transformer network itself"

2020-02-11 About

Adam Roberts sur Twitter : "New preprint: How Much Knowledge Can You Pack into the Parameters of a Language Model?..."

Tags:

2020-02-11 About

[1911.05507] Compressive Transformers for Long-Range Sequence Modelling

Tags:

2020-02-11 About

Semantic Text Matching for Long-Form Documents (2019)

Tags:

2020-01-23 About

Building a real-time embeddings similarity matching system | Solutions | Google Cloud

Tags:

2020-01-11 About

Tags:

2020-01-10 About

Elasticsearch meets BERT: Building Search Engine with Elasticsearch and BERT

Tags:

2020-01-10 About

[2003.05473] Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking (CoNNL 2019)

Tags:

Training BERT-base-uncased on English Wikipedia and then fine-tuned and evaluating it
on an entity linking (EL) benchmark (EL implemented as a token classification over the entity vocabulary)

> BERT+Entity is a straightforward extension on top
of BERT, i.e. we initialize BERT with the publicly
available weights from the BERT-base-uncased
model and add an output classification layer on
top of the architecture. Given a contextualized token,
the classifier computes the probability of an
entity link for each entry in the entity vocabulary.

Can BERT’s architecture learn all entity
linking steps jointly? To answer:

> an extreme
simplification of the **entity linking setup that
works surprisingly well**: simply cast it as **a
per token classification over the entire entity
vocabulary** (over 700K classes in our case).

> the model
is the first that performs entity linking without any
pipeline or any heuristics, compared to all prior
approaches. We found that with our approach we
can learn additional entity knowledge in BERT that
helps in entity linking. **However, we also found
that almost none of the downstream tasks really
required entity knowledge**.

### Related work

- > [Durrett and Klein (2014)](/doc/2020/01/a_joint_model_for_entity_analys) were the first to propose
jointly modelling Mention detection, Candidate generation and Entity disambiguation in a graphical
model and could show that each of those steps are
interdependent and benefit from a joint objective

This paper uses neural techniques instead of CRF.

- > [Yamada](/showprop.do?pptyuri=http%3A%2F%2Fwww.semanlink.net%2F2001%2F00%2Fsemanlink-schema%23arxiv_author&pptyval=Ikuya%2BYamada) (2016, 2017) was the first to
investigate neural text representations and entity
linking, but their approach is limited to ED.

cf. [#Wikipedia2Vec](tag:wikipedia2vec). Compare with [newer work by Yamada](doc:2020/09/1909_01259_neural_attentive_b)

2020-01-09 About

Named Entity Recognition with Bert – Depends on the definition

Tags:

2020-01-09 About

Joint Intent Classification and Slot Filling with Transformers (Jupyter Notebook Viewer)

Tags:

2020-01-09 About

[1902.10909] BERT for Joint Intent Classification and Slot Filling

Tags:

2020-01-09 About

Natural Language Understanding with Sequence to Sequence Models

Tags:

2020-01-09 About

Lecture 14 – Contextual Vectors | Stanford CS224U: Natural Language Understanding | Spring 2019

Tags:

2020-01-05 About

Détection d'intention: application industrielle d'un projet de recherche

Tags:

2019-12-14 About

Named Entity Recognition with Pytorch Transformers – Pierre-Yves Vandenbussche

Tags:

2019-12-11 About - Local Copy

NLP at Scale for Maintenance and Supply Chain Management

Tags:

2019-12-07 About

Meetup Paris #40 : Beyond plain text: elasticsearch’s annotated text field type (en anglais) - YouTube

Tags:

2019-11-24 About

[1911.01464] Emerging Cross-lingual Structure in Pretrained Language Models

Tags:

2019-11-06 About

BERT is now part of Google Search, so let’s understand how it reasons

Tags:

2019-10-31 About

Improving long-form question answering by compressing search results

Tags:

2019-10-24 About

Language and Perception in Deep Learning - Florian Strub DeepMind, Univ. Lille, Inria

Tags:

2019-10-07 About

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 About

Open-sourcing hyperparameter autotuning for fastText

Tags:

FastText

2019-08-27 About

Nandana Mihindukulasooriya sur Twitter : "I wonder if there is a service for disambiguation and entity linking for @wikidata where I can pass a text and get Wikidata entities mentioned in it!

Tags:

2019-07-13 About

[1905.10070] Label-aware Document Representation via Hybrid Attention for Extreme Multi-Label Text Classification

Tags:

2019-06-22 About

Combining Knowledge with Deep Convolutional Neural Networks for Short Text Classification (2017)

Tags:

2019-05-20 About

[1810.09164] Named Entity Disambiguation using Deep Learning on Graphs

Tags:

2019-04-26 About

Dandelion API - Semantic Text Analytics as a service

Tags:

2019-04-24 About

[1904.08398] DocBERT: BERT for Document Classification

Tags:

2019-04-18 About

Wikify!: linking documents to encyclopedic knowledge (2007)

Tags:

2019-04-16 About

A Neural QA Model for DBpedia (GSoC 2019)

Tags:

2019-03-26 About

[1901.11504] Multi-Task Deep Neural Networks for Natural Language Understanding

Tags:

2019-02-17 About

Keywords2vec

Tags:

2019-02-09 About

Cheatsheet · fastText

Tags:

2019-01-29 About

Using FastText models (not vectors) for robust embeddings | Kaggle

Tags:

2019-01-29 About

[1601.01343] Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation

Tags:

2019-01-27 About

Romain Vial (Hyperlex) at Paris NLP meetup, slides

Tags:

2019-01-24 About

The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar

Tags:

2018-12-03 About

Tutorial on Text Classification (NLP) using ULMFiT and fastai Library in Python - Analytics Vidhya

Tags:

2018-12-01 About

[1811.05370] Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents

Tags:

2018-11-20 About

Teaching Machines to Understand Natural Language (2018)

Tags:

2018-10-26 About

[1810.04805] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Tags:

2018-10-12 About

Unsupervised Machine Translation. G. Lample (slides)

Tags:

2018-09-29 About

Paris NLP Season 3 Meetup #1 | Meetup

Tags:

2018-09-27 About

[Learning Note] StarSpace For Multi-label Text Classification

Tags:

2018-09-11 About

Google AI Blog: Transformer: A Novel Neural Network Architecture for Language Understanding

Tags:

2018-08-17 About

Using Machine Learning to Support Continuous Ontology Development (2010)

Tags:

2018-08-07 About

Practical guide to text classification | Google Developers

Tags:

2018-07-23 About

Studying the Spatio-Temporal Dynamics of Small-Scale Events in Twitter

Tags:

2018-07-12 About

Tutorial on Poincaré Embeddings (Jupyter Notebook )

Tags:

2018-05-20 About

Implementing Poincaré Embeddings | RARE Technologies

Tags:

2018-05-20 About

Under the hood: Multilingual embeddings | Engineering Blog | Facebook Code

Tags:

2018-05-11 About

GitHub - Babylonpartners/fastText_multilingual: Multilingual word vectors

Tags:

2018-05-11 About

Text Classification with TensorFlow Estimators

Tags:

2018-04-17 About

Semantic hashing using tags and topic modeling (2013)

Tags:

2018-03-22 About

[1603.01360] Neural Architectures for Named Entity Recognition

Tags:

2018-03-05 About

[1710.04099] Wembedder: Wikidata entity embedding web service

Tags:

2018-02-13 About

[1712.09405] Advances in Pre-Training Distributed Word Representations

Tags:

2017-12-29 About

gensim/WMD_tutorial.ipynb

Tags:

2017-12-23 About

facebookresearch/MUSE: A library for Multilingual Unsupervised or Supervised word Embeddings

Tags:

2017-12-22 About

(fastText) Euclidean distance instead of cosine-similarity?

Tags:

2017-12-07 About

How can we get the vector of a paragraph? · Issue #26 · facebookresearch/fastText

Tags:

2017-11-25 About

How to use pre-trained word vectors from Facebook’s fastText

Tags:

2017-11-25 About

Implementing a CNN for Text Classification in TensorFlow – WildML

Tags:

2017-11-06 About

Spell Checker using Word2vec | Kaggle

Tags:

2017-11-03 About

Using Gensim Word2Vec Embeddings in Keras | Ben Bolte's Blog

Tags:

2017-10-23 About

[1710.04087] Word Translation Without Parallel Data

Tags:

2017-10-14 About

TensorFlow Neural Machine Translation (seq2seq) Tutorial

Tags:

2017-09-18 About

[1607.01759] Bag of Tricks for Efficient Text Classification

Tags:

2017-09-10 About

Research Blog: Transformer: A Novel Neural Network Architecture for Language Understanding

Tags:

2017-09-01 About

How to spot first stories on Twitter using Storm | Michael Vogiatzis

Tags:

2017-07-26 About

Tags:

2017-07-19 About

Tags:

2017-07-19 About

gensim/FastText_Tutorial.ipynb

Tags:

2017-06-28 About

facebookresearch/fastText: Library for fast text representation and classification.

Tags:

2017-06-28 About

An Intuitive Understanding of Word Embeddings: From Count Vectors to Word2Vec

Tags:

2017-06-09 About

word2vec-api

Tags:

2017-06-09 About

gensim: models.word2vec – Deep learning with word2vec

Tags:

2017-06-01 About

Word2vec in gensim Tutorial | RaRe Technologies

Tags:

2017-06-01 About

Improving Topic Models with Latent Feature Word Representations | Nguyen | Transactions of the Association for Computational Linguistics

Tags:

2017-05-20 About

Using Word2Vec for topic modeling - Stack Overflow

Tags:

2017-05-19 About

Text Classification With Word2Vec - DS lore (2016)

Tags:

2017-05-18 About

Create a Domain Text Classifier Using Cognonto | Frederick Giasson

Tags:

2016-10-25 About

La propagande des algorithmes ? Vraiment ? | InternetActu

Tags:

2016-09-18 About

Extraction de la semantique

Tags:

2014-06-18 About

nltk - hierarchical classification + topic model training data for internet articles and social media - Stack Overflow

Tags:

2014-04-23 About

Real-Time Topic Modeling of Microblogs

Tags:

2014-04-22 About

Bing - Knowledge Widget (Beta)

Tags:

2014-04-04 About

NERD: Named Entity Recognition and Disambiguation

Tags:

2013-07-10 About

Wikilinks - Information Extraction and Synthesis Laboratory

Tags:

Wikilinks Corpus

2013-03-12 About

Learning from Big Data: 40 Million Entities in Context

Tags:

Wikilinks Corpus

2013-03-12 About

From Linked Data to Linked Entities: a Migration Path - Giovanni Bartolomeo, Stefano Salsano

Tags:

2012-04-20 About

LINDEN: Linking Named Entities with Knowledge Base via Semantic Knowledge

Tags:

2012-04-19 About

A Spectrometry of Linked Data

Tags:

Entity mining is still a troublesome open problem. In past years many approaches allowed to automate the generation of equivalence links between references using schema matching or various heuristics based on the recognition of similar property values. In contrast, few of them considered the analysis of the network of equivalence links (“equivalence network”) as an indication of the likelihood and strength of the equivalence.

Could a URI reference (URIRef) be thought as exactly “attached” to its referent? Could it make sense to talk about entity “identifiers” or would it be better to talk about more ambiguous “references”, i.e., placeholders for any model that satisfies the formal semantics of the Semantic Web (Hayes)? Booth observes that the aforementioned question, which in the past has been often regarded as fundamental in the debate about identity on the Web, is relatively unimportant. As long as an entity, identified by whatsoever URIRef, is associated to at least one description containing machine understandable information, this information can be automatically processed and used by applications.

2012-04-16 About

Automated interlinking of speech radio archives

Tags: