Semanlink - Internet AND Knowledge bases

a language
model with access to information available in a
KB, no assumptions
about the availability of additional components
(such as Named Entity Taggers) or annotations

> While classes of
named entities (e.g., person or location) occur frequently,
each individual name (e.g, Atherton or
Zhouzhuang) may be observed infrequently even
in a very large corpus of text. As a result language
models learn to represent accurately only the most
popular named entities

> knowing that Alice is a name
used to refer to a person should give ample information
about the context in which the word may
occur (e.g., Bob visited Alice).

> ---

> extends a traditional **RNN LM**

> we enhance a traditional LM with a
gating mechanism that controls whether a particular
word is modeled as a general word or as a reference
to an entity
>
> We train the model end-to-end
with only the traditional predictive language modeling
perplexity objective
>
> KALM is trained end-to-end using
a predictive objective on large corpus of text.

> To the best of our knowledge, KALM is the first
unsupervised neural NER approach.

> KALM extends a traditional, RNN-based neural
LM.

2022-08-31 About

[2208.03299] Few-shot Learning with Retrieval Augmented Language Model

Tags:

2022-08-08 About

[2205.00820] Entity-aware Transformers for Entity Search

Tags:

2022-07-12 About

[2205.15952] Knowledge Graph -- Deep Learning: A Case Study in Question Answering in Aviation Safety Domain

Tags:

2022-06-11 About

[2205.08184] SKILL: Structured Knowledge Infusion for Large Language Models

Tags:

2022-05-18 About

[2205.04260] EASE: Entity-Aware Contrastive Learning of Sentence Embedding

Tags:

2022-05-11 About

[2110.08151] mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

Tags:

[Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_)

> Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**.
>
> the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features.

[Github](https://github.com/studio-ousia/luke)

> Entity representations are known to enhance
language models in mono-lingual settings
(Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019:  [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al.,
2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context))
presumably by introducing real-world knowledge.
We show that using entity representations facilitates
cross-lingual transfer by providing languageindependent
features.
>
> Multilingual extension of LUKE. The model is trained with the multilingual
masked language modeling (MLM) task as well
as the masked entity prediction (MEP) task with
Wikipedia entity embeddings

> We investigate two ways of using the entity representations
in cross-lingual transfer tasks:
> 1. perform
entity linking for the input text, and append
the detected entity tokens to the input sequence.
The entity tokens are expected to provide language independent
features to the model
> 2. use the entity
[MASK] token from the MEP task as a languageindependent
feature extractor.

2022-04-17 About

Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."

Tags:

2022-04-13 About

Jason Weston sur Twitter : "SeeKeR: An open source search-augmented language model"

Tags:

2022-03-25 About

Studio Ousia sur Twitter : "Now using LUKE is easier than ever!" / Twitter

Tags:

2022-03-15 About

Wiki History Game

Tags:

2022-01-16 About

Improving Language Models by Retrieving from Trillions of Tokens | DeepMind

Tags:

2021-12-09 About

facebookresearch/DrQA: Reading Wikipedia to Answer Open-Domain Questions

Tags:

2021-12-08 About

Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine

Tags:

2021-12-05 About

5 Reasons Knowledge Graph will never bloom | by Dongsheng Wang | Medium

Tags:

2021-12-02 About

Andrea Volpini sur Twitter : New blog post on #ContentHub for #Semantic #SEO. How to cluster content using Knowledge Graph Embeddings to detect: ✓ Clusters ✓ Sub-topics ✓ Author's topical authority

Tags:

2021-11-16 About

Enterprise Knowledge Graph Foundation

Tags:

2021-10-02 About

A Moderate Proposal for Radically Better AI-powered Web Search

Tags:

2021-07-10 About

[2102.07043] Reasoning Over Virtual Knowledge Bases With Open Predicate Relations

Tags:

> a method for constructing **a virtual KB (VKB) trained entirely from text**

Open Predicate Query Language (OPQL): constructing a virtual knowledge base (VKB) that supports KB reasoning & open-domain QA, tackling the incompleteness of knowledge bases by constructing a virtual KB only from text

> OPQL constructs
a VKB by **encoding and indexing a set of
relation mentions** in a way that naturally enables
reasoning and can be trained without any structured
supervision.

> can be used
as an **external memory integrated into a language
model**

cf. this earlier paper [[2002.10640] Differentiable Reasoning over a Virtual Knowledge Base](doc:2020/07/2002_10640_differentiable_rea). But does not require an initial structured KB for distant
supervision.

> The key idea in constructing the OPQL VKB is to use a
dual-encoder pre-training process, similar to 
[[1906.03158] Matching the Blanks: Distributional Similarity for Relation Learning](doc:2021/05/1906_03158_matching_the_blank)

Related work section refers to [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced). Also refers to [[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge](doc:2020/07/2007_00849_facts_as_experts_) (some authors in common)

2021-06-20 About

[1410.5859] Towards a Model Theory for Distributed Representations

Tags:

2021-06-10 About

Matching and mining in knowledge graphs of the Web of data Applications in pharmacogenomics

Tags:

2021-06-10 About

Enterprise Knowledge Graph Solutions | eccenca

Tags:

2021-05-19 About

Projet:Requêter le Wiktionnaire — Wiktionnaire

Tags:

2021-04-12 About

Pulling Turtle RDF triples from the Google Knowledge Graph

Tags:

2021-03-30 About

ringgaard/sling: SLING - A natural language frame semantics parser

Tags:

2021-03-08 About

Wikidata browser

Tags:

Wikidata browser

2021-03-06 About

Hugging Face sur Twitter : "Transformers release of the Retrieval-Augmented Generation model in collaboration with @facebookai!"

Tags:

2021-02-23 About

Linking Entities with Knowledge Graphs | by Sigurd Berglann | Strise | Medium

Tags:

2021-02-05 About

The complexity to construct knowledge graphs and how low code tools can help or hurt you | Medium

Tags:

2021-01-30 About

[1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Tags:

> way of **injecting factual knowledge about entities into the pretrained BERT model**.

(Feeding entity vectors
into BERT as if they
were wordpiece vectors without additional encoder
pretraining)

>
> **We align [Wikipedia2Vec](tag:wikipedia2vec) entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors**. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to [ERNIE](tag:ernie) (Zhang et al., 2019) and [KnowBert](tag:knowbert) (Peters et al., 2019), but it **requires no expensive further pretraining of the BERT encoder**.
>
> Our vector space alignment strategy is inspired by
cross-lingual word vector alignment

Related work on Entity-enhanced BERT:

> ([ERNIE](doc:2019/08/_1905_07129_ernie_enhanced_la) and [Knowbert](doc:2020/05/1909_04164_knowledge_enhanced)) are based on the design principle
that BERT be adapted to entity vectors. They introduce
new encoder layers to feed pretrained entity
vectors into the Transformer, and they require additional
pretraining to integrate the new parameters.
In contrast, E-BERT’s design principle is that entity
vectors be adapted to BERT.
>
> Two other knowledge-enhanced MLMs are [KEPLER](doc:2020/11/1911_06136_kepler_a_unified_)
(Wang et al., 2019c) and K-Adapter (Wang
et al., 2020)... Their factual knowledge
does not stem from entity vectors – instead, they
are trained in a multi-task setting on relation classification
and knowledge base completion.

Not to be cofounded with [[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](doc:2020/12/2009_02835_e_bert_a_phrase_a)

2021-01-12 About

[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce

Tags:

2020-12-14 About

[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training

Tags:

**Augment language model pre-training with a retriever module**, which
is trained using the masked language modeling objective.

> To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. **For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner**, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents

Hum, #TODO: parallel to be drawn with techniques in [KG-augmented Language Models](tag:knowledge_graph_augmented_language_models) which focus "on the problem of capturing declarative knowledge in the learned parameters of a language model."

[Google AI Blog Post](doc:2020/08/google_ai_blog_realm_integrat)

[Summary](https://joeddav.github.io/blog/2020/03/03/REALM.html) for the [Hugging Face awesome-papers reading group](doc:2021/03/huggingface_awesome_papers_pap)

2020-12-12 About

[2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering

Tags:

> a method to train an information retrieval module for downstream tasks, **without using pairs of queries and documents as annotations**.

Uses two models (standard pipeline for open-domain QA):

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 About

Supporting content decision makers with machine learning | Dec, 2020 | Netflix TechBlog

Tags:

2020-12-11 About

Exploration Engines - the koodos collective

Tags:

2020-12-10 About

Drew Tada sur Twitter : "Officially launching giantgra.ph A search engine for knowledge graphs..."

Tags:

2020-12-10 About

FP Servant sur Twitter : "constructing a personal knowledge graph as a support for learning (and a metaphor of the learning experience)..."

Tags:

2020-12-03 About

[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

Tags:

2020-11-26 About

raphaelsty/ckb: Contextual knowledge bases

Tags:

2020-11-09 About

[2010.03496] Inductive Entity Representations from Text via Link Prediction

Tags:

BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a
link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training)

> a method for **learning representations
of entities**, that uses a **pre-trained Transformer** based
architecture as an entity encoder, and
**link prediction training on a knowledge graph
with textual entity descriptions**.

> using entity descriptions,
an entity encoder is trained for link prediction in
a knowledge graph. The encoder can then be used
without fine-tuning to obtain features for entity classification
and information retrieval

Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their
objective targeted exclusively for link prediction (and not an objective that combines language modeling
and link prediction as Kepler)
performs better than Kepler's more complex one.

2020-11-03 About

TAGME: on-the-fly annotation of short text fragments!

Tags:

2020-10-11 About

[2010.00402] From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering

Tags:

2020-10-03 About

[2009.00318] More is not Always Better: The Negative Impact of A-box Materialization on RDF2vec Knowledge Graph Embeddings

Tags:

2020-09-02 About

[1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification

Tags:

A model that performs **text classification using entities in a knowledge base**.

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 About

[1812.06280] Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia

Tags:

2020-09-02 About

RDF2vec.org

Tags:

RDF2VEC

2020-09-02 About

Google AI Blog: REALM: Integrating Retrieval into Language Representation Models

Tags:

2020-08-13 About

[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision

Tags:

2020-07-11 About

[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge

Tags:

> a neural language model that includes **an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge.**... **The model can be updated without re-training by manipulating its symbolic representations**. In particular this model allows us to add new facts and overwrite existing ones.

> a **neural language model which learns to access information
in a symbolic knowledge graph.**

> This
model builds on the recently-proposed [Entities as
Experts](doc:2020/07/2004_07202_entities_as_expert) (EaE) language model (Févry et al., 2020),
which extends the same transformer (Vaswani
et al., 2017) architecture of BERT (Devlin et al., 2019) with an additional external memory for entities.
>
> After training EaE, the embedding associated
with an entity will (ideally) capture information
about the textual context in which that
entity appears, and by inference, the entity’s semantic
properties
>
> we include an additional
memory called a fact memory, which encodes
triples from a symbolic KB.
>
> This combination results in a
neural language model which learns to access information
in a the symbolic knowledge graph.

TODO:

- read again IBM's [Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) ("an effort to avoid encoding general knowledge in the transformer network itself")
- compare with [[1907.05242] Large Memory Layers with Product Keys](doc:2019/07/_1907_05242_large_memory_layer)
- how does it relate with [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)?

2020-07-09 About

Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge (ACL Anthology 2020)

Tags:

Aim to leverage both contextual representation of input text (deep LMs)  and  knowledge derived
from curated KBs ([Wordnet](tag:wordnet)) to improve [slot tagging](tag:slot_tagging) in the presence of [out-of-vocab](tag:oov) words ([few-shot scenario](tag:few_shot_learning))

Method:

1. retrieve potentially relevant KB entities and
encode them into distributed representations that
describe global graph-structured information
2. BERT encoder
layer to capture context-aware representations of
the sequence and attend to the KB embeddings
using multi-level graph attention
3. integrate
BERT embeddings and the KB embeddings
to predict the slot type

Contributions:

1. feasibility of applying lexical ontology
to facilitate recognizing OOV words. First to consider the large-scale background
knowledge for enhancing context-aware
slot tagging models.
2. a knowledge integration mechanism that uses multi-level graph
attention to model explicit lexical relations.
3.experiments on two benchmark datasets

> our method makes a notable difference in a
scenario where samples are linguistically diverse,
and large vocab exists.

(Better improvements when using RNN than BERT, because BERT already contains a lot of background knowledge)

2020-07-04 About

Abstract Wikipedia/July 2020 announcement

Tags:

2020-07-03 About

Lord of the Wiki Ring: Importing Wikidata into Neo4j and analyzing family trees

Tags:

2020-06-17 About

facebookresearch/faiss: A library for efficient similarity search and clustering of dense vectors.

Tags:

2020-06-09 About

Label unstructured data using Enterprise Knowledge Graphs 2

Tags:

2020-05-22 About

[1909.04164] Knowledge Enhanced Contextual Word Representations

Tags:

2020-05-13 About

[1912.08422] Distilling Structured Knowledge into Embeddings for Explainable and Accurate Recommendation

Tags:

2020-05-12 About

Disambiguating KBpedia Knowledge Graph Concepts

Tags:

2020-05-03 About

Cognonto - Data. Structure. Meaning.

Tags:

KBPedia

2020-05-03 About

[1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Tags:

2020-05-02 About

Sebastian Riedel sur Twitter : Happy to introduce BLINK, the @facebookai open-source entity linker!...

Tags:

2020-05-02 About

[1909.03193] KG-BERT: BERT for Knowledge Graph Completion

Tags:

2020-03-22 About

[1911.02168] CoKE: Contextualized Knowledge Graph Embedding

Tags:

2020-03-22 About

Martynas Jusevicius sur Twitter : "Is there a solution for entity recognition that would use a local #KnowledgeGraph to look for matches? Ideally any SPARQL datasource..."

Tags:

2020-03-13 About

[1909.07606] K-BERT: Enabling Language Representation with Knowledge Graph

Tags:

2020-03-08 About

[2003.02320] Knowledge Graphs

Tags:

2020-03-07 About

[2002.11402] Detecting Potential Topics In News Using BERT, CRF and Wikipedia

Tags:

2020-02-27 About

[1911.00172] Generalization through Memorization: Nearest Neighbor Language Models

Tags:

2019-12-20 About

A Brief History of Knowledge Graph's Main Ideas: A tutorial, Claudio Gutierrez and Juan F. Sequeda

Tags:

2019-10-27 About

Improving long-form question answering by compressing search results

Tags:

2019-10-24 About

[1910.03524] Beyond Vector Spaces: Compact Data Representation as Differentiable Weighted Graphs

Tags:

2019-10-09 About

[1909.01066] Language Models as Knowledge Bases?

Tags:

2019-09-05 About

[1905.07129] ERNIE: Enhanced Language Representation with Informative Entities

Tags:

2019-08-05 About

A2N: Attending to Neighbors for Knowledge Graph Inference - ACL 2019

Tags:

2019-07-31 About

Nandana Mihindukulasooriya sur Twitter : "I wonder if there is a service for disambiguation and entity linking for @wikidata where I can pass a text and get Wikidata entities mentioned in it!

Tags:

2019-07-13 About

A Common Sense View of Knowledge Graphs | AI3:::Adaptive Information

Tags:

2019-07-03 About

[1905.10070] Label-aware Document Representation via Hybrid Attention for Extreme Multi-Label Text Classification

Tags:

2019-06-22 About

Virtual Knowledge Graphs: An Overview of Systems and Use Cases | MIT Press Journals (2019)

Tags:

2019-05-28 About

Knowledge Graph Technology and Applications 2019 - Simia

Tags:

2019-05-23 About

Supercharging Elasticsearch for extended Knowledge Graph use cases: Data science + business analytics training: Strata Data Conference

Tags:

2019-05-16 About

Knowledge Graph Conference 2019, Day 1 - Simia

Tags:

2019-05-09 About

Technical usability of Wikidata’s linked data

Tags:

2019-05-07 About

Learning to Map Wikidata Entities To Predefined Topics

Tags:

2019-05-03 About

open-knowledge-embeddings.pdf - (2019)

Tags:

2019-04-29 About

Federated KBs at AKBC2019

Tags:

2019-04-29 About

[1810.09164] Named Entity Disambiguation using Deep Learning on Graphs

Tags:

2019-04-26 About

Dandelion API - Semantic Text Analytics as a service

Tags:

2019-04-24 About

Wikify!: linking documents to encyclopedic knowledge (2007)

Tags:

2019-04-16 About

2019 Knowledge Graph Conference - Knowledge Graphs for AI in the Enterprise

Tags:

Knowledge Graph Conference 2019

2019-04-11 About

[1901.11504] Multi-Task Deep Neural Networks for Natural Language Understanding

Tags:

2019-02-17 About

BPEmb: Subword Embeddings

Tags:

2019-01-31 About

[1601.01343] Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation

Tags:

2019-01-27 About

Romain Vial (Hyperlex) at Paris NLP meetup, slides

Tags:

2019-01-24 About

Wikipedia2Vec

Tags:

2018-12-23 About

Capturing meaning: Toward an abstract Wikipedia (ISWC 2018)

Tags:

2018-10-16 About

HyperE: Hyperbolic Embeddings for Entities

Tags:

2018-07-27 About

Extending NLP - Stardog

Tags:

Stardog

2018-06-14 About

D2KLab/entity2rec: entity2rec generates item recommendation from knowledge graphs

Tags:

2018-06-04 About

Stardog: The Enterprise Knowledge Graph Platform

Tags:

Enterprise Knowledge Graph

2018-05-31 About

Tutorial on Poincaré Embeddings (Jupyter Notebook )

Tags:

2018-05-20 About

Implementing Poincaré Embeddings | RARE Technologies

Tags:

2018-05-20 About

[1804.04526] EventKG: A Multilingual Event-Centric Temporal Knowledge Graph

Tags:

2018-04-15 About

ML-knn: A lazy learning approach to multi-label learning (2007)

Tags:

a lazy learning approach derived from the traditional [k-nearest neighbors algorithm](tag:k_nearest_neighbors_algorithm)

> for each unseen instance, its K nearest neighbors in the training set are firstly identified. After that, based on statistical information gained from the label sets of these neighboring instances, i.e. the number of neighboring instances belonging to each possible class, [Maximum a posteriori (MAP)](tag:maximum_a_posteriori_estimation) principle is utilized to determine the label set for the unseen instance.

Implemented in [scikit-multilearn](http://scikit.ml/api/skmultilearn.adapt.mlknn.html), in [java](https://github.com/lefman/mulan-extended/blob/master/mulan/src/mulan/classifier/lazy/MLkNN.java)

> the first lazy approach proposed specifically for multi-label classification. This is also a binary relevance approach which considers each label independently as a binary classification problem. Instead of a standard k-NN method, however, MLkNN uses the maximum a-posteriori (MAP) (Kelleher et al., 2015) approach combined with k-NN. [src](https://pdfs.semanticscholar.org/af9b/33da37d290c063cd826ab5923d96892a9767.pdf)

2018-03-18 About

Gaze of time | Timelines and Chronologies curated by the community

Tags:

2018-03-06 About

[1710.04099] Wembedder: Wikidata entity embedding web service

Tags: