Semanlink - Domain Knowledge + Deep Learning

BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a
link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training)

> a method for **learning representations
of entities**, that uses a **pre-trained Transformer** based
architecture as an entity encoder, and
**link prediction training on a knowledge graph
with textual entity descriptions**.

> using entity descriptions,
an entity encoder is trained for link prediction in
a knowledge graph. The encoder can then be used
without fine-tuning to obtain features for entity classification
and information retrieval

Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their
objective targeted exclusively for link prediction (and not an objective that combines language modeling
and link prediction as Kepler)
performs better than Kepler's more complex one.

2020-11-03 About

[2010.11967] Language Models are Open Knowledge Graphs

Tags:

2020-10-26 About

raphaelsty/mkb: Knowledge Base Embedding By Cooperative Knowledge Distillation

Tags:

2020-07-24 About

[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision

Tags:

2020-07-11 About

[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge

Tags:

> a neural language model that includes **an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge.**... **The model can be updated without re-training by manipulating its symbolic representations**. In particular this model allows us to add new facts and overwrite existing ones.

> a **neural language model which learns to access information
in a symbolic knowledge graph.**

> This
model builds on the recently-proposed [Entities as
Experts](doc:2020/07/2004_07202_entities_as_expert) (EaE) language model (Févry et al., 2020),
which extends the same transformer (Vaswani
et al., 2017) architecture of BERT (Devlin et al., 2019) with an additional external memory for entities.
>
> After training EaE, the embedding associated
with an entity will (ideally) capture information
about the textual context in which that
entity appears, and by inference, the entity’s semantic
properties
>
> we include an additional
memory called a fact memory, which encodes
triples from a symbolic KB.
>
> This combination results in a
neural language model which learns to access information
in a the symbolic knowledge graph.

TODO:

- read again IBM's [Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) ("an effort to avoid encoding general knowledge in the transformer network itself")
- compare with [[1907.05242] Large Memory Layers with Product Keys](doc:2019/07/_1907_05242_large_memory_layer)
- how does it relate with [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)?

2020-07-09 About

facebookresearch/faiss: A library for efficient similarity search and clustering of dense vectors.

Tags:

2020-06-09 About

[1909.04164] Knowledge Enhanced Contextual Word Representations

Tags:

2020-05-13 About

[1907.04829] BAM! Born-Again Multi-Task Networks for Natural Language Understanding

Tags:

2020-05-12 About

[1912.08422] Distilling Structured Knowledge into Embeddings for Explainable and Accurate Recommendation

Tags:

2020-05-12 About

[1807.08447] LinkNBed: Multi-Graph Representation Learning with Entity Linkage

Tags:

2020-05-11 About

Bootstrapping Entity Alignment with Knowledge Graph Embedding | IJCAI

Tags:

2020-05-11 About

[1706.00384] Deep Mutual Learning

Tags:

2020-05-11 About

[1906.07241] Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling

Tags:

2020-05-11 About

Deepak Nathani | Pay Attention, Relations are Important

Tags:

2020-04-30 About

[1906.01195] Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs

Tags:

2020-04-30 About

CEUR-WS.org/Vol-2377 - Workshop on Deep Learning for Knowledge Graphs 2019

Tags:

2020-04-29 About

[1911.02168] CoKE: Contextualized Knowledge Graph Embedding

Tags:

2020-03-22 About

Combining knowledge graphs, quickly and accurately (2020)

Tags:

2020-03-19 About

[1909.07606] K-BERT: Enabling Language Representation with Knowledge Graph

Tags:

2020-03-08 About

[1902.10197] RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space

Tags:

2020-03-03 About

Advancing Natural Language Processing (NLP) for Enterprise Domains

Tags:

2020-01-07 About

[1909.01066] Language Models as Knowledge Bases?

Tags:

2019-09-05 About

[1905.07854] KGAT: Knowledge Graph Attention Network for Recommendation

Tags:

2019-08-23 About

Neural Knowledge Acquisition via Mutual Attention between Knowledge Graph and Text (2018)

Tags:

2019-08-23 About

The knowledge graph as the default data model for learning on heterogeneous knowledge (2017)

Tags:

2019-08-22 About

[1905.07129] ERNIE: Enhanced Language Representation with Informative Entities

Tags:

2019-08-05 About

Reasoning With Neural Tensor Networks for Knowledge Base Completion (2013)

Tags:

**Predicting the likely truth of additional facts based on existing facts in the knowledge base.**

> we introduce an expressive neural
tensor network suitable for reasoning over relationships between two entities.

Most similar work: [Bordes et al.](http://127.0.0.1:8080/semanlink/doc/2019/08/learning_structured_embeddings_) (2011)

Contributions:

1. new neural tensor
network (**NTN**) suitable for reasoning over relationships between two entities. Generalizes several previous neural network models and provides a more
powerful way to model relational information than a standard neural network layer.
2. a new way to represent entities in knowledge bases, as the
average of  their constituting word vectorss, allowing the sharing of statistical strength between the words describing
each entity (e.g., Bank of China and China).
3. incorporation of word vectors which are trained on large unlabeled text

> We **learn to modify word representations
via grounding in world knowledge**. This essentially allows us to analyze word embeddings and
query them for specific relations. Furthermore, the resulting vectors could be used in other tasks
such as named entity recognition or relation classification in natural language

**Makes use of entity name**: NTN
first learns word vectors from an auxiliary news corpus, and
then initializes the representation of each entity by averaging
the vectors of words contained in its name. For example,
**the embedding of AlfredHitchcock is initialized by the
average word vectors of “alfred” and “hitchcock”**... This kind of methods model textual information
separately from KG facts, and hence fail to leverage
interactions between them.[src](doc:2019/05/knowledge_graph_embedding_a_su)

2019-08-03 About

Knowledge Graph Embedding: A Survey of Approaches and Applications - IEEE (2017)

Tags:

- "facts only" based embeddings
    - Translational Distance Models, distance-based scoring functions
       - TransE and co: entities and relations as points in vector space
           - TransH: relation-specific hyperplanes.
       - Gaussian Embeddings: entities and relations as random variables
    - Semantic matching models, similarity-based scoring functions
        - RESCAL (entities as a vector, relation as a matrix) and familiy
            - DistMult (relation as diagonal matrix)
            - Holographic Embeddings (HolE)
            - etc.
        - Matching with Neural Networks
            - SME, etc
- incorporating other info: entity types, relation paths,
textual descriptions, as well as logical rules
    - entity types
        - semantically smooth embedding
(SSE):  entities of the same type to stay close
to each other in the embedding space
        - type-embodied knowledge representation
learning (TKRL), which can handle hierarchical
entity categories and multiple category labels
- Applications in downstream tasks

2019-05-27 About

Integrating Domain-Knowledge into Deep Learning (2019)

Tags:

2019-03-03 About

[1901.11504] Multi-Task Deep Neural Networks for Natural Language Understanding

Tags: