Semanlink - AI + Knowledge Bases

Parents:

AI + Knowledge Bases

Related Tags:

NN / Symbolic AI hybridation

Descendants

Knowledge Graph + Deep Learning
- Attention + Knowledge Graphs
  - Graph Transformers
- KG-augmented Language Models
  - ERNIE
- Language Models as Knowledge Bases
- Thèse IRIT-Renault NLP-KB
  - KD-MKB
    - KD-MKB biblio
    - KD-MKB paper
    - KD-MKB related
Knowledge Graphs and NLP
- BERT + KB
  - BLP
    - CKB
  - KnowBert
  - LUKE
- Knowledge Graphs in NLP
  - BERT + KB
  - ERNIE
  - KG-augmented Language Models
  - KnowBert
  - Microsoft Concept Graph
- LM + KG
  - BERT + KB
  - LLM + KG
    - LLM generated Knowledge Graphs
    - RAG + KG
  - LLMs and basic Renault Knowledge
- SPIKE-KBC
- Text, KG and embeddings
  - Text-Aware KG embedding
    - BLP
    - DKRL
    - LUKE
      [2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
      
      Tags:
      
      Arxiv Doc
      BERT + KB
      EMNLP 2020
      Entity embeddings
      Ikuya Yamada
      LUKE
      Self-Attention
      Text-Aware KG embedding
      
      > LUKE is based on bidirectional Transformer, treats words and entities in a text as independent tokens, and outputs contextualized representations of them. The representations can be used to address downstream tasks similarly to BERT. [src](https://twitter.com/ikuyamada/status/1312947499141750786) > LUKE is trained using a novel pretraining task that involves predicting randomly masked words (equivalent to BERT’s masked language model) and entities in an entity-annotated corpus obtained from Wikipedia. (Hum, ça me rappelle quelque chose) > LUKE also uses a new *entity-aware* self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores. [github](https://github.com/studio-ousia/luke), [at Hugging Face](https://twitter.com/AkariAsai/status/1389428550298525696), [doc](https://huggingface.co/transformers/model_doc/luke.html), [tweet](https://twitter.com/ikuyamada/status/1392742990586683392?s=20)
      
      2020-11-26 About
      Text-Enhanced Representation Learning for Knowledge Graph (IJCAI 2016)
      
      Tags:
      
      IJCAI
      Text-Aware KG embedding
      
      2020-11-03 About
      [2010.03496] Inductive Entity Representations from Text via Link Prediction
      
      Tags:
      
      Arxiv Doc
      Attention + Knowledge Graphs
      BLP
      Discuté avec Raphaël
      Entity embeddings
      Good
      Link Prediction
      Text-Aware KG embedding
      TheWebConf 2021
      
      BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training) > a method for **learning representations of entities**, that uses a **pre-trained Transformer** based architecture as an entity encoder, and **link prediction training on a knowledge graph with textual entity descriptions**. > using entity descriptions, an entity encoder is trained for link prediction in a knowledge graph. The encoder can then be used without fine-tuning to obtain features for entity classification and information retrieval Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their objective targeted exclusively for link prediction (and not an objective that combines language modeling and link prediction as Kepler) performs better than Kepler's more complex one.
      
      2020-11-03 About
      SSP: Semantic Space Projection for Knowledge Graph Embedding with Text Descriptions (AAAI 2017)
      
      Tags:
      
      Roosevelt
      Text-Aware KG embedding
      
      > jointly learns from the symbolic triples and textual descriptions > The data involved in our model are the knowledge triples and the textual descriptions of entities. In experiments, we adopt the “entity descriptions” of Freebase and the textual definitions of Wordnet as textual information. Obvious but very good remark about link prediction in facts-only KG: > the triple (Anna Roosevelt, Parents, Franklin Roosevelt), indicates “Franklin Roosevelt” is the parent of “Anna Roosevelt”. However, it’s quite difficult to infer this fact merely from other symbolic triples.
      
      2020-10-02 About
      Representation learning of knowledge graphs with entity descriptions (AAAI 2016)
      
      Tags:
      
      DKRL
      Text-Aware KG embedding
      Zero shot
      
      "Description-Embodied Knowledge Representation Learning" (DKRL) > In most knowledge graphs there are usually concise descriptions for entities, which cannot be well utilized by existing methods... Experimental results on real-world datasets show that, our method outperforms other baselines on the knowledge graph completion and entity classification tasks, especially under the zero-shot setting, which indicates that **our method is capable of building representations for novel entities according to their descriptions**. [Source code on github](https://github.com/xrb92/DKRL). For fact triples: TransE. Meanwhile, given an entity we will also learn to maximize the likelihood of predicting its description (using either CBOW or CNN encoder) (head + relation = tail, also in "text space") Two types of representations for entities: structure-based representations and description-based representations. They are learned simultaneously into the same vector space but not forced to be unified **so that novel entities with only descriptions can be represented**.
      
      2020-10-02 About
      [1910.12507] A Survey on Knowledge Graph Embeddings with Literals: Which model links better Literal-ly?
      
      Tags:
      
      Arxiv Doc
      Knowledge Graph Embeddings
      Survey / Review
      Text-Aware KG embedding
      
      2020-05-04 About
      Iterative Entity Alignment with Improved Neural Attribute Embedding
      
      Tags:
      
      Entity alignment
      Entity embeddings
      Text-Aware KG embedding
      
      2020-04-29 About
      A Comprehensive Survey of Knowledge Graph Embeddings with Literals: Techniques and Applications
      
      Tags:
      
      Knowledge Graph Embeddings
      Survey / Review
      Text-Aware KG embedding
      
      > survey of the KG embedding models which consider the structured information of the graph as well as the unstructured information in form of literals such as text, numerical values etc A le mérite de poser la question de l'utilisation de littéraux dans les embeddings de KG [Newer and longer version](/doc/2020/05/1910_12507_a_survey_on_knowle)
      
      2020-04-29 About
      [2001.09522] TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network
      
      Tags:
      
      Arxiv Doc
      Discuté avec Raphaël
      Graph neural networks
      Knowledge Graph Completion
      Microsoft Research
      NLP@Microsoft
      Taxonomies
      Taxonomy expansion task
      Text-Aware KG embedding
      TheWebConf 2020
      
      how to add a set of new concepts to an existing taxonomy. [Tweet](https://twitter.com/mickeyjs6/status/1253772146142216194?s=20) [GitHub](https://github.com/mickeystroller/TaxoExpan) > we study the taxonomy expansion task: given an existing taxonomy and a set of new emerging concepts, we aim to automatically expand the taxonomy to incorporate these new concepts (without changing the existing relations in the given taxonomy). > To the best of our knowledge, this is the first study on **how to expand an existing directed acyclic graph (as we model a taxonomy as a DAG) using self-supervised learning**. Self-supervised framework, the existing taxonomy being used as training data: it learns a model to predict whether a query concept is the direct hyponym of an anchor concept. > 2 techniques: > > 1. a **position-enhanced graph neural network that encodes the local structure of an anchor concept** in the existing taxonomy, > 2. a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data. Regarding 1: uses [GNN](/tag/graph_neural_networks.html) to model the "ego network" of concepts (potential “siblings” and “grand parents” of the query concept). > Regular GNNs fail to distinguish nodes with different relative positions to the query (i.e., some nodes are grand parents of the query while the others are siblings of the query). To address this limitation, we present a simple but effective enhancement to inject such position information into GNNs using position embedding. We show that such embedding can be easily integrated with existing GNN architectures (e.g., [GCN](/tag/graph_convolutional_networks) and GAT) and significantly boosts the prediction performance Regarding point 2: uses InfoNCE loss, cf. [Contrastive Predictive Coding](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1807.03748) > Instead of predicting whether each individual ⟨query concept, anchor concept⟩ pair is positive or not, we first group all pairs sharing the same query concept into a single training instance and learn a model to select the positive pair among other negative ones from the group. (Hum, ça me rappelle quelque chose) > assume each concept (in existing taxonomy + set of new concepts) has an initial embedding vector learned from some text associated with this concept. To keep things tractable, only attempts to find a single parent node of each new concept.
      
      2020-04-25 About
      Reasoning With Neural Tensor Networks for Knowledge Base Completion (2013)
      
      Tags:
      
      Andrew Ng
      Chris Manning
      Knowledge Graph + Deep Learning
      Knowledge Graph Completion
      Richard Socher
      Text-Aware KG embedding
      
      **Predicting the likely truth of additional facts based on existing facts in the knowledge base.** > we introduce an expressive neural tensor network suitable for reasoning over relationships between two entities. Most similar work: [Bordes et al.](http://127.0.0.1:8080/semanlink/doc/2019/08/learning_structured_embeddings_) (2011) Contributions: 1. new neural tensor network (**NTN**) suitable for reasoning over relationships between two entities. Generalizes several previous neural network models and provides a more powerful way to model relational information than a standard neural network layer. 2. a new way to represent entities in knowledge bases, as the average of their constituting word vectorss, allowing the sharing of statistical strength between the words describing each entity (e.g., Bank of China and China). 3. incorporation of word vectors which are trained on large unlabeled text > We **learn to modify word representations via grounding in world knowledge**. This essentially allows us to analyze word embeddings and query them for specific relations. Furthermore, the resulting vectors could be used in other tasks such as named entity recognition or relation classification in natural language **Makes use of entity name**: NTN first learns word vectors from an auxiliary news corpus, and then initializes the representation of each entity by averaging the vectors of words contained in its name. For example, **the embedding of AlfredHitchcock is initialized by the average word vectors of “alfred” and “hitchcock”**... This kind of methods model textual information separately from KG facts, and hence fail to leverage interactions between them.[src](doc:2019/05/knowledge_graph_embedding_a_su)
      
      2019-08-03 About
  - Using literal descriptions of entities
  - Word + Entity embeddings
    - Wikipedia2Vec
- Text to KG
Knowledge mining
Language Models + Knowledge
- Entities and LM
  - LUKE
  - Masked entity prediction
  - Transformer-Based Entity Linking
    - BERT-based entity linking
- Knowledge-augmented Language Models
  - KG-augmented Language Models
  - Retrieval augmented LM
    - ColBERT
    - DSPy (Demonstrate-Search-Predict)
    - REALM
    - Retrieval-based language models
- Language Models as Knowledge Bases
- LM + KG
- Not Encoding Factual Knowledge in Language Model
  - REALM
  - Retrieval augmented LM
    François Chollet sur Twitter : "I'm pretty optimistic that the LLM reliability / factualness issue can be fixed. The key is to use LLMs as a dialog interface and not as a store of knowledge..."
    Tags:
    
    François Chollet
    Knowledge Graphs
    Not Encoding Factual Knowledge in Language Model
    Tweet
    Twitter
    > **LLMs as the query layer** between a human user an a knowledge graph with sources (which can be hybrid generated/curated).
    
    2023-01-24 About
    Akari Asai sur Twitter : "Can we solely rely on LLMs’ memories (eg replace search w ChatGPT)? Probably not... Our analysis shows how retrieval is complementary to LLMs’ parametric knowledge..."
    Tags:
    
    ChatGPT
    ChatGPT+Search
    Information retrieval
    Language Models + Knowledge
    NLP@Meta
    Not Encoding Factual Knowledge in Language Model
    Tweet
    2022-12-21 About
    David Chapman sur Twitter : "AI labs should compete to build the smallest possible language models, which “know” as little as possible..."
    Tags:
    
    Not Encoding Factual Knowledge in Language Model
    Tweet
    2022-10-01 About
    [2206.06520] Memory-Based Model Editing at Scale
    Tags:
    
    Arxiv Doc
    Chris Manning
    Language Model
    Not Encoding Factual Knowledge in Language Model
    Editing knowledge of a Language Model without retraining it.
    
    2022-07-07 About
    Building Scalable, Explainable, and Adaptive NLP Models with Retrieval | SAIL Blog
    Tags:
    
    ColBERT
    Good
    NLP@Stanford
    Not Encoding Factual Knowledge in Language Model
    Retrieval augmented LM
    Retrieval-based NLP
    > The black-box nature of large language models like T5 and GPT-3 makes them inefficient to train and deploy, opaque in their knowledge representations and in backing their claims with provenance, and static in facing a constantly evolving world and diverse downstream contexts. **This post explores retrieval-based NLP, where models retrieve information pertinent to solving their tasks from a plugged-in text corpus**. > > Retrieval-based NLP methods view tasks as “open-book” exams: knowledge is encoded explicitly in the form of a text corpus like Wikipedia, the medical literature, or a software’s API documentation. When solving a language task, **the model learns to search for pertinent passages** and to then use the retrieved information for crafting knowledgeable responses. In doing so, **retrieval helps decouple the capacity that language models have for understanding text from how they store knowledge**
    
    2021-10-07 About
    Hugging Face sur Twitter : "Transformers release of the Retrieval-Augmented Generation model in collaboration with @facebookai!"
    Tags:
    
    huggingface / transformers
    NLP@Facebook
    Not Encoding Factual Knowledge in Language Model
    RAG (Retrieval-Augmented Generation)
    Retrieval augmented LM
    Tweet
    > the **RAG model is trained end-to-end for retrieval-in-the-loop generation**, a new paradigm that allows a model to go find useful information in a text corpus when generating. **No need to try to encode all of that knowledge in a trillion parameters any more ;)**
    
    2021-02-23 About
    [2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
    Tags:
    
    Arxiv Doc
    Knowledge-augmented Language Models
    Neural Search
    NLP@Google
    Not Encoding Factual Knowledge in Language Model
    REALM
    Retrieval augmented LM
    Retriever-Reader
    **Augment language model pre-training with a retriever module**, which is trained using the masked language modeling objective. > To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. **For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner**, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents Hum, #TODO: parallel to be drawn with techniques in [KG-augmented Language Models](tag:knowledge_graph_augmented_language_models) which focus "on the problem of capturing declarative knowledge in the learned parameters of a language model." [Google AI Blog Post](doc:2020/08/google_ai_blog_realm_integrat) [Summary](https://joeddav.github.io/blog/2020/03/03/REALM.html) for the [Hugging Face awesome-papers reading group](doc:2021/03/huggingface_awesome_papers_pap)
    
    2020-12-12 About
    [2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision
    Tags:
    
    Arxiv Doc
    KG-augmented Language Models
    Memory networks
    NLP@Google
    Not Encoding Factual Knowledge in Language Model
    Question Answering
    Transformers
    > We focus on the problem of **capturing declarative knowledge in the learned parameters of a language model**... > Entities as Experts (EaE) can access distinct memories of the entities mentioned in a piece of text; > To understand the motivation for distinct and independent entity representations: A traditional Transformer would need to build an internal representation of Charles Darwin from the words “Charles” and “Darwin”... Conversely, EAE can access a dedicated representation of “Charles Darwin”, which is a memory of all of the contexts in which this entity has previously been mentioned.... Having retrieved and re-integrated this memory it is much easier for EAE to relate the question to the answer > EaE's entity representations are learned directly from text. Correct identification, and representation, of entities is essential to EaE's performance Based on transformer architecture Extension: [Facts as Experts](doc:2020/07/2007_00849_facts_as_experts_)
    
    2020-07-11 About
    [2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
    Tags:
    
    AI + Knowledge Bases
    Arxiv Doc
    Google Research
    Knowledge Graph + Deep Learning
    Neural Memory
    NLP@Google
    NN / Symbolic AI hybridation
    Not Encoding Factual Knowledge in Language Model
    > a neural language model that includes **an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge.**... **The model can be updated without re-training by manipulating its symbolic representations**. In particular this model allows us to add new facts and overwrite existing ones. > a **neural language model which learns to access information in a symbolic knowledge graph.** > This model builds on the recently-proposed [Entities as Experts](doc:2020/07/2004_07202_entities_as_expert) (EaE) language model (Févry et al., 2020), which extends the same transformer (Vaswani et al., 2017) architecture of BERT (Devlin et al., 2019) with an additional external memory for entities. > > After training EaE, the embedding associated with an entity will (ideally) capture information about the textual context in which that entity appears, and by inference, the entity’s semantic properties > > we include an additional memory called a fact memory, which encodes triples from a symbolic KB. > > This combination results in a neural language model which learns to access information in a the symbolic knowledge graph. TODO: - read again IBM's [Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) ("an effort to avoid encoding general knowledge in the transformer network itself") - compare with [[1907.05242] Large Memory Layers with Product Keys](doc:2019/07/_1907_05242_large_memory_layer) - how does it relate with [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)?
    
    2020-07-09 About
    [1909.04120] Span Selection Pre-training for Question Answering
    Tags:
    
    Arxiv Doc
    Knowledge-augmented Language Models
    Language Models + Knowledge
    Memory in deep learning
    Michael Glass
    NLP@IBM
    Not Encoding Factual Knowledge in Language Model
    Question Answering
    Transformers
    > a **new pre-training task inspired by reading comprehension** and an **effort to avoid encoding general knowledge in the transformer network itself** Current transformer architectures store general knowledge -> large models, long pre-training time. Better to offload the requirement of general knowledge to a sparsely activated network. "Span selection" as an additional auxiliary task: the query is a sentence drawn from a corpus with a term replaced with a special token: [BLANK]. The term replaced by the blank is the answer term. The passage is relevant as determined by a BM25 search, and answer-bearing (containing the answer term). Unlike BERT’s cloze task, where the answer must be drawn from the model itself, the answer is found in a passage using language understanding. > **We hope to progress to a model of general purpose language modeling that uses an indexed long term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.**
    
    2019-09-18 About

1 Document (Long List)

[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge

Tags:

> a neural language model that includes **an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge.**... **The model can be updated without re-training by manipulating its symbolic representations**. In particular this model allows us to add new facts and overwrite existing ones.

> a **neural language model which learns to access information
in a symbolic knowledge graph.**

> This
model builds on the recently-proposed [Entities as
Experts](doc:2020/07/2004_07202_entities_as_expert) (EaE) language model (Févry et al., 2020),
which extends the same transformer (Vaswani
et al., 2017) architecture of BERT (Devlin et al., 2019) with an additional external memory for entities.
>
> After training EaE, the embedding associated
with an entity will (ideally) capture information
about the textual context in which that
entity appears, and by inference, the entity’s semantic
properties
>
> we include an additional
memory called a fact memory, which encodes
triples from a symbolic KB.
>
> This combination results in a
neural language model which learns to access information
in a the symbolic knowledge graph.

TODO:

- read again IBM's [Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) ("an effort to avoid encoding general knowledge in the transformer network itself")
- compare with [[1907.05242] Large Memory Layers with Product Keys](doc:2019/07/_1907_05242_large_memory_layer)
- how does it relate with [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)?

2020-07-09 About

Properties

sl:creationDate : 2020-07-09
sl:creationTime : 2020-07-09T23:58:23Z
rdf:type : sl:Tag
skos:prefLabel : AI + Knowledge Bases