Semanlink - NLP: using Knowledge

Parents:

NLP: using Knowledge

Related Tags:

Descendants

Knowledge Graphs in NLP
- BERT + KB
  - BLP
    - CKB
  - KnowBert
  - LUKE
- ERNIE
- KG-augmented Language Models
  - ERNIE
    Specializing Static and Contextual Embeddings in the Medical Domain Using Knowledge Graphs: Let’s Keep It Simple - ACL Anthology (2022)
    Tags:
    
    ACL
    Discuté avec M. Labeau
    KG-augmented Language Models
    Node2Vec
    Olivier Ferret
    > a simple approach to embedding specialization that relies on knowledge graph embeddings and concatenation > we first demonstrate that static embeddings can indeed be improved through concatenation with in-domain node2vec representations. Then, we validate this approach on contextual models and generalize it further by proposing a variant of BERT that incorporates knowledge embeddings within its hidden states through the same process of concatenation.
    
    2023-10-11 About
    [2205.08184] SKILL: Structured Knowledge Infusion for Large Language Models
    Tags:
    
    Arxiv Doc
    KG-augmented Language Models
    NLP@Google
    Structured knowledge
    T5 (Text-To-Text Transfer Transformer)
    > a method to infuse structured knowledge into LLMs, by directly training T5 models on factual triples of knowledge graphs > The models pre-trained on factual triples compare competitively with the ones on natural language sentences that contain the same knowledge. > The proposed method has an advantage that no alignment between the knowledge graph and text corpus is required
    
    2022-05-18 About
    Knowledge Graphs in NLP @ EMNLP 2020 | by Michael Galkin | Nov, 2020 | Medium
    Tags:
    
    EMNLP 2020
    KG-augmented Language Models
    Knowledge Graphs and NLP
    About "Entity Representations in LMs", refers to: - [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context) by [Yamada](/showprop.do?pptyuri=http%3A%2F%2Fwww.semanlink.net%2F2001%2F00%2Fsemanlink-schema%23arxiv_author&pptyval=Ikuya%2BYamada) - Shen et al employ a background KG in their GLM (Graph-guided Masked Language Model): the graph supplies a vocabulary of named entities with their connectivity patterns (reachable entities in k-hops) - [[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision](doc:2020/07/2004_07202_entities_as_expert) - Poerner et al make use of [Wikipedia2Vec](tag:wikipedia2vec) (by Yamada) in their E-BERT Autoregressive KG-augmented LMs: generation process of LMs is conditioned by or enriched with structured knowledge like small subgraphs!
    
    2020-11-24 About
    [2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision
    Tags:
    
    Arxiv Doc
    KG-augmented Language Models
    Memory networks
    NLP@Google
    Not Encoding Factual Knowledge in Language Model
    Question Answering
    Transformers
    > We focus on the problem of **capturing declarative knowledge in the learned parameters of a language model**... > Entities as Experts (EaE) can access distinct memories of the entities mentioned in a piece of text; > To understand the motivation for distinct and independent entity representations: A traditional Transformer would need to build an internal representation of Charles Darwin from the words “Charles” and “Darwin”... Conversely, EAE can access a dedicated representation of “Charles Darwin”, which is a memory of all of the contexts in which this entity has previously been mentioned.... Having retrieved and re-integrated this memory it is much easier for EAE to relate the question to the answer > EaE's entity representations are learned directly from text. Correct identification, and representation, of entities is essential to EaE's performance Based on transformer architecture Extension: [Facts as Experts](doc:2020/07/2007_00849_facts_as_experts_)
    
    2020-07-11 About
    [1909.04164] Knowledge Enhanced Contextual Word Representations
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Contextualized word representations
    EMNLP 2019
    Entities and LM
    Good
    Grounded Language Learning
    KD-MKB biblio
    KG-augmented Language Models
    KnowBert
    Knowledge-augmented Language Models
    Knowledge-driven embeddings
    Knowledge Graphs in NLP
    Multiple Knowledge Bases
    General method to **embed multiple knowledge bases into pre-trained language models** (KB in the sense as fixed collection of entity nodes) > The key idea is to explicitly model entity spans in the input text and use an **entity linker** to retrieve relevant entity embeddings from a KB to form knowledge enhanced entity-span representations. > Then, update contextual word representations via a form of **word-to-entity attention**. > In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that **combines a small amount of entity linking supervision with a large amount of raw text**.
    
    2020-05-13 About
    [1906.07241] Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    KD-MKB biblio
    KG-augmented Language Models
    Knowledge Graphs and NLP
    > a **neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context**. These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. > The KGLM maintains a dynamically growing local knowledge graph, a subset of the knowledge graph that contains entities that have already been mentioned in the text, and their related entities.
    
    2020-05-11 About
- KnowBert
- Microsoft Concept Graph

2 Documents (Long List)

Materials for ACL-2022 tutorial: Knowledge-Augmented Methods for NLP

Tags:

2022-08-26 About

[1812.00417] Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale

Tags:

2019-06-28 About

Properties

sl:creationDate : 2019-06-28
sl:creationTime : 2019-06-28T00:35:14Z
rdf:type : sl:Tag
skos:prefLabel : NLP: using Knowledge