Semanlink - Language Models + Knowledge

Parents:

Language Models + Knowledge

Related Tags:

Knowledge Graphs and NLP

Descendants

Entities and LM
- LUKE
- Masked entity prediction
- Transformer-Based Entity Linking
  - BERT-based entity linking
Knowledge-augmented Language Models
- KG-augmented Language Models
  - ERNIE
- Retrieval augmented LM
  - ColBERT
  - DSPy (Demonstrate-Search-Predict)
  - REALM
  - Retrieval-based language models
Language Models as Knowledge Bases
LM + KG
- BERT + KB
  - BLP
    - CKB
  - KnowBert
  - LUKE
    [2205.00820] Entity-aware Transformers for Entity Search
    Tags:
    
    Arxiv Doc
    BERT + KB
    Discuté avec Raphaël
    Entities and LM
    Entity Retrieval
    KG-aware low-resource learning
    SIGIR 2022
    > **Do BERT-based entity retrieval models benefit from additional entity information stored in knowledge graphs?** To address this research question, we map entity embeddings into the same input space as a pre-trained BERT model and inject these entity embeddings into the BERT model. This entity-enriched language model is then employed on the entity retrieval task. > we observe empirically that the entity-enriched BERT models **enable fine-tuning on limited training data**, which otherwise would not be feasible due to the known instabilities of BERT in few-sample fine-tuning Uses [Wikipedia2Vec](tag:wikipedia2vec) as graph embedding method
    
    2022-07-12 About
    [1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
    Tags:
    
    Arxiv Doc
    BERT
    BERT + KB
    Discuté avec Raphaël
    Entity embeddings
    Text, KG and embeddings
    > way of **injecting factual knowledge about entities into the pretrained BERT model**. (Feeding entity vectors into BERT as if they were wordpiece vectors without additional encoder pretraining) > > **We align [Wikipedia2Vec](tag:wikipedia2vec) entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors**. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to [ERNIE](tag:ernie) (Zhang et al., 2019) and [KnowBert](tag:knowbert) (Peters et al., 2019), but it **requires no expensive further pretraining of the BERT encoder**. > > Our vector space alignment strategy is inspired by cross-lingual word vector alignment Related work on Entity-enhanced BERT: > ([ERNIE](doc:2019/08/_1905_07129_ernie_enhanced_la) and [Knowbert](doc:2020/05/1909_04164_knowledge_enhanced)) are based on the design principle that BERT be adapted to entity vectors. They introduce new encoder layers to feed pretrained entity vectors into the Transformer, and they require additional pretraining to integrate the new parameters. In contrast, E-BERT’s design principle is that entity vectors be adapted to BERT. > > Two other knowledge-enhanced MLMs are [KEPLER](doc:2020/11/1911_06136_kepler_a_unified_) (Wang et al., 2019c) and K-Adapter (Wang et al., 2020)... Their factual knowledge does not stem from entity vectors – instead, they are trained in a multi-task setting on relation classification and knowledge base completion. Not to be cofounded with [[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](doc:2020/12/2009_02835_e_bert_a_phrase_a)
    
    2021-01-12 About
    [2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce
    Tags:
    
    Arxiv Doc
    Aspect Detection
    BERT + KB
    Domain-Specific BERT
    Product data & NLP
    E-BERT, pre-training framework for product data. 1. to benefit from phrase-level knowledge: Adaptive Hybrid Masking, a new masking strategy, which allows the model to adaptively switch from learning preliminary word knowledge to learning complex phrases 2. leveraging product-level knowledge: training E-BERT to predict a product’s associated neighbors (product association) Resources used: - description of millions of products from the amazon dataset (title, description, reviews) - e-commerce phrases: extracted from above dataset using [AutoPhrase](doc:2020/12/autophrase_automated_phrase_mi) - product association graph: pairs of substitutable and complementary products extracted from amazon dataset Not to be confounded with [[1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT](doc:2021/01/1911_03681_e_bert_efficient_)
    
    2020-12-14 About
    Supporting content decision makers with machine learning | Dec, 2020 | Netflix TechBlog
    Tags:
    
    BERT + KB
    Entity embeddings
    Link Prediction
    Netflix
    Transfer learning
    2020-12-11 About
    [2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
    Tags:
    
    Arxiv Doc
    BERT + KB
    EMNLP 2020
    Entity embeddings
    Ikuya Yamada
    LUKE
    Self-Attention
    Text-Aware KG embedding
    > LUKE is based on bidirectional Transformer, treats words and entities in a text as independent tokens, and outputs contextualized representations of them. The representations can be used to address downstream tasks similarly to BERT. [src](https://twitter.com/ikuyamada/status/1312947499141750786) > LUKE is trained using a novel pretraining task that involves predicting randomly masked words (equivalent to BERT’s masked language model) and entities in an entity-annotated corpus obtained from Wikipedia. (Hum, ça me rappelle quelque chose) > LUKE also uses a new *entity-aware* self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores. [github](https://github.com/studio-ousia/luke), [at Hugging Face](https://twitter.com/AkariAsai/status/1389428550298525696), [doc](https://huggingface.co/transformers/model_doc/luke.html), [tweet](https://twitter.com/ikuyamada/status/1392742990586683392?s=20)
    
    2020-11-26 About
    Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge (ACL Anthology 2020)
    Tags:
    
    ACL 2020
    BERT + KB
    Few-shot learning
    Knowledge Graphs in NLP
    OOV
    Slot filling
    Aim to leverage both contextual representation of input text (deep LMs) and knowledge derived from curated KBs ([Wordnet](tag:wordnet)) to improve [slot tagging](tag:slot_tagging) in the presence of [out-of-vocab](tag:oov) words ([few-shot scenario](tag:few_shot_learning)) Method: 1. retrieve potentially relevant KB entities and encode them into distributed representations that describe global graph-structured information 2. BERT encoder layer to capture context-aware representations of the sequence and attend to the KB embeddings using multi-level graph attention 3. integrate BERT embeddings and the KB embeddings to predict the slot type Contributions: 1. feasibility of applying lexical ontology to facilitate recognizing OOV words. First to consider the large-scale background knowledge for enhancing context-aware slot tagging models. 2. a knowledge integration mechanism that uses multi-level graph attention to model explicit lexical relations. 3.experiments on two benchmark datasets > our method makes a notable difference in a scenario where samples are linguistically diverse, and large vocab exists. (Better improvements when using RNN than BERT, because BERT already contains a lot of background knowledge)
    
    2020-07-04 About
- LLM + KG
  - LLM generated Knowledge Graphs
  - RAG + KG
- LLMs and basic Renault Knowledge
Not Encoding Factual Knowledge in Language Model
- REALM
- Retrieval augmented LM
  François Chollet sur Twitter : "I'm pretty optimistic that the LLM reliability / factualness issue can be fixed. The key is to use LLMs as a dialog interface and not as a store of knowledge..."
  Tags:
  > **LLMs as the query layer** between a human user an a knowledge graph with sources (which can be hybrid generated/curated).
  
  2023-01-24 About
  Akari Asai sur Twitter : "Can we solely rely on LLMs’ memories (eg replace search w ChatGPT)? Probably not... Our analysis shows how retrieval is complementary to LLMs’ parametric knowledge..."
  Tags:
  2022-12-21 About
  David Chapman sur Twitter : "AI labs should compete to build the smallest possible language models, which “know” as little as possible..."
  Tags:
  - Not Encoding Factual Knowledge in Language Model
  - Tweet
  2022-10-01 About
  [2206.06520] Memory-Based Model Editing at Scale
  Tags:
  Editing knowledge of a Language Model without retraining it.
  
  2022-07-07 About
  Building Scalable, Explainable, and Adaptive NLP Models with Retrieval | SAIL Blog
  Tags:
  > The black-box nature of large language models like T5 and GPT-3 makes them inefficient to train and deploy, opaque in their knowledge representations and in backing their claims with provenance, and static in facing a constantly evolving world and diverse downstream contexts. **This post explores retrieval-based NLP, where models retrieve information pertinent to solving their tasks from a plugged-in text corpus**. > > Retrieval-based NLP methods view tasks as “open-book” exams: knowledge is encoded explicitly in the form of a text corpus like Wikipedia, the medical literature, or a software’s API documentation. When solving a language task, **the model learns to search for pertinent passages** and to then use the retrieved information for crafting knowledgeable responses. In doing so, **retrieval helps decouple the capacity that language models have for understanding text from how they store knowledge**
  
  2021-10-07 About
  Hugging Face sur Twitter : "Transformers release of the Retrieval-Augmented Generation model in collaboration with @facebookai!"
  Tags:
  > the **RAG model is trained end-to-end for retrieval-in-the-loop generation**, a new paradigm that allows a model to go find useful information in a text corpus when generating. **No need to try to encode all of that knowledge in a trillion parameters any more ;)**
  
  2021-02-23 About
  [2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
  Tags:
  **Augment language model pre-training with a retriever module**, which is trained using the masked language modeling objective. > To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. **For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner**, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents Hum, #TODO: parallel to be drawn with techniques in [KG-augmented Language Models](tag:knowledge_graph_augmented_language_models) which focus "on the problem of capturing declarative knowledge in the learned parameters of a language model." [Google AI Blog Post](doc:2020/08/google_ai_blog_realm_integrat) [Summary](https://joeddav.github.io/blog/2020/03/03/REALM.html) for the [Hugging Face awesome-papers reading group](doc:2021/03/huggingface_awesome_papers_pap)
  
  2020-12-12 About
  [2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision
  Tags:
  > We focus on the problem of **capturing declarative knowledge in the learned parameters of a language model**... > Entities as Experts (EaE) can access distinct memories of the entities mentioned in a piece of text; > To understand the motivation for distinct and independent entity representations: A traditional Transformer would need to build an internal representation of Charles Darwin from the words “Charles” and “Darwin”... Conversely, EAE can access a dedicated representation of “Charles Darwin”, which is a memory of all of the contexts in which this entity has previously been mentioned.... Having retrieved and re-integrated this memory it is much easier for EAE to relate the question to the answer > EaE's entity representations are learned directly from text. Correct identification, and representation, of entities is essential to EaE's performance Based on transformer architecture Extension: [Facts as Experts](doc:2020/07/2007_00849_facts_as_experts_)
  
  2020-07-11 About
  [2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
  Tags:
  > a neural language model that includes **an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge.**... **The model can be updated without re-training by manipulating its symbolic representations**. In particular this model allows us to add new facts and overwrite existing ones. > a **neural language model which learns to access information in a symbolic knowledge graph.** > This model builds on the recently-proposed [Entities as Experts](doc:2020/07/2004_07202_entities_as_expert) (EaE) language model (Févry et al., 2020), which extends the same transformer (Vaswani et al., 2017) architecture of BERT (Devlin et al., 2019) with an additional external memory for entities. > > After training EaE, the embedding associated with an entity will (ideally) capture information about the textual context in which that entity appears, and by inference, the entity’s semantic properties > > we include an additional memory called a fact memory, which encodes triples from a symbolic KB. > > This combination results in a neural language model which learns to access information in a the symbolic knowledge graph. TODO: - read again IBM's [Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) ("an effort to avoid encoding general knowledge in the transformer network itself") - compare with [[1907.05242] Large Memory Layers with Product Keys](doc:2019/07/_1907_05242_large_memory_layer) - how does it relate with [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)?
  
  2020-07-09 About
  [1909.04120] Span Selection Pre-training for Question Answering
  Tags:
  > a **new pre-training task inspired by reading comprehension** and an **effort to avoid encoding general knowledge in the transformer network itself** Current transformer architectures store general knowledge -> large models, long pre-training time. Better to offload the requirement of general knowledge to a sparsely activated network. "Span selection" as an additional auxiliary task: the query is a sentence drawn from a corpus with a term replaced with a special token: [BLANK]. The term replaced by the blank is the answer term. The passage is relevant as determined by a BM25 search, and answer-bearing (containing the answer term). Unlike BERT’s cloze task, where the answer must be drawn from the model itself, the answer is found in a passage using language understanding. > **We hope to progress to a model of general purpose language modeling that uses an indexed long term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.**
  
  2019-09-18 About

5 Documents (Long List)

[2002.01808] K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Tags:

2023-01-12 About

Akari Asai sur Twitter : "Can we solely rely on LLMs’ memories (eg replace search w ChatGPT)? Probably not... Our analysis shows how retrieval is complementary to LLMs’ parametric knowledge..."

Tags:

2022-12-21 About

cs224n Lecture 13: Integrating Knowledge in Language Models

Tags:

2022-07-08 About

Language Models are Open Knowledge Graphs... but are hard to mine - Towards Data Science

Tags:

2021-01-17 About

[1909.04120] Span Selection Pre-training for Question Answering

Tags:

> a **new pre-training task inspired by reading
comprehension** and an **effort to avoid encoding general knowledge in the transformer network itself**

Current transformer architectures store general knowledge -> large models, long pre-training time. Better to offload the requirement of general knowledge to a sparsely activated network.

"Span selection" as an additional auxiliary task: the query is a sentence drawn from a corpus
with a term replaced with a special token: [BLANK]. The term replaced by the blank is the answer term. The passage is
relevant as determined by a BM25 search, and answer-bearing (containing the answer
term). Unlike BERT’s cloze task, where the answer must be drawn from the model itself, the answer is found in a passage
using language understanding.

> **We hope to progress to a model of general purpose language modeling that uses an indexed long
term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.**

2019-09-18 About

Properties

sl:creationDate : 2020-07-11
sl:creationTime : 2020-07-11T15:10:34Z
rdf:type : sl:Tag
skos:prefLabel : Language Models + Knowledge