Semanlink - AI + Knowledge

Parents:

AI + Knowledge

Related Tags:

Descendants

AI + Knowledge Bases
- Knowledge Graph + Deep Learning
  - Attention + Knowledge Graphs
    - Graph Transformers
  - KG-augmented Language Models
    - ERNIE
  - Language Models as Knowledge Bases
  - Thèse IRIT-Renault NLP-KB
    - KD-MKB
      - KD-MKB biblio
        
        facebookresearch/faiss: A library for efficient similarity search and clustering of dense vectors.
        
        Tags:
        
        Facebook FAIR
        faiss
        GitHub project
        KD-MKB biblio
        Locality Sensitive Hashing
        Nearest neighbor search
        
        Algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. Faiss is written in C++ with complete wrappers for Python/numpy. [paper](https://arxiv.org/abs/1702.08734)
        
        2020-06-09 About
        
        [1909.04164] Knowledge Enhanced Contextual Word Representations
        
        Tags:
        
        Allen Institute for AI (AI2)
        Arxiv Doc
        Contextualized word representations
        EMNLP 2019
        Entities and LM
        Good
        Grounded Language Learning
        KD-MKB biblio
        KG-augmented Language Models
        KnowBert
        Knowledge-augmented Language Models
        Knowledge-driven embeddings
        Knowledge Graphs in NLP
        Multiple Knowledge Bases
        
        General method to **embed multiple knowledge bases into pre-trained language models** (KB in the sense as fixed collection of entity nodes) > The key idea is to explicitly model entity spans in the input text and use an **entity linker** to retrieve relevant entity embeddings from a KB to form knowledge enhanced entity-span representations. > Then, update contextual word representations via a form of **word-to-entity attention**. > In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that **combines a small amount of entity linking supervision with a large amount of raw text**.
        
        2020-05-13 About
        
        [1907.04829] BAM! Born-Again Multi-Task Networks for Natural Language Understanding
        
        Tags:
        
        Arxiv Doc
        Chris Manning
        KD-MKB biblio
        Knowledge distillation
        Multitask Learning in NLP
        Quoc V. Le
        
        > **knowledge distillation where single-task models teach a multi-task model.** We enhance this training with **teacher annealing**, a novel method that gradually transitions the model from distillation to supervised learning, helping the multi-task model surpass its single-task teachers
        
        2020-05-12 About
        
        [1807.08447] LinkNBed: Multi-Graph Representation Learning with Entity Linkage
        
        Tags:
        
        AI@Amazon
        Arxiv Doc
        Combining knowledge graphs
        Graph Embeddings
        KD-MKB biblio
        
        > a deep relational learning framework that **learns entity and relationship representations across multiple graphs**. We identify entity linkage across graphs as a vital component to achieve our goal. We design a novel objective that leverage entity linkage and build an efficient multi-task training procedure > > We posit that **combining graph alignment task with deep representation learning across multi-relational graphs** has potential to induce a synergistic effect on both tasks
        
        2020-05-11 About
        
        Bootstrapping Entity Alignment with Knowledge Graph Embedding | IJCAI
        
        Tags:
        
        Entity alignment
        IJCAI
        KD-MKB biblio
        Knowledge Graph Embeddings
        
        Embedding-based entity alignment: finds entity alignment by measuring the similarities between entity embeddings > Existing approaches are challenged by the lack of enough prior alignment as labeled training data. > A bootstrapping approach: it iteratively labels likely entity alignment as training data for learning alignment-oriented KG embeddings. [GitHub](https://github.com/nju-websoft/BootEA)
        
        2020-05-11 About
        
        [1706.00384] Deep Mutual Learning
        
        Tags:
        
        Arxiv Doc
        KD-MKB biblio
        Knowledge distillation
        Mutual Learning
        
        > In this paper we explore a different but related idea to model distillation – that of mutual learning. Distillation starts with a powerful large and pre-trained teacher network and performs one-way knowledge transfer to a small untrained student. In contrast, in mutual learning we start with a pool of untrained students who learn simultaneously to solve the task together. [critic here](doc:2020/06/1804_03235_large_scale_distri): > Zhang et al. (2017) reported a benefit in quality over basic distillation, but they compare distilling model M1 into model M2 with training model M1 and model M2 using codistillation; they do not compare to distilling an ensemble of models M1 and M2 into model M3. > > ... > > we can achieve the 70.7% they report for online distillation using traditional offline distillation.
        
        2020-05-11 About
        
        [1906.07241] Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling
        
        Tags:
        
        Allen Institute for AI (AI2)
        Arxiv Doc
        KD-MKB biblio
        KG-augmented Language Models
        Knowledge Graphs and NLP
        
        > a **neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context**. These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. > The KGLM maintains a dynamically growing local knowledge graph, a subset of the knowledge graph that contains entities that have already been mentioned in the text, and their related entities.
        
        2020-05-11 About
        
        [1902.10197] RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
        
        Tags:
        
        Arxiv Doc
        KD-MKB biblio
        Knowledge Graph Completion
        Knowledge Graph Embeddings
        RotatE
        
        > We study the problem of learning representations of entities and relations in knowledge graphs for predicting missing links.
        
        2020-03-03 About
        
        Knowledge Graph Embedding: A Survey of Approaches and Applications - IEEE (2017)
        
        Tags:
        
        KD-MKB biblio
        Knowledge Graph Embeddings
        Survey / Review
        
        - "facts only" based embeddings - Translational Distance Models, distance-based scoring functions - TransE and co: entities and relations as points in vector space - TransH: relation-specific hyperplanes. - Gaussian Embeddings: entities and relations as random variables - Semantic matching models, similarity-based scoring functions - RESCAL (entities as a vector, relation as a matrix) and familiy - DistMult (relation as diagonal matrix) - Holographic Embeddings (HolE) - etc. - Matching with Neural Networks - SME, etc - incorporating other info: entity types, relation paths, textual descriptions, as well as logical rules - entity types - semantically smooth embedding (SSE): entities of the same type to stay close to each other in the embedding space - type-embodied knowledge representation learning (TKRL), which can handle hierarchical entity categories and multiple category labels - Applications in downstream tasks
        
        2019-05-27 About
        
        [1901.11504] Multi-Task Deep Neural Networks for Natural Language Understanding
        
        Tags:
        
        Arxiv Doc
        BERT
        KD-MKB biblio
        Multi-task learning
        NLP@Microsoft
        NLU
        Pre-Trained Language Models
        
        outperforms BERT in nine of eleven benchmark NLP tasks
        
        2019-02-17 About
      - KD-MKB paper
        
        Knowledge Base Embedding By Cooperative Knowledge Distillation - ACL Anthology
        
        Tags:
        
        KD-MKB paper
        
        2020-12-05 About
      - KD-MKB related
        
        [1912.08422] Distilling Structured Knowledge into Embeddings for Explainable and Accurate Recommendation
        
        Tags:
        
        AI@Facebook
        Arxiv Doc
        Explainable AI
        KD-MKB related
        Knowledge distillation
        Recommender Systems
        
        distilling structured knowledge from a differentiable path-based recommendation model. > proposed framework can achieve state-of-the-art recommendation performance and meanwhile provide interpretable recommendation reasons
        
        2020-05-12 About
- Knowledge Graphs and NLP
  - BERT + KB
    - BLP
      - CKB
        
        raphaelsty/ckb: Contextual knowledge bases
        
        Tags:
        
        CKB
        GitHub project
        Raphaël Sourty
        Semanlink Tag Finder
        
        Une implémentation de [BLP](tag:blp) [[2010.03496] Inductive Entity Representations from Text via Link Prediction](doc:2020/11/2010_03496_inductive_entity_r)
        
        2020-11-09 About
    - KnowBert
    - LUKE
  - Knowledge Graphs in NLP
    - BERT + KB
    - ERNIE
    - KG-augmented Language Models
    - KnowBert
    - Microsoft Concept Graph
  - LM + KG
    - BERT + KB
    - LLM + KG
      - LLM generated Knowledge Graphs
        
        GraphRAG: Unlocking LLM discovery on narrative private data - Microsoft Research
        
        Tags:
        
        LLM generated Knowledge Graphs
        Microsoft Research
        NLP@Microsoft
        RAG + KG
        
        > GraphRAG uses **LLM generated knowledge graphs** to provide substantial improvements in question-and-answer performance when conducting document analysis of complex information. > power of **prompt augmentation** when performing **discovery** on private datasets (data that the LLM is not trained on and has never seen before, such as an enterprise’s proprietary research, business documents..) > GraphRAG uses the LLM to **create a knowledge graph based on the private dataset**. This graph is then used alongside graph machine learning to perform **prompt augmentation** at query time. > the GraphRAG approach [can] **discover entities in the query**. This allows the LLM to ground itself in the graph and results in superior answer that contains provenance through links to the original supporting text GraphRAG can answer queries such as "**what are the top five themes in the data?**"
        
        2024-03-07 About
      - RAG + KG
        
        Enhancing RAG-based application accuracy by constructing and leveraging knowledge graphs
        
        Tags:
        
        LangChain
        Neo4j
        RAG + KG
        
        2024-03-16 About
        
        GraphRAG: Unlocking LLM discovery on narrative private data - Microsoft Research
        
        Tags:
        
        LLM generated Knowledge Graphs
        Microsoft Research
        NLP@Microsoft
        RAG + KG
        
        > GraphRAG uses **LLM generated knowledge graphs** to provide substantial improvements in question-and-answer performance when conducting document analysis of complex information. > power of **prompt augmentation** when performing **discovery** on private datasets (data that the LLM is not trained on and has never seen before, such as an enterprise’s proprietary research, business documents..) > GraphRAG uses the LLM to **create a knowledge graph based on the private dataset**. This graph is then used alongside graph machine learning to perform **prompt augmentation** at query time. > the GraphRAG approach [can] **discover entities in the query**. This allows the LLM to ground itself in the graph and results in superior answer that contains provenance through links to the original supporting text GraphRAG can answer queries such as "**what are the top five themes in the data?**"
        
        2024-03-07 About
        
        Knowledge Graph RAG Query Engine - LlamaIndex
        
        Tags:
        
        LlamaIndex
        LLM + KG
        RAG (Retrieval-Augmented Generation)
        RAG + KG
        
        > Typically, this is to build context based on entities’ SubGraph related to the task.
        
        2023-08-17 About
        
        Jerry Liu on X: “Graph RAG” in @llama_index: a new method of augmenting LLMs with context from a graph database
        
        Tags:
        
        Graph database
        Jerry Liu
        LlamaIndex
        LLM + KG
        RAG (Retrieval-Augmented Generation)
        RAG + KG
        Tweet
        
        [Knowledge Graph RAG Query Engine - LlamaIndex](doc:2023/08/knowledge_graph_rag_query_engin)
        
        2023-08-17 About
        
        Knowledge Graphs & LLMs: Fine-Tuning Vs. Retrieval-Augmented Generation | by Tomaz Bratanic | Neo4j Developer Blog (June 6 2023)
        
        Tags:
        
        LLM + KG
        LLM Fine-tuning vs RAG
        LM Fine-tuning
        Neo4j
        RAG (Retrieval-Augmented Generation)
        RAG + KG
        
        > What are the limitations of LLMs, and how to overcome them? > We haven’t found any recent articles describing **how to use a knowledge graph to prepare good question-answer pairs that can be used to fine-tune an LLM**. This is an area that we plan to explore during the [NaLLM project](doc:2023/08/github_neo4j_nallm_repositor) > we recommend updating the internal knowledge of an LLM through fine-tuning techniques only for slowly changing or updating data.
        
        2023-08-13 About
    - LLMs and basic Renault Knowledge
  - SPIKE-KBC
  - Text, KG and embeddings
    - Text-Aware KG embedding
      - BLP
      - DKRL
        
        Representation learning of knowledge graphs with entity descriptions (AAAI 2016)
        
        Tags:
        
        DKRL
        Text-Aware KG embedding
        Zero shot
        
        "Description-Embodied Knowledge Representation Learning" (DKRL) > In most knowledge graphs there are usually concise descriptions for entities, which cannot be well utilized by existing methods... Experimental results on real-world datasets show that, our method outperforms other baselines on the knowledge graph completion and entity classification tasks, especially under the zero-shot setting, which indicates that **our method is capable of building representations for novel entities according to their descriptions**. [Source code on github](https://github.com/xrb92/DKRL). For fact triples: TransE. Meanwhile, given an entity we will also learn to maximize the likelihood of predicting its description (using either CBOW or CNN encoder) (head + relation = tail, also in "text space") Two types of representations for entities: structure-based representations and description-based representations. They are learned simultaneously into the same vector space but not forced to be unified **so that novel entities with only descriptions can be represented**.
        
        2020-10-02 About
      - LUKE
        
        [2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
        
        Tags:
        
        Arxiv Doc
        BERT + KB
        EMNLP 2020
        Entity embeddings
        Ikuya Yamada
        LUKE
        Self-Attention
        Text-Aware KG embedding
        
        > LUKE is based on bidirectional Transformer, treats words and entities in a text as independent tokens, and outputs contextualized representations of them. The representations can be used to address downstream tasks similarly to BERT. [src](https://twitter.com/ikuyamada/status/1312947499141750786) > LUKE is trained using a novel pretraining task that involves predicting randomly masked words (equivalent to BERT’s masked language model) and entities in an entity-annotated corpus obtained from Wikipedia. (Hum, ça me rappelle quelque chose) > LUKE also uses a new *entity-aware* self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores. [github](https://github.com/studio-ousia/luke), [at Hugging Face](https://twitter.com/AkariAsai/status/1389428550298525696), [doc](https://huggingface.co/transformers/model_doc/luke.html), [tweet](https://twitter.com/ikuyamada/status/1392742990586683392?s=20)
        
        2020-11-26 About
        
        Text-Enhanced Representation Learning for Knowledge Graph (IJCAI 2016)
        
        Tags:
        
        IJCAI
        Text-Aware KG embedding
        
        2020-11-03 About
        
        [2010.03496] Inductive Entity Representations from Text via Link Prediction
        
        Tags:
        
        Arxiv Doc
        Attention + Knowledge Graphs
        BLP
        Discuté avec Raphaël
        Entity embeddings
        Good
        Link Prediction
        Text-Aware KG embedding
        TheWebConf 2021
        
        BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training) > a method for **learning representations of entities**, that uses a **pre-trained Transformer** based architecture as an entity encoder, and **link prediction training on a knowledge graph with textual entity descriptions**. > using entity descriptions, an entity encoder is trained for link prediction in a knowledge graph. The encoder can then be used without fine-tuning to obtain features for entity classification and information retrieval Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their objective targeted exclusively for link prediction (and not an objective that combines language modeling and link prediction as Kepler) performs better than Kepler's more complex one.
        
        2020-11-03 About
        
        SSP: Semantic Space Projection for Knowledge Graph Embedding with Text Descriptions (AAAI 2017)
        
        Tags:
        
        Roosevelt
        Text-Aware KG embedding
        
        > jointly learns from the symbolic triples and textual descriptions > The data involved in our model are the knowledge triples and the textual descriptions of entities. In experiments, we adopt the “entity descriptions” of Freebase and the textual definitions of Wordnet as textual information. Obvious but very good remark about link prediction in facts-only KG: > the triple (Anna Roosevelt, Parents, Franklin Roosevelt), indicates “Franklin Roosevelt” is the parent of “Anna Roosevelt”. However, it’s quite difficult to infer this fact merely from other symbolic triples.
        
        2020-10-02 About
        
        Representation learning of knowledge graphs with entity descriptions (AAAI 2016)
        
        Tags:
        
        DKRL
        Text-Aware KG embedding
        Zero shot
        
        "Description-Embodied Knowledge Representation Learning" (DKRL) > In most knowledge graphs there are usually concise descriptions for entities, which cannot be well utilized by existing methods... Experimental results on real-world datasets show that, our method outperforms other baselines on the knowledge graph completion and entity classification tasks, especially under the zero-shot setting, which indicates that **our method is capable of building representations for novel entities according to their descriptions**. [Source code on github](https://github.com/xrb92/DKRL). For fact triples: TransE. Meanwhile, given an entity we will also learn to maximize the likelihood of predicting its description (using either CBOW or CNN encoder) (head + relation = tail, also in "text space") Two types of representations for entities: structure-based representations and description-based representations. They are learned simultaneously into the same vector space but not forced to be unified **so that novel entities with only descriptions can be represented**.
        
        2020-10-02 About
        
        [1910.12507] A Survey on Knowledge Graph Embeddings with Literals: Which model links better Literal-ly?
        
        Tags:
        
        Arxiv Doc
        Knowledge Graph Embeddings
        Survey / Review
        Text-Aware KG embedding
        
        2020-05-04 About
        
        Iterative Entity Alignment with Improved Neural Attribute Embedding
        
        Tags:
        
        Entity alignment
        Entity embeddings
        Text-Aware KG embedding
        
        2020-04-29 About
        
        A Comprehensive Survey of Knowledge Graph Embeddings with Literals: Techniques and Applications
        
        Tags:
        
        Knowledge Graph Embeddings
        Survey / Review
        Text-Aware KG embedding
        
        > survey of the KG embedding models which consider the structured information of the graph as well as the unstructured information in form of literals such as text, numerical values etc A le mérite de poser la question de l'utilisation de littéraux dans les embeddings de KG [Newer and longer version](/doc/2020/05/1910_12507_a_survey_on_knowle)
        
        2020-04-29 About
        
        [2001.09522] TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network
        
        Tags:
        
        Arxiv Doc
        Discuté avec Raphaël
        Graph neural networks
        Knowledge Graph Completion
        Microsoft Research
        NLP@Microsoft
        Taxonomies
        Taxonomy expansion task
        Text-Aware KG embedding
        TheWebConf 2020
        
        how to add a set of new concepts to an existing taxonomy. [Tweet](https://twitter.com/mickeyjs6/status/1253772146142216194?s=20) [GitHub](https://github.com/mickeystroller/TaxoExpan) > we study the taxonomy expansion task: given an existing taxonomy and a set of new emerging concepts, we aim to automatically expand the taxonomy to incorporate these new concepts (without changing the existing relations in the given taxonomy). > To the best of our knowledge, this is the first study on **how to expand an existing directed acyclic graph (as we model a taxonomy as a DAG) using self-supervised learning**. Self-supervised framework, the existing taxonomy being used as training data: it learns a model to predict whether a query concept is the direct hyponym of an anchor concept. > 2 techniques: > > 1. a **position-enhanced graph neural network that encodes the local structure of an anchor concept** in the existing taxonomy, > 2. a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data. Regarding 1: uses [GNN](/tag/graph_neural_networks.html) to model the "ego network" of concepts (potential “siblings” and “grand parents” of the query concept). > Regular GNNs fail to distinguish nodes with different relative positions to the query (i.e., some nodes are grand parents of the query while the others are siblings of the query). To address this limitation, we present a simple but effective enhancement to inject such position information into GNNs using position embedding. We show that such embedding can be easily integrated with existing GNN architectures (e.g., [GCN](/tag/graph_convolutional_networks) and GAT) and significantly boosts the prediction performance Regarding point 2: uses InfoNCE loss, cf. [Contrastive Predictive Coding](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1807.03748) > Instead of predicting whether each individual ⟨query concept, anchor concept⟩ pair is positive or not, we first group all pairs sharing the same query concept into a single training instance and learn a model to select the positive pair among other negative ones from the group. (Hum, ça me rappelle quelque chose) > assume each concept (in existing taxonomy + set of new concepts) has an initial embedding vector learned from some text associated with this concept. To keep things tractable, only attempts to find a single parent node of each new concept.
        
        2020-04-25 About
        
        Reasoning With Neural Tensor Networks for Knowledge Base Completion (2013)
        
        Tags:
        
        Andrew Ng
        Chris Manning
        Knowledge Graph + Deep Learning
        Knowledge Graph Completion
        Richard Socher
        Text-Aware KG embedding
        
        **Predicting the likely truth of additional facts based on existing facts in the knowledge base.** > we introduce an expressive neural tensor network suitable for reasoning over relationships between two entities. Most similar work: [Bordes et al.](http://127.0.0.1:8080/semanlink/doc/2019/08/learning_structured_embeddings_) (2011) Contributions: 1. new neural tensor network (**NTN**) suitable for reasoning over relationships between two entities. Generalizes several previous neural network models and provides a more powerful way to model relational information than a standard neural network layer. 2. a new way to represent entities in knowledge bases, as the average of their constituting word vectorss, allowing the sharing of statistical strength between the words describing each entity (e.g., Bank of China and China). 3. incorporation of word vectors which are trained on large unlabeled text > We **learn to modify word representations via grounding in world knowledge**. This essentially allows us to analyze word embeddings and query them for specific relations. Furthermore, the resulting vectors could be used in other tasks such as named entity recognition or relation classification in natural language **Makes use of entity name**: NTN first learns word vectors from an auxiliary news corpus, and then initializes the representation of each entity by averaging the vectors of words contained in its name. For example, **the embedding of AlfredHitchcock is initialized by the average word vectors of “alfred” and “hitchcock”**... This kind of methods model textual information separately from KG facts, and hence fail to leverage interactions between them.[src](doc:2019/05/knowledge_graph_embedding_a_su)
        
        2019-08-03 About
    - Using literal descriptions of entities
    - Word + Entity embeddings
      - Wikipedia2Vec
        
        [1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification
        
        Tags:
        
        Arxiv Doc
        Entities
        Entity salience
        Good
        Ikuya Yamada
        Text Classification
        Transformers
        Wikipedia2Vec
        
        A model that performs **text classification using entities in a knowledge base**. > Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**. 2 steps: 1. Entity detection 2. Classification using the detected entities (+text) as inputs Regarding entity linking, a local model which uses cosine similarity between the embedding of the target entity and the word-based representation of the document to capture the relevance of an entity given a document. Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities close to one another in a unified vector space) Model using attention, with 2 features : - cosine similarity between the embedding of the entity and the word based representation of the document - the probability that the entity name refers to the entity in KB. Somewhat [related](doc:2020/01/investigating_entity_knowledge_) ### Conclusion: >a neural network model that performs text classification using entities in Wikipedia. We combined simple dictionary-based entity detection with a neural attention mechanism to enable the model to focus on a small number of unambiguous and relevant entities in a document.
        
        2020-09-02 About
        
        [1812.06280] Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia
        
        Tags:
        
        Arxiv Doc
        Ikuya Yamada
        Wikipedia2Vec
        
        2020-09-02 About
        
        [1601.01343] Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation
        
        Tags:
        
        Arxiv Doc
        Entity linking
        Ikuya Yamada
        Named Entity Recognition
        Thèse IRIT-Renault: biblio initiale
        Wikipedia2Vec
        Word + Entity embeddings
        
        > An embedding method specifically **designed for NED** that jointly **maps words and entities into the same continuous vector space**. > We extend the skip-gram model by using two models. The KB graph model learns the relatedness of entities using the link structure of the KB, whereas the anchor context model aims to align vectors such that similar words and entities occur close to one another in the vector space by leveraging KB anchors and their context words Technique later used in [Wikipedia2Vec](doc:?uri=https%3A%2F%2Fwikipedia2vec.github.io%2Fwikipedia2vec%2F), by the same team. [Neural Attentive Bag-of-Entities Model for Text Classification](doc:2020/09/1909_01259_neural_attentive_b) uses Wikipedia2Vec model.
        
        2019-01-27 About
        
        Wikipedia2Vec
        
        Tags:
        
        Wikipedia2Vec
        Word + Entity embeddings
        
        [GitHub](https://github.com/wikipedia2vec/wikipedia2vec) [Paper](doc:2020/09/1812_06280_wikipedia2vec_an_)
        
        2018-12-23 About
  - Text to KG
- Knowledge mining
- Language Models + Knowledge
  - Entities and LM
    - LUKE
    - Masked entity prediction
    - Transformer-Based Entity Linking
      - BERT-based entity linking
        
        [1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval
        
        Tags:
        
        Arxiv Doc
        BERT-based entity linking
        BLINK
        Entity Retrieval
        NLP@Facebook
        Retriever-Reader
        Zero-shot Entity Linking
        
        > a two stage approach, based on fine-tuned BERT architectures. In the first stage, we do retrieval in a dense space defined by a bi-encoder that independently embeds the mention context and the entity descriptions (Humeau et al., 2019; Gillick et al., 2019). Each retrieved candidate is then examined more carefully with a cross-encoder that concatenates the mention and entity text,
        
        2020-05-02 About
  - Knowledge-augmented Language Models
    - KG-augmented Language Models
    - Retrieval augmented LM
      - ColBERT
        
        [2404.13950] SPLATE: Sparse Late Interaction Retrieval
        
        Tags:
        
        Arxiv Doc
        ColBERT
        Sparse retrieval
        SPLADE
        
        Adapts the ColBERTv2 model to map its embeddings to a sparse space, enabling efficient sparse retrieval for candidate generation in the late interaction paradigm.
        
        2024-04-23 About
        
        ColBERT gist:c1182551fa609736d47df4af82f7c5ab
        
        Tags:
        
        ColBERT
        Sample code
        
        > a quick gist that does synthetic data gen, fine-tuning, eval. Just add your own documents, or try it on a PG essay. @JoshPurtell
        
        2024-03-08 About
        
        ColBERT Inference in the Browser
        
        Tags:
        
        ColBERT
        Demo
        Vespa
        
        Demo of ColBERT query-passage scoring interpretability - try with the following: "what are the mentioned EICPS?" and passage "There is a security risk related to EICPS 67" - MaxSim Score: 20.71 - Estimated Relevance: 64.71% - highlights: There related - then "what are the mentioned animals?" and "There is a security risk related to lions" - MaxSim Score: 9.18 - Estimated Relevance: 28.68% - highlights: related lions ``` Effects of climate change on marine ecosystems MaxSim Score: 27.90 Estimated Relevance: 87.17% Effects of global warming on marine ecosystems MaxSim Score: 24.62 Estimated Relevance: 76.94% Effects of global warming on life in the oceans MaxSim Score: 19.64 Estimated Relevance: 61.39% Effects of global warming on life on Mars MaxSim Score: 13.65 Estimated Relevance: 42.65% ```
        
        2024-03-08 About
        
        Announcing Vespa Long-Context ColBERT | Vespa Blog
        
        Tags:
        
        @jobergum
        ColBERT
        Long documents
        Vespa
        
        2024-03-03 About
        
        Omar Khattab sur X : "ColBERT in 81 languages by generalizing from English training! ..."
        
        Tags:
        
        ColBERT
        Multilingual NLP
        Omar Khattab
        Tweet
        
        2024-02-28 About
        
        Omar Khattab sur X : "A thread on late interaction, how it works efficiently, and why/where it's been shown to improve quality"
        
        Tags:
        
        ColBERT
        Omar Khattab
        Twitter thread
        
        2024-02-05 About
        
        Craig Macdonald sur X : "a thread of our main ColBERT research findings"
        
        Tags:
        
        ColBERT
        Twitter thread
        
        2024-02-01 About
        
        long document ranking · Issue #11 · stanford-futuredata/ColBERT
        
        Tags:
        
        ColBERT
        Long documents
        
        [Instructions on using ColBERT](https://github.com/stanford-futuredata/ColBERT/issues/73) passage and query length, etc
        
        2024-01-31 About
        
        Exploring ColBERT with RAGatouille | Simon Willison’s TILs
        
        Tags:
        
        ColBERT
        RAGatouille
        
        2024-01-29 About
        
        Jo Kristian Bergum sur X : (on "why using ColBERT?")
        
        Tags:
        
        @jobergum
        ColBERT
        Twitter thread
        
        > The idea that you can accurately boil down the nuances of ~256 tokens (2/3rds of a page) into a single vector is a pretty wild proposition > Inspired by @lateinteraction , we hacked into the ColBERT model's contextualized late-interaction similarities to produce [interpretable snippets](https://x.com/jobergum/status/1750282246072746178?s=20)!
        
        2024-01-28 About
        
        bclavie/RAGatouille
        
        Tags:
        
        Benjamin Clavié
        ColBERT
        GitHub project
        RAG (Retrieval-Augmented Generation)
        RAGatouille
        
        > RAGatouille's purpose is make it easy to use state-of-the-art methods in your RAG pipeline, without having to worry about the details or the years of literature! At the moment, RAGatouille focuses on making ColBERT simple to use. [Using ColBERT in-memory: Index-Free Encodings & Search](https://github.com/bclavie/RAGatouille/blob/0.0.5b1/examples/06-index_free_use.ipynb) ``` from ragatouille import RAGPretrainedModel RAG = RAGPretrainedModel.from_pretrained "colbert-ir/colbertv2. 0" ) # Your documents, a plain old list of chunked strings. documents = [...] # In-memory indexing supports metadata too! meta = ['attribute': ' really cool value'}...] # All the magic happens here RAG.encode documents, document_metadatas=meta) # Query your in-memory index RAG. search_encoded_docs(query = "A great question", k=3) # All further encode() calls add to the existing documents... RAG.encode(extra_documents, document_metadatas=extra_meta) # ... until you clear them RAG.clear_encoded ```
        
        2024-01-26 About
        
        Omar Khattab sur X : "A🧵on beating the hardware lottery for retrieval: the internals of the late interaction stack. ColBERT..."
        
        Tags:
        
        ColBERT
        Omar Khattab
        Twitter thread
        
        2023-12-29 About
        
        raphaelsty/neural-cherche: Neural Search
        
        Tags:
        
        Cherche (Raphaël)
        ColBERT
        GitHub project
        Neural Search
        Raphaël Sourty
        SPLADE
        
        > a library to fine-tune neural search models such as Splade, ColBERT, and SparseEmbed on a specific dataset
        
        2023-11-17 About
        
        Omar Khattab sur X : "This isn't the main point of this great new paper by @sophiaalthammer et al. But it's incredible how ColBERT at 1000 training queries is better than DPR trained at *50,000* queries!"
        
        Tags:
        
        ColBERT
        Omar Khattab
        Tweet
        
        mentions [Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection](doc:2023/09/2309_06131_annotating_data_fo) > why is ColBERT so data-efficient? > Answer: > > - ColBERT neither needs to learn how to condense each document (unlike DPR) > - nor how to do matching (unlike MonoBERT). > - Just needs to learn contextual term representations—a much lower burden on the encoders.
        
        2023-09-14 About
        
        LangChain Retrieval Webinar - YouTube
        
        Tags:
        
        ColBERT
        DSPy (Demonstrate-Search-Predict)
        LangChain
        Omar Khattab
        Webinar
        YouTube video
        
        [ColBERT](tag:colbert) retrieval model and the [DSP](doc:2023/02/stanfordnlp_dsp_𝗗𝗦𝗣_demons) programming model
        
        2023-05-27 About
        
        [2304.01982] Rethinking the Role of Token Retrieval in Multi-Vector Retrieval
        
        Tags:
        
        Arxiv Doc
        ColBERT
        Multi-Vector Retrieval
        Neural Search
        NLP@Google
        
        > Multi-vector retrievers like [ColBERT](tag:colbert) are powerful, but they come at the cost of complicated inference. In this paper, we ask: "can token retrieval alone achieve great performance in multi-vector retrieval?" [tweet](https://twitter.com/leejnhk/status/1643632578824396805?s=20) > The key insight of XTR is that the token-retrieval in multi-vector models should be **trained to retrieve the most salient and informative document tokens**, so that the score between a query and document can be computed using only the retrieved information, just like how single-vector retrieval models work > This is an *amazing* way to re-engineer the scoring mechanism of late interaction / ColBERT retrievers! [src: ColBERT's author Omar Khattab](https://twitter.com/lateinteraction/status/1643439889902637056?s=20) - scoring using only retrieved document terms - imputing missing token scores using their upper bound
        
        2023-04-05 About
        
        Omar Khattab sur Twitter : "New features in 𝗖𝗼𝗹𝗕𝗘𝗥𝗧 ‣ Index Updater. Simple API to add/remove documents..."
        
        Tags:
        
        ColBERT
        Tweet
        
        2023-01-30 About
        
        [2203.13088] Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction
        
        Tags:
        
        Arxiv Doc
        ColBERT
        
        [tweet](https://twitter.com/s_hofstaetter/status/1508803785317138435)
        
        2022-03-30 About
        
        Integrate ORQA and REALM for Open Domain Question Answering · Issue #312 · deepset-ai/haystack
        
        Tags:
        
        ColBERT
        Haystack
        REALM
        
        mntions [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
        
        2022-01-12 About
        
        [2004.12832] ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
        
        Tags:
        
        Arxiv Doc
        ColBERT
        
        (The 1st Colbert paper)
        
        2022-01-12 About
        
        [2007.00814] Relevance-guided Supervision for OpenQA with ColBERT
        
        Tags:
        
        Arxiv Doc
        ColBERT
        
        Colbert-QA
        
        2022-01-07 About
        
        [2112.01488] ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
        
        Tags:
        
        Arxiv Doc
        ColBERT
        Domain specific corpora
        
        2021-12-05 About
        
        Building Scalable, Explainable, and Adaptive NLP Models with Retrieval | SAIL Blog
        
        Tags:
        
        ColBERT
        Good
        NLP@Stanford
        Not Encoding Factual Knowledge in Language Model
        Retrieval augmented LM
        Retrieval-based NLP
        
        > The black-box nature of large language models like T5 and GPT-3 makes them inefficient to train and deploy, opaque in their knowledge representations and in backing their claims with provenance, and static in facing a constantly evolving world and diverse downstream contexts. **This post explores retrieval-based NLP, where models retrieve information pertinent to solving their tasks from a plugged-in text corpus**. > > Retrieval-based NLP methods view tasks as “open-book” exams: knowledge is encoded explicitly in the form of a text corpus like Wikipedia, the medical literature, or a software’s API documentation. When solving a language task, **the model learns to search for pertinent passages** and to then use the retrieved information for crafting knowledgeable responses. In doing so, **retrieval helps decouple the capacity that language models have for understanding text from how they store knowledge**
        
        2021-10-07 About
        
        A Moderate Proposal for Radically Better AI-powered Web Search
        
        Tags:
        
        ColBERT
        Neural Search
        NLP@Stanford
        Web search
        
        2021-07-10 About
      - DSPy (Demonstrate-Search-Predict)
        
        NER using DSPy
        
        Tags:
        
        DSPy (Demonstrate-Search-Predict)
        Named Entity Recognition
        
        2024-04-12 About
        
        Leonie sur X : "Ollama allows you to run open source LLMs LOCALLY..."
        
        Tags:
        
        DSPy (Demonstrate-Search-Predict)
        Mistral
        Open-source LLM
        Weaviate
        
        Ollama: good name, anyway
        
        2024-03-30 About
        
        DSPy Cheatsheet | DSPy
        
        Tags:
        
        Cheat sheet
        DSPy (Demonstrate-Search-Predict)
        
        2024-03-30 About
        
        Krista Opsahl-Ong sur X : "Got a pipeline with **multiple prompts**, like a DSPy program? ... Introducing MIPRO, a Multi-prompt Instruction Proposal Optimizer...."
        
        Tags:
        
        DSPy (Demonstrate-Search-Predict)
        Twitter thread
        
        2024-03-09 About
        
        Intro to DSPy: Goodbye Prompting, Hello Programming! | by Leonie Monigatti | Feb, 2024 | Towards Data Science
        
        Tags:
        
        DSPy (Demonstrate-Search-Predict)
        
        2024-03-01 About
        
        Omar Khattab sur X : "...Let's use 30 lines of DSPy—without any hand-written prompts or any calls to OpenAI ($0)—to teach..."
        
        Tags:
        
        DSPy (Demonstrate-Search-Predict)
        Omar Khattab
        Prompts
        Sample code
        Twitter thread
        
        2024-01-01 About
        
        Inside DSPy: The New Language Model Programming Framework You Need… – Towards AI
        
        Tags:
        
        DSPy (Demonstrate-Search-Predict)
        
        2023-09-06 About
        
        Omar Khattab sur X : "Announcing 𝗗𝗦𝗣y..."
        
        Tags:
        
        DSPy (Demonstrate-Search-Predict)
        Omar Khattab
        Tweet
        
        2023-08-24 About
        
        [2212.14024] Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
        
        Tags:
        
        Arxiv Doc
        DSPy (Demonstrate-Search-Predict)
        Omar Khattab
        
        2023-06-23 About
        
        Jerry Liu sur Twitter : "The DSP project carries a lot of insights for improving RAG..."
        
        Tags:
        
        DSPy (Demonstrate-Search-Predict)
        J'y étais
        Jerry Liu
        Omar Khattab
        RAG (Retrieval-Augmented Generation)
        Tweet
        Webinar
        
        > - value of few-shot ex’s > - declarative modules > - compile an optimized system with distilled LM’s
        
        2023-06-18 About
        
        LangChain Retrieval Webinar - YouTube
        
        Tags:
        
        ColBERT
        DSPy (Demonstrate-Search-Predict)
        LangChain
        Omar Khattab
        Webinar
        YouTube video
        
        [ColBERT](tag:colbert) retrieval model and the [DSP](doc:2023/02/stanfordnlp_dsp_𝗗𝗦𝗣_demons) programming model
        
        2023-05-27 About
        
        stanfordnlp/dspy: 𝗗𝗦𝗣: Demonstrate-Search-Predict. A framework for composing retrieval and language models for knowledge-intensive NLP.
        
        Tags:
        
        DSPy (Demonstrate-Search-Predict)
        GitHub project
        Knowledge-Intensive NLP Tasks
        LM + Search
        NLP@Stanford
        NLP based IR
        Omar Khattab
        Prompting/In-context learning
        Retriever-Reader
        
        (initially called DSP, rebranded as DSPy) > The DSP framework provides a programming abstraction for building grounded AI systems. In a few lines of code, a DSP program expresses rich interactions between retrieval models (RMs) and language models (LMs) to tackle difficult knowledge-intensive NLP tasks (e.g., complex question answering or conversational search). > DSP discourages ["prompt engineering"](tag:prompted_models), which we view much the same way as hyperparameter tuning in traditional ML [@matei_zaharia](https://twitter.com/matei_zaharia/status/1626705622585716737?s=20): >Who are the World Cup champions? I knew ChatGPT would get it wrong when it launched, but it's surprising that all the new search+LLM engines do too. > > **Combining retrieval+LMs won't just be a matter of prompting**. That's why we've been building tools like DSP at Stanford to do it.
        
        2023-02-18 About
      - REALM
        
        [2207.06300] Re2G: Retrieve, Rerank, Generate
        
        Tags:
        
        Arxiv Doc
        Discuté avec Raphaël
        Knowledge distillation
        Michael Glass
        Question Answering
        RAG (Retrieval-Augmented Generation)
        REALM
        Retriever-Reader
        Slot filling
        Zero shot
        
        > Recent models such as RAG and REALM have introduced retrieval into conditional generation. These models incorporate neural initial retrieval from a corpus of passages. We build on this line of research, proposing Re2G, which combines both neural initial retrieval and reranking into a BART-based sequence-to-sequence generation. Our reranking approach also permits merging retrieval results from sources with incomparable scores, enabling an ensemble of BM25 and neural initial retrieval. > > To train our system end-to-end, we introduce a novel variation of knowledge distillation to train the initial retrieval, reranker, and generation using only ground truth on the target sequence output. > > Large gains in four diverse tasks: zero-shot slot filling, question answering, fact checking and dialog, with relative gains of 9% to 34% over the previous SotA on the KILT leaderboard. [Code]( ibm/kgi-slot-filling)
        
        2022-07-14 About
        
        REALM: Retrieval-Augmented Language Model Pre-Training (Paper Explained) - YouTube
        
        Tags:
        
        REALM
        YouTube video
        
        - A new pretraining method - separate language and world knowledge - pre-training is MLM
        
        2022-01-23 About
        
        Integrate ORQA and REALM for Open Domain Question Answering · Issue #312 · deepset-ai/haystack
        
        Tags:
        
        ColBERT
        Haystack
        REALM
        
        mntions [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
        
        2022-01-12 About
        
        [2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
        
        Tags:
        
        Arxiv Doc
        Knowledge-augmented Language Models
        Neural Search
        NLP@Google
        Not Encoding Factual Knowledge in Language Model
        REALM
        Retrieval augmented LM
        Retriever-Reader
        
        **Augment language model pre-training with a retriever module**, which is trained using the masked language modeling objective. > To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. **For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner**, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents Hum, #TODO: parallel to be drawn with techniques in [KG-augmented Language Models](tag:knowledge_graph_augmented_language_models) which focus "on the problem of capturing declarative knowledge in the learned parameters of a language model." [Google AI Blog Post](doc:2020/08/google_ai_blog_realm_integrat) [Summary](https://joeddav.github.io/blog/2020/03/03/REALM.html) for the [Hugging Face awesome-papers reading group](doc:2021/03/huggingface_awesome_papers_pap)
        
        2020-12-12 About
        
        Google AI Blog: REALM: Integrating Retrieval into Language Representation Models
        
        Tags:
        
        Knowledge-augmented Language Models
        Neural Search
        NLP@Google
        REALM
        
        > a new open-source method for language model pre-training that uses a supplemental knowledge retriever that enables it to perform well on knowledge-intensive tasks without billions of parameters. > > The key intuition of REALM is that a retrieval system should improve the model's ability to fill in missing words [Paper: REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
        
        2020-08-13 About
      - Retrieval-based language models
        
        [2108.13934] Robust Retrieval Augmented Generation for Zero-shot Slot Filling
        
        Tags:
        
        Arxiv Doc
        Discuté avec Raphaël
        EMNLP 2021
        Hard negative mining
        Knowledge Extraction
        Michael Glass
        NLP@IBM
        RAG (Retrieval-Augmented Generation)
        Retrieval-based language models
        Slot filling
        Zero shot
        
        > "Knowledge Graph Induction", a system for slot filling based on advanced training strategies for both Dense Passage Retrieval (DPR) and Retrieval Augmented Generation (RAG) see [[1909.04120] Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) (same first author) [GitHub](https://github.com/IBM/kgi-slot-filling)
        
        2022-01-19 About
  - Language Models as Knowledge Bases
  - LM + KG
  - Not Encoding Factual Knowledge in Language Model
    - REALM
    - Retrieval augmented LM
      François Chollet sur Twitter : "I'm pretty optimistic that the LLM reliability / factualness issue can be fixed. The key is to use LLMs as a dialog interface and not as a store of knowledge..."
      
      Tags:
      
      François Chollet
      Knowledge Graphs
      Not Encoding Factual Knowledge in Language Model
      Tweet
      Twitter
      
      > **LLMs as the query layer** between a human user an a knowledge graph with sources (which can be hybrid generated/curated).
      
      2023-01-24 About
      Akari Asai sur Twitter : "Can we solely rely on LLMs’ memories (eg replace search w ChatGPT)? Probably not... Our analysis shows how retrieval is complementary to LLMs’ parametric knowledge..."
      
      Tags:
      
      ChatGPT
      ChatGPT+Search
      Information retrieval
      Language Models + Knowledge
      NLP@Meta
      Not Encoding Factual Knowledge in Language Model
      Tweet
      
      2022-12-21 About
      David Chapman sur Twitter : "AI labs should compete to build the smallest possible language models, which “know” as little as possible..."
      
      Tags:
      
      Not Encoding Factual Knowledge in Language Model
      Tweet
      
      2022-10-01 About
      [2206.06520] Memory-Based Model Editing at Scale
      
      Tags:
      
      Arxiv Doc
      Chris Manning
      Language Model
      Not Encoding Factual Knowledge in Language Model
      
      Editing knowledge of a Language Model without retraining it.
      
      2022-07-07 About
      Building Scalable, Explainable, and Adaptive NLP Models with Retrieval | SAIL Blog
      
      Tags:
      
      ColBERT
      Good
      NLP@Stanford
      Not Encoding Factual Knowledge in Language Model
      Retrieval augmented LM
      Retrieval-based NLP
      
      > The black-box nature of large language models like T5 and GPT-3 makes them inefficient to train and deploy, opaque in their knowledge representations and in backing their claims with provenance, and static in facing a constantly evolving world and diverse downstream contexts. **This post explores retrieval-based NLP, where models retrieve information pertinent to solving their tasks from a plugged-in text corpus**. > > Retrieval-based NLP methods view tasks as “open-book” exams: knowledge is encoded explicitly in the form of a text corpus like Wikipedia, the medical literature, or a software’s API documentation. When solving a language task, **the model learns to search for pertinent passages** and to then use the retrieved information for crafting knowledgeable responses. In doing so, **retrieval helps decouple the capacity that language models have for understanding text from how they store knowledge**
      
      2021-10-07 About
      Hugging Face sur Twitter : "Transformers release of the Retrieval-Augmented Generation model in collaboration with @facebookai!"
      
      Tags:
      
      huggingface / transformers
      NLP@Facebook
      Not Encoding Factual Knowledge in Language Model
      RAG (Retrieval-Augmented Generation)
      Retrieval augmented LM
      Tweet
      
      > the **RAG model is trained end-to-end for retrieval-in-the-loop generation**, a new paradigm that allows a model to go find useful information in a text corpus when generating. **No need to try to encode all of that knowledge in a trillion parameters any more ;)**
      
      2021-02-23 About
      [2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
      
      Tags:
      
      Arxiv Doc
      Knowledge-augmented Language Models
      Neural Search
      NLP@Google
      Not Encoding Factual Knowledge in Language Model
      REALM
      Retrieval augmented LM
      Retriever-Reader
      
      **Augment language model pre-training with a retriever module**, which is trained using the masked language modeling objective. > To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. **For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner**, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents Hum, #TODO: parallel to be drawn with techniques in [KG-augmented Language Models](tag:knowledge_graph_augmented_language_models) which focus "on the problem of capturing declarative knowledge in the learned parameters of a language model." [Google AI Blog Post](doc:2020/08/google_ai_blog_realm_integrat) [Summary](https://joeddav.github.io/blog/2020/03/03/REALM.html) for the [Hugging Face awesome-papers reading group](doc:2021/03/huggingface_awesome_papers_pap)
      
      2020-12-12 About
      [2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision
      
      Tags:
      
      Arxiv Doc
      KG-augmented Language Models
      Memory networks
      NLP@Google
      Not Encoding Factual Knowledge in Language Model
      Question Answering
      Transformers
      
      > We focus on the problem of **capturing declarative knowledge in the learned parameters of a language model**... > Entities as Experts (EaE) can access distinct memories of the entities mentioned in a piece of text; > To understand the motivation for distinct and independent entity representations: A traditional Transformer would need to build an internal representation of Charles Darwin from the words “Charles” and “Darwin”... Conversely, EAE can access a dedicated representation of “Charles Darwin”, which is a memory of all of the contexts in which this entity has previously been mentioned.... Having retrieved and re-integrated this memory it is much easier for EAE to relate the question to the answer > EaE's entity representations are learned directly from text. Correct identification, and representation, of entities is essential to EaE's performance Based on transformer architecture Extension: [Facts as Experts](doc:2020/07/2007_00849_facts_as_experts_)
      
      2020-07-11 About
      [2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
      
      Tags:
      
      AI + Knowledge Bases
      Arxiv Doc
      Google Research
      Knowledge Graph + Deep Learning
      Neural Memory
      NLP@Google
      NN / Symbolic AI hybridation
      Not Encoding Factual Knowledge in Language Model
      
      > a neural language model that includes **an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge.**... **The model can be updated without re-training by manipulating its symbolic representations**. In particular this model allows us to add new facts and overwrite existing ones. > a **neural language model which learns to access information in a symbolic knowledge graph.** > This model builds on the recently-proposed [Entities as Experts](doc:2020/07/2004_07202_entities_as_expert) (EaE) language model (Févry et al., 2020), which extends the same transformer (Vaswani et al., 2017) architecture of BERT (Devlin et al., 2019) with an additional external memory for entities. > > After training EaE, the embedding associated with an entity will (ideally) capture information about the textual context in which that entity appears, and by inference, the entity’s semantic properties > > we include an additional memory called a fact memory, which encodes triples from a symbolic KB. > > This combination results in a neural language model which learns to access information in a the symbolic knowledge graph. TODO: - read again IBM's [Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) ("an effort to avoid encoding general knowledge in the transformer network itself") - compare with [[1907.05242] Large Memory Layers with Product Keys](doc:2019/07/_1907_05242_large_memory_layer) - how does it relate with [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)?
      
      2020-07-09 About
      [1909.04120] Span Selection Pre-training for Question Answering
      
      Tags:
      
      Arxiv Doc
      Knowledge-augmented Language Models
      Language Models + Knowledge
      Memory in deep learning
      Michael Glass
      NLP@IBM
      Not Encoding Factual Knowledge in Language Model
      Question Answering
      Transformers
      
      > a **new pre-training task inspired by reading comprehension** and an **effort to avoid encoding general knowledge in the transformer network itself** Current transformer architectures store general knowledge -> large models, long pre-training time. Better to offload the requirement of general knowledge to a sparsely activated network. "Span selection" as an additional auxiliary task: the query is a sentence drawn from a corpus with a term replaced with a special token: [BLANK]. The term replaced by the blank is the answer term. The passage is relevant as determined by a BM25 search, and answer-bearing (containing the answer term). Unlike BERT’s cloze task, where the answer must be drawn from the model itself, the answer is found in a passage using language understanding. > **We hope to progress to a model of general purpose language modeling that uses an indexed long term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.**
      
      2019-09-18 About
Cognitive computing
- Cognition-as-a-Service
- Cognitive Search
  - Azure Cognitive Search
  - Goldfire
  - Sinequa
Cognitive Search
Domain Knowledge + Deep Learning
- Knowledge Graph + Deep Learning
  A review of some techniques for inclusion of domain‐knowledge into deep neural networks
  Tags:
  2022-05-19 About
  Integrating Domain-Knowledge into Deep Learning (2019)
  Tags:
  2019-03-03 About
Knowledge-based AI
- KBPedia

1 Document (Long List)

Weak Supervision: A New Programming Paradigm for Machine Learning | SAIL Blog

Tags:

2019-03-12 About

Properties

sl:creationDate : 2019-08-07
sl:creationTime : 2019-08-07T01:21:13Z
rdf:type : sl:Tag
skos:altLabel : Domain Knowledge in AI
skos:prefLabel : AI + Knowledge