Semanlink - Graph

Parents:

Graph

Related Tags:

Descendants

Graph-based Semi-Supervised Learning
Graph-based Text Representations
Graph database
- Graph database and NLP
- GraphDB
- Neo4j
- Property Graphs
  - RDF and Property Graphs
    - RDF*
Graph Editor
Graph Parsing
Graphs+Machine Learning
- Attention in Graphs
  - Attention + Knowledge Graphs
    - Graph Transformers
  - Graph Attention Networks
- Graph-based Semi-Supervised Learning
- Graph Embeddings
  - Knowledge Graph Embeddings
    - Convolutional Knowledge Graph Embeddings
    - Dismult
    - Entity type representation
    - Hierarchy-Aware KG Embeddings
      - Poincaré Embeddings
        
        [2010.00402] From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering
        
        Tags:
        
        AI@Google
        AI@Stanford
        Arxiv Doc
        Hierarchical Clustering
        Poincaré Embeddings
        
        > The key idea of our method, HypHC, is showing a direct correspondence from discrete trees to continuous representations (via the hyperbolic embeddings of their leaf nodes) and back (via a decoding algorithm that maps leaf embeddings to a dendrogram), **allowing us to search the space of discrete binary trees with continuous optimization**. Cites [Dasgupta: A cost function for similarity-based hierarchical clustering](https://arxiv.org/abs/1510.05043)
        
        2020-10-03 About
        
        [1910.03524] Beyond Vector Spaces: Compact Data Representation as Differentiable Weighted Graphs
        
        Tags:
        
        Arxiv Doc
        Geometry of language embeddings
        Graphs+Machine Learning
        Poincaré Embeddings
        Vector space model
        Yandex
        
        > In this paper, we aim to eliminate the inductive bias imposed by the embedding space geometry. Namely, we propose to map data into more general non-vector metric spaces: a weighted graph with a shortest path distance. By design, such graphs can model arbitrary geometry with a proper configuration of edges and weights. Our main contribution is PRODIGE (Probabilistic Differentiable Graph Embeddings): a method that learns a weighted graph representation of data end-to-end by gradient descent. [Github](https://github.com/stanis-morozov/prodige)
        
        2019-10-09 About
        
        HyperE: Hyperbolic Embeddings for Entities
        
        Tags:
        
        Entity embeddings
        Poincaré Embeddings
        
        hyperbolic entity embeddings for 100 Wikidata relationships
        
        2018-07-27 About
        
        Tutorial on Poincaré Embeddings (Jupyter Notebook )
        
        Tags:
        
        gensim
        Poincaré Embeddings
        Tutorial
        
        2018-05-20 About
        
        Implementing Poincaré Embeddings | RARE Technologies
        
        Tags:
        
        gensim
        Poincaré Embeddings
        
        2018-05-20 About
        
        [1705.08039] Poincaré Embeddings for Learning Hierarchical Representations
        
        Tags:
        
        Arxiv Doc
        NLP@Facebook
        Poincaré Embeddings
        Thèse IRIT-Renault: biblio initiale
        
        > While complex symbolic datasets often exhibit a latent hierarchical structure, state-of-the-art methods typically learn embeddings in Euclidean vector spaces, which do not account for this property. For this purpose, we introduce a new approach for learning hierarchical representations of symbolic data by embedding them into hyperbolic space
        
        2017-12-16 About
    - KG Embeddings Library
    - RDF embeddings
      - RDF2VEC
        
        [2009.00318] More is not Always Better: The Negative Impact of A-box Materialization on RDF2vec Knowledge Graph Embeddings
        
        Tags:
        
        Arxiv Doc
        Knowledge Graph Completion
        Knowledge Graph Embeddings
        RDF2VEC
        
        > we argue that despite the huge body of work devoted on completing missing information in knowledge graphs, such missing implicit information is actually a signal, not a defect
        
        2020-09-02 About
        
        RDF2vec.org
        
        Tags:
        
        RDF2VEC
        
        2020-09-02 About
        
        RDF2Vec: RDF Graph Embeddings for Data Mining - (2016)
        
        Tags:
        
        RDF2VEC
        
        > In our approach, we adapt neural language models for RDF graph embeddings. Such approaches take advantage of the word order in text documents, explicitly modeling the assumption that closer words in the word sequence are statistically more dependent. In the case of RDF graphs, we consider entities and relations between entities instead of word sequences. Thus, in order to apply such approaches on RDF graph data, we first have to transform the graph data into sequences of entities, which can be considered as sentences. Using those sentences, we can train the same neural language models to represent each entity in the RDF graph as a vector of numerical values in a latent feature space.
        
        2018-01-03 About
        
        RDF2Vec: RDF Graph Embeddings for Data Mining
        
        Tags:
        
        RDF2VEC
        
        2018-01-03 About
    - RotatE
    - Text, KG and embeddings
      - Text-Aware KG embedding
        BLP
        CKB
        
        raphaelsty/ckb: Contextual knowledge bases
        
        Tags:
        
        CKB
        GitHub project
        Raphaël Sourty
        Semanlink Tag Finder
        
        Une implémentation de [BLP](tag:blp) [[2010.03496] Inductive Entity Representations from Text via Link Prediction](doc:2020/11/2010_03496_inductive_entity_r)
        
        2020-11-09 About
        
        Daniel Daza sur Twitter : "BioBLP, a method for learning embeddings on multimodal knowledge graphs...."
        
        Tags:
        
        BLP
        Tweet
        
        2023-06-07 About
        
        [2010.03496] Inductive Entity Representations from Text via Link Prediction
        
        Tags:
        
        Arxiv Doc
        Attention + Knowledge Graphs
        BLP
        Discuté avec Raphaël
        Entity embeddings
        Good
        Link Prediction
        Text-Aware KG embedding
        TheWebConf 2021
        
        BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training) > a method for **learning representations of entities**, that uses a **pre-trained Transformer** based architecture as an entity encoder, and **link prediction training on a knowledge graph with textual entity descriptions**. > using entity descriptions, an entity encoder is trained for link prediction in a knowledge graph. The encoder can then be used without fine-tuning to obtain features for entity classification and information retrieval Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their objective targeted exclusively for link prediction (and not an objective that combines language modeling and link prediction as Kepler) performs better than Kepler's more complex one.
        
        2020-11-03 About
        
        DKRL
        
        Representation learning of knowledge graphs with entity descriptions (AAAI 2016)
        
        Tags:
        
        DKRL
        Text-Aware KG embedding
        Zero shot
        
        "Description-Embodied Knowledge Representation Learning" (DKRL) > In most knowledge graphs there are usually concise descriptions for entities, which cannot be well utilized by existing methods... Experimental results on real-world datasets show that, our method outperforms other baselines on the knowledge graph completion and entity classification tasks, especially under the zero-shot setting, which indicates that **our method is capable of building representations for novel entities according to their descriptions**. [Source code on github](https://github.com/xrb92/DKRL). For fact triples: TransE. Meanwhile, given an entity we will also learn to maximize the likelihood of predicting its description (using either CBOW or CNN encoder) (head + relation = tail, also in "text space") Two types of representations for entities: structure-based representations and description-based representations. They are learned simultaneously into the same vector space but not forced to be unified **so that novel entities with only descriptions can be represented**.
        
        2020-10-02 About
        
        LUKE
        
        [2110.08151] mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models
        
        Tags:
        
        Arxiv Doc
        Entities and LM
        Ikuya Yamada
        LUKE
        Masked entity prediction
        Multilingual NLP
        
        [Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_) > Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**. > > the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features. [Github](https://github.com/studio-ousia/luke) > Entity representations are known to enhance language models in mono-lingual settings (Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019: [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al., 2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context)) presumably by introducing real-world knowledge. We show that using entity representations facilitates cross-lingual transfer by providing languageindependent features. > > Multilingual extension of LUKE. The model is trained with the multilingual masked language modeling (MLM) task as well as the masked entity prediction (MEP) task with Wikipedia entity embeddings > We investigate two ways of using the entity representations in cross-lingual transfer tasks: > 1. perform entity linking for the input text, and append the detected entity tokens to the input sequence. The entity tokens are expected to provide language independent features to the model > 2. use the entity [MASK] token from the MEP task as a languageindependent feature extractor.
        
        2022-04-17 About
        
        Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."
        
        Tags:
        
        Entities and LM
        Ikuya Yamada
        LUKE
        Multilingual Language Models
        Tweet
        
        [[2110.08151] mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models](doc:2022/04/2110_08151_mluke_the_power_o) > mLUKE, an extension of [LUKE](tag:luke) based on 1M Wikidata entity embeddings shared across languages > mLUKE solves downstream tasks by using its language-agnostic entity embeddings as inputs. > entity representations are shared across languages during pretraining -> they are much more language-agnostic than word representations
        
        2022-04-13 About
        
        Studio Ousia sur Twitter : "Now using LUKE is easier than ever!" / Twitter
        
        Tags:
        
        LUKE
        Named Entity Recognition
        Tweet
        
        2022-03-15 About
        
        [2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
        
        Tags:
        
        Arxiv Doc
        BERT + KB
        EMNLP 2020
        Entity embeddings
        Ikuya Yamada
        LUKE
        Self-Attention
        Text-Aware KG embedding
        
        > LUKE is based on bidirectional Transformer, treats words and entities in a text as independent tokens, and outputs contextualized representations of them. The representations can be used to address downstream tasks similarly to BERT. [src](https://twitter.com/ikuyamada/status/1312947499141750786) > LUKE is trained using a novel pretraining task that involves predicting randomly masked words (equivalent to BERT’s masked language model) and entities in an entity-annotated corpus obtained from Wikipedia. (Hum, ça me rappelle quelque chose) > LUKE also uses a new *entity-aware* self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores. [github](https://github.com/studio-ousia/luke), [at Hugging Face](https://twitter.com/AkariAsai/status/1389428550298525696), [doc](https://huggingface.co/transformers/model_doc/luke.html), [tweet](https://twitter.com/ikuyamada/status/1392742990586683392?s=20)
        
        2020-11-26 About
        
        [2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
        
        Tags:
        
        Arxiv Doc
        BERT + KB
        EMNLP 2020
        Entity embeddings
        Ikuya Yamada
        LUKE
        Self-Attention
        Text-Aware KG embedding
        
        > LUKE is based on bidirectional Transformer, treats words and entities in a text as independent tokens, and outputs contextualized representations of them. The representations can be used to address downstream tasks similarly to BERT. [src](https://twitter.com/ikuyamada/status/1312947499141750786) > LUKE is trained using a novel pretraining task that involves predicting randomly masked words (equivalent to BERT’s masked language model) and entities in an entity-annotated corpus obtained from Wikipedia. (Hum, ça me rappelle quelque chose) > LUKE also uses a new *entity-aware* self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores. [github](https://github.com/studio-ousia/luke), [at Hugging Face](https://twitter.com/AkariAsai/status/1389428550298525696), [doc](https://huggingface.co/transformers/model_doc/luke.html), [tweet](https://twitter.com/ikuyamada/status/1392742990586683392?s=20)
        
        2020-11-26 About
        
        Text-Enhanced Representation Learning for Knowledge Graph (IJCAI 2016)
        
        Tags:
        
        IJCAI
        Text-Aware KG embedding
        
        2020-11-03 About
        
        [2010.03496] Inductive Entity Representations from Text via Link Prediction
        
        Tags:
        
        Arxiv Doc
        Attention + Knowledge Graphs
        BLP
        Discuté avec Raphaël
        Entity embeddings
        Good
        Link Prediction
        Text-Aware KG embedding
        TheWebConf 2021
        
        BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training) > a method for **learning representations of entities**, that uses a **pre-trained Transformer** based architecture as an entity encoder, and **link prediction training on a knowledge graph with textual entity descriptions**. > using entity descriptions, an entity encoder is trained for link prediction in a knowledge graph. The encoder can then be used without fine-tuning to obtain features for entity classification and information retrieval Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their objective targeted exclusively for link prediction (and not an objective that combines language modeling and link prediction as Kepler) performs better than Kepler's more complex one.
        
        2020-11-03 About
        
        SSP: Semantic Space Projection for Knowledge Graph Embedding with Text Descriptions (AAAI 2017)
        
        Tags:
        
        Roosevelt
        Text-Aware KG embedding
        
        > jointly learns from the symbolic triples and textual descriptions > The data involved in our model are the knowledge triples and the textual descriptions of entities. In experiments, we adopt the “entity descriptions” of Freebase and the textual definitions of Wordnet as textual information. Obvious but very good remark about link prediction in facts-only KG: > the triple (Anna Roosevelt, Parents, Franklin Roosevelt), indicates “Franklin Roosevelt” is the parent of “Anna Roosevelt”. However, it’s quite difficult to infer this fact merely from other symbolic triples.
        
        2020-10-02 About
        
        Representation learning of knowledge graphs with entity descriptions (AAAI 2016)
        
        Tags:
        
        DKRL
        Text-Aware KG embedding
        Zero shot
        
        "Description-Embodied Knowledge Representation Learning" (DKRL) > In most knowledge graphs there are usually concise descriptions for entities, which cannot be well utilized by existing methods... Experimental results on real-world datasets show that, our method outperforms other baselines on the knowledge graph completion and entity classification tasks, especially under the zero-shot setting, which indicates that **our method is capable of building representations for novel entities according to their descriptions**. [Source code on github](https://github.com/xrb92/DKRL). For fact triples: TransE. Meanwhile, given an entity we will also learn to maximize the likelihood of predicting its description (using either CBOW or CNN encoder) (head + relation = tail, also in "text space") Two types of representations for entities: structure-based representations and description-based representations. They are learned simultaneously into the same vector space but not forced to be unified **so that novel entities with only descriptions can be represented**.
        
        2020-10-02 About
        
        [1910.12507] A Survey on Knowledge Graph Embeddings with Literals: Which model links better Literal-ly?
        
        Tags:
        
        Arxiv Doc
        Knowledge Graph Embeddings
        Survey / Review
        Text-Aware KG embedding
        
        2020-05-04 About
        
        Iterative Entity Alignment with Improved Neural Attribute Embedding
        
        Tags:
        
        Entity alignment
        Entity embeddings
        Text-Aware KG embedding
        
        2020-04-29 About
        
        A Comprehensive Survey of Knowledge Graph Embeddings with Literals: Techniques and Applications
        
        Tags:
        
        Knowledge Graph Embeddings
        Survey / Review
        Text-Aware KG embedding
        
        > survey of the KG embedding models which consider the structured information of the graph as well as the unstructured information in form of literals such as text, numerical values etc A le mérite de poser la question de l'utilisation de littéraux dans les embeddings de KG [Newer and longer version](/doc/2020/05/1910_12507_a_survey_on_knowle)
        
        2020-04-29 About
        
        [2001.09522] TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network
        
        Tags:
        
        Arxiv Doc
        Discuté avec Raphaël
        Graph neural networks
        Knowledge Graph Completion
        Microsoft Research
        NLP@Microsoft
        Taxonomies
        Taxonomy expansion task
        Text-Aware KG embedding
        TheWebConf 2020
        
        how to add a set of new concepts to an existing taxonomy. [Tweet](https://twitter.com/mickeyjs6/status/1253772146142216194?s=20) [GitHub](https://github.com/mickeystroller/TaxoExpan) > we study the taxonomy expansion task: given an existing taxonomy and a set of new emerging concepts, we aim to automatically expand the taxonomy to incorporate these new concepts (without changing the existing relations in the given taxonomy). > To the best of our knowledge, this is the first study on **how to expand an existing directed acyclic graph (as we model a taxonomy as a DAG) using self-supervised learning**. Self-supervised framework, the existing taxonomy being used as training data: it learns a model to predict whether a query concept is the direct hyponym of an anchor concept. > 2 techniques: > > 1. a **position-enhanced graph neural network that encodes the local structure of an anchor concept** in the existing taxonomy, > 2. a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data. Regarding 1: uses [GNN](/tag/graph_neural_networks.html) to model the "ego network" of concepts (potential “siblings” and “grand parents” of the query concept). > Regular GNNs fail to distinguish nodes with different relative positions to the query (i.e., some nodes are grand parents of the query while the others are siblings of the query). To address this limitation, we present a simple but effective enhancement to inject such position information into GNNs using position embedding. We show that such embedding can be easily integrated with existing GNN architectures (e.g., [GCN](/tag/graph_convolutional_networks) and GAT) and significantly boosts the prediction performance Regarding point 2: uses InfoNCE loss, cf. [Contrastive Predictive Coding](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1807.03748) > Instead of predicting whether each individual ⟨query concept, anchor concept⟩ pair is positive or not, we first group all pairs sharing the same query concept into a single training instance and learn a model to select the positive pair among other negative ones from the group. (Hum, ça me rappelle quelque chose) > assume each concept (in existing taxonomy + set of new concepts) has an initial embedding vector learned from some text associated with this concept. To keep things tractable, only attempts to find a single parent node of each new concept.
        
        2020-04-25 About
        
        Reasoning With Neural Tensor Networks for Knowledge Base Completion (2013)
        
        Tags:
        
        Andrew Ng
        Chris Manning
        Knowledge Graph + Deep Learning
        Knowledge Graph Completion
        Richard Socher
        Text-Aware KG embedding
        
        **Predicting the likely truth of additional facts based on existing facts in the knowledge base.** > we introduce an expressive neural tensor network suitable for reasoning over relationships between two entities. Most similar work: [Bordes et al.](http://127.0.0.1:8080/semanlink/doc/2019/08/learning_structured_embeddings_) (2011) Contributions: 1. new neural tensor network (**NTN**) suitable for reasoning over relationships between two entities. Generalizes several previous neural network models and provides a more powerful way to model relational information than a standard neural network layer. 2. a new way to represent entities in knowledge bases, as the average of their constituting word vectorss, allowing the sharing of statistical strength between the words describing each entity (e.g., Bank of China and China). 3. incorporation of word vectors which are trained on large unlabeled text > We **learn to modify word representations via grounding in world knowledge**. This essentially allows us to analyze word embeddings and query them for specific relations. Furthermore, the resulting vectors could be used in other tasks such as named entity recognition or relation classification in natural language **Makes use of entity name**: NTN first learns word vectors from an auxiliary news corpus, and then initializes the representation of each entity by averaging the vectors of words contained in its name. For example, **the embedding of AlfredHitchcock is initialized by the average word vectors of “alfred” and “hitchcock”**... This kind of methods model textual information separately from KG facts, and hence fail to leverage interactions between them.[src](doc:2019/05/knowledge_graph_embedding_a_su)
        
        2019-08-03 About
      - Using literal descriptions of entities
        
        [1806.06478] Co-training Embeddings of Knowledge Graphs and Entity Descriptions for Cross-lingual Entity Alignment
        
        Tags:
        
        Arxiv Doc
        Co-training
        Cross-lingual NLP
        Entity alignment
        Knowledge Graph Embeddings
        Using literal descriptions of entities
        
        > Since many multilingual KGs also provide literal descriptions of entities, in this paper, we introduce an embedding-based approach which leverages a weakly aligned multilingual KG for semi-supervised cross-lingual learning using entity descriptions
        
        2020-09-06 About
      - Word + Entity embeddings
        Wikipedia2Vec
        
        [1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification
        
        Tags:
        
        Arxiv Doc
        Entities
        Entity salience
        Good
        Ikuya Yamada
        Text Classification
        Transformers
        Wikipedia2Vec
        
        A model that performs **text classification using entities in a knowledge base**. > Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**. 2 steps: 1. Entity detection 2. Classification using the detected entities (+text) as inputs Regarding entity linking, a local model which uses cosine similarity between the embedding of the target entity and the word-based representation of the document to capture the relevance of an entity given a document. Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities close to one another in a unified vector space) Model using attention, with 2 features : - cosine similarity between the embedding of the entity and the word based representation of the document - the probability that the entity name refers to the entity in KB. Somewhat [related](doc:2020/01/investigating_entity_knowledge_) ### Conclusion: >a neural network model that performs text classification using entities in Wikipedia. We combined simple dictionary-based entity detection with a neural attention mechanism to enable the model to focus on a small number of unambiguous and relevant entities in a document.
        
        2020-09-02 About
        
        [1812.06280] Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia
        
        Tags:
        
        Arxiv Doc
        Ikuya Yamada
        Wikipedia2Vec
        
        2020-09-02 About
        
        [1601.01343] Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation
        
        Tags:
        
        Arxiv Doc
        Entity linking
        Ikuya Yamada
        Named Entity Recognition
        Thèse IRIT-Renault: biblio initiale
        Wikipedia2Vec
        Word + Entity embeddings
        
        > An embedding method specifically **designed for NED** that jointly **maps words and entities into the same continuous vector space**. > We extend the skip-gram model by using two models. The KB graph model learns the relatedness of entities using the link structure of the KB, whereas the anchor context model aims to align vectors such that similar words and entities occur close to one another in the vector space by leveraging KB anchors and their context words Technique later used in [Wikipedia2Vec](doc:?uri=https%3A%2F%2Fwikipedia2vec.github.io%2Fwikipedia2vec%2F), by the same team. [Neural Attentive Bag-of-Entities Model for Text Classification](doc:2020/09/1909_01259_neural_attentive_b) uses Wikipedia2Vec model.
        
        2019-01-27 About
        
        Wikipedia2Vec
        
        Tags:
        
        Wikipedia2Vec
        Word + Entity embeddings
        
        [GitHub](https://github.com/wikipedia2vec/wikipedia2vec) [Paper](doc:2020/09/1812_06280_wikipedia2vec_an_)
        
        2018-12-23 About
        
        [1601.01343] Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation
        
        Tags:
        
        Arxiv Doc
        Entity linking
        Ikuya Yamada
        Named Entity Recognition
        Thèse IRIT-Renault: biblio initiale
        Wikipedia2Vec
        Word + Entity embeddings
        
        > An embedding method specifically **designed for NED** that jointly **maps words and entities into the same continuous vector space**. > We extend the skip-gram model by using two models. The KB graph model learns the relatedness of entities using the link structure of the KB, whereas the anchor context model aims to align vectors such that similar words and entities occur close to one another in the vector space by leveraging KB anchors and their context words Technique later used in [Wikipedia2Vec](doc:?uri=https%3A%2F%2Fwikipedia2vec.github.io%2Fwikipedia2vec%2F), by the same team. [Neural Attentive Bag-of-Entities Model for Text Classification](doc:2020/09/1909_01259_neural_attentive_b) uses Wikipedia2Vec model.
        
        2019-01-27 About
        
        Wikipedia2Vec
        
        Tags:
        
        Wikipedia2Vec
        Word + Entity embeddings
        
        [GitHub](https://github.com/wikipedia2vec/wikipedia2vec) [Paper](doc:2020/09/1812_06280_wikipedia2vec_an_)
        
        2018-12-23 About
        
        A Tri-Partite Neural Document Language Model for Semantic Information Retrieval (2018 - ESWC conference)
        
        Tags:
        
        Embeddings in Information Retrieval
        ESWC
        Laure Soulier
        Lynda Tamine
        Thèse IRIT-Renault: biblio initiale
        Word + Entity embeddings
        Word sense / Lexical ambiguity
        
        from the abstract: Previous work in information retrieval have shown that using evidence, such as concepts and relations, from external knowledge sources could enhance the retrieval performance... This paper presents a new tri-partite neural document language framework that leverages explicit knowledge to jointly constrain word, concept, and document learning representations to tackle a number of issues including polysemy and granularity mismatch.
        
        2018-06-08 About
        
        [1804.01486] Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data
        
        Tags:
        
        Arxiv Doc
        Embeddings
        Medical Data
        Word + Entity embeddings
        
        2018-04-14 About
        
        Modèle Neuronal de Recherche d'Information Augmenté par une Ressource Sémantique (2017)
        
        Tags:
        
        Embeddings in Information Retrieval
        IRIT
        Laure Soulier
        LIP6
        Lynda Tamine
        Word + Entity embeddings
        
        Modèle neuronal pour la recherche d'information qui combine sémantique symbolique (représentation explicite d'entités et de leur relations, issue de ressources externes) et sémantique distributionnelle (représentation sémantique latente calculée à partir des textes)
        
        2018-02-12 About
        
        Knowledge Graph and Text Jointly Embedding (2014)
        
        Tags:
        
        Knowledge Graph Completion
        Knowledge Graph Embeddings
        Microsoft Research
        NLP@Microsoft
        Thèse IRIT-Renault: biblio initiale
        Word + Entity embeddings
        
        method of **jointly embedding knowledge graphs and a text corpus** so that **entities and words/phrases are represented in the same vector space**. Promising improvement in the accuracy of predicting facts, compared to separately embedding knowledge graphs and text (in particular, enables the prediction of facts containing entities out of the knowledge graph) [cité par J. Moreno](/doc/?uri=https%3A%2F%2Fhal.archives-ouvertes.fr%2Fhal-01626196%2Fdocument)
        
        2018-01-05 About
        
        Learning Concept-Driven Document Embeddings for Medical Information Search (2017)
        
        Tags:
        
        Document embeddings
        IRIT
        Laure Soulier
        Lynda Tamine
        Medical Data
        Medical Information Search
        Semantic gap
        Thèse IRIT-Renault: biblio initiale
        Word + Entity embeddings
        
        In this paper, we study how to optimize the document representation by leveraging neural-based approaches to capture latent representations built upon both validated medical concepts specified in an external resource as well as the used words. **Document vectors are learned so they allow predicting concepts in their context**
        
        2018-01-03 About
        
        Combining word and entity embeddings for entity linking (ESWC 2017)
        
        Tags:
        
        CEA, LIST
        Embeddings
        Entity embeddings
        Entity linking
        ESWC
        IRIT
        José Moreno
        Thèse IRIT-Renault: biblio initiale
        Word + Entity embeddings
        
        The general approach for the entity linking task is to generate, for a given mention, a set of candidate entities from the base and, in a second step, determine which is the best one. This paper proposes a novel method for the second step which is based on the **joint learning of embeddings for the words in the text and the entities in the knowledge base**.
        
        2017-12-30 About
        
        Enriching Word Embeddings Using Knowledge Graph for Semantic Tagging in Conversational Dialog Systems - Microsoft Research (2015)
        
        Tags:
        
        Microsoft Research
        NLP@Microsoft
        Semantic tagging
        Word + Entity embeddings
        
        > new simple, yet effective approaches to learn domain specific word embeddings. ## Intro > Adapting word embeddings, such as jointly capturing syntactic and semantic information, can further enrich semantic word representations for several tasks, e.g., sentiment analysis (Tang et al. 2014), named entity recognition (Lebret, Legrand, and Collobert 2013), entity-relation extraction (Weston et al. 2013), etc. (Yu and Dredze 2014) has introduced a lightly supervised word embedding learning extending word2vec. They incorporate prior information to the objective function as a regularization term considering synonymy relations between words from Wordnet (Fellbaum 1999). > In this work, we go one step further and investigate if enriching the word2vec word embeddings trained on unstructured/ unlabeled text with domain specific semantic relations obtained from knowledge sources (e.g., knowledge graphs, search query logs, etc.) can help to discover relation aware word embeddings. Unlike earlier work, **we encode the information about the relations between phrases, thereby, entities and relation mentions are all embedded into a low dimensional vector space**. ## Related work (Learning Word Embeddings with Priors) - word2vec - Relational Constrained Model (RTM) (Yu and Dredze 2014) While CBOW learns lexical word embeddings from provided text, the RTM learns embeddings of words based on their similarity to other words provided by a knowledge resource (eg. wordnet) - Joint model (Yu and Dredze 2014) combines CBOW and RTM through linear combination
        
        2017-10-02 About
    - Text-Aware KG embedding
    - TransE
    - Transformer based KG embeddings
  - Node Embeddings
    - DeepWalk
    - Node2Vec
  - OWL2Vec
- Graph neural networks
  - Graph Attention Networks
  - Graph Convolutional Networks
  - Graph message passing
- Graphs + NLP
  - Knowledge Graphs and NLP
    - BERT + KB
      - BLP
      - KnowBert
        
        [1909.04164] Knowledge Enhanced Contextual Word Representations
        
        Tags:
        
        Allen Institute for AI (AI2)
        Arxiv Doc
        Contextualized word representations
        EMNLP 2019
        Entities and LM
        Good
        Grounded Language Learning
        KD-MKB biblio
        KG-augmented Language Models
        KnowBert
        Knowledge-augmented Language Models
        Knowledge-driven embeddings
        Knowledge Graphs in NLP
        Multiple Knowledge Bases
        
        General method to **embed multiple knowledge bases into pre-trained language models** (KB in the sense as fixed collection of entity nodes) > The key idea is to explicitly model entity spans in the input text and use an **entity linker** to retrieve relevant entity embeddings from a KB to form knowledge enhanced entity-span representations. > Then, update contextual word representations via a form of **word-to-entity attention**. > In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that **combines a small amount of entity linking supervision with a large amount of raw text**.
        
        2020-05-13 About
      - LUKE
        
        [2205.00820] Entity-aware Transformers for Entity Search
        
        Tags:
        
        Arxiv Doc
        BERT + KB
        Discuté avec Raphaël
        Entities and LM
        Entity Retrieval
        KG-aware low-resource learning
        SIGIR 2022
        
        > **Do BERT-based entity retrieval models benefit from additional entity information stored in knowledge graphs?** To address this research question, we map entity embeddings into the same input space as a pre-trained BERT model and inject these entity embeddings into the BERT model. This entity-enriched language model is then employed on the entity retrieval task. > we observe empirically that the entity-enriched BERT models **enable fine-tuning on limited training data**, which otherwise would not be feasible due to the known instabilities of BERT in few-sample fine-tuning Uses [Wikipedia2Vec](tag:wikipedia2vec) as graph embedding method
        
        2022-07-12 About
        
        [1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
        
        Tags:
        
        Arxiv Doc
        BERT
        BERT + KB
        Discuté avec Raphaël
        Entity embeddings
        Text, KG and embeddings
        
        > way of **injecting factual knowledge about entities into the pretrained BERT model**. (Feeding entity vectors into BERT as if they were wordpiece vectors without additional encoder pretraining) > > **We align [Wikipedia2Vec](tag:wikipedia2vec) entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors**. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to [ERNIE](tag:ernie) (Zhang et al., 2019) and [KnowBert](tag:knowbert) (Peters et al., 2019), but it **requires no expensive further pretraining of the BERT encoder**. > > Our vector space alignment strategy is inspired by cross-lingual word vector alignment Related work on Entity-enhanced BERT: > ([ERNIE](doc:2019/08/_1905_07129_ernie_enhanced_la) and [Knowbert](doc:2020/05/1909_04164_knowledge_enhanced)) are based on the design principle that BERT be adapted to entity vectors. They introduce new encoder layers to feed pretrained entity vectors into the Transformer, and they require additional pretraining to integrate the new parameters. In contrast, E-BERT’s design principle is that entity vectors be adapted to BERT. > > Two other knowledge-enhanced MLMs are [KEPLER](doc:2020/11/1911_06136_kepler_a_unified_) (Wang et al., 2019c) and K-Adapter (Wang et al., 2020)... Their factual knowledge does not stem from entity vectors – instead, they are trained in a multi-task setting on relation classification and knowledge base completion. Not to be cofounded with [[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](doc:2020/12/2009_02835_e_bert_a_phrase_a)
        
        2021-01-12 About
        
        [2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce
        
        Tags:
        
        Arxiv Doc
        Aspect Detection
        BERT + KB
        Domain-Specific BERT
        Product data & NLP
        
        E-BERT, pre-training framework for product data. 1. to benefit from phrase-level knowledge: Adaptive Hybrid Masking, a new masking strategy, which allows the model to adaptively switch from learning preliminary word knowledge to learning complex phrases 2. leveraging product-level knowledge: training E-BERT to predict a product’s associated neighbors (product association) Resources used: - description of millions of products from the amazon dataset (title, description, reviews) - e-commerce phrases: extracted from above dataset using [AutoPhrase](doc:2020/12/autophrase_automated_phrase_mi) - product association graph: pairs of substitutable and complementary products extracted from amazon dataset Not to be confounded with [[1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT](doc:2021/01/1911_03681_e_bert_efficient_)
        
        2020-12-14 About
        
        Supporting content decision makers with machine learning | Dec, 2020 | Netflix TechBlog
        
        Tags:
        
        BERT + KB
        Entity embeddings
        Link Prediction
        Netflix
        Transfer learning
        
        2020-12-11 About
        
        [2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
        
        Tags:
        
        Arxiv Doc
        BERT + KB
        EMNLP 2020
        Entity embeddings
        Ikuya Yamada
        LUKE
        Self-Attention
        Text-Aware KG embedding
        
        > LUKE is based on bidirectional Transformer, treats words and entities in a text as independent tokens, and outputs contextualized representations of them. The representations can be used to address downstream tasks similarly to BERT. [src](https://twitter.com/ikuyamada/status/1312947499141750786) > LUKE is trained using a novel pretraining task that involves predicting randomly masked words (equivalent to BERT’s masked language model) and entities in an entity-annotated corpus obtained from Wikipedia. (Hum, ça me rappelle quelque chose) > LUKE also uses a new *entity-aware* self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores. [github](https://github.com/studio-ousia/luke), [at Hugging Face](https://twitter.com/AkariAsai/status/1389428550298525696), [doc](https://huggingface.co/transformers/model_doc/luke.html), [tweet](https://twitter.com/ikuyamada/status/1392742990586683392?s=20)
        
        2020-11-26 About
        
        Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge (ACL Anthology 2020)
        
        Tags:
        
        ACL 2020
        BERT + KB
        Few-shot learning
        Knowledge Graphs in NLP
        OOV
        Slot filling
        
        Aim to leverage both contextual representation of input text (deep LMs) and knowledge derived from curated KBs ([Wordnet](tag:wordnet)) to improve [slot tagging](tag:slot_tagging) in the presence of [out-of-vocab](tag:oov) words ([few-shot scenario](tag:few_shot_learning)) Method: 1. retrieve potentially relevant KB entities and encode them into distributed representations that describe global graph-structured information 2. BERT encoder layer to capture context-aware representations of the sequence and attend to the KB embeddings using multi-level graph attention 3. integrate BERT embeddings and the KB embeddings to predict the slot type Contributions: 1. feasibility of applying lexical ontology to facilitate recognizing OOV words. First to consider the large-scale background knowledge for enhancing context-aware slot tagging models. 2. a knowledge integration mechanism that uses multi-level graph attention to model explicit lexical relations. 3.experiments on two benchmark datasets > our method makes a notable difference in a scenario where samples are linguistically diverse, and large vocab exists. (Better improvements when using RNN than BERT, because BERT already contains a lot of background knowledge)
        
        2020-07-04 About
    - Knowledge Graphs in NLP
      - BERT + KB
      - ERNIE
        
        [1905.07129] ERNIE: Enhanced Language Representation with Informative Entities
        
        Tags:
        
        ACL 2019
        Arxiv Doc
        BERT
        Discuté avec Raphaël
        ERNIE
        Knowledge Graphs in NLP
        Text, KG and embeddings
        Transformers
        
        > We argue that informative entities in **KGs can enhance language representation with external knowledge**. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. > ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks [GitHub](https://github.com/thunlp/ERNIE) WARNING, there is another ERNIE (by [NLP@Baidu](tag:nlp_baidu)): Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. This doesn't happen when you choose François-Paul as the name for your child.
        
        2019-08-05 About
      - KG-augmented Language Models
        ERNIE
        
        Specializing Static and Contextual Embeddings in the Medical Domain Using Knowledge Graphs: Let’s Keep It Simple - ACL Anthology (2022)
        
        Tags:
        
        ACL
        Discuté avec M. Labeau
        KG-augmented Language Models
        Node2Vec
        Olivier Ferret
        
        > a simple approach to embedding specialization that relies on knowledge graph embeddings and concatenation > we first demonstrate that static embeddings can indeed be improved through concatenation with in-domain node2vec representations. Then, we validate this approach on contextual models and generalize it further by proposing a variant of BERT that incorporates knowledge embeddings within its hidden states through the same process of concatenation.
        
        2023-10-11 About
        
        [2205.08184] SKILL: Structured Knowledge Infusion for Large Language Models
        
        Tags:
        
        Arxiv Doc
        KG-augmented Language Models
        NLP@Google
        Structured knowledge
        T5 (Text-To-Text Transfer Transformer)
        
        > a method to infuse structured knowledge into LLMs, by directly training T5 models on factual triples of knowledge graphs > The models pre-trained on factual triples compare competitively with the ones on natural language sentences that contain the same knowledge. > The proposed method has an advantage that no alignment between the knowledge graph and text corpus is required
        
        2022-05-18 About
        
        Knowledge Graphs in NLP @ EMNLP 2020 | by Michael Galkin | Nov, 2020 | Medium
        
        Tags:
        
        EMNLP 2020
        KG-augmented Language Models
        Knowledge Graphs and NLP
        
        About "Entity Representations in LMs", refers to: - [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context) by [Yamada](/showprop.do?pptyuri=http%3A%2F%2Fwww.semanlink.net%2F2001%2F00%2Fsemanlink-schema%23arxiv_author&pptyval=Ikuya%2BYamada) - Shen et al employ a background KG in their GLM (Graph-guided Masked Language Model): the graph supplies a vocabulary of named entities with their connectivity patterns (reachable entities in k-hops) - [[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision](doc:2020/07/2004_07202_entities_as_expert) - Poerner et al make use of [Wikipedia2Vec](tag:wikipedia2vec) (by Yamada) in their E-BERT Autoregressive KG-augmented LMs: generation process of LMs is conditioned by or enriched with structured knowledge like small subgraphs!
        
        2020-11-24 About
        
        [2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision
        
        Tags:
        
        Arxiv Doc
        KG-augmented Language Models
        Memory networks
        NLP@Google
        Not Encoding Factual Knowledge in Language Model
        Question Answering
        Transformers
        
        > We focus on the problem of **capturing declarative knowledge in the learned parameters of a language model**... > Entities as Experts (EaE) can access distinct memories of the entities mentioned in a piece of text; > To understand the motivation for distinct and independent entity representations: A traditional Transformer would need to build an internal representation of Charles Darwin from the words “Charles” and “Darwin”... Conversely, EAE can access a dedicated representation of “Charles Darwin”, which is a memory of all of the contexts in which this entity has previously been mentioned.... Having retrieved and re-integrated this memory it is much easier for EAE to relate the question to the answer > EaE's entity representations are learned directly from text. Correct identification, and representation, of entities is essential to EaE's performance Based on transformer architecture Extension: [Facts as Experts](doc:2020/07/2007_00849_facts_as_experts_)
        
        2020-07-11 About
        
        [1909.04164] Knowledge Enhanced Contextual Word Representations
        
        Tags:
        
        Allen Institute for AI (AI2)
        Arxiv Doc
        Contextualized word representations
        EMNLP 2019
        Entities and LM
        Good
        Grounded Language Learning
        KD-MKB biblio
        KG-augmented Language Models
        KnowBert
        Knowledge-augmented Language Models
        Knowledge-driven embeddings
        Knowledge Graphs in NLP
        Multiple Knowledge Bases
        
        General method to **embed multiple knowledge bases into pre-trained language models** (KB in the sense as fixed collection of entity nodes) > The key idea is to explicitly model entity spans in the input text and use an **entity linker** to retrieve relevant entity embeddings from a KB to form knowledge enhanced entity-span representations. > Then, update contextual word representations via a form of **word-to-entity attention**. > In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that **combines a small amount of entity linking supervision with a large amount of raw text**.
        
        2020-05-13 About
        
        [1906.07241] Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling
        
        Tags:
        
        Allen Institute for AI (AI2)
        Arxiv Doc
        KD-MKB biblio
        KG-augmented Language Models
        Knowledge Graphs and NLP
        
        > a **neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context**. These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. > The KGLM maintains a dynamically growing local knowledge graph, a subset of the knowledge graph that contains entities that have already been mentioned in the text, and their related entities.
        
        2020-05-11 About
      - KnowBert
      - Microsoft Concept Graph
        
        Microsoft Concept Graph and Concept Tagging Release
        
        Tags:
        
        Microsoft Concept Graph
        
        2019-05-28 About
        
        Microsoft Concept Graph: Mining Semantic Concepts for Short Text Understanding | MIT Press Journals (2019)
        
        Tags:
        
        Microsoft Concept Graph
        NLP: short texts
        
        Microsoft Concept Graph, a knowledge graph engine that provides concept tagging APIs to facilitate the understanding of human languages. Built upon Probase, a universal probabilistic taxonomy consisting of instances and concepts mined from the Web. The conceptualization model (also known as the Concept Tagging Model) aims to map text into semantic concept categories with some probabilities
        
        2019-05-28 About
    - LM + KG
      - BERT + KB
      - LLM + KG
        LLM generated Knowledge Graphs
        
        GraphRAG: Unlocking LLM discovery on narrative private data - Microsoft Research
        
        Tags:
        
        LLM generated Knowledge Graphs
        Microsoft Research
        NLP@Microsoft
        RAG + KG
        
        > GraphRAG uses **LLM generated knowledge graphs** to provide substantial improvements in question-and-answer performance when conducting document analysis of complex information. > power of **prompt augmentation** when performing **discovery** on private datasets (data that the LLM is not trained on and has never seen before, such as an enterprise’s proprietary research, business documents..) > GraphRAG uses the LLM to **create a knowledge graph based on the private dataset**. This graph is then used alongside graph machine learning to perform **prompt augmentation** at query time. > the GraphRAG approach [can] **discover entities in the query**. This allows the LLM to ground itself in the graph and results in superior answer that contains provenance through links to the original supporting text GraphRAG can answer queries such as "**what are the top five themes in the data?**"
        
        2024-03-07 About
        
        RAG + KG
        
        Enhancing RAG-based application accuracy by constructing and leveraging knowledge graphs
        
        Tags:
        
        LangChain
        Neo4j
        RAG + KG
        
        2024-03-16 About
        
        GraphRAG: Unlocking LLM discovery on narrative private data - Microsoft Research
        
        Tags:
        
        LLM generated Knowledge Graphs
        Microsoft Research
        NLP@Microsoft
        RAG + KG
        
        > GraphRAG uses **LLM generated knowledge graphs** to provide substantial improvements in question-and-answer performance when conducting document analysis of complex information. > power of **prompt augmentation** when performing **discovery** on private datasets (data that the LLM is not trained on and has never seen before, such as an enterprise’s proprietary research, business documents..) > GraphRAG uses the LLM to **create a knowledge graph based on the private dataset**. This graph is then used alongside graph machine learning to perform **prompt augmentation** at query time. > the GraphRAG approach [can] **discover entities in the query**. This allows the LLM to ground itself in the graph and results in superior answer that contains provenance through links to the original supporting text GraphRAG can answer queries such as "**what are the top five themes in the data?**"
        
        2024-03-07 About
        
        Knowledge Graph RAG Query Engine - LlamaIndex
        
        Tags:
        
        LlamaIndex
        LLM + KG
        RAG (Retrieval-Augmented Generation)
        RAG + KG
        
        > Typically, this is to build context based on entities’ SubGraph related to the task.
        
        2023-08-17 About
        
        Jerry Liu on X: “Graph RAG” in @llama_index: a new method of augmenting LLMs with context from a graph database
        
        Tags:
        
        Graph database
        Jerry Liu
        LlamaIndex
        LLM + KG
        RAG (Retrieval-Augmented Generation)
        RAG + KG
        Tweet
        
        [Knowledge Graph RAG Query Engine - LlamaIndex](doc:2023/08/knowledge_graph_rag_query_engin)
        
        2023-08-17 About
        
        Knowledge Graphs & LLMs: Fine-Tuning Vs. Retrieval-Augmented Generation | by Tomaz Bratanic | Neo4j Developer Blog (June 6 2023)
        
        Tags:
        
        LLM + KG
        LLM Fine-tuning vs RAG
        LM Fine-tuning
        Neo4j
        RAG (Retrieval-Augmented Generation)
        RAG + KG
        
        > What are the limitations of LLMs, and how to overcome them? > We haven’t found any recent articles describing **how to use a knowledge graph to prepare good question-answer pairs that can be used to fine-tune an LLM**. This is an area that we plan to explore during the [NaLLM project](doc:2023/08/github_neo4j_nallm_repositor) > we recommend updating the internal knowledge of an LLM through fine-tuning techniques only for slowly changing or updating data.
        
        2023-08-13 About
        
        Frank van Harmelen sur X : "...GenAI is rapidly becoming the best motivation for symbolic AI in a long time!"
        
        Tags:
        
        Andrew Ng
        Frank van Harmelen
        LLM + KG
        
        2024-03-15 About
        
        KGC23 Keynote: The Future of Knowledge Graphs in a World of LLMs — Denny Vrandečić, Wikimedia - YouTube
        
        Tags:
        
        Denny Vrandečić
        LLM + KG
        The Knowledge Graph Conference
        Wikimedia
        YouTube video
        
        2024-03-07 About
        
        [2311.07509] A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases
        
        Tags:
        
        Arxiv Doc
        Benchmark
        Dean Allemang
        Juan Sequeda
        Knowledge Graphs
        LLM + KG
        LLM+SQL
        
        2023-11-21 About
        
        Jerry Liu sur X : "seven full ways to query knowledge graphs with LLMs..."
        
        Tags:
        
        Jerry Liu
        LLM + KG
        
        seven full ways to query knowledge graphs with LLMs
        
        2023-09-30 About
        
        [2309.15427] Graph Neural Prompting with Large Language Models
        
        Tags:
        
        Arxiv Doc
        LLM + KG
        NLP@Amazon
        
        > Can we learn beneficial knowledge from KGs and integrate them into pre-trained LLMs? > we propose to leverage the factual knowledge from KGs to enhance LLMs, while still benefiting from circumventing the burdensome training expenses by using pre-trained LLMs > Graph Neural Prompting (GNP), a plug-and-play method to assist pre-trained LLMs in learning beneficial knowledge from KGs > > GNP encodes the pertinent grounded knowledge and complex structural information to derive Graph Neural Prompt, an embedding vector that can be sent into LLMs to provide guidance and instructions > - GNP first utilizes a GNN to capture and encode the intricate graph knowledge into **entity/node embeddings**. > - Then, a cross-modality pooling module is present to determine the **most relevant node embeddings in relation to the text input**, and consolidate these node embeddings into **a holistic graph-level embedding**. > - After that, GNP encompasses a **domain projector** to bridge the inherent disparities between the graph and text domains. > - Finally, a **self-supervised link prediction objective** is introduced to enhance the model comprehension of relationships between entities and capture graph knowledge in a self-supervised manner.
        
        2023-09-28 About
        
        Knowledge Graph RAG Query Engine - LlamaIndex
        
        Tags:
        
        LlamaIndex
        LLM + KG
        RAG (Retrieval-Augmented Generation)
        RAG + KG
        
        > Typically, this is to build context based on entities’ SubGraph related to the task.
        
        2023-08-17 About
        
        Jerry Liu on X: “Graph RAG” in @llama_index: a new method of augmenting LLMs with context from a graph database
        
        Tags:
        
        Graph database
        Jerry Liu
        LlamaIndex
        LLM + KG
        RAG (Retrieval-Augmented Generation)
        RAG + KG
        Tweet
        
        [Knowledge Graph RAG Query Engine - LlamaIndex](doc:2023/08/knowledge_graph_rag_query_engin)
        
        2023-08-17 About
        
        GitHub - neo4j/NaLLM: Repository for the NaLLM project
        
        Tags:
        
        GitHub project
        LLM + KG
        Neo4j
        
        > synergies between Neo4j and Large Language Models (LLMs). As a part of our ongoing project, we are focusing on three primary use cases: > - a Natural Language Interface to a Knowledge Graph, > - Creating a Knowledge Graph from Unstructured Data > - and Generate a Report using both static data and data from LLM.
        
        2023-08-14 About
        
        Knowledge Graphs & LLMs: Fine-Tuning Vs. Retrieval-Augmented Generation | by Tomaz Bratanic | Neo4j Developer Blog (June 6 2023)
        
        Tags:
        
        LLM + KG
        LLM Fine-tuning vs RAG
        LM Fine-tuning
        Neo4j
        RAG (Retrieval-Augmented Generation)
        RAG + KG
        
        > What are the limitations of LLMs, and how to overcome them? > We haven’t found any recent articles describing **how to use a knowledge graph to prepare good question-answer pairs that can be used to fine-tune an LLM**. This is an area that we plan to explore during the [NaLLM project](doc:2023/08/github_neo4j_nallm_repositor) > we recommend updating the internal knowledge of an LLM through fine-tuning techniques only for slowly changing or updating data.
        
        2023-08-13 About
        
        ML Blog - Improve ChatGPT with Knowledge Graphs
        
        Tags:
        
        ChatGPT
        LLM + KG
        
        2023-07-04 About
        
        [2306.08302] Unifying Large Language Models and Knowledge Graphs: A Roadmap
        
        Tags:
        
        Arxiv Doc
        LLM + KG
        Survey / Review
        
        1. **KG-enhanced LLMs**, which incorporate KGs during the pre-training and inference phases of LLMs, or for the purpose of enhancing understanding of the knowledge learned by LLMs; #[KG-augmented Language Models](tag:knowledge_graph_augmented_language_models) 2. **LLM-augmented KGs**, that leverage LLMs for different KG tasks such as embedding, completion, construction, graph-to-text generation, and question answering; 3. **Synergized LLMs + KGs**, in which LLMs and KGs play equal roles and work in a mutually beneficial way to enhance both LLMs and KGs for bidirectional reasoning driven by both data and knowledge.
        
        2023-06-18 About
      - LLMs and basic Renault Knowledge
        
        Ravi Theja sur X : "𝐀𝐜𝐭𝐢𝐯𝐞𝐑𝐀𝐆: 𝐑𝐞𝐯𝐞𝐚𝐥𝐢𝐧𝐠 𝐭𝐡𝐞 𝐓𝐫𝐞𝐚𝐬𝐮𝐫𝐞𝐬 𝐨𝐟 𝐊𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 𝐯𝐢𝐚 𝐀𝐜𝐭𝐢𝐯𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠..."
        
        Tags:
        
        LLMs and basic Renault Knowledge
        LM + KG
        RAG (Retrieval-Augmented Generation)
        
        2024-02-25 About
    - SPIKE-KBC
    - Text, KG and embeddings
    - Text to KG
Graphs + NLP
Graph visualization
- Graphviz
- Sigma.js
- TouchGraph
Knowledge Graphs
- Combining knowledge graphs
  - Entity alignment
- Enterprise Knowledge Graph
  - eccenca
  - Enterprise Knowledge Graph Platform
    - eccenca
    - Stardog
  - Knowledge Graph Conference 2019
- Google Knowledge Graph
- KBPedia
- KG: tasks
  - Combining knowledge graphs
  - Entities to topics
  - Knowledge Graph Completion
    - Knowledge Graph Construction
      - SPIKE-KBC
      - Text to KG
        
        [2302.05019] A Comprehensive Survey on Automatic Knowledge Graph Construction
        
        Tags:
        
        Arxiv Doc
        Knowledge Graph Construction
        Survey / Review
        
        2023-02-15 About
        
        KGC Workshop
        
        Tags:
        
        ESWC 2022
        Knowledge Graph Construction
        Workshop
        
        2022-05-30 About
        
        SPIKE for Knowledge Base Construction
        
        Tags:
        
        Extractive Search
        Knowledge Graph Construction
        SPIKE-KBC
        
        a platform for knowledge base construction based on the SPIKE extractive search engine
        
        2022-02-22 About
        
        Yosi Shamay sur Twitter : "a new platform for rapid ad-hoc knowledgebase construction using extractive search...."
        
        Tags:
        
        Extractive Search
        Human in the loop
        Knowledge Graph Construction
        SPIKE-KBC
        Tweet
        Yoav Goldberg
        
        > a fully functional human-machine hybrid tool for rapid construction of knowledgebases (KB) in biomedicine. [Tweet](https://twitter.com/yoavgo/status/1495868946393800715) de [Yoav Goldberg](tag:yoav_goldberg) > This means that now we have a protocol, and a supporting toolset, by which researchers can create personalized, ad-hoc knowledge-basses in their fields of expertise, or in a field they want to get into, in hours. This is a great productivity boost to science. > How do you construct a KB with ES? > > 1. choose a topic. > 2. define a set of allowed relations between entity classes. > 3. extract entities+relations with the powerful NLP extraction engine-SPIKE. > 4. Import relations to the app and annotate/edit
        
        2022-02-22 About
        
        The complexity to construct knowledge graphs and how low code tools can help or hurt you | Medium
        
        Tags:
        
        François Scharffe
        Knowledge Graph Construction
        Knowledge Graphs
        
        2021-01-30 About
        
        [2008.08995] Constructing a Knowledge Graph from Unstructured Documents without External Alignment
        
        Tags:
        
        Arxiv Doc
        Knowledge Graph Construction
        Knowledge Graphs and NLP
        Virtual knowledge graph
        
        Building a virtual KG from unstructured documents > we first extract knowledge tuples in their surface form from unstructured documents, encode them using a pre-trained language model, and link the surface-entities via the encoding to form the graph structure.
        
        2020-08-21 About
        
        Trip Report: AKBC 2019 (1st Conference on Automated Knowledge Base Construction)
        
        Tags:
        
        Knowledge bases
        Knowledge Graph Construction
        
        2019-06-03 About
        
        Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction - ACL Anthology
        
        Tags:
        
        AI 4 IP
        EMNLP 2018
        Knowledge Graph + Deep Learning
        Knowledge Graph Construction
        Multitask Learning in NLP
        
        Attempting to answer questions such as: "What is the task described in this paper?", "what method was used in solving the task?", "what dataset did the paper use?". The multi-task setup reduces cascading errors between tasks and leverages cross-sentence relations through coreference links.
        
        2019-02-09 About
        
        Romain Vial (Hyperlex) at Paris NLP meetup, slides
        
        Tags:
        
        BERT
        Good
        Information extraction
        Knowledge Graph Completion
        Knowledge Graph Construction
        Long documents
        NLP + juridique
        Paris NLP meetup
        Slides
        
        > Hyperlex is a contract analytics and management solution powered by artificial intelligence. Hyperlex helps companies manage and make the most of their contract portfolio by identifying relevant information and data to manage key contractual commitments. > Take-home message: > > - Sentence representation starts to be well understood empirically > - Large document representation is still an open (and interesting) problem!
        
        2019-01-24 About
        
        Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction
        
        Tags:
        
        EMNLP 2018
        Good
        Knowledge Graph Construction
        Knowledge Graphs
        Scientific information extraction
        
        > A multi-task setup of identifying and classifying entities, relations, and coreference clusters in scientific articles. > The framework supports **construction of a scientific knowledge graph** [http://nlp.cs.washington.edu/sciIE/](http://nlp.cs.washington.edu/sciIE/)
        
        2018-11-04 About
    - Link Prediction
      - BLP
      - LM + Link Prediction
        
        [2210.09338] Deep Bidirectional Language-Knowledge Graph Pretraining
        
        Tags:
        
        Arxiv Doc
        Chris Manning
        Jure Leskovec
        Knowledge Graphs and NLP
        LM + Link Prediction
        NLP@Stanford
        
        > DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised method to pretrain a deeply joint language-knowledge foundation model from text and KG at scale. > > Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities.
        
        2022-10-23 About
    - Relation Extraction
    - RotatE
    - Taxonomy expansion task
    - TransE
      [2308.00081] Towards Semantically Enriched Embeddings for Knowledge Graph Completion
      
      Tags:
      
      Arxiv Doc
      Knowledge Graph Completion
      Knowledge Graph Embeddings
      Survey / Review
      
      nothing new, but not a bad review of the current state of work
      
      2023-08-02 About
      [2304.02711] Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning
      
      Tags:
      
      Arxiv Doc
      ChatGPT: use-case
      Knowledge Extraction
      Knowledge Graph Completion
      Relation Extraction
      Zero-Shot Learning
      
      > Structured Prompt Interrogation and Recursive Extraction of Semantics (SPIRES), a Knowledge Extraction approach that relies on the ability of Large Language Models (LLMs) to perform zero-shot learning (ZSL) and general-purpose query answering from flexible prompts and return information conforming to a specified schema.
      
      2023-04-07 About
      StATIK: Structure and Text for Inductive Knowledge Graph Completion - ACL Anthology (2022)
      
      Tags:
      
      Knowledge Graph Completion
      LM + KG
      
      > StATIK uses Language Models to extract the semantic information from text descriptions, while using Message Passing Neural Networks to capture the structural information. > Structure is incorporated through a Message Passing Neural Network (MPNN) (Gilmer et al., 2017) that aggregates information from a neighborhood defined around each entity, while textual information is incorporated through a pretrained language model such as BERT KG are dynamic (new entities are added) -> we want an inductive KG completion model (able to generalize to unseen entities)
      
      2022-07-17 About
      [2009.07938] Type-augmented Relation Prediction in Knowledge Graphs
      
      Tags:
      
      AI@IBM
      Arxiv Doc
      Knowledge Graph Completion
      Link Prediction
      
      2020-09-19 About
      [2009.00318] More is not Always Better: The Negative Impact of A-box Materialization on RDF2vec Knowledge Graph Embeddings
      
      Tags:
      
      Arxiv Doc
      Knowledge Graph Completion
      Knowledge Graph Embeddings
      RDF2VEC
      
      > we argue that despite the huge body of work devoted on completing missing information in knowledge graphs, such missing implicit information is actually a signal, not a defect
      
      2020-09-02 About
      [1911.03903] A Re-evaluation of Knowledge Graph Completion Methods
      
      Tags:
      
      Arxiv Doc
      Critical evaluation
      Knowledge Graph Completion
      
      2020-07-28 About
      Peter Bloem sur Twitter : "One of the messages from Ruffinelli et al 2020..."
      
      Tags:
      
      Knowledge Graph Completion
      Negative Sampling
      Noise contrastive estimation
      Peter Bloem
      Tweet
      
      [about this paper](doc:2020/05/you_can_teach_an_old_dog_new_tr) "You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings" > in knowledge base completion, negative sampling works better if you then take the softmax + multiclass CE over the whole sample (as opposed to sigmoids and binary CE) @Mniepert: > My take: softmax + multi-class CE better under “open world” assumption.
      
      2020-05-19 About
      [2003.08001] Realistic Re-evaluation of Knowledge Graph Completion Methods: An Experimental Study
      
      Tags:
      
      Arxiv Doc
      Critical evaluation
      Embedding evaluation
      Graph Embeddings
      Knowledge Graph Completion
      Link Prediction
      
      data redundancy (reverse relations), Cartesian product relations > A more fundamental defect of these models is that the link prediction scenario, given such data, is non-existent in the real-world
      
      2020-05-15 About
      [2001.09522] TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network
      
      Tags:
      
      Arxiv Doc
      Discuté avec Raphaël
      Graph neural networks
      Knowledge Graph Completion
      Microsoft Research
      NLP@Microsoft
      Taxonomies
      Taxonomy expansion task
      Text-Aware KG embedding
      TheWebConf 2020
      
      how to add a set of new concepts to an existing taxonomy. [Tweet](https://twitter.com/mickeyjs6/status/1253772146142216194?s=20) [GitHub](https://github.com/mickeystroller/TaxoExpan) > we study the taxonomy expansion task: given an existing taxonomy and a set of new emerging concepts, we aim to automatically expand the taxonomy to incorporate these new concepts (without changing the existing relations in the given taxonomy). > To the best of our knowledge, this is the first study on **how to expand an existing directed acyclic graph (as we model a taxonomy as a DAG) using self-supervised learning**. Self-supervised framework, the existing taxonomy being used as training data: it learns a model to predict whether a query concept is the direct hyponym of an anchor concept. > 2 techniques: > > 1. a **position-enhanced graph neural network that encodes the local structure of an anchor concept** in the existing taxonomy, > 2. a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data. Regarding 1: uses [GNN](/tag/graph_neural_networks.html) to model the "ego network" of concepts (potential “siblings” and “grand parents” of the query concept). > Regular GNNs fail to distinguish nodes with different relative positions to the query (i.e., some nodes are grand parents of the query while the others are siblings of the query). To address this limitation, we present a simple but effective enhancement to inject such position information into GNNs using position embedding. We show that such embedding can be easily integrated with existing GNN architectures (e.g., [GCN](/tag/graph_convolutional_networks) and GAT) and significantly boosts the prediction performance Regarding point 2: uses InfoNCE loss, cf. [Contrastive Predictive Coding](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1807.03748) > Instead of predicting whether each individual ⟨query concept, anchor concept⟩ pair is positive or not, we first group all pairs sharing the same query concept into a single training instance and learn a model to select the positive pair among other negative ones from the group. (Hum, ça me rappelle quelque chose) > assume each concept (in existing taxonomy + set of new concepts) has an initial embedding vector learned from some text associated with this concept. To keep things tractable, only attempts to find a single parent node of each new concept.
      
      2020-04-25 About
      Chengkai Li sur Twitter : "Link prediction methods on knowledge graphs don't work..."
      
      Tags:
      
      Critical evaluation
      Knowledge Graph Completion
      Knowledge Graph Embeddings
      Link Prediction
      Tweet
      
      [paper](/doc/2020/05/2003_08001_realistic_re_evalu)
      
      2020-03-29 About
      [1909.03193] KG-BERT: BERT for Knowledge Graph Completion
      
      Tags:
      
      Arxiv Doc
      BERT
      Discuté avec Raphaël
      Knowledge Graph Completion
      Pre-Trained Language Models
      Transformers
      
      Pre-trained language models for knowledge graph completion. **Triples are treated as textual sequences**. (Hum, j'ai déjà vu ça quelque part. Ah, peut-être [RDF2VEC](tag:rdf2vec)? // TODO à voir) Takes entity and relation descriptions of a triple as input and computes scoring function of the triple with the KG-BERT language model > we first treat entities, relations and triples as textual sequences and turn knowledge graph completion into a sequence classification problem. We then fine-tune BERT model on these sequences for predicting the plausibility of a triple or a relation. The method [GitHub](https://github.com/yao8839836/kg-bert)
      
      2020-03-22 About
      [1902.10197] RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
      
      Tags:
      
      Arxiv Doc
      KD-MKB biblio
      Knowledge Graph Completion
      Knowledge Graph Embeddings
      RotatE
      
      > We study the problem of learning representations of entities and relations in knowledge graphs for predicting missing links.
      
      2020-03-03 About
      Reasoning With Neural Tensor Networks for Knowledge Base Completion (2013)
      
      Tags:
      
      Andrew Ng
      Chris Manning
      Knowledge Graph + Deep Learning
      Knowledge Graph Completion
      Richard Socher
      Text-Aware KG embedding
      
      **Predicting the likely truth of additional facts based on existing facts in the knowledge base.** > we introduce an expressive neural tensor network suitable for reasoning over relationships between two entities. Most similar work: [Bordes et al.](http://127.0.0.1:8080/semanlink/doc/2019/08/learning_structured_embeddings_) (2011) Contributions: 1. new neural tensor network (**NTN**) suitable for reasoning over relationships between two entities. Generalizes several previous neural network models and provides a more powerful way to model relational information than a standard neural network layer. 2. a new way to represent entities in knowledge bases, as the average of their constituting word vectorss, allowing the sharing of statistical strength between the words describing each entity (e.g., Bank of China and China). 3. incorporation of word vectors which are trained on large unlabeled text > We **learn to modify word representations via grounding in world knowledge**. This essentially allows us to analyze word embeddings and query them for specific relations. Furthermore, the resulting vectors could be used in other tasks such as named entity recognition or relation classification in natural language **Makes use of entity name**: NTN first learns word vectors from an auxiliary news corpus, and then initializes the representation of each entity by averaging the vectors of words contained in its name. For example, **the embedding of AlfredHitchcock is initialized by the average word vectors of “alfred” and “hitchcock”**... This kind of methods model textual information separately from KG facts, and hence fail to leverage interactions between them.[src](doc:2019/05/knowledge_graph_embedding_a_su)
      
      2019-08-03 About
      A2N: Attending to Neighbors for Knowledge Graph Inference - ACL 2019
      
      Tags:
      
      ACL 2019
      Attention in Graphs
      Google Research
      Knowledge Graph Completion
      Knowledge Graphs
      NLP@Google
      
      > State-of-the-art models for knowledge graph completion aim at learning a fixed embedding representation of entities in a multi-relational graph which can generalize to infer unseen entity relationships at test time. This can be sub-optimal as it requires memorizing and generalizing to all possible entity relationships using these fixed representations. We thus propose a novel **attention-based method to learn query-dependent representation of entities** which adaptively combines the relevant graph neighborhood of an entity leading to more accurate KG completion.
      
      2019-07-31 About
      Romain Vial (Hyperlex) at Paris NLP meetup, slides
      
      Tags:
      
      BERT
      Good
      Information extraction
      Knowledge Graph Completion
      Knowledge Graph Construction
      Long documents
      NLP + juridique
      Paris NLP meetup
      Slides
      
      > Hyperlex is a contract analytics and management solution powered by artificial intelligence. Hyperlex helps companies manage and make the most of their contract portfolio by identifying relevant information and data to manage key contractual commitments. > Take-home message: > > - Sentence representation starts to be well understood empirically > - Large document representation is still an open (and interesting) problem!
      
      2019-01-24 About
      Knowledge base completion by learning pairwise-interaction differentiated embeddings | SpringerLink (2015)
      
      Tags:
      
      Knowledge Graph Completion
      Knowledge Graph Embeddings
      LIP6
      Patrick Gallinari
      
      Embedding entities and relations in the knowledge base to low dimensional vector representations and then predict the possible truth of additional facts to extend the knowledge base
      
      2018-01-27 About
      Knowledge Graph and Text Jointly Embedding (2014)
      
      Tags:
      
      Knowledge Graph Completion
      Knowledge Graph Embeddings
      Microsoft Research
      NLP@Microsoft
      Thèse IRIT-Renault: biblio initiale
      Word + Entity embeddings
      
      method of **jointly embedding knowledge graphs and a text corpus** so that **entities and words/phrases are represented in the same vector space**. Promising improvement in the accuracy of predicting facts, compared to separately embedding knowledge graphs and text (in particular, enables the prediction of facts containing entities out of the knowledge graph) [cité par J. Moreno](/doc/?uri=https%3A%2F%2Fhal.archives-ouvertes.fr%2Fhal-01626196%2Fdocument)
      
      2018-01-05 About
      [1506.01094] Traversing Knowledge Graphs in Vector Space
      
      Tags:
      
      Arxiv Doc
      Embeddings
      Knowledge Graph Completion
      Knowledge Graph Embeddings
      Knowledge Graphs
      Path queries
      
      Knowledge graphs often have missing facts (edges) which disrupts path queries. Recent models for knowledge base completion impute missing facts by embedding knowledge graphs in vector spaces. We show that these models can be recursively applied to answer path queries, but that they suffer from cascading errors. This motivates a new "compositional" training objective, which dramatically improves all models' ability to answer path queries, in some cases more than doubling accuracy.
      
      2015-10-31 About
  - Knowledge Graph Construction
  - Triple Classification
- Knowledge Graph + ML
  - KG-aware low-resource learning
  - Knowledge Graph + Deep Learning
    - Attention + Knowledge Graphs
    - KG-augmented Language Models
    - Language Models as Knowledge Bases
    - Thèse IRIT-Renault NLP-KB
      - KD-MKB
        KD-MKB biblio
        
        facebookresearch/faiss: A library for efficient similarity search and clustering of dense vectors.
        
        Tags:
        
        Facebook FAIR
        faiss
        GitHub project
        KD-MKB biblio
        Locality Sensitive Hashing
        Nearest neighbor search
        
        Algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. Faiss is written in C++ with complete wrappers for Python/numpy. [paper](https://arxiv.org/abs/1702.08734)
        
        2020-06-09 About
        
        [1909.04164] Knowledge Enhanced Contextual Word Representations
        
        Tags:
        
        Allen Institute for AI (AI2)
        Arxiv Doc
        Contextualized word representations
        EMNLP 2019
        Entities and LM
        Good
        Grounded Language Learning
        KD-MKB biblio
        KG-augmented Language Models
        KnowBert
        Knowledge-augmented Language Models
        Knowledge-driven embeddings
        Knowledge Graphs in NLP
        Multiple Knowledge Bases
        
        General method to **embed multiple knowledge bases into pre-trained language models** (KB in the sense as fixed collection of entity nodes) > The key idea is to explicitly model entity spans in the input text and use an **entity linker** to retrieve relevant entity embeddings from a KB to form knowledge enhanced entity-span representations. > Then, update contextual word representations via a form of **word-to-entity attention**. > In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that **combines a small amount of entity linking supervision with a large amount of raw text**.
        
        2020-05-13 About
        
        [1907.04829] BAM! Born-Again Multi-Task Networks for Natural Language Understanding
        
        Tags:
        
        Arxiv Doc
        Chris Manning
        KD-MKB biblio
        Knowledge distillation
        Multitask Learning in NLP
        Quoc V. Le
        
        > **knowledge distillation where single-task models teach a multi-task model.** We enhance this training with **teacher annealing**, a novel method that gradually transitions the model from distillation to supervised learning, helping the multi-task model surpass its single-task teachers
        
        2020-05-12 About
        
        [1807.08447] LinkNBed: Multi-Graph Representation Learning with Entity Linkage
        
        Tags:
        
        AI@Amazon
        Arxiv Doc
        Combining knowledge graphs
        Graph Embeddings
        KD-MKB biblio
        
        > a deep relational learning framework that **learns entity and relationship representations across multiple graphs**. We identify entity linkage across graphs as a vital component to achieve our goal. We design a novel objective that leverage entity linkage and build an efficient multi-task training procedure > > We posit that **combining graph alignment task with deep representation learning across multi-relational graphs** has potential to induce a synergistic effect on both tasks
        
        2020-05-11 About
        
        Bootstrapping Entity Alignment with Knowledge Graph Embedding | IJCAI
        
        Tags:
        
        Entity alignment
        IJCAI
        KD-MKB biblio
        Knowledge Graph Embeddings
        
        Embedding-based entity alignment: finds entity alignment by measuring the similarities between entity embeddings > Existing approaches are challenged by the lack of enough prior alignment as labeled training data. > A bootstrapping approach: it iteratively labels likely entity alignment as training data for learning alignment-oriented KG embeddings. [GitHub](https://github.com/nju-websoft/BootEA)
        
        2020-05-11 About
        
        [1706.00384] Deep Mutual Learning
        
        Tags:
        
        Arxiv Doc
        KD-MKB biblio
        Knowledge distillation
        Mutual Learning
        
        > In this paper we explore a different but related idea to model distillation – that of mutual learning. Distillation starts with a powerful large and pre-trained teacher network and performs one-way knowledge transfer to a small untrained student. In contrast, in mutual learning we start with a pool of untrained students who learn simultaneously to solve the task together. [critic here](doc:2020/06/1804_03235_large_scale_distri): > Zhang et al. (2017) reported a benefit in quality over basic distillation, but they compare distilling model M1 into model M2 with training model M1 and model M2 using codistillation; they do not compare to distilling an ensemble of models M1 and M2 into model M3. > > ... > > we can achieve the 70.7% they report for online distillation using traditional offline distillation.
        
        2020-05-11 About
        
        [1906.07241] Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling
        
        Tags:
        
        Allen Institute for AI (AI2)
        Arxiv Doc
        KD-MKB biblio
        KG-augmented Language Models
        Knowledge Graphs and NLP
        
        > a **neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context**. These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. > The KGLM maintains a dynamically growing local knowledge graph, a subset of the knowledge graph that contains entities that have already been mentioned in the text, and their related entities.
        
        2020-05-11 About
        
        [1902.10197] RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
        
        Tags:
        
        Arxiv Doc
        KD-MKB biblio
        Knowledge Graph Completion
        Knowledge Graph Embeddings
        RotatE
        
        > We study the problem of learning representations of entities and relations in knowledge graphs for predicting missing links.
        
        2020-03-03 About
        
        Knowledge Graph Embedding: A Survey of Approaches and Applications - IEEE (2017)
        
        Tags:
        
        KD-MKB biblio
        Knowledge Graph Embeddings
        Survey / Review
        
        - "facts only" based embeddings - Translational Distance Models, distance-based scoring functions - TransE and co: entities and relations as points in vector space - TransH: relation-specific hyperplanes. - Gaussian Embeddings: entities and relations as random variables - Semantic matching models, similarity-based scoring functions - RESCAL (entities as a vector, relation as a matrix) and familiy - DistMult (relation as diagonal matrix) - Holographic Embeddings (HolE) - etc. - Matching with Neural Networks - SME, etc - incorporating other info: entity types, relation paths, textual descriptions, as well as logical rules - entity types - semantically smooth embedding (SSE): entities of the same type to stay close to each other in the embedding space - type-embodied knowledge representation learning (TKRL), which can handle hierarchical entity categories and multiple category labels - Applications in downstream tasks
        
        2019-05-27 About
        
        [1901.11504] Multi-Task Deep Neural Networks for Natural Language Understanding
        
        Tags:
        
        Arxiv Doc
        BERT
        KD-MKB biblio
        Multi-task learning
        NLP@Microsoft
        NLU
        Pre-Trained Language Models
        
        outperforms BERT in nine of eleven benchmark NLP tasks
        
        2019-02-17 About
        
        KD-MKB paper
        
        Knowledge Base Embedding By Cooperative Knowledge Distillation - ACL Anthology
        
        Tags:
        
        KD-MKB paper
        
        2020-12-05 About
        
        KD-MKB related
        
        [1912.08422] Distilling Structured Knowledge into Embeddings for Explainable and Accurate Recommendation
        
        Tags:
        
        AI@Facebook
        Arxiv Doc
        Explainable AI
        KD-MKB related
        Knowledge distillation
        Recommender Systems
        
        distilling structured knowledge from a differentiable path-based recommendation model. > proposed framework can achieve state-of-the-art recommendation performance and meanwhile provide interpretable recommendation reasons
        
        2020-05-12 About
        
        raphaelsty/mkb: Knowledge Base Embedding By Cooperative Knowledge Distillation
        
        Tags:
        
        GitHub project
        KD-MKB
        Raphaël Sourty
        
        2020-07-24 About
- Knowledge Graph Embeddings
- Knowledge Graphs and NLP
- Knowledge graph search engine
- Machine Learning + Semantic Web
- Microsoft Concept Graph
- Open Knowledge Network
- Personal Knowledge Graph
  - Semanlink
    - About Semanlink
      - LinkTo Semanlink
        
        Helping online communities to semantically enrich folksonomies
        
        Tags:
        
        Folksonomy
        LinkTo Semanlink
        
        2012-02-20 About
        
        Linking Folksonomies and Ontologies for Supporting Knowledge Sharing: a State of the Art
        
        Tags:
        
        folksonomies ontologies
        LinkTo Semanlink
        Semantic tagging
        
        Social tagging systems have recently become very popular as a means to classify large sets of resources shared among on-line communities over the social Web. However, the folksonomies resulting from the use of these systems revealed limitations: tags are ambiguous and their spelling may vary, and folksonomies are difficult to exploit in order to retrieve or exchange information. This report compares the recent attempts to overcome these limitations and to support the use of folksonomies with formal languages and ontologies from the Semantic Web.<br/> Projet ISICIL : Intégration Sémantique de l'Information par des Communautés d'Intelligence en Ligne
        
        2009-11-03 About
        
        Manageability - Open Source Social Networking Applications Written in Java
        
        Tags:
        
        LinkTo Semanlink
        
        2008-05-17 About
        
        Semanlink - semantische Bookmarks
        
        Tags:
        
        LinkTo Semanlink
        
        2008-04-10 About
        
        Social Book marking Script - Clones of Digg.com
        
        Tags:
        
        LinkTo Semanlink
        
        2008-02-15 About
        
        semanlink 소개 - Mimul's Developer World
        
        Tags:
        
        LinkTo Semanlink
        
        2007-12-30 About
        
        Nouveautés sur le site hyperSOLutions
        
        Tags:
        
        Archive
        hyperSOLutions
        LinkTo Semanlink
        
        Archive datée de 1998 mentionnant Semanlink
        
        2007-11-13 About
        
        web - Tom Heath's Displacement Activities
        
        Tags:
        
        LinkTo Semanlink
        Tom Heath
        
        2007-09-18 About
        
        Juan Sequeda - Meetings
        
        Tags:
        
        LinkTo Semanlink
        
        2007-09-18 About
        
        Applications Built on Jena - Tom Heath's Displacement Activities - by tomheath
        
        Tags:
        
        Jena
        LinkTo Semanlink
        Tom Heath
        
        2007-08-22 About
        
        ConTag A Tagging System: Linking the Semantic Desktop with Web 2.0
        
        Tags:
        
        LinkTo Semanlink
        Semantic Desktop
        Tagging
        Web 2.0
        
        Diploma Thesis qui cite Semanlink
        
        2007-04-20 About
        
        Folksonomies e tagging/3 « emmeesse
        
        Tags:
        
        LinkTo Semanlink
        
        2007-04-03 About
        
        Semanlink on Netvouz
        
        Tags:
        
        LinkTo Semanlink
        
        Netvouz is an online social bookmark manager
        
        2006-05-28 About
        
        chillyinside.com » Blog Archive » 2006 Jena User Conference
        
        Tags:
        
        Jena User Conference
        LinkTo Semanlink
        
        2006-05-26 About
        
        Jena User Conference - some interesting UI related papers/presentations from Shabajee, Paul on 2006-05-17 (public-semweb-ui@w3.org from May 2006)
        
        Tags:
        
        Jena User Conference
        LinkTo Semanlink
        
        2006-05-25 About
        
        Peter Hale PhD Research - User Driven Programming
        
        Tags:
        
        LinkTo Semanlink
        User Driven Modelling
        
        My current research is on a technique of 'User Driven Modelling/Programming'. My intention is to enable non-programmers to create software from a user interface that allows them to model a particular problem or scenario.
        
        2006-05-13 About
        
        dowhatimean.net » [juc] François-Paul Servant – Semanlink
        
        Tags:
        
        LinkTo Semanlink
        Richard Cyganiak
        
        > Unlike most tagging systems, Semanlink lets you arrange tags into a concept hierarchy > > It’s a del.icio.us on steroids. The UI is not yet quite streamlined enough for my taste, but it looks usable. It’s technologically simple – Jena memory model, file-based persistence – so it should be hackable. Quite cool.
        
        2006-05-13 About
        
        Semanlink on del.icio.us
        
        Tags:
        
        LinkTo Semanlink
        
        2006-05-13 About
    - NLP 4 Semanlink
      - Semanlink Tag Finder
        
        raphaelsty/ckb: Contextual knowledge bases
        
        Tags:
        
        CKB
        GitHub project
        Raphaël Sourty
        Semanlink Tag Finder
        
        Une implémentation de [BLP](tag:blp) [[2010.03496] Inductive Entity Representations from Text via Link Prediction](doc:2020/11/2010_03496_inductive_entity_r)
        
        2020-11-09 About
    - Semanlink: archives
    - Semanlink2
      - NLP 4 Semanlink
    - Semanlink dev
    - SL GUI
- Product Knowledge Graph
- The Knowledge Graph Conference
  - Knowledge Graph Conference 2019
    KGC23 Keynote: The Future of Knowledge Graphs in a World of LLMs — Denny Vrandečić, Wikimedia - YouTube
    Tags:
    
    Denny Vrandečić
    LLM + KG
    The Knowledge Graph Conference
    Wikimedia
    YouTube video
    2024-03-07 About
    The Knowledge Graph Conference 2023
    Tags:
    
    The Knowledge Graph Conference
    2023-02-24 About
    KGC 2021 Call for Presentations | The Knowledge Graph Conference
    Tags:
    
    The Knowledge Graph Conference
    2021-01-30 About
- Virtual knowledge graph
Property Graphs