Semanlink - [2008.12813] HittER: Hierarchical Transformers for Knowledge Graph Embeddings

[2008.12813] HittER: Hierarchical Transformers for Knowledge Graph Embeddings

Tags:

Au sujet de ce document

sl:arxiv_author :
sl:arxiv_firstAuthor : Sanxing Chen
sl:arxiv_num : 2008.12813
sl:arxiv_published : 2020-08-28T18:58:15Z
sl:arxiv_summary : This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical Transformer model to jointly learn Entity-relation composition and Relational contextualization based on a source entity's neighborhood. Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block. We further design a masked entity prediction task to balance information from the relational context and the source entity itself. Experimental results show that HittER achieves new state-of-the-art results on multiple link prediction datasets. We additionally propose a simple approach to integrate HittER into BERT and demonstrate its effectiveness on two Freebase factoid question answering datasets.@en
sl:arxiv_title : HittER: Hierarchical Transformers for Knowledge Graph Embeddings@en
sl:arxiv_updated : 2021-10-06T04:52:07Z
sl:bookmarkOf : https://arxiv.org/abs/2008.12813
sl:creationDate : 2022-06-30
sl:creationTime : 2022-06-30T18:33:10Z

Infos sur le fichier

Bookmark of: https://arxiv.org/abs/2008.12813

Documents with similar tags (experimental)

[2002.06275] TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval

Tags:

2023-08-27 A propos

[2307.08621] Retentive Network: A Successor to Transformer for Large Language Models

Tags:

2023-07-20 A propos

[2306.07174] Augmenting Language Models with Long-Term Memory

Tags:

2023-06-13 A propos

[2305.15294] Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy

Tags:

2023-05-26 A propos

[2106.09685] LoRA: Low-Rank Adaptation of Large Language Models

Tags:

2023-03-21 A propos

[2206.02743] A Neural Corpus Indexer for Document Retrieval

Tags:

2023-01-18 A propos

[2002.01808] K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Tags:

2023-01-12 A propos

[2212.02623] Unifying Vision, Text, and Layout for Universal Document Processing

Tags:

2022-12-07 A propos

[2205.08012] CascadER: Cross-Modal Cascading for Knowledge Graph Link Prediction

Tags:

2022-07-07 A propos

[2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Tags:

2022-03-31 A propos

[2108.13934] Robust Retrieval Augmented Generation for Zero-shot Slot Filling

Tags:

2022-01-19 A propos

[1911.02655] Towards Domain Adaptation from Limited Data for Question Answering Using Deep Neural Networks

Tags:

2021-11-19 A propos

[2106.13474] Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Tags:

2021-10-21 A propos

[2004.09095] The State and Fate of Linguistic Diversity and Inclusion in the NLP World

Tags:

2021-10-03 A propos

[2109.08133] Phrase Retrieval Learns Passage Retrieval, Too

Tags:

2021-09-30 A propos

[2104.06979] TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

Tags:

> The most
successful previous approaches like InferSent (Conneau
et al., 2017), Universial Sentence Encoder
(USE) (Cer et al., 2018) and SBERT (Reimers and
Gurevych, 2019) heavily relied on labeled data to
train sentence embedding models.
>
> TSDAE can
achieve up to 93.1% of the performance of indomain
supervised approaches. Further, we
show that TSDAE is **a strong domain adaptation
and pre-training method for sentence
embeddings**, significantly outperforming other
approaches like Masked Language Model.

> During training, TSDAE
encodes corrupted sentences into fixed-sized
vectors and requires the decoder to reconstruct the
original sentences from this sentence embedding.

- <https://www.sbert.net/examples/unsupervised_learning/TSDAE/README.html>
- [github](https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning/TSDAE)
- [UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet](doc:2020/07/ukplab_sentence_transformers_s)
- [twitter](https://twitter.com/KexinWang2049/status/1433361957579538432):

> **TSDAE can learn domain-specific sentence embeddings with unlabeled sentences**
>
> Most importantly, instead of STS (Semantic Textual Similarity), **we suggest evaluating unsupervised sentence embeddings on the domain-specific tasks&datasets, which is the real use case for them**. Actually, STS scores do not correlate with performance on specific tasks.

2021-09-01 A propos

[2007.15779] Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Tags:

2021-04-11 A propos

[2010.03496] Inductive Entity Representations from Text via Link Prediction

Tags:

BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a
link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training)

> a method for **learning representations
of entities**, that uses a **pre-trained Transformer** based
architecture as an entity encoder, and
**link prediction training on a knowledge graph
with textual entity descriptions**.

> using entity descriptions,
an entity encoder is trained for link prediction in
a knowledge graph. The encoder can then be used
without fine-tuning to obtain features for entity classification
and information retrieval

Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their
objective targeted exclusively for link prediction (and not an objective that combines language modeling
and link prediction as Kepler)
performs better than Kepler's more complex one.

2020-11-03 A propos

[2009.07938] Type-augmented Relation Prediction in Knowledge Graphs

Tags:

2020-09-19 A propos

[2003.08001] Realistic Re-evaluation of Knowledge Graph Completion Methods: An Experimental Study

Tags:

2020-05-15 A propos

[2001.09522] TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network

Tags:

how to add a set of new concepts to an existing taxonomy.

[Tweet](https://twitter.com/mickeyjs6/status/1253772146142216194?s=20) [GitHub](https://github.com/mickeystroller/TaxoExpan)

> we study the taxonomy expansion task: given an
existing taxonomy and a set of new emerging concepts, we aim
to automatically expand the taxonomy to incorporate these new
concepts (without changing the existing relations in the given taxonomy).

> To the best of our knowledge, this is the first study on **how to
expand an existing directed acyclic graph (as we model a taxonomy
as a DAG) using self-supervised learning**.

Self-supervised framework, the existing taxonomy being used as training data: it learns a model to predict whether a query concept is the direct hyponym of an anchor concept.

> 2 techniques:
>
> 1. a **position-enhanced graph neural network that encodes the local structure of an anchor concept** in the existing taxonomy,
> 2. a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data.

Regarding 1: uses [GNN](/tag/graph_neural_networks.html) to model the "ego network" of concepts (potential “siblings”
and “grand parents” of the query concept).

> Regular
GNNs fail to distinguish nodes with different relative positions to
the query (i.e., some nodes are grand parents of the query while
the others are siblings of the query). To address this limitation, we
present a simple but effective enhancement to inject such position
information into GNNs using position embedding. We show that
such embedding can be easily integrated with existing GNN architectures
(e.g., [GCN](/tag/graph_convolutional_networks) and GAT) and significantly boosts the
prediction performance

Regarding point 2: uses InfoNCE loss, cf. [Contrastive Predictive Coding](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1807.03748)

> Instead of predicting
whether each individual ⟨query concept, anchor concept⟩ pair
is positive or not, we first group all pairs sharing the same query
concept into a single training instance and learn a model to select
the positive pair among other negative ones from the group.

(Hum, ça me rappelle quelque chose)

> assume each concept (in existing taxonomy + set of new concepts) has an initial embedding
vector learned from some text associated with this concept.

To keep things tractable, only attempts to find a single parent node of each new concept.

2020-04-25 A propos

[1911.02168] CoKE: Contextualized Knowledge Graph Embedding

Tags: