Semanlink - NLP tasks / problems ; Neural networks AND Knowledge Graphs

Ravi Theja sur X : "𝐀𝐜𝐭𝐢𝐯𝐞𝐑𝐀𝐆: 𝐑𝐞𝐯𝐞𝐚𝐥𝐢𝐧𝐠 𝐭𝐡𝐞 𝐓𝐫𝐞𝐚𝐬𝐮𝐫𝐞𝐬 𝐨𝐟 𝐊𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 𝐯𝐢𝐚 𝐀𝐜𝐭𝐢𝐯𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠..."

Tags:

2024-02-25 About

ML Blog - Improve ChatGPT with Knowledge Graphs

Tags:

2023-07-04 About

[2304.02711] Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning

Tags:

2023-04-07 About

[2205.00820] Entity-aware Transformers for Entity Search

Tags:

2022-07-12 About

[2205.15952] Knowledge Graph -- Deep Learning: A Case Study in Question Answering in Aviation Safety Domain

Tags:

2022-06-11 About

[2110.08151] mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

Tags:

[Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_)

> Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**.
>
> the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features.

[Github](https://github.com/studio-ousia/luke)

> Entity representations are known to enhance
language models in mono-lingual settings
(Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019:  [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al.,
2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context))
presumably by introducing real-world knowledge.
We show that using entity representations facilitates
cross-lingual transfer by providing languageindependent
features.
>
> Multilingual extension of LUKE. The model is trained with the multilingual
masked language modeling (MLM) task as well
as the masked entity prediction (MEP) task with
Wikipedia entity embeddings

> We investigate two ways of using the entity representations
in cross-lingual transfer tasks:
> 1. perform
entity linking for the input text, and append
the detected entity tokens to the input sequence.
The entity tokens are expected to provide language independent
features to the model
> 2. use the entity
[MASK] token from the MEP task as a languageindependent
feature extractor.

2022-04-17 About

Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."

Tags:

2022-04-13 About

Domain adaptation of word embeddings through the exploitation of in-domain corpora and knowledge bases (PhD Thesis 2021)

Tags:

Thèse de Hicham El Boukkouri, univ Paris-Saclay

[Github](https://github.com/helboukkouri/phd-code)

### Goal

Given a target specialized domain, improve the quality of general-domain
word representations using in-domain corpora and/or knowledge bases

### Contributions

#### a method for specializing general-domain embeddings in a [Low-Resource](tag:nlp_low_resource_scenarios) context.

> - train static representations on the task corpus, 
> - resume the
pre-training of general-domain contextual embeddings on the same task corpus,
> - finally, combine both static and contextual representations into
one final model

#### we tackle the issue of using a general-domain vocabulary in a specialized domain

#### Evaluation of  re-training vs training from scratch on specialized corpora using a specialized vocabulary

training from scratch better, but not that much: re-training from a general model
is still appropriate as it is less expensive and leads to comparable, although
slightly lower, performance

#### Regarding subword-based tokenization systems
> we argue that they are inconvenient in practice -> CharacterBERT, a variant
of BERT that uses ELMo’s character-based system instead of WordPieces. More convenient ti use, superior robustness to misspellings

#### Ways to specialize general-domain representations using knowledge bases

a strong baseline using a simple
method relying on graph embeddings and concatenation, using only is_a relation

> both static and contextual embeddings may effectively be
specialized using this simple approach

#### Knowledge Injection Modules (KIM) that
inject the knowledge representations directly within the BERT-like models' architecture

### Notes

> our experiments focused on a single
setting (i.e. the medical domain and the English language)

> meta-embeddings, an approach that consists in combining
different sets of representations for achieving improved performance

2022-03-23 About

Studio Ousia sur Twitter : "Now using LUKE is easier than ever!" / Twitter

Tags:

2022-03-15 About

[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce

Tags:

2020-12-14 About

raphaelsty/ckb: Contextual knowledge bases

Tags:

2020-11-09 About

[1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification

Tags:

A model that performs **text classification using entities in a knowledge base**.

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 About

[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision

Tags:

2020-07-11 About

[2002.10640] Differentiable Reasoning over a Virtual Knowledge Base

Tags:

2020-07-11 About

Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge (ACL Anthology 2020)

Tags:

Aim to leverage both contextual representation of input text (deep LMs)  and  knowledge derived
from curated KBs ([Wordnet](tag:wordnet)) to improve [slot tagging](tag:slot_tagging) in the presence of [out-of-vocab](tag:oov) words ([few-shot scenario](tag:few_shot_learning))

Method:

1. retrieve potentially relevant KB entities and
encode them into distributed representations that
describe global graph-structured information
2. BERT encoder
layer to capture context-aware representations of
the sequence and attend to the KB embeddings
using multi-level graph attention
3. integrate
BERT embeddings and the KB embeddings
to predict the slot type

Contributions:

1. feasibility of applying lexical ontology
to facilitate recognizing OOV words. First to consider the large-scale background
knowledge for enhancing context-aware
slot tagging models.
2. a knowledge integration mechanism that uses multi-level graph
attention to model explicit lexical relations.
3.experiments on two benchmark datasets

> our method makes a notable difference in a
scenario where samples are linguistically diverse,
and large vocab exists.

(Better improvements when using RNN than BERT, because BERT already contains a lot of background knowledge)

2020-07-04 About

[1909.04164] Knowledge Enhanced Contextual Word Representations

Tags: