Semanlink - Google Research AND Language Model

Recherche de Mot-clé

Recherche de Doc

Préférences...

Mots-clés liés

AI + Knowledge Bases - Arxiv Doc - fast.ai - Google Research - Language Model - Nearest neighbor search - Neural Memory - NLP@Google - NLP and Search - NN / Symbolic AI hybridation - Search Engines - Text Classification - Tutorial -

Google Research AND Language Model

Descendants partagés

4 Documents

[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge

Tags:

> a neural language model that includes **an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge.**... **The model can be updated without re-training by manipulating its symbolic representations**. In particular this model allows us to add new facts and overwrite existing ones.

> a **neural language model which learns to access information
in a symbolic knowledge graph.**

> This
model builds on the recently-proposed [Entities as
Experts](doc:2020/07/2004_07202_entities_as_expert) (EaE) language model (Févry et al., 2020),
which extends the same transformer (Vaswani
et al., 2017) architecture of BERT (Devlin et al., 2019) with an additional external memory for entities.
>
> After training EaE, the embedding associated
with an entity will (ideally) capture information
about the textual context in which that
entity appears, and by inference, the entity’s semantic
properties
>
> we include an additional
memory called a fact memory, which encodes
triples from a symbolic KB.
>
> This combination results in a
neural language model which learns to access information
in a the symbolic knowledge graph.

TODO:

- read again IBM's [Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) ("an effort to avoid encoding general knowledge in the transformer network itself")
- compare with [[1907.05242] Large Memory Layers with Product Keys](doc:2019/07/_1907_05242_large_memory_layer)
- how does it relate with [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)?

2020-07-09 A propos

Building a Search Engine with BERT and TensorFlow - Towards Data Science

Tags:

2020-01-12 A propos

Tutorial on Text Classification (NLP) using ULMFiT and fastai Library in Python - Analytics Vidhya

Tags:

2018-12-01 A propos

A Billion Words: Because today's language modeling standard should be higher

Tags:

2014-05-02 A propos