Semanlink - Transformers

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 A propos

Amit Chaudhary sur Twitter : "How to learn transformers:..."

Tags:

2020-08-23 A propos

Aran Komatsuzaki sur Twitter : "Big Bird: Transformers for Longer Sequences..."

Tags:

2020-07-29 A propos

[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision

Tags:

2020-07-11 A propos

Dr Simon Osindero sur Twitter : "Neat! Transformers as RNNs"

Tags:

2020-07-02 A propos

[1909.03193] KG-BERT: BERT for Knowledge Graph Completion

Tags:

2020-03-22 A propos

Chaitanya Joshi sur Twitter : "Excited to share a blog post on the connection between #Transformers for NLP and #GraphNeuralNetworks"

Tags:

2020-03-01 A propos

Transformers are Graph Neural Networks | NTU Graph Deep Learning Lab

Tags:

2020-03-01 A propos

[2002.05867] Transformers as Soft Reasoners over Language

Tags:

2020-02-17 A propos

Hugging Face: How to train a new language model from scratch using Transformers and Tokenizers

Tags:

2020-02-16 A propos

[1911.05507] Compressive Transformers for Long-Range Sequence Modelling

Tags:

2020-02-11 A propos

Is the future of Neural Networks Sparse? An Introduction

Tags:

2020-02-05 A propos

Semantic Text Matching for Long-Form Documents (2019)

Tags:

2020-01-23 A propos

Named Entity Recognition with Pytorch Transformers – Pierre-Yves Vandenbussche

Tags:

2019-12-11 A propos - Copie locale

Fastai with Transformers (BERT, RoBERTa, XLNet, XLM, DistilBERT)

Tags:

2019-11-30 A propos

huggingface/transformers: 🤗 Transformers: State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch.

Tags:

2019-11-16 A propos

[1909.04120] Span Selection Pre-training for Question Answering

Tags:

> a **new pre-training task inspired by reading
comprehension** and an **effort to avoid encoding general knowledge in the transformer network itself**

Current transformer architectures store general knowledge -> large models, long pre-training time. Better to offload the requirement of general knowledge to a sparsely activated network.

"Span selection" as an additional auxiliary task: the query is a sentence drawn from a corpus
with a term replaced with a special token: [BLANK]. The term replaced by the blank is the answer term. The passage is
relevant as determined by a BM25 search, and answer-bearing (containing the answer
term). Unlike BERT’s cloze task, where the answer must be drawn from the model itself, the answer is found in a passage
using language understanding.

> **We hope to progress to a model of general purpose language modeling that uses an indexed long
term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.**

2019-09-18 A propos

Evolution of Representations in the Transformer (2019)

Tags:

2019-09-16 A propos

CTRL: A CONDITIONAL TRANSFORMER LANGUAGE MODEL FOR CONTROLLABLE GENERATION

Tags:

2019-09-12 A propos

[1909.03186] On Extractive and Abstractive Neural Document Summarization with Transformer Language Models

Tags:

2019-09-11 A propos

[1905.07129] ERNIE: Enhanced Language Representation with Informative Entities

Tags:

2019-08-05 A propos

huggingface/pytorch-transformers: A library of state-of-the-art pretrained models for NLP

Tags:

2019-07-27 A propos

[1907.05242] Large Memory Layers with Product Keys

Tags:

> **a structured memory which can be easily integrated into a neural network.** The memory is very large by design and therefore significantly increases the capacity of the architecture, by up to a billion parameters with a negligible computational overhead. Its design and access pattern is based on **product keys**, which enable fast and exact nearest neighbor search. The ability to increase the number of parameters while keeping the same computational budget lets the overall system strike a better trade-off between prediction accuracy and computation efficiency both at training and test time.

> a key-value memory layer that can increase model capacity for a negligible computational cost. A 12-layer transformer with a memory outperforms a 24-layer transformer, and is 2x faster!

[Implementation](/doc/2019/08/product_key_memory_pkm_minima)

TODO: compare with [[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge](doc:2020/07/2007_00849_facts_as_experts_)

2019-07-13 A propos

[1903.05823] Deep Patent Landscaping Model Using Transformer and Graph Embedding

Tags: