Semanlink - Technical girls and guys ; Software AND Deep Learning

- Intelligence has emerged: why? how?
- Let's study this with *controlled experiments* and *toy models*
- Clean and clear insights that peer slightly behind the magic curtain

emergence; rise of scale. Phase transition: fct(compute time for training: nb of params, dataset size)

(12:25) Transformers: the jump when compared to traditional NN: instead on oparating on single input X, operates on a set of inputs. (cf. training to say whether an image contains 2 elements of the same kind). **The aptitude to compare elements in the input sequence enables analogies** - the essence of reasoning.(Classical NN treats one high dimensional vector, tranformers hanlde sets of vectors. Attention layer replaces learned filters W of classical NN by the other elements in the sequence. Attention module compares part of input against other part of input. "Relative machines rather than absolute". Sets are a very powerful level of abstraction)

emergence
> The truth is that nobody has a clue what is going on"
>
> "**Something unknown is doing we don't know what**" (A. Eddington)

being inspired by physics' methodology: controled experiments and toy mathematical models

2023-03-28 About

Unveiling Transformers with LEGO - YouTube

Tags:

2022-06-30 About

Lenka Zdeborova sur Twitter : "Amazing lecture of @SebastienBubeck giving intuition on transformers and how to approach them theoretically..."

Tags:

2022-06-30 About

[1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification

Tags:

A model that performs **text classification using entities in a knowledge base**.

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 About

Scientists See Advances in Deep Learning, a Part of Artificial Intelligence - NYTimes.com

Tags:

2012-11-30 About