Semanlink - Mat Kelcey : The map interpretation of attention

Tags:

Au sujet de ce document

sl:bookmarkOf : https://www.youtube.com/watch?v=7wMQgveLiQ4
sl:creationDate : 2022-10-24
sl:creationTime : 2022-10-24T13:48:58Z

Infos sur le fichier

Bookmark of: https://www.youtube.com/watch?v=7wMQgveLiQ4

Documents with similar tags (experimental)

Tags:

- Intelligence has emerged: why? how?
- Let's study this with *controlled experiments* and *toy models*
- Clean and clear insights that peer slightly behind the magic curtain

emergence; rise of scale. Phase transition: fct(compute time for training: nb of params, dataset size)

(12:25) Transformers: the jump when compared to traditional NN: instead on oparating on single input X, operates on a set of inputs. (cf. training to say whether an image contains 2 elements of the same kind). **The aptitude to compare elements in the input sequence enables analogies** - the essence of reasoning.(Classical NN treats one high dimensional vector, tranformers hanlde sets of vectors. Attention layer replaces learned filters W of classical NN by the other elements in the sequence. Attention module compares part of input against other part of input. "Relative machines rather than absolute". Sets are a very powerful level of abstraction)

emergence
> The truth is that nobody has a clue what is going on"
>
> "**Something unknown is doing we don't know what**" (A. Eddington)

being inspired by physics' methodology: controled experiments and toy mathematical models

2023-03-28 A propos

Unveiling Transformers with LEGO - YouTube

Tags:

2022-06-30 A propos