Semanlink - Transformers from Scratch

Transformers from Scratch

Tags:

Transformers explained

Au sujet de ce document

sl:bookmarkOf : https://e2eml.school/transformers.html#one_hot
sl:creationDate : 2022-08-28
sl:creationTime : 2022-08-28T22:50:30Z

Infos sur le fichier

Bookmark of: https://e2eml.school/transformers.html#one_hot

Documents with similar tags (experimental)

Physics of AI - YouTube

Tags:

- Intelligence has emerged: why? how?
- Let's study this with *controlled experiments* and *toy models*
- Clean and clear insights that peer slightly behind the magic curtain

emergence; rise of scale. Phase transition: fct(compute time for training: nb of params, dataset size)

(12:25) Transformers: the jump when compared to traditional NN: instead on oparating on single input X, operates on a set of inputs. (cf. training to say whether an image contains 2 elements of the same kind). **The aptitude to compare elements in the input sequence enables analogies** - the essence of reasoning.(Classical NN treats one high dimensional vector, tranformers hanlde sets of vectors. Attention layer replaces learned filters W of classical NN by the other elements in the sequence. Attention module compares part of input against other part of input. "Relative machines rather than absolute". Sets are a very powerful level of abstraction)

emergence
> The truth is that nobody has a clue what is going on"
>
> "**Something unknown is doing we don't know what**" (A. Eddington)

being inspired by physics' methodology: controled experiments and toy mathematical models

2023-03-28 A propos

Sebastian Raschka sur Twitter : "A question that often comes up when introducing colleagues to the attention mechanism: how are attention scores different from weights in a fully-connected layer?..."

Tags:

2023-02-26 A propos

Transformers from Scratch

Tags: