Semanlink - Attention mechanism ; NLP@Meta AND Technical girls and guys

Printer friendly

Search Tag:

Search Doc:

Preferences...

Tag Cloud

Attention mechanism ; NLP@Meta AND Technical girls and guys

Common descendants

9 Documents

Jeremy Howard sur X : "FSDP/QDoRA with Llama3 : I believe that this combination is likely to create better task-specific models than anything else at any cost."

Tags:

2024-04-23 About

Easily Finetune Llama 2 for Your Text-to-SQL Applications | by Jerry Liu | Aug, 2023

Tags:

2023-08-18 About

Andrej Karpathy sur Twitter : "How is LLaMa.cpp possible?"

Tags:

2023-08-16 About

Andrej Karpathy sur Twitter : "My fun weekend hack: llama2.c Lets you train a baby Llama 2 model in PyTorch, then inference it with one 500-line file with no dependencies, in pure C."

Tags:

2023-07-24 About

Jeremy Howard sur Twitter : "There's a lot of folks under the misunderstanding that it's now possible to run a 30B param LLM in <6GB, based on this GitHub discussion. This is not the case..." (thread)

Tags:

2023-04-03 About

Guillaume Lample sur Twitter : "Today we release LLaMA, 4 foundation models ranging from 7B to 65B parameters..."

Tags:

2023-02-25 About

Guillaume Lample sur Twitter : "Last year, we showed that you can outperform a 24-layer transformer in language modeling with just...

Tags:

2020-10-10 About

[1911.00172] Generalization through Memorization: Nearest Neighbor Language Models

Tags:

2019-12-20 About

[1907.05242] Large Memory Layers with Product Keys

Tags:

> **a structured memory which can be easily integrated into a neural network.** The memory is very large by design and therefore significantly increases the capacity of the architecture, by up to a billion parameters with a negligible computational overhead. Its design and access pattern is based on **product keys**, which enable fast and exact nearest neighbor search. The ability to increase the number of parameters while keeping the same computational budget lets the overall system strike a better trade-off between prediction accuracy and computation efficiency both at training and test time.

> a key-value memory layer that can increase model capacity for a negligible computational cost. A 12-layer transformer with a memory outperforms a 24-layer transformer, and is 2x faster!

[Implementation](/doc/2019/08/product_key_memory_pkm_minima)

TODO: compare with [[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge](doc:2020/07/2007_00849_facts_as_experts_)

2019-07-13 About