Semanlink

> **Part of what makes LoRA so effective is that ... it’s stackable.**
>
> By contrast, training giant models from scratch not only
throws away the pretraining, but also any iterative
improvements that have been made on top.

> LoRA updates are very cheap to produce (~$100) for the
most popular model sizes.

> Many of these projects are saving time by training on
small, highly curated datasets...
> These
datasets are built using synthetic methods (e.g. filtering
the best responses from an existing model) and
scavenging from other projects

> Directly Competing With Open Source
Is a Losing Proposition

> Paradoxically, the one clear winner in all of this is Meta.
Because the leaked model was theirs ([LLaMA](tag:llama)), they have
effectively garnered an entire planet's worth of free labor.
Since most open source innovation is happening on top of
their architecture, there is nothing stopping them from
directly incorporating it into their products.

2023-05-04 A propos

Yann Le Cun, directeur à Meta : « L’idée même de vouloir ralentir la recherche sur l’IA s’apparente à un nouvel obscurantisme »

Tags:

Interviews concomitantes de [Bengio](doc:2023/04/yoshua_bengio_chercheur_«_au) et LeCun par le Monde

> Ce qui est en cause dans les phénomènes de
désinformation, c’est moins le volume de production des
contenus problématiques que leur capacité à être
diffusés... La production ne
contribue pas à la dissémination.

(pour la dissémination, on a déjà Facebook)

> En matière de désinformation, l’IA n’est pas le
problème mais la solution

(good point)

[Pause Giant AI Experiments](tag:pause_giant_ai_experiments)? obscurantisme, et irréaliste

> le progrès, qu’il soit
scientifique ou social, dépend étroitement de
l’intelligence

oui, il faut mieux miser sur l'intelligence - mais pas forcément sur celle des humains qui auront entre leurs mains le pouvoir conféré par ces super-IA

> Ma priorité, en tant que chercheur, c’est de trouver le
moyen de rendre ces systèmes pilotables. (#[AI alignment](tag:ai_alignment))

> faut-il
continuer la recherche ouverte ? En ce qui me concerne,
la réponse est « oui ».

2023-04-29 A propos

Aran Komatsuzaki sur Twitter : "Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System Enables LLMs to outperform ChatGPT in scenarios involving ultra-long document summarization or long-term conversations."

Tags:

2023-04-27 A propos

ibiscp/LLM-IMDB: Proof of concept app using LangChain and LLMs to retrieve information from graphs, built with the IMDB dataset

Tags:

2023-04-10 A propos

[2303.17651] Self-Refine: Iterative Refinement with Self-Feedback

Tags:

2023-04-03 A propos

whitead/paper-qa: LLM Chain for answering questions from documents with citations

Tags:

2023-03-29 A propos

Jim Fan sur Twitter : "...NVIDIA AI Foundations "

Tags:

2023-03-24 A propos

Andrej Karpathy sur Twitter : "Base LLMs (non-finetuned) make very strong few-shot classifiers. Describe task in English, give few examples, read off the label probabilities on test example. No gradient-based optimization necessary. It brings a cannon to a knife fight but is fast, convenient, strong baseline." / Twitter

Tags:

2023-03-19 A propos

Jerry Liu sur Twitter : "A key goal of @gpt_index is to enable end users to ask an LLM *any* questions over their own data..."

Tags:

2023-02-27 A propos

LLM Powered Assistants for Complex Interfaces - Nick Arner

Tags:

2023-02-26 A propos

abacaj/awesome-transformers: A curated list of awesome transformer models.

Tags:

2023-02-25 A propos

Guillaume Lample sur Twitter : "Today we release LLaMA, 4 foundation models ranging from 7B to 65B parameters..."

Tags:

2023-02-25 A propos

Maria Khalusova @maria@recsys.social sur Twitter : "Did you know that you can tweak the text output generated by a LLM without changing any of the trainable parameters?..."

Tags:

2023-02-23 A propos

[2302.08091] Do We Still Need Clinical Language Models?

Tags:

2023-02-17 A propos

[2203.14465] STaR: Bootstrapping Reasoning With Reasoning

Tags:

2023-02-07 A propos

Yann LeCun sur Twitter : "On the highway towards Human-Level AI, Large Language Model is an off-ramp."

Tags:

2023-02-05 A propos

deepset sur Twitter : " Generative models have taken the world of NLP by storm. But LLMs do not know about your personal data. This makes personal assistants, enterprise knowledge management and many other applications challenging. Retrieval augmented pipelines are the answer"

Tags:

2023-02-02 A propos

hwchase17/langchain: ⚡ Building applications with LLMs through composability ⚡

Tags:

LLM

2023-02-02 A propos

The Flan Collection: Advancing open source methods for instruction tuning – Google AI Blog

Tags:

2023-02-02 A propos

An empirical analysis of compute-optimal large language model training

Tags:

2023-01-26 A propos

Andrej Karpathy sur Twitter : "Great post (5mo ago) "chinchilla's wild implications" giving context to LLM goldrush shifting from model size to dataset size..."

Tags:

2023-01-05 A propos

Some remarks on Large Language Models

Tags:

> There turned out to be a phase shift somewhere between 60B parameters and 175B parameters, that made language models super impressive.

> **The performance of current days language models are not obtained by language modeling**
>
>    - [Traditional] LMs are not [grounded](tag:grounded_language_learning)
> 
> **3 conceptual steps between GPT-3 and chatGPT: Instructions, code, RLHF.** The last one is, I think, the least interesting despite getting the most attention
>
> Instruction tuning: For example, the human annotators would write something like "please summarize this text", followed by some text they got, followed by a summary they produced of this text. -> Some symbols ("summarize", "translate", "formal") are used in a consistent way together with the concept/task they denote. And they always appear in the beginning of the text. -> the act of producing a summary grounded to the human concept of "summary"
>
> code: programming language code data, and specifically data that contains both natural language instructions or descriptions (in the form of code comments) and the corresponding programming language code. This produced another very direct form of grounding. the human language describes concepts (or intents), which are then realized in the form of the corresponding programs.
>
> "[RL with Human Feedback](tag:reinforcement_learning_from_human_feedback)". This is a fancy way of saying that the model now observes two humans in a conversation, one playing the role of a user, and another playing the role of "the AI", demonstrating how the AI should respond in different situations. This clearly helps the model learn how dialogs work, and how to keep track of information across dialog states (something that is very hard to learn from just "found" data).

2023-01-03 A propos