Semanlink - sl:creationTime : 2023-04-23T11:35:38Z

Search Tag:

Search Doc:

Preferences...

Tag Cloud

ChatGPT - Reinforcement learning - Yoav Goldberg -

sl:creationTime : 2023-04-23T11:35:38Z

Reinforcement Learning for Language Models

Tags:

> I was puzzled for a while as to why we need RL for LM training, rather than just using supervised instruct tuning. I now have a convincing argument, which is also reflected in a recent talk by @johnschulman2.

1st convincing argument:

> supervised learning allows only positive feedback (we show the model a series of questions and their correct answers) while **RL allows also for negative feedback** (the model is allowed to generate an answer an get a feedback saying "this is not correct")...if you as a learner are allowed to form your own hypotheses and ask the teacher if they are correct (as in the RL setting), even an adversarial teacher can no longer trick you into latching on to a wrong hypothesis.

2nd convincing argument is about knowledge-seeking queries

> we want to encourage the model to answer based on its internal knowledge, but we don't know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer... But if we are succeed in training the model to generalize in [the cases it doesn't know], then we essentially teaches the model to make stuff up! it actively encourages the model to "lie".

2023-04-23 About