http://www.semanlink.net/tag/reinforcement_learning Documents tagged with Reinforcement learning http://www.semanlink.net/doc/2025/02/deepseek_r1_model_by_deepseek_a > DeepSeek-R1 is a first-generation **reasoning model trained using large-scale reinforcement learning** (RL) to solve complex reasoning tasks across domains such as math, code, and language. The model leverages RL to develop reasoning capabilities, which are further enhanced through supervised fine-tuning (SFT) to improve readability and coherence. 2025-02-24T13:34:19Z http://www.semanlink.net/doc/2025/02/diffuse_one > There is an emerging pattern of fine-tuning a small language model followed by reinforcement learning. > A reasoning model is a large language model that is trained to output both a chain of thought and a response. The chain of thought should be relatively long ( > 1,000 tokens) and the reasoning should improve its performance relative to a similar-sized non-reasoning models. This is sometimes called "test-time" or "inference-time" scaling because reasoning models emit more tokens per completion and gain some performance as a result. 2025-02-24T13:21:09Z http://www.semanlink.net/doc/2025/02/autotelic_agents_with_intrinsic > Building autonomous machines that can explore open-ended environments, discover possible interactions and build repertoires of skills is a general objective of artificial intelligence. Developmental approaches argue that this can only be achieved by autotelic agents: intrinsically motivated learning agents that can learn to represent, generate, select and solve their own problems. In recent years, the convergence of developmental approaches with deep reinforcement learning (rl) methods has been leading to the emergence of a new field: developmental reinforcement learning. 2025-02-09T15:56:41Z http://www.semanlink.net/doc/2023/05/archit_sharma_sur_twitter_ev 2023-05-31T18:30:01Z http://www.semanlink.net/doc/2023/05/eric_sur_twitter_rlhf_is_the 2023-05-31T18:16:59Z http://www.semanlink.net/doc/2023/05/peter_j_liu_sur_twitter_her > TL;DR: Works as well as RLHF, but a lot simpler. About as easy and efficient as fine-tuning. Much better than simply fine-tuning on good examples. 2023-05-18T09:53:46Z http://www.semanlink.net/doc/2023/05/hyung_won_chung_sur_twitter_ 2023-05-18T09:47:49Z http://www.semanlink.net/doc/2023/04/aran_komatsuzaki_sur_twitter__1 2023-04-27T08:13:24Z http://www.semanlink.net/doc/2023/04/rl_for_llms_md > I was puzzled for a while as to why we need RL for LM training, rather than just using supervised instruct tuning. I now have a convincing argument, which is also reflected in a recent talk by @johnschulman2. 1st convincing argument: > supervised learning allows only positive feedback (we show the model a series of questions and their correct answers) while **RL allows also for negative feedback** (the model is allowed to generate an answer an get a feedback saying "this is not correct")...if you as a learner are allowed to form your own hypotheses and ask the teacher if they are correct (as in the RL setting), even an adversarial teacher can no longer trick you into latching on to a wrong hypothesis. 2nd convincing argument is about knowledge-seeking queries > we want to encourage the model to answer based on its internal knowledge, but we don't know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer... But if we are succeed in training the model to generalize in [the cases it doesn't know], then we essentially teaches the model to make stuff up! it actively encourages the model to "lie". 2023-04-23T11:35:38Z http://www.semanlink.net/doc/2023/02/shayne_longpre_sur_twitter_ 2023-02-27T23:18:48Z http://www.semanlink.net/doc/2023/02/prompting_instruction_finetuni 2023-02-16T23:12:04Z http://www.semanlink.net/doc/2023/01/some_remarks_on_large_language_ > There turned out to be a phase shift somewhere between 60B parameters and 175B parameters, that made language models super impressive. > **The performance of current days language models are not obtained by language modeling** > > - [Traditional] LMs are not [grounded](tag:grounded_language_learning) > > **3 conceptual steps between GPT-3 and chatGPT: Instructions, code, RLHF.** The last one is, I think, the least interesting despite getting the most attention > > Instruction tuning: For example, the human annotators would write something like "please summarize this text", followed by some text they got, followed by a summary they produced of this text. -> Some symbols ("summarize", "translate", "formal") are used in a consistent way together with the concept/task they denote. And they always appear in the beginning of the text. -> the act of producing a summary grounded to the human concept of "summary" > > code: programming language code data, and specifically data that contains both natural language instructions or descriptions (in the form of code comments) and the corresponding programming language code. This produced another very direct form of grounding. the human language describes concepts (or intents), which are then realized in the form of the corresponding programs. > > "[RL with Human Feedback](tag:reinforcement_learning_from_human_feedback)". This is a fancy way of saying that the model now observes two humans in a conversation, one playing the role of a user, and another playing the role of "the AI", demonstrating how the AI should respond in different situations. This clearly helps the model learn how dialogs work, and how to keep track of information across dialog states (something that is very hard to learn from just "found" data). 2023-01-03T09:15:16Z http://www.semanlink.net/doc/2022/12/tanishq_mathew_abraham_sur_twit 2022-12-28T17:44:47Z http://www.semanlink.net/doc/2022/12/illustrating_reinforcement_lear 2022-12-10T11:51:09Z http://www.semanlink.net/doc/2022/10/prithviraj_raj_ammanabrolu_su 2022-10-06T01:56:53Z http://www.semanlink.net/doc/2019/12/meta_reinforcement_learning 2019-12-07T11:26:22Z http://www.semanlink.net/doc/2019/09/machine_learning_for_humans_pa 2019-09-23T23:36:26Z https://spinningup.openai.com/en/latest/spinningup/keypapers.html 2018-11-09T13:56:17Z https://sermanet.github.io/imitate/ Self-supervised approach for learning representations and robotic behaviors entirely from unlabeled videos recorded from multiple viewpoints, and study how this representation can be used in two robotic imitation settings: imitating object interactions from videos of humans, and imitating human poses. > We train our representations using a metric learning loss, where multiple simultaneous viewpoints of the same observation are attracted in the embedding space, while being repelled from temporal neighbors which are often visually similar but functionally different. In other words, the model simultaneously learns to recognize what is common between different-looking images, and what is different between similar-looking images. > This signal causes our model to discover attributes that do not change across viewpoint, but do change across time, while ignoring nuisance variables such as occlusions, motion blur, lighting and background. We demonstrate that this representation can be used by a robot to directly mimic human poses without an explicit correspondence, and that it can be used as a reward function within a reinforcement learning algorithm. 2018-10-27T14:59:43Z https://blog.insightdatascience.com/reinforcement-learning-from-scratch-819b65f074d8 2018-06-09T09:26:53Z http://amid.fish/reproducing-deep-rl 2018-04-10T13:33:49Z https://deepmind.com/blog/learning-to-generate-images/ This ability to interpret objects through the tools that created them gives us a richer understanding of the world and is an important aspect of our intelligence. 2018-03-28T12:11:42Z http://www.wildml.com/2018/02/introduction-to-learning-to-trade-with-reinforcement-learning/ 2018-02-11T12:20:30Z https://blog.openai.com/evolution-strategies/ 2018-01-06T15:11:28Z https://eng.uber.com/deep-neuroevolution/ > a suite of five papers that support the emerging realization that neuroevolution, where neural networks are optimized through evolutionary algorithms, is also an effective method to train deep neural networks for reinforcement learning (RL) problems. 2017-12-19T09:26:01Z https://deepmind.com/blog/alphago-zero-learning-scratch/ 2017-10-18T22:43:19Z