About This Document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Yutao Sun
- sl:arxiv_num : 2307.08621
- sl:arxiv_published : 2023-07-17T16:40:01Z
- sl:arxiv_summary : In this work, we propose Retentive Network (RetNet) as a foundation
architecture for large language models, simultaneously achieving training
parallelism, low-cost inference, and good performance. We theoretically derive
the connection between recurrence and attention. Then we propose the retention
mechanism for sequence modeling, which supports three computation paradigms,
i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel
representation allows for training parallelism. The recurrent representation
enables low-cost $O(1)$ inference, which improves decoding throughput, latency,
and GPU memory without sacrificing performance. The chunkwise recurrent
representation facilitates efficient long-sequence modeling with linear
complexity, where each chunk is encoded parallelly while recurrently
summarizing the chunks. Experimental results on language modeling show that
RetNet achieves favorable scaling results, parallel training, low-cost
deployment, and efficient inference. The intriguing properties make RetNet a
strong successor to Transformer for large language models. Code will be
available at https://aka.ms/retnet.@en
- sl:arxiv_title : Retentive Network: A Successor to Transformer for Large Language Models@en
- sl:arxiv_updated : 2023-07-19T05:56:42Z
- sl:bookmarkOf : https://arxiv.org/abs/2307.08621
- sl:creationDate : 2023-07-20
- sl:creationTime : 2023-07-20T23:43:53Z
Documents with similar tags (experimental)
Preferences...