Semanlink - OpenAI GPT AND LMs: context length

Philipp Schmid sur X : “YaRN” allows you to scale LLMs like llama 2 to over 100k context!...

Tags:

2023-09-01 About

State of GPT (Andrej Karpathy at Microsoft Build)

Tags:

2023-05-24 About

Aran Komatsuzaki sur Twitter : "Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System Enables LLMs to outperform ChatGPT in scenarios involving ultra-long document summarization or long-term conversations."

Tags:

2023-04-27 About

Does ChatGPT remember what happened earlier in the conversation? | OpenAI Help Center

Tags:

[aistackexchange](https://ai.stackexchange.com/questions/38150/how-does-chatgpt-retain-the-context-of-previous-questions)

> [me]I heard people using the phrase "context length" about you. Is it the same thing as the number of tokens you just spoke of?
>
> [ChatGPT]"Context length" and "number of tokens" are related concepts, but they refer to slightly different things. The number of tokens refers to the total number of individual units of meaning (e.g. words or subwords) in a given input text.
>
> Context length, on the other hand, refers to the number of tokens used as a context for generating a response. The context is the information the model uses to generate a response. So, context length is the number of tokens that the model will take into account to generate a response.
>
> In other words, the context length is a subset of the number of tokens, it's the number of tokens used to generate the response. **It's usually a sliding window of tokens that goes back and forth on the input text**. The larger the context length, the more information the model has to work with, which can lead to more accurate and relevant responses but also requires more computational resources and can make the model slower.
>
> [me][surprised]

2023-01-27 About