Semanlink - Transformers ; LLM AND LMs: context length

Printer friendly

Search Tag:

Search Doc:

Preferences...

Tag Cloud

Andrej Karpathy - Arxiv Doc - LLM - LLMs: extended context window - LMs: context length - Long documents - NLP@Meta - NLP@Microsoft - OpenAI GPT - Transformers - Transformers: Quadratic complexity - Tweet -

Transformers ; LLM AND LMs: context length

Common descendants

8 Documents

Albert Gu sur X : "Quadratic attention has been indispensable for information-dense modalities such as language... until now..."

Tags:

2023-12-05 About

Philipp Schmid sur X : “YaRN” allows you to scale LLMs like llama 2 to over 100k context!...

Tags:

2023-09-01 About

[2307.02486] LongNet: Scaling Transformers to 1,000,000,000 Tokens

Tags:

2023-07-06 About

[2305.07185] MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

Tags:

2023-07-01 About

State of GPT (Andrej Karpathy at Microsoft Build)

Tags:

2023-05-24 About

Aran Komatsuzaki sur Twitter : "Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System Enables LLMs to outperform ChatGPT in scenarios involving ultra-long document summarization or long-term conversations."

Tags:

2023-04-27 About

Aran Komatsuzaki sur Twitter : "Scaling Transformer to 1M tokens and beyond with Recurrent Memory Transformer..."

Tags:

2023-04-25 About

Does ChatGPT remember what happened earlier in the conversation? | OpenAI Help Center

Tags:

[aistackexchange](https://ai.stackexchange.com/questions/38150/how-does-chatgpt-retain-the-context-of-previous-questions)

> [me]I heard people using the phrase "context length" about you. Is it the same thing as the number of tokens you just spoke of?
>
> [ChatGPT]"Context length" and "number of tokens" are related concepts, but they refer to slightly different things. The number of tokens refers to the total number of individual units of meaning (e.g. words or subwords) in a given input text.
>
> Context length, on the other hand, refers to the number of tokens used as a context for generating a response. The context is the information the model uses to generate a response. So, context length is the number of tokens that the model will take into account to generate a response.
>
> In other words, the context length is a subset of the number of tokens, it's the number of tokens used to generate the response. **It's usually a sliding window of tokens that goes back and forth on the input text**. The larger the context length, the more information the model has to work with, which can lead to more accurate and relevant responses but also requires more computational resources and can make the model slower.
>
> [me][surprised]

2023-01-27 About